Embeddings: useful geometry, not truth

Section 01

Vectors that capture relationships

In Lesson 00b, you saw that embeddings are vectors — lists of numbers — where tokens with similar meanings map to nearby points. That was an introduction. Now we examine embeddings as a system: how they are created, how similarity is measured, what proximity means, and where it misleads.

Embeddings are the foundation of semantic search, RAG, clustering, and duplicate detection. They are also one of the most over-trusted tools in the LLM stack. A high similarity score feels like a strong signal, but it is not a probability, not a guarantee of relevance, and not evidence of truth.

Section 02

Stones in a deep tray

Imagine a deep, open tray filled with hundreds of small ceramic stones. Each stone is a token, sentence, or document, positioned so that stones with similar meanings cluster together. The tray is deep — it has hundreds or thousands of dimensions you cannot see.

Above the tray, a translucent sheet catches the shadow of the stones. The shadow looks like a 2D map of the clusters, but it is a projection: distances are distorted, some clusters overlap in shadow that are separate in the full space, and some stones that appear close in shadow are far apart in reality.

That shadow is what you see when someone shows you a 2D or 3D plot of embeddings. It is useful for intuition, nothing more. The actual distances that matter are computed in the full high-dimensional space, using mathematical similarity functions — not by eyeballing a projection.

Where the analogy breaks: in the physical tray, 'nearby' has a single meaning (physical distance). In embedding space, the choice of similarity function (cosine, dot product, Euclidean) affects which neighbors are 'nearby'. Different metrics can produce different rankings.

Section 03

Cosine similarity and nearest neighbors

An embedding model maps input to a numeric vector optimized to preserve relationships useful for its training objective. The vectors have hundreds or thousands of dimensions. Individual dimensions usually do not map cleanly to interpretable concepts like 'royalty' or 'gender' — those stories come from selective analysis, not from reading off axis labels.

The most common similarity function is cosine similarity, which compares the direction of two vectors and ignores their magnitude.

cosine(A, B) = (A . B) / (||A|| * ||B||)

A . B    = sum of element-wise products (dot product)
||A||   = sqrt(sum of A_i^2)  (magnitude / Euclidean norm)

Result range: [-1, 1]
  1  = identical direction
  0  = orthogonal (no relationship)
 -1  = opposite direction

// If vectors are normalized (magnitude = 1),
// cosine similarity equals the dot product.

Section 04

Nearest-neighbor search: exact vs approximate

To find the most similar items to a query, you compute the query's embedding and search for the nearest vectors in your index. Exact search compares the query against every vector — accurate but slow for large collections.

Approximate nearest neighbor (ANN) indexes trade perfect recall for speed. HNSW (Hierarchical Navigable Small World), IVF (Inverted File), and product quantization are common ANN methods. Each introduces tunable quality-latency-memory tradeoffs. The right choice depends on your collection size, latency budget, and acceptable recall loss.

Metadata filters often matter as much as dense similarity. A query for 'Python error handling' may match documents about Python and documents about error handling — but you usually want documents about both. Hybrid search (combining dense similarity with lexical or metadata filtering) is standard in production RAG systems.

query_vec = embed(query_text)

# Exact search (O(n) comparisons)
scores = [(cosine(query_vec, v), id) for id, v in index]
results = sorted(scores, reverse=True)[:k]

# ANN search (HNSW or IVF)
# Trades perfect recall for sub-linear search time
results = ann_index.search(query_vec, k=k, filter=metadata)

Section 05

Versioning, normalization, and thresholds

Vectors from different embedding models or versions are generally not interchangeable. A vector from `text-embedding-3-small` cannot be compared with a vector from `text-embedding-ada-002`. If you change the embedding model, you must re-embed the entire collection. Store the model name and version with every vector.

Normalization affects which metric to use. If vectors are normalized to unit length, dot product equals cosine similarity, and you can use the faster dot product. If vectors are not normalized, use cosine similarity explicitly. Check your embedding model's documentation — many return pre-normalized vectors.

Thresholds must be calibrated on labeled task data. There is no universal 'similarity > 0.8 means relevant' rule. What counts as relevant depends on your domain, your query distribution, and your tolerance for false positives. Monitor thresholds after model or version changes.

Section 06

Where embeddings mislead

Analogy arithmetic — examples like king - man + woman = queen — is a historical intuition from specific embedding models. It is not a guaranteed API contract. It can encode social bias present in the training data. Presenting it as literal semantic algebra misleads.

A 2D or 3D projection of embeddings is a lossy view. Clusters that look separate in 2D may overlap in high-dimensional space. Distances in the projection are distorted. Use projections for intuition; use the actual high-dimensional distances for decisions.

Semantic proximity can reflect topical association rather than factual entailment. Two documents about 'databases' will have high similarity regardless of whether one answers the other's question. RAG systems that rely on raw similarity without reranking or verification retrieve topically-related-but-unhelpful results.

Section 07

Unscored lab: build a semantic search prototype

Create a semantic search prototype for a small collection of engineering runbooks (10-20 documents).

1. Choose an embedding model. Record its name and version.
2. Chunk each runbook into meaningful units (sections, paragraphs). Define your chunk unit.
3. Embed each chunk. Store the model version with each vector.
4. Implement cosine similarity search. Query for 'how to restart the cache layer'.
5. Inspect the top 5 results. Are they actually helpful? Or just topically related?
6. Find one query where semantic proximity is misleading — where the most similar chunk does not answer the question.
7. Document: what would make this better? Reranking? Metadata filters? Smaller chunks?