Embeddings and vector space: turning meaning into geometry

We have spent eleven posts treating the model as a frozen function over tokens — integer IDs going in, a probability distribution over the vocabulary coming out. But there is a step we have quietly skipped, and it is the one that makes everything else possible. Between "token ID 5432" and "matrix multiplications," something has to turn that integer into something a network can actually compute with. That something is an embedding.

An embedding is text mapped to a fixed-length list of real numbers — a vector — chosen so that text with similar meaning lands close together in the resulting space. That is the entire idea, and it is worth saying slowly, because everything downstream leans on it: an embedding turns meaning into geometry. Once meaning is geometry, "how similar are these two pieces of text?" becomes "how close are these two points?" — and closeness is something a computer can measure in microseconds.

If you want one sentence to carry the rest of this post: if you can measure closeness between vectors, you can approximate similarity between meanings. That approximation is the foundation of semantic search, RAG, classification, clustering, and — as we will see in the next post — the attention mechanism itself.

The core idea: a vector is just a list of numbers

A vector, for our purposes, is nothing exotic — it is a list of numbers, like [0.12, -0.34, 0.88, ...]. The length of that list is the number of dimensions. Real embedding models produce vectors with hundreds to thousands of dimensions: 384, 768, 1024, 1536, and 3072 are all sizes you will meet in practice.

The property that makes an embedding useful is not the numbers themselves — no single number means anything on its own — but the arrangement. A good embedding places "the cat sat on the mat" and "a feline rested on the rug" near each other, even though they share almost no words, while "TCP retransmission timeouts" lands far from both. Meaning, not surface spelling, drives the geometry.

"Near" needs a definition, and there are three you will see. Cosine similarity measures the angle between two vectors and ignores their length; it ranges from −1 to 1, and it is the most common choice for text because only direction carries meaning. Dot product is cosine similarity's unnormalized cousin — if the vectors are already unit length, the two are identical, and the dot product is cheaper to compute. Euclidean (L2) distance is ordinary straight-line distance; it shows up less often for text but is not wrong. Which one to use is not a matter of taste — it depends on how the model was trained, and the model card will tell you.

A useful mental model for the software engineer: an embedding is a lossy hash for meaning. Unlike a cryptographic hash, where similar inputs deliberately produce wildly different outputs, an embedding is a hash where similar inputs collide on purpose — they land near each other. And cosine similarity is the join key you use to match them.

Why this works: distributional semantics

Why should a list of numbers capture meaning at all? The answer predates neural networks by decades. In 1957 the linguist J.R. Firth wrote the line every NLP course now quotes: "You shall know a word by the company it keeps." The idea — the distributional hypothesis, also credited to Zellig Harris's 1954 "Distributional Structure" — is that words appearing in similar contexts tend to have similar meanings. "Coffee" and "tea" show up around "cup," "hot," and "morning"; that shared company is evidence they are related.

Modern embedding models operationalize this by training on enormous text corpora to predict context, or to tell similar and dissimilar pairs apart. The vectors that fall out of that training encode a genuinely rich picture of meaning. This is a real step up from older surface-level methods like bag-of-words or TF-IDF, which count words but cannot tell that "feline" and "cat" are related unless the exact strings match.

The king − man + woman ≈ queen story, told honestly

No discussion of embeddings escapes the most famous demo in the field. In 2013, Mikolov and colleagues at Google introduced word2vec ("Efficient Estimation of Word Representations in Vector Space," and its companion "Distributed Representations of Words and Phrases"), and showed something that looked like magic: take the vector for "king," subtract "man," add "woman," and the nearest vector is "queen." Arithmetic on meanings. Directions in the space seemed to correspond to human concepts — a "gender" direction, a "plural" direction, a "verb tense" direction.

The intuition is real and worth internalizing: directions in embedding space carry semantic meaning. But the specific party trick has been oversold, and honesty demands the caveat. The famous result only works if you exclude the input words ("king," "man," "woman") from the pool of possible answers — otherwise the nearest vector to the result is usually just "king" again. Tal Linzen's 2016 paper "Issues in evaluating semantic spaces using word analogies" showed that the offset method's reliance on cosine similarity conflates genuine analogy structure with irrelevant neighborhood effects, and that simple baselines (like just returning the nearest neighbor of "woman") do suspiciously well. Nissim, van Noord, and van der Goot's 2020 "Fair Is Better than Sensational: Man Is to Doctor as Woman Is to Doctor" made the point sharply for bias claims like "man is to computer programmer as woman is to homemaker" — these implementations quietly force the answer to differ from the inputs, so the sensational output is partly an artifact of the method.

So: king − queen is the origin story, not the whole story. Directions encode meaning — that part survives. The clean four-word arithmetic is a simplification that breaks under scrutiny.

Two very different things both called "embeddings"

Here is the distinction that trips up almost every beginner, and the single most useful thing this post can give you. Two completely different artifacts share the name "embedding."

Kind A: token embeddings, inside the model. Recall from the inference post that a forward pass starts with integer token IDs. The very first thing a transformer does is a lookup: each token ID indexes into an embedding matrix of shape [vocab_size × hidden_dim]. So if the vocabulary has 50,000 tokens and the model's hidden dimension is 4096, that matrix is 50,000 rows by 4096 columns, and token 5432 simply becomes row 5432 — a specific 4096-dimensional vector. These vectors are learned during training, exactly like any other weight, via backpropagation. They are the input to the rest of the network, not the output. Everything after that first lookup — attention, feed-forward layers — is vectors transforming vectors. This is why it is fair to say the transformer speaks in vectors; text is just the boundary condition at the very edge.

Kind B: text/sentence/document embeddings, as a standalone product. These are separate models whose entire job is to take an arbitrary span of text and emit a single vector representing the whole thing, for use in search, RAG, classification, and clustering. This is the "embeddings API" you buy or download. Examples, with verified current specs:

• OpenAI text-embedding-3-small (1536 dimensions) and text-embedding-3-large (3072 dimensions), launched January 2024. Both support shortening via the dimensions parameter; OpenAI states that "on the MTEB benchmark, a text-embedding-3-large embedding can be shortened to a size of 256 while still outperforming an unshortened text-embedding-ada-002 embedding with a size of 1536."
• Cohere Embed 4 (embed-v4.0), a multimodal model supporting configurable output dimensions (256, 512, 1024, 1536) and 100+ languages.
• Voyage voyage-3-large (default 1024 dimensions, also 256/512/2048). Per Voyage's January 2025 announcement, "it outperforms OpenAI-v3-large and Cohere-v3-English by an average of 9.74% and 20.71%, respectively, across 100 datasets, spanning eight diverse domains, including law, finance, and code."
• Open-source families: SBERT / sentence-transformers (typically 384–1024 dimensions), plus the BGE, E5, and mxbai lines.

The two kinds share the underlying idea — text becomes a similarity-preserving vector — but they are different models trained for different objectives. The most common concrete confusion: OpenAI's text-embedding-3 is not the same thing as GPT-4o's internal token embeddings. One is a purpose-built retrieval model; the other is the input layer of a generative model. Same word, different artifact.

And you cannot cheaply cross the gap. A tempting shortcut is to grab a base language model's token embeddings and just average them to get a sentence vector. This is a known weak baseline. The SBERT paper measured it directly: averaging BERT's output embeddings scores 54.81 on a standard suite of semantic-similarity tasks, and using BERT's [CLS] token scores just 29.19 — both worse than averaging plain GloVe word vectors at 61.32. As the authors put it, this "common practice yields rather bad sentence embeddings, often worse than averaging GloVe embeddings." A model that is genuinely good at sentence-level similarity has to be trained for it.

How sentence-embedding models are trained

The training recipe that produces good text embeddings is contrastive learning. You show the model many pairs of texts, some labeled "similar" (positive pairs) and some "dissimilar" (negatives). The loss function pulls positives together in vector space and pushes negatives apart. Do this over hundreds of millions of pairs and the geometry organizes itself so that closeness tracks meaning.

The architecture is usually a dual encoder (also called a bi-encoder): two copies of the same encoder, sharing weights, process the query and the document independently, each producing one vector; similarity is then just their dot product. Because the two sides are independent, you can embed your whole corpus once, offline, and at query time embed only the query — which is exactly what makes billion-scale search feasible.

SBERT (Reimers & Gurevych, 2019, "Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks") is the seminal work here; it fine-tuned BERT in a siamese setup on natural-language-inference pairs and made semantic similarity search practical. The paper reports that finding the most similar pair in a collection of 10,000 sentences dropped "from 65 hours with BERT to the computation of 10,000 sentence embeddings (~5 seconds with SBERT) and computing cosine-similarity (~0.01 seconds)." The training data typically comes from NLI datasets, parallel/paraphrase corpora, and query–passage pairs mined from search logs. Microsoft's E5 ("Text Embeddings by Weakly-Supervised Contrastive Pre-training," 2022) pushed the weakly-supervised version of this and was the first model to beat the classical BM25 baseline on the BEIR benchmark without labeled data. The through-line: sentence-embedding models are almost always fine-tuned from base language models, then specialized for the similarity objective.

Dimensionality and the Matryoshka trick

More dimensions mean more capacity to represent nuance, and often better quality — but they cost more storage, memory, and compute on every single query, multiplied across millions of vectors. That tension used to force a hard choice at model-selection time.

Matryoshka Representation Learning (Kusupati et al., 2022) softened it. The idea — named for nested Russian dolls — is to train the vector so that its first N dimensions are independently useful. You can truncate a 3072-dimensional vector down to 512 or 256 and lose quality gracefully rather than catastrophically. This is what lets OpenAI's text-embedding-3 models, Voyage's voyage-3 line, and Cohere's Embed 4 offer a dimensions knob: pick your point on the quality-versus-cost curve without retraining or running a second model. One caveat worth remembering: truncating a normalized (unit-length) vector leaves it no longer normalized, so you renormalize before comparing.

Practical realities that will bite you

Embeddings are simple in principle and full of operational sharp edges in practice.

Vectors drift between model versions, and spaces are not aligned across models. A vector from text-embedding-3 is meaningless when compared to one from the older ada-002; they live in unrelated coordinate systems. Upgrade your embedding model and you must re-embed your entire corpus — a real, budgeted operational cost, not a footnote. For the same reason, different providers' embeddings are not interchangeable: you cannot mix OpenAI and Cohere vectors in one index and expect the distances to mean anything.

Cosine versus dot product depends on the model, and normalization matters. Some models emit unit-length vectors (making cosine and dot product equivalent); some do not. Read the card; do not assume.

Hybrid search usually beats pure vector search. This is an active best practice, not a hedge. Dense vector search is superb at paraphrase and synonym matching but "smooths over" exact tokens — rare names, error codes, statute numbers, technical identifiers. BM25, the classical keyword-ranking algorithm (Robertson & Zaragoza, 2009), nails those exact matches. Because the two methods fail in complementary ways, combining them — typically by fusing the two ranked lists with Reciprocal Rank Fusion — recovers documents either method alone would miss. A 2026 benchmark on financial documents (T2-RAGBench) even found BM25 outperforming a strong dense model on most metrics, precisely because those documents are dense with identifiers and exact numbers. Treat "dense-only is enough" as an assumption to test, not a default.

Vector databases exist because brute force does not scale. Computing cosine similarity against millions of vectors one by one is too slow for interactive queries, so vector stores — pgvector, Pinecone, Weaviate, Qdrant, Chroma, LanceDB — implement approximate nearest neighbor (ANN) search. Most use a graph-based index called HNSW (Hierarchical Navigable Small World), which trades a small, tunable amount of recall for enormous speedups, giving sub-linear query time as the index grows. The internals (HNSW parameters, IVF, product quantization) are their own rabbit hole; for a basics series, it is enough to know the operation is "find the nearest vectors, fast, approximately."

How you compare embedding models: the standard artifact is the MTEB leaderboard (Massive Text Embedding Benchmark, Muennighoff et al.), which "spans 8 embedding tasks covering a total of 58 datasets and 112 languages." A key finding from the MTEB paper is both sobering and useful: "no particular text embedding method dominates across all tasks," and "the field has yet to converge on a universal text embedding method." The right model depends on your data and your workload — which is why the discipline's real advice is to build a small evaluation set from your own queries and measure.

What embeddings power — and what they don't

The short tour, since several of these get their own posts later:

• Semantic search / retrieval: embed a query, find nearby vectors in a corpus. This is the retrieval half of RAG.
• Classification: feed an embedding to a small classifier head, or do zero-shot classification by cosine similarity to label embeddings.
• Clustering and topic modeling; deduplication; recommender systems (user and item embeddings).
• Attention — next post — where the model computes internal "query" and "key" vectors and uses their dot products to decide what to attend to. Same machinery, learned and living inside the forward pass.

And the boundaries, which matter just as much:

• Embeddings capture similarity, not truth. Two confidently false statements that are worded alike will sit near each other. Distance is not correctness.
• Embeddings do not "understand." They are a compressed representation optimized for a similarity objective — nothing more.
• They are not interpretable dimension by dimension. The 512th number does not mean "royalty." Meaning lives in high-dimensional directions and combinations, not in individual coordinates.
• They inherit bias from training data. Bolukbasi et al. (2016) documented gender stereotypes sitting as measurable directions in word-embedding space; the similarity structure faithfully reproduces the correlations — including the ugly ones — in its training corpus.

Where this leaves us

An embedding is the bridge from text to geometry: a fixed-length vector, arranged so that similar meanings sit close, compared with a cheap distance metric. There are two kinds — the token-embedding lookup that opens every forward pass, and the standalone text-embedding models you use for search and RAG — and keeping them straight is most of what separates confusion from fluency here.

The next post stays inside the model and asks the obvious follow-up: once every token is a vector, how does the model decide which other tokens matter? That is attention — the mechanism that reads these vectors and lets each position pull in the context relevant to it. Later, in the Extending section, we will come back to the standalone kind of embedding when we build RAG, where the whole point is to embed your documents, nearest-neighbor over them, and inject the closest chunks into the context window — the only write surface a frozen model has.

The transformer speaks in vectors. This post was the phrasebook.

Further reading

Foundations

• Mikolov et al. (2013). Efficient Estimation of Word Representations in Vector Space and Distributed Representations of Words and Phrases and their Compositionality — word2vec; the origin of vector arithmetic on words.
• J.R. Firth (1957), "A synopsis of linguistic theory, 1930–1955," and Zellig Harris (1954), "Distributional Structure" — the distributional hypothesis: "you shall know a word by the company it keeps."
• Linzen (2016). Issues in evaluating semantic spaces using word analogies — the critical reappraisal of king − man + woman.
• Nissim, van Noord, van der Goot (2020). Fair Is Better than Sensational: Man Is to Doctor as Woman Is to Doctor.

Models & training

• Reimers & Gurevych (2019). Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks — averaged BERT 54.81, [CLS] 29.19, averaged GloVe 61.32; the 65 hours → ~5 seconds result.
• Wang et al. (2022). Text Embeddings by Weakly-Supervised Contrastive Pre-training (E5).
• Kusupati et al. (2022). Matryoshka Representation Learning.

Benchmarks & retrieval

• Muennighoff, Tazi, Magne, Reimers (2022). MTEB: Massive Text Embedding Benchmark — "no particular text embedding method dominates across all tasks."
• Malkov & Yashunin (2016). Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs (HNSW) — the index behind most vector databases.
• Robertson & Zaragoza (2009). "The Probabilistic Relevance Framework: BM25 and Beyond" — the keyword baseline hybrid search leans on.

Bias & limits

• Bolukbasi et al. (2016). Man is to Computer Programmer as Woman is to Homemaker? Debiasing Word Embeddings.

Products & docs

• OpenAI. New embedding models and API updates (Jan 2024).
• Voyage AI. voyage-3-large (Jan 2025).
• Cohere Embed documentation; the sentence-transformers (SBERT) library.