What a vector holds

Two courses have now treated the embedding as a given: text goes in, 1024 floats come out, and near vectors mean related text. That worked, and it left the interesting question unasked.

Why does it work at all? Nothing in [0.021, -0.113, 0.008, ...] looks like meaning.

The representations that came before

The oldest way to turn a word into a vector is to give every word in the vocabulary its own dimension and set only that one:

vocabulary: [cat, dog, subscription, billing]

"cat"          →  [1, 0, 0, 0]
"dog"          →  [0, 1, 0, 0]
"subscription" →  [0, 0, 1, 0]

This is one-hot encoding. It is honest, and useless for search. Every pair of distinct words is the same distance apart: cat is as far from dog as it is from subscription. The representation holds nothing the vocabulary list didn't already have.

Bag-of-words extends it to documents by counting words. BM25 refines that count. Because it matches the words themselves, lexical search works so well on exact terms. But it inherits the same blindness. "cancel my subscription" and "stop being billed" share no words, so their vectors are orthogonal: mathematically, as unrelated as two documents can be.

The dimensionality is also brutal. A realistic vocabulary has hundreds of thousands of words, so every document becomes a vector of hundreds of thousands of numbers, almost all zero.

Let the numbers be learned

A dense embedding drops the idea that a dimension corresponds to a word. Instead:

• The vector is short: 256 to 4096 numbers rather than 300,000.
• Every dimension is used for every input; nothing is mostly zeros.
• No dimension has a name. The values are learned, not assigned.

That last point trips people up, so here it is plainly: dimension 42 is not "formality" or "is about money". Occasionally someone finds an interpretable direction, but the model was never asked to build one. Training produces a space where distances carry meaning. The individual axes are an arbitrary basis for that space.

flowchart TB
  subgraph Sparse["One-hot / bag-of-words"]
    S1["300,000 dimensions"]
    S2["Assigned: one word per dimension"]
    S3["Unrelated words are equidistant"]
  end
  subgraph Dense["Learned embedding"]
    D1["1,024 dimensions"]
    D2["Learned: no dimension has a name"]
    D3["Related meanings land close"]
  end

Meaning as geometry

Think of a map. On a map of Brazil, the coordinates of Recife are two numbers, and on their own they say nothing about the city. They only mean something next to the coordinates of other cities. The map is useful because cities that are close in reality are close on the page.

An embedding space works the same way, with a thousand axes instead of two. "cancel my subscription" and "how do I stop being billed" share no words, yet they end up near each other. The model was trained on data where phrases like these were used interchangeably.

This also explains the limitation you met in the RAG course. A map places Rio near Niterói, and it can't tell you which one to visit. In the same way, "the Pro plan includes audit logs" and "the Pro plan does not include audit logs" are about the same subject, so they land in nearly the same place. The geometry encodes what a text is about. It says nothing about whether the text is true.

Why more dimensions can hold more distinctions

With two dimensions you can arrange a handful of concepts so that related ones sit close together. Add a third and you keep those relationships while gaining a new axis of variation. Each dimension adds room for one more distinction without disturbing the ones already encoded.

High-dimensional space also behaves strangely in a way worth knowing: as the number of dimensions grows, randomly chosen vectors become almost perpendicular to one another. That is good news for capacity: a thousand dimensions leave room for a huge number of nearly independent directions, so many distinctions fit without interfering. Trained models don't spread text out at random, though. Each model puts unrelated pairs in its own band of scores, near zero for some and 0.7 or higher for others (the e5 family, for example). Only the order of the scores carries meaning, which is why the pgvector course derived its thresholds per model and corpus instead of guessing them.

What this course covers

1. What a vector holds (you are here).
2. From text to one vector: tokens, context, and pooling.
3. How embedding models are trained: contrastive learning.
4. Bi-encoders and cross-encoders: the architecture behind retrieve-then-rerank.
5. Dimensions, Matryoshka training, and truncation.
6. Choosing a model, and why leaderboards mislead.
7. Adapting to your domain, and when fine-tuning is the wrong tool.

Next: what happens between your sentence and those 1024 numbers.