The same idea, one level down

You already did this once. The pgvector course stored embeddings as halfvec to halve the index, and as one bit per dimension for a first-pass search. Both traded a little accuracy for a lot of memory.

Quantization of model weights is the same trade applied to the model itself. The numbers you shrink are the parameters rather than your vectors. The payoff is not a smaller index. It is whether the model runs on hardware you can afford at all.

Why this decides everything

A model's weights have to be in memory to run. That one sentence is the whole subject:

A 70-billion-parameter model:
  32-bit floats    70B × 4 bytes  =  280 GB
  16-bit floats    70B × 2 bytes  =  140 GB
  8-bit integers   70B × 1 byte   =   70 GB
  4-bit integers   70B × 0.5 byte =   35 GB

Read that column as a list of different machines, not as a cost curve. At 140 GB you need data-centre hardware: two 80 GB accelerators, or one of the newest parts with more memory than that. At 35 GB the same model fits on a single 48 GB workstation card, or on two 24 GB consumer cards (the largest consumer card today has 32 GB). That is the difference between a rented server and a workstation, and often the difference between the project happening and not.

Cutting to four bits reduces memory by roughly 75%. That is what turns "this needs a multi-GPU server" into "this runs on one card".

What quantization is

A weight is a number like 0.0293841. Stored as a 32-bit float, it costs four bytes and carries far more precision than the model's behavior depends on.

Quantization stores it in fewer bits by mapping a range of real values onto a small set of integers:

float32 weights   -0.42 ... 0.38     (4 bytes each, ~4 billion distinct values)
int8 quantized    0 ... 255          (1 byte each, 256 distinct values)

flowchart LR
  V["Embedding vectors"] --> H["halfvec, binary"] --> S1["Smaller index"]
  W["Model weights"] --> Q["INT8, INT4"] --> S2["Model fits the hardware"]

The mapping is arithmetic, a scale and an offset, and lesson 4 works through it. What matters here is that it is lossy and structured: you lose precision within a range you chose, rather than losing arbitrary information.

Why it works at all

It is not obvious that throwing away most of a number's precision leaves the model intact. Three reasons it largely does.

First, the precision was never load-bearing. Weights were trained with noise everywhere: dropout, batching, stochastic gradients. A model that depended on the eighth decimal place of a weight would not have survived training.

Second, the errors are many and small. Quantization error per weight is tiny and roughly independent. Across billions of weights the effects tend to cancel rather than compound.

Third, the model's behavior depends on relationships between values, as with embeddings. Scaling everything consistently preserves those relationships.

The same three explain why quantization fails when it fails: a few weights that do carry the load, errors that stop being independent, and ranges distorted by outliers. Lesson 4 is about that.

Where this applies to you

There are two situations, and they frame the whole course.

You are serving an open-weight model. Quantization is how you fit it on the hardware you have. The format and the bit width are decisions you make and measure.

You are calling an API. Then this is not your decision, because the provider serves what it serves. It is still worth understanding. Parameters times bytes is what any model costs to hold in memory, so it explains a large part of why a small model is cheaper to serve. And the moment a self-hosted option is on the table, this is the arithmetic that decides it.

What this course covers

1. The same idea, one level down (you are here).
2. The arithmetic: what fits.
3. Precision formats, and where quality falls off.
4. How quantization works, and how it breaks.
5. Methods and formats: GPTQ, AWQ, GGUF and the rest.
6. Measuring the loss honestly.
7. Putting it together: choosing, serving and deciding.

Key takeaways

• Quantization of weights is the same trade as halfvec and binary vectors, applied to the model instead of your index.
• Weights must be in memory to run, so bits per parameter decides which hardware can serve a model at all.
• Four-bit quantization cuts memory by roughly 75%, which often turns a multi-GPU requirement into a single card.
• It works because trained weights never depended on that precision, errors are small and independent, and relative relationships survive.
• It matters directly when you self-host, and indirectly when you call an API: memory per model is a large part of what serving costs.