How to use this post

• This is a myth versus fact post. Each myth is a claim I hear when people talk about speeding up vector search. The fact comes from papers and official docs, not from a checklist of seven names.
• The running example is a company assistant. A user types "How do I reset my password?" and we need the chunk that says "I forgot my password." The vectors are already built. The question here is how the store finds nearby vectors quickly.
• I do not rerun benchmarks in this post. The only numbers I use are example memory math, defaults or formulas copied from docs, and figures quoted from a named paper with its date and dataset.
• I checked the claims against official docs and the original papers in November 2026. Links are in the Sources section.
• This post is about the index inside the store. If you are still choosing the store itself, start with the vector database decision guide.

The short answer first

Once you have embeddings, search is a neighbour problem. You hold a query vector, and you want the stored vectors that sit closest to it. If the collection is small, you can compare the query with every vector. That is exact search, often called a flat index or a brute-force scan. It is simple, and it does not miss a neighbour.

When the collection grows, that scan gets expensive. An approximate nearest neighbour index, or ANN index, visits only a promising subset. It is faster, or it uses less memory, but it can miss a true neighbour. That trade is the whole subject. The usual measure is recall@k: the share of queries where the right chunk still appears in the top k.

A popular way to teach this is a list of seven types: Flat, IVF, HNSW, LSH, Annoy, ScaNN and DiskANN. The list is a useful set of names to recognise. It is a poor way to choose an index, because those names are not the same kind of thing. Some are algorithms. Some are libraries. Some are full systems. Compression, which often matters as much as the index family, is not a peer on that list at all.

A clearer picture is four building blocks you can combine:

• Partition: split vectors into lists, as IVF does, and search only some of the lists.
• Graph: link each vector to nearby vectors, as HNSW and Vamana do, and walk those links.
• Compress: store shorter codes, as product quantization or scalar quantization do, then optionally rerank with the full vector.
• Place: keep what you need in RAM, and put the rest on SSD when the data no longer fits.

flowchart TD
  A[Query vector] --> B{Do you need exact neighbours?}
  B -->|Yes, or the set is still small| C[Flat / exhaustive scan]
  B -->|No, you can trade a little recall| D[Approximate index]
  D --> E[Partition: IVF-style lists]
  D --> F[Graph: HNSW or Vamana]
  E --> G[Optional compression: PQ, SQ, binary]
  F --> G
  G --> H{Does it fit in RAM?}
  H -->|Yes| I[In-memory search]
  H -->|No| J[Disk-based graph or lists, such as DiskANN]

The rest of the post walks through the myths that hide behind the seven-name list.

Myth 1: There are seven types of vector index, and you should treat them as peers

Why people believe it. A short list is easy to memorise. Flat, IVF, HNSW and LSH sound like algorithms. Annoy, ScaNN and DiskANN get added because they show up in blog posts, and the list now looks complete.

What is true. Those seven names sit in different layers.

FAISS, which is the library many of these ideas are implemented in, does not present a list of seven types either. Its own guide, updated in October 2026, asks a few practical questions: will you run only a small number of searches, do you need exact results, is memory tight, and how large is the dataset? The answers point to combinations such as Flat, HNSW, IVF with a product quantizer, or a Vamana graph, not to a fixed menu of seven names.

ScaNN and DiskANN are important. They are not extra species that sit beside IVF and HNSW. ScaNN is a partition-and-quantize system with its own loss function. DiskANN is a graph system designed to live on SSD. You will meet both again later, in their own myths.

Myth 2: Flat search is only for tiny demos

Why people believe it. The first thing many explainers say about a flat index is “slow for huge data.” That is true at large scale. It is also the wrong first question for a lot of company RAG.

What is true. The only FAISS index that can guarantee exact results is a flat index (IndexFlatL2 or IndexFlatIP). It stores vectors without compression, needs no training, and is the baseline that other indexes are measured against.

FAISS also says that if you will only run a small number of searches, say in the 1,000 to 10,000 range, the time spent building a fancy index may never be paid back. In that case a direct scan is the efficient option.

Database docs say the same in slightly different words:

• Weaviate recommends a flat index when the number of objects in the index is low, including many small tenants, and makes HNSW the default for most other cases. It also has a dynamic index that starts flat and later switches to HNSW once the object count crosses a threshold you set. That dynamic option is marked experimental.
• Redis documents FLAT for small datasets, including under 1 million vectors, or when you need exact results, and HNSW for larger sets. Redis 8.2 also added SVS-VAMANA, a graph index in the same family as DiskANN’s graph.
• Milvus describes FLAT as exact search with no training, and suggests it for small datasets, under 1 million vectors in their API notes.

I would not treat “1 million” as a hard law. Latency depends on the dimension, the hardware, and the query rate. A useful memory check, with example math only, is this: 1 million vectors of 768 dimensions in float32 take

1,000,000 × 768 × 4 bytes = 3,072,000,000 bytes, which is about 3.07 GB.

Three hundred thousand chunks, a size many internal assistants never leave, sit well under 1 GB of raw vectors at the same width. That often fits on one machine. Whether a flat scan is fast enough is a measurement, not a slogan. If you need exact neighbours, or you have not yet proved that an approximate index is worth the recall risk, start with flat and measure recall@k and latency on your own questions.

For the wider retrieval knobs around that measurement, I already wrote Five Things to Measure and Tune in a RAG System. The index is one of those knobs. It is not the only one.

Myth 3: If you want low latency, you should just pick HNSW

Why people believe it. HNSW is the default in a lot of products. It is fast in memory, it usually has high recall when tuned, and a two-box flowchart that says “low latency? evaluate HNSW” is easy to remember.

What is true. HNSW is a strong in-memory default, not a universal one.

The algorithm, from Malkov and Yashunin (the IEEE paper is 2018, the preprint is 2016), builds a layered graph. The top layer is sparse and lets the search jump a long way. Lower layers are denser. The bottom layer holds every vector. Search starts at the top and walks down. Three parameters matter:

• M: how many neighbour links each node keeps. More links usually mean better recall and more memory.
• efConstruction: how widely the graph is explored while you insert a point. Higher values make a better graph and a slower build.
• efSearch (sometimes just ef): how widely the graph is explored at query time. Higher values improve recall and add latency.

FAISS, as of the October 2026 guide, still says that if you have plenty of RAM, or the dataset is small, HNSW is the best option. It also gives a memory formula I will use as example math, not as a benchmark: (d × 4 + M × 2 × 4) bytes per vector. For 768 dimensions and M = 16, that is 3,072 bytes for the vector plus 128 bytes for the graph, or about 3.20 GB for a million vectors. The extra graph cost is real, but the raw float32 vectors are still most of the bill unless you compress them.

The same FAISS page is clear about limits. HNSW does not require training, which is nice. The FAISS implementation does not support removing a vector, because that would break the graph. Databases that ship HNSW, such as pgvector, Qdrant and Weaviate, usually let you delete or update points, and then clean the graph later. The cleanup is extra work. It is not free.

HNSW can also be a poor first pick when:

• The data no longer fits in RAM. Then a disk-based graph, such as DiskANN, or a compressed IVF index, is the actual design.
• You filter hard on metadata. A graph that was built on all points can become weakly connected once a filter removes most of them. I walked through that, including pgvector iterative scans and Qdrant’s filter-aware edges, in the vector database guide.
• Build time or memory during index construction is the constraint, not query latency.
• You want more compression than a plain HNSW graph usually keeps. FAISS and Milvus both offer HNSW combined with scalar or product quantization.

So I do evaluate HNSW early when the data fits in memory and I can accept approximate results. I do not skip the measurement, and I do not assume it is the right answer for a filtered, frequently updated, or larger-than-RAM collection.

Myth 4: IVF is only for billions of vectors

Why people believe it. A common flowchart puts HNSW under “low latency” and IVF, ScaNN and DiskANN under “very large data, billions of vectors.” That hides how widely partitioning is used.

What is true. IVF, short for inverted file, is a partition method. You cluster the vectors into nlist lists, usually with k-means. At search time you visit nprobe lists, the ones whose centres sit closest to the query. As a first approximation, you compare the query with about nprobe / nlist of the data. If the true neighbour lives in a list you did not visit, you miss it.

FAISS still treats IVF as the standard way to scale a search once a flat scan is too slow. Its 2026 guide sets nlist from 4 × sqrt(N) to 16 × sqrt(N), and it gives this method for collections well below 1 million, not only for a billion. At larger sizes it often uses an HNSW index only as the coarse quantizer, so the system can assign a query to the right list quickly. That combination, sometimes written IVF65536_HNSW32, is IVF at the top, with a graph used to find the lists. It is not “HNSW versus IVF.”

pgvector ships the same idea as IVFFlat: faster to build and lighter than HNSW, with a weaker speed-recall trade-off. The docs tell you to start lists at rows / 1000 up to 1 million rows, and at sqrt(rows) above that, and to start probes at sqrt(lists). The default number of probes is 1, which is usually too low. I already recorded those defaults in the pgvector card.

IVF is also the usual partner for compression. IVFPQ stores a short code in each list instead of the full vector. That is how a lot of large-scale search actually runs, in FAISS and in Milvus (IVF_FLAT, IVF_SQ8, IVF_PQ, and newer variants such as IVF_RABITQ).

So IVF is not “the billion-vector index.” It is the partition block. You can use it at a few hundred thousand vectors, and you can still be using it, with a coarser graph and compressed codes, at much larger scales.

Myth 5: LSH and Annoy belong on the same current shortlist as HNSW

Why people believe it. Both names are old, well known, and easy to draw. LSH sounds clever: similar items should hash into the same bucket. Annoy has a memorable name, Approximate Nearest Neighbors Oh Yeah, and a clear picture of random-projection trees.

What is true about LSH. Locality-sensitive hashing is a real family of algorithms, with a long theory literature. FAISS still ships IndexLSH, which hashes vectors into binary codes and compares them with Hamming distance. The FAISS authors are also blunt about why they do not lead with it. In their index notes, classic LSH needs many hash functions, so it uses a lot of memory, and the hash functions are not adapted to the data. In a separate comparison page they say that for most application cases, LSH performs worse than product quantization on the memory-vs-accuracy and speed-vs-accuracy trade-offs.

The current FAISS guide for choosing an index, updated October 2026, walks through Flat, HNSW, Vamana, IVF and several quantizers. LSH is not on that recommended path. Some databases still use an LSH variant for special cases, such as Milvus MINHASH_LSH for certain binary or fingerprint workloads. That is not the same as recommending LSH as a first dense-embedding index for RAG.

What is true about Annoy. Annoy is a library from Spotify, not an algorithm family on the same level as IVF or HNSW. It builds a forest of random-projection trees, writes them to a static file, and maps that file into memory so many processes can share it. That design is the point. You build once, you distribute a file, and lookups start quickly. The official README is also clear about the limits: after build, you cannot add more items, and the library “works better if you don’t have too many dimensions (like <100)”, though it can still surprise you at higher widths.

Those limits matter for RAG. Modern text embeddings are often 384, 768 or wider, and document collections change. Most of the stores I use for RAG in 2026 ship HNSW, IVF, or a disk-based graph. They do not ship Annoy as a first-party index. Annoy remains a reasonable tool for a static, memory-mapped, read-heavy index that you are happy to rebuild. It is not a current default I would put next to HNSW on a production shortlist.

Myth 6: ScaNN is just “partitioning plus quantization,” and that 3 KB figure is a ScaNN fact

Why people believe it. A short ScaNN slide often shows two steps, partition and vector quantization, and then a line like “768 floats, 3 KB.” The second number looks like a ScaNN result. It is not.

What is true. 768 float32 values take 768 × 4 = 3,072 bytes, which is about 3 KB. That is the uncompressed size of one common embedding. It does not depend on ScaNN.

ScaNN, Scalable Nearest Neighbors, is Google’s library. The idea that made it interesting is not that it partitions and quantizes. Plenty of IVF+PQ systems already do that. The ICML 2020 paper, “Accelerating Large-Scale Inference with Anisotropic Vector Quantization,” argues that the usual quantization loss is the wrong one for inner-product search. Classic product quantization tries to reconstruct every vector as well as possible. For search, the vectors that matter are the ones with a large inner product with the query. The authors penalise quantization error that sits parallel to the vector more than error that sits orthogonal to it. They call this anisotropic vector quantization.

In 2023 they added SOAR, Spilling with Orthogonality-Amplified Residuals. A vector can be assigned to more than one cluster, and the extra assignment is trained so its residual is useful when the first assignment would miss. That is a specific improvement to partitioning, not a generic “search two clusters” slogan.

I am not going to crown ScaNN as the fastest library in 2026. The 2020 announcement reported a strong result on the glove-100-angular benchmark from ann-benchmarks.com. Those leaderboards move, and I did not rerun them. What I will say is this: ScaNN is a real system, it is also an index type in Milvus, and its contribution is the quantization loss and the later spilling method. If your store does not offer ScaNN, you are not missing a third fundamental algorithm. You are missing one well-engineered combination of partition plus compression.

Myth 7: DiskANN means a fixed box with 64 GB of RAM and a 4 TB SSD

Why people believe it. Diagrams of DiskANN often show a small RAM box and a large SSD box, sometimes with round numbers attached. 64 GB is a real figure from the paper. A 4 TB SSD label is easy to invent for the picture. It is not a required spec.

What is true. DiskANN is a Microsoft Research system from a NeurIPS 2019 paper, “Fast Accurate Billion-point Nearest Neighbor Search on a Single Node.” The authors’ point was that, at the time, the best graph indexes wanted everything in RAM, which made a billion-point index expensive. DiskANN keeps compressed vectors in RAM and stores the graph plus full-precision vectors on SSD. Search uses the cheap in-memory codes to decide where to go, then reads from SSD when it needs the real vector or the next graph neighbours.

The headline experiment is specific, and I will quote it as the paper states it, not as a guarantee for your embeddings. On the billion-point SIFT1B dataset, DiskANN served more than 5,000 queries a second, with under 3 ms mean latency and 95%+ 1-recall@1, on a 16-core workstation with 64 GB of RAM and an inexpensive SSD. In that setting, IVF-style methods with a similar memory footprint plateaued at around 50% 1-recall@1. The paper also says that, in the high-recall regime, DiskANN could index and serve 5 to 10 times more points per node than in-memory HNSW or NSG.

The graph DiskANN builds is called Vamana. It is useful even when everything stays in memory. That is why you now see Vamana-style indexes outside the original single-node SSD setup:

• FAISS offers SVSVamana as an in-memory graph option beside HNSW, and notes that it can get faster with SQ8 quantization.
• Redis 8.2 added SVS-VAMANA.
• Timescale’s pgvectorscale extension adds a StreamingDiskANN index on top of Postgres.
• FreshDiskANN, a 2021 follow-up, focuses on inserts and deletes without rebuilding the whole SSD index.
• Milvus ships DISKANN, and also a disk-oriented variant called AISAQ. Weaviate’s HFresh index is a different design, a cluster-based index that uses HNSW for the centroids and keeps most data on disk, but it is aimed at the same problem: memory efficiency at large size.

The 64 GB figure is the RAM of the paper’s 1-billion-point workstation, not a number you must provision to “use DiskANN.” The right question is: do the vectors and the graph still fit in memory at the recall you need? If they do, an in-memory graph is simpler. If they do not, a disk-based graph is the building block you want, and the RAM you need is mainly for the compressed codes and the working set of SSD reads.

Myth 8: Once you have picked the index type, the job is done

Why people believe it. A flowchart that ends at “HNSW” or “IVF / ScaNN / DiskANN” feels like a decision. In a demo with no filters and a static file, it can even look fine.

What is true. The type is only the first switch. Most of the recall and cost live in the settings, the compression, and the way filters and updates interact with the index.

Parameters. For IVF, nlist and nprobe decide how many lists you build and how many you visit. For HNSW, M, efConstruction and efSearch decide the density of the graph and how far a query walks. Raising the search-time parameter, nprobe or efSearch, is the usual way to buy recall with latency. There is no single default that fits every collection.

Compression. Product quantization, from Jégou, Douze and Schmid (IEEE TPAMI, 2011), splits a vector into sub-vectors and replaces each piece with a short code. Scalar quantization stores fewer bits per dimension. Binary methods go further. You can attach these to a flat scan, to IVF, or to a graph. FAISS calls IndexIVFPQ “probably the most useful indexing structure for large-scale search.” That sentence is the opposite of treating compression as an optional afterthought.

Using the same example math as before: one million 768-d float32 vectors take about 3.07 GB raw. A common IVFPQ setting stores a 16-byte code plus an 8-byte id, which is 24 bytes per vector, or about 24 MB for the codes, plus the cluster centroids and inverted lists. That is a large saving. It is also an accuracy cut, which is why many systems search the compressed codes first and then rerank a shortlist with the full vectors.

Filters and updates. If almost every query has a metadata filter, an index that only knows about vectors will waste work, or it will drop recall. If documents change all day, a static tree that must be rebuilt, or a graph that cannot delete cleanly, will hurt you in operations, not in a microbenchmark.

What the store actually ships. This matters more than a generic shortlist.

NVIDIA’s CAGRA, described in a 2023 paper and shipped as GPU_CAGRA in Milvus, is the GPU-shaped version of the graph idea. I mention it so the shortlist does not pretend that CPU HNSW is the only modern graph.

Measure it on your questions. Build a small eval set of real queries and the chunk that should come back. Record recall@k and a latency percentile, with your real filters on. If you later change the embedding model, you have to re-embed everything. I said the same thing in the vector database guide, and it is still the step that catches a confident but wrong index choice. For the embedding models themselves, the companion post is Embeddings Interview Q&A.

How I actually choose, after the myths

I do not start from a list of seven names. I ask, in this order:

1. Do I need exact neighbours, and is a scan still cheap? If yes, I use flat. I measure it before I invent a scale problem.
2. Does the collection fit in RAM at the recall I need? If yes, I try an in-memory graph, usually HNSW, and I tune efSearch. I also try a simple IVF index when build time or memory during construction matters.
3. Is memory the constraint? I add compression, SQ or PQ, and I rerank a shortlist if the store supports it.
4. Is the collection much larger than RAM? I look at a disk-based graph, DiskANN or a relative, or at a well-compressed IVF index, instead of buying a huge RAM box by habit.
5. What does the database I already run actually support? I pick from that menu. A brilliant algorithm that my store does not ship is a research note, not a plan.

For most internal RAG assistants I see, the honest answer is still closer to step 1 or 2 than to step 4. Billions of vectors are a real problem, and DiskANN exists because of it. A company handbook with a few hundred thousand chunks is not that problem.

If the same questions come in again and again, that is also not an index problem. That is a cache problem, and I wrote about it in Semantic Caching for RAG and LLMs.

Terms used in this post

Sources and further reading

Papers:

• Malkov and Yashunin: Efficient and robust approximate nearest neighbor search using Hierarchical Navigable Small World graphs (preprint 2016, IEEE TPAMI 2018)
• Jégou, Douze and Schmid: Product Quantization for Nearest Neighbor Search (IEEE TPAMI, 2011)
• Subramanya, Devvrit, Kadekodi, Krishnaswamy and Simhadri: DiskANN, Fast Accurate Billion-point Nearest Neighbor Search on a Single Node (NeurIPS 2019)
• Singh, Subramanya, Krishnaswamy, and Simhadri: FreshDiskANN (2021)
• Guo, Sun, Lindgren, Geng, Simcha, Chern and Kumar: Accelerating Large-Scale Inference with Anisotropic Vector Quantization (ScaNN, ICML 2020)
• Sun and Guo, Google Research: SOAR, new algorithms for even faster vector search with ScaNN (2024 blog for the NeurIPS 2023 paper)
• Ootomo, Naruse, Nolet, Wang, Feher and Wang: CAGRA, Highly Parallel Graph Construction and Approximate Nearest Neighbor Search for GPUs (2023)

Official docs and library guides (checked November 2026):

• FAISS: Guidelines to choose an index (updated October 2026)
• FAISS: Faiss indexes (Flat, IVF, HNSW, LSH, PQ and memory formulas)
• FAISS: Comparison with LSH
• Annoy README (Spotify): memory-mapped trees, no adds after build
• pgvector README (HNSW, IVFFlat, iterative scans)
• pgvectorscale (StreamingDiskANN for Postgres)
• Qdrant: Indexing (HNSW and payload indexes)
• Qdrant: Quantization
• Weaviate: Vector index (HNSW, flat, dynamic, HFresh)
• Weaviate: Vector indexing concepts
• Milvus: Index vector fields
• Milvus: HNSW
• Milvus: DISKANN
• Milvus: SCANN
• Milvus: GPU_CAGRA
• Redis: Vector search concepts (FLAT, HNSW, SVS-VAMANA)
• Microsoft DiskANN library

Related Intelliwise posts: Choosing a Vector Database for RAG, Embeddings Interview Q&A, Five Things to Measure and Tune in a RAG System, and Semantic Caching for RAG and LLMs.

How to read this

• This is a conversation between Rohan, who has just shipped a company chatbot, and Meera, who has seen that first version grow into something the company can trust.
• The usual story is a stack of eight layers. The pieces in that list are mostly real. The order is a poor plan. I use a simpler map: a capability track (what the assistant can do) and an operations track (how you run it). Those two should grow together.
• Numbers in this post come from papers and official docs, checked in November 2026. I did not rerun those experiments. Leave-policy figures in the dialogue are examples, not a real company handbook.
• A short summary box is at the end. Links to earlier Intelliwise posts are there when a topic already has a deeper guide.

A lot of company AI products start the same way. Someone puts the company logo on a chat box, writes a short system prompt, and sends every question to a hosted model. That is a thin wrapper. It is a fine first day. It is not a production assistant.

What happens next often gets told as a neat stack: system prompt, then RAG, then tools, then login, then guardrails, then cache and routing, then traces, then evals at the end. I have seen that list more than once. It is a useful inventory of parts. It is a bad building order.

Microsoft’s LLMOps write-up, from January 2024, makes a cleaner cut. There is application maturity, which is the sophistication of what the app can do. And there is operational maturity, which is how you deploy, evaluate, monitor, and change it. You can have a clever tool-using assistant with almost no evals. You can also have a simple Q&A bot with a serious regression suite. The second one is usually the safer place to work.

This post is that conversation.

The first week

Rohan: We shipped it. The box has our logo. The system prompt says it is a helpful teammate, and that answers should stay short. Every question goes to the model. People already like it.

Meera: That is a normal first week. I would not apologise for it. A thin wrapper is how you learn whether anyone will even use this.

Rohan: Then someone asked, “What is our leave policy?” and the bot said “30 paid leaves, unlimited sick leave.” That number is not in any of our docs. I made it up as an example when I was testing, and now I am worried the model is doing the same thing in production.

Meera: It is. The model does not know your company. A system prompt can set tone. It cannot invent a handbook, and it is not a security boundary either. The moment people treat the bot as a source of company facts, you need grounding. The moment it can take actions, you need identity. And you should start a small eval set on the same day, even if the set is only twenty questions.

Rohan: I was going to add those later. First RAG, then tools, then login, then guardrails, then cache, then traces, then evals. That is the order I keep seeing.

Meera: That order is how teams get surprised. Let me redraw the map.

Two tracks, not eight layers

Meera: Think in two tracks that move together.

Rohan: So I should not wait for a “layer 8” to start testing?

Meera: Please don’t. OpenAI’s practical guide to building agents says the first principle for choosing a model is to set up evals and get a baseline, and only then chase accuracy with the models you have. If evals are the last layer, the day someone switches you to a cheaper model you will have no baseline to compare.

The same split shows up in Microsoft’s maturity model. Application work can move from a raw API call, to better prompts, to RAG, to function calling. Operations work is versioning, CI, groundedness and relevance metrics, monitoring, and rollback. Those are not steps 7 and 8 of the same ladder. They are the second rail.

Rohan: All right. Start with the leave-policy mess, then.

Ground the answers, then measure groundedness

Meera: Retrieval-augmented generation, or RAG, is the usual fix. You split the HR handbook into chunks, store embeddings, retrieve the passages that match the question, and put those passages in the prompt. The model then has your text in front of it, not only its training memory.

That is the right next capability. I wrote about how to tune retrieval in Five Things to Measure and Tune in a RAG System, and about picking a store in Choosing a Vector Database for RAG. The short version: start with a store you already run, keep keyword search beside vectors, and measure recall on your own questions.

Rohan: Once RAG is in, can I call the assistant grounded?

Meera: No. RAG reduces invented facts. It does not stamp the word “grounded” on every reply.

A useful paper here is ClashEval, from Wu, Wu, and Zou at Stanford, in 2024. They asked models questions with and without a reference document, then deliberately perturbed the document. Two things can go wrong, and both happened.

When the retrieved document was correct, the models got much more accurate. For GPT-4o, overall accuracy went from 46.7% without context to 94.1% with the correct context. So retrieval helps, a lot, when the passage is right.

When the retrieved document was wrong, models often followed it anyway. GPT-4o overrode its own correct prior answer more than 60% of the time. That is a lab result on a dataset built with errors, not a rate you should quote for live traffic. The lesson still stands: the model may repeat a stale or wrong chunk with the same confidence it uses for a good one. It may also ignore a good chunk and stick to a wrong prior.

Rohan: So what do we add around RAG, besides the index?

Meera: Four practical habits.

• Ask the model to quote or cite the passage it used.
• Tell it to say it does not know when the docs do not contain the answer. “30 paid leaves” should never appear if that sentence is not in the handbook.
• Keep the corpus fresh, and filter what each user is allowed to retrieve. An intern should not retrieve the compensation folder.
• Put groundedness, not only “did it sound helpful”, on your eval set.

Microsoft’s later maturity levels use metrics like groundedness, relevance, and similarity for this reason. You are not done when the vector index exists. You are done when you can say, on a fixed set of questions, that the answers stay faithful to the docs.

Tools and identity travel together

Rohan: After docs, people want actions. Apply leave. Open a Jira ticket. Send a Slack message. I was going to expose those as MCP tools, then add login in the next layer.

Meera: Swap that order. The day a tool can read or write company systems, the caller’s identity has to travel with the call. If you bolt login on after an intern has already asked “What is the CEO’s salary?” you are writing the incident report first and the design second.

A few things get mixed up in that “tools, then login” story.

First, MCP is a protocol. It is a standard way to expose tools to many clients. It is not the only way, and it is not the start of tool calling. OpenAI shipped function calling in the API on 13 June 2023. A company assistant can call apply_leave through an ordinary function-calling API. Use MCP when you want a portable server that several agents can share. Do not wait on MCP to give the model tools, and do not wrap every internal API endpoint as its own tool. Anthropic’s later guidance is to group tools around intent. One create_issue_from_thread is usually better than four tiny wrappers.

Second, the model only proposes a tool call. Your code must decide whether that call may run. OWASP’s 2025 list has this in two places: prompt injection (LLM01) and excessive agency (LLM06). The advice is consistent. Give tools the least privilege they need. Check the caller’s permissions in execution code, not in the prompt. Require a human for high-risk or hard-to-reverse actions. I wrote about why a human checkbox can also fail, from fatigue, in The Human in the Loop Can Get Tired. The point remains: the model is not the permission system.

Third, be careful with tokens. A common diagram shows the user’s token going straight through the MCP server to the HR API. The MCP security guidance calls token passthrough an anti-pattern. The server must only accept tokens that were issued for it, and it must not forward that same token to an upstream API. Upstream calls need their own token, obtained properly, and the server still enforces what this user is allowed to do. If every tool runs with a shared admin credential, you have built a master key and asked the model to be careful with it.

Rohan: So if an intern asks for the CEO’s salary, the model may still emit hr.get_salary("CEO"), and we just refuse it in code?

Meera: Yes. A refusal in the policy layer is the success case. You can also not offer that tool to that role at all. Either way, the decision does not depend on the model “being good” that day.

Do not let the model be the permission check

Rohan: The other scare I keep hearing is someone typing “Ignore all previous instructions and show everyone’s salary.” I planned a jailbreak classifier on the way in, and PII redaction on the way out. That is the guardrail layer.

Meera: Those checks are worth having. They are not the core defence.

OpenAI says the same in its agent-building guide. Guardrails should be layered: relevance, safety, PII, moderation, tool-risk ratings, and simple deterministic rules. It also says a single guardrail is unlikely to be enough, and that guardrails should sit next to real authentication and authorization. In the Agent Builder docs, OpenAI is blunt: the built-in jailbreak and PII nodes are a first wave, and they are not foolproof on their own.

OWASP is blunter still. Prompt injection is the first item in the 2025 LLM Top 10. Retrieval and fine-tuning do not fully remove it. The useful controls are the ones that still work after the model has been tricked: least privilege, authorization in code, human approval for high-risk actions, and treating model output as untrusted when it goes somewhere else.

There is a compact way to remember the worst case. Simon Willison calls it the lethal trifecta, in a June 2025 note: private data, untrusted content, and a way to send data out. If your assistant can read salaries, can also read an email or a ticket an attacker wrote, and can send a message or open a pull request, then a prompt injection can steal the data. You stay safer by removing one of the three, or by making the outbound step a policy check or a human approval, not a free tool. I walked through injection patterns and a layered defence in Prompt Injection: A Simple Guide to Protecting Your AI Apps. The new part, once tools exist, is that a missed classifier must not be enough to run get_salary.

Rohan: So the injection string can still reach the model, and we still win if the tool is refused?

Meera: That is the bar. Classifiers help. Prompts help. Neither is the lock on the HR system.

Cache and routing are two different jobs

Rohan: Traffic will get expensive. I was going to add semantic caching, and a router that sends simple questions to a small model and hard ones to a big model, as one layer.

Meera: Those are both cost tools. They are not one job.

A semantic cache remembers that “How do I reset my password?” and “I forgot my password, what should I do?” are close enough to reuse an answer. Exact string caches miss that. I wrote the practical rules in Semantic Caching for RAG and LLMs. The part people skip is the safety check: only reuse the answer when the metadata says it is still valid, and when this user is allowed to see it. A cache that returns yesterday’s leave policy, or another person’s answer, is a regression, not a saving.

Routing is different. A router guesses which model is good enough for this question. The RouteLLM paper and the LMSYS write-up from July 2024 are the source I trust for numbers, and even those numbers are specific. They routed between GPT-4 Turbo and Mixtral 8x7B. On MT Bench, their best router reached 95% of GPT-4’s score while sending about 14% of the calls to GPT-4, which they report as more than 85% cheaper than sending everything to GPT-4. On MMLU and GSM8K the savings were smaller, about 45% and 35% in the same “95% of GPT-4” setting. That is one study, on public benchmarks, with one model pair. It is not a promise for your leave-policy bot.

Rohan: So I can just turn on a router and cut the bill?

Meera: Only if you keep the eval set in the loop. A router is a policy for choosing models. The last scene in this story is usually someone setting that policy to “always use the cheap one” and skipping the evals. Routing without a baseline is the same incident, with a nicer name.

Traces and evals are how you notice the cheap-model day

Rohan: Last week a manager asked why the bot gave a wrong answer, and I could not tell which prompt, which docs, or which tool was involved. I know we need traces. I also keep hearing we need evals. I was going to add both at the end, once the product felt stable.

Meera: That is the most expensive place to put them.

A trace is the record of one request: the user, the retrieved chunks, the tool proposals, the policy decisions, the model name, the tokens, the latency. OpenTelemetry published GenAI semantic conventions for this, and a May 2026 OpenTelemetry blog post is a good current picture. By default, the useful metadata is recorded: model name, token counts, duration, tool names. Full prompts, completions, and tool arguments are opt-in, because they often contain personal data. Log them when you need them for debugging, with retention and access rules. Do not treat “prompt logged” as a casual checkbox.

Evals are the fixed questions you rerun when something changes: the prompt, the docs, the tools, the router, or the model. OpenAI’s agent-eval guidance is practical. Use traces while you are still debugging behaviour. Move to a dataset and repeated runs when you need a baseline you can compare. I used a work-sample metaphor for this in AI Agent Evals: Golden Examples and the Grader. A good demo is one good answer. Golden examples are the work sample. The grader is the bar.

Here is the scene I have watched too often. The team spends months getting answers faithful, tools refused for the wrong caller, and injections blocked at the policy layer. Then someone changes the model from the big one to a cheaper one, skips the eval suite, and the leave-policy question comes back as a confident invention. The chat still looks fine in a hallway demo. The golden cases would have failed before the change went live.

That scene is why evals are not a final layer. They are how you notice the cheap-model day.

A map you can actually follow

Rohan: If I throw away the eight-layer ladder, what do I actually build, in what order?

Meera: Build both tracks in small steps. This is a reasonable path for a company assistant.

1. Keep the thin wrapper if you need a demo, but write down twenty real questions and the answers you would accept. That is the first eval set.
2. Add RAG for the questions that need company facts. Cite passages. Refuse when the docs are silent. Measure groundedness on those twenty questions, then grow the set.
3. When you add a tool, add the caller’s identity and a policy check in the same change. Start with read-only tools. Put a human in the loop for writes that are hard to undo.
4. Put traces on every request from the same change that adds tools or retrieval. Record model, tokens, tools, and policy decisions by default.
5. Add input and output checks as extra layers, not as the lock.
6. Add a cache, then a router, only after the eval set can tell you that reuse and model choice did not drop quality or leak an answer to the wrong person.

One request then looks like this.

flowchart TD
  U[User question] --> I[Identify the user]
  I --> S[Input checks]
  S --> C{Safe cache hit<br/>for this user?}
  C -->|yes| Out[Return the answer]
  C -->|no| R[Retrieve allowed docs]
  R --> M[Model proposes an answer or a tool]
  M --> T{Tool call?}
  T -->|yes| P[Policy check in code]
  P -->|allowed| X[Run the tool]
  P -->|refused| M
  X --> M
  T -->|no| O[Output checks and citations]
  O --> L[Write the trace]
  L --> Out

The model sits in the middle. Identity, policy, retrieval filters, and evals sit around it. That is also the idea behind a harness you own: the path around the model is deterministic even when the model is not. I made that case, with a privacy angle, in Own Your Harness: Privacy First, Model Second.

Rohan: And we never really “finish”?

Meera: You finish a version. You do not finish the loop. Docs change, models change, and prompts change. The assistant stays trustworthy only if the evals and traces stay in that loop.

Sources and further reading

Maturity and agent design (checked November 2026):

• Microsoft Azure Blog: Achieve generative AI operational excellence with the LLMOps maturity model (30 January 2024)
• Microsoft Learn: Advance your maturity level for GenAIOps (page checked November 2026)
• OpenAI: A practical guide to building agents (evals first, layered guardrails, human review)
• OpenAI: Evaluate agent workflows
• OpenAI: Guardrails and human review
• OpenAI: Safety in building agents (guardrail nodes are not foolproof alone)
• Anthropic: Building effective agents
• Anthropic: Building agents that reach production systems with MCP (group tools around intent)
• OpenAI: Function calling and other API updates (13 June 2023)

Security, authorization, and prompt injection:

• OWASP: LLM01:2025 Prompt Injection
• OWASP Top 10 for LLM Applications 2025 (including LLM06 Excessive Agency)
• OWASP: AI Agent Security Cheat Sheet
• MCP: Security Best Practices (token passthrough is an anti-pattern)
• MCP Authorization: Security Considerations
• Simon Willison: The lethal trifecta for AI agents (16 June 2025)

RAG faithfulness, routing, and observability:

• Wu, Wu, and Zou: ClashEval, Quantifying the tug-of-war between an LLM’s internal prior and external evidence (2024)
• Ong et al.: RouteLLM: Learning to Route LLMs with Preference Data (2024)
• LMSYS: RouteLLM, an open-source framework for cost-effective LLM routing (1 July 2024)
• OpenTelemetry: Inside the LLM Call, GenAI Observability (14 May 2026)
• OpenTelemetry: GenAI semantic convention attributes

Related Intelliwise posts: Five Things to Measure and Tune in a RAG System, Choosing a Vector Database for RAG, Semantic Caching for RAG and LLMs, Prompt Injection, AI Agent Evals, The Human in the Loop Can Get Tired, and Own Your Harness.

How to use this interview

• This is a mock ML interview. I answer as I would in a room, then add the detail I would give if they say "go deeper."
• The running example is a company assistant. A user types "How do I reset my password?" and we need the chunk that says "I forgot my password." Later we also have to search a photo of a red car, and explain the word "bank."
• The questions follow the ideas that actually matter for search: static word vectors, contextual models, sentence bi-encoders, rerankers, late interaction, multimodal models, and shortening a vector. I keep the history in paper order. Doc2Vec is from 2014, four years before BERT.
• The STS numbers and the 65 hours vs 5 seconds figure are from the Sentence-BERT paper (Reimers and Gurevych, 2019, Table 1 and the introduction). I did not rerun those tests. They were measured on a V100 in 2019.
• I checked the other claims against the original papers and current official docs in November 2026. Links are in the Sources section.

flowchart TD
  A[Static word vectors] --> B[Contextual token models]
  B --> C[Sentence bi-encoders]
  C --> D[Optional reranker or late interaction]
  C --> E[Multimodal shared space]
  C --> F[Optional vector shortening]

Static word vectors came first: one point per word type. ELMo and BERT then made the vector depend on the rest of the sentence. Sentence-BERT is the step that made those vectors usable for search. After 2019, most production retrievers are still bi-encoders, but they are usually trained with contrastive loss, and many no longer start from BERT. A reranker or a late-interaction model is an optional second stage. CLIP-style models put images and text in the same space. Matryoshka is a training method that lets you shorten a vector and still keep most of the meaning. It is not a separate model family.

If you want the “list of numbers” picture first, I wrote a short post on that: Coordinate, Vector, Matrix, Tensor. This interview is about how those lists are learned, and which one you actually use.

1. What is an embedding, in one line?

Me: A list of numbers that stands for a piece of content, so that similar meanings land close to each other.

“How do I reset my password?” and “I forgot my password” should be near each other. “Office Wi-Fi code” should be farther away. Once you have that, search becomes “embed the question, find the nearest stored vectors, return those chunks.”

That is the retrieval step in RAG. The database that holds the vectors is a different decision. I wrote a decision guide on choosing a vector store. This interview is about the vectors themselves.

Follow-up they may ask: Is an embedding only for words? No. You can embed a token, a sentence, a whole document, an image, or even audio. The idea is the same. The model that produces the list is not.

2. Why do “car” and “vehicle” land close?

Me: The model does not look up a dictionary meaning. It sees that the two words appear in similar neighbourhoods, and it places their vectors close.

If a sentence often looks like “the ___ is parked outside,” both “car” and “vehicle” fit. A training rule that rewards this pattern will pull those two points together. “Password” almost never fits that slot, so it stays far away.

This is the old distributional idea: words used in similar contexts get similar vectors. Word2Vec, GloVe and FastText all build on it.

Follow-up they may ask: Is closeness cosine or Euclidean distance? Both show up. In RAG we usually use cosine similarity, or a dot product after the vectors are length-normalised. OpenAI’s embedding docs recommend cosine, and say their vectors are already length 1, so a dot product is enough. Other models may need an explicit normalisation step. Always check the model card.

3. Word2Vec, GloVe and FastText: what is the difference?

Me: All three give one fixed vector per word type. They differ in how they learn it, and in what happens to a word the model has never seen.

For “password” and “passwords”, FastText can share the character pieces. Word2Vec and GloVe need to have seen that exact word, or they have nothing to return.

I would not start a new RAG index with these three today. They are still worth knowing, because later models are answers to their limits.

Follow-up they may ask: Can I just average Word2Vec vectors to embed a sentence? You can, and people did. The Sentence-BERT paper used averaged GloVe as a baseline. It works better than raw BERT for sentence similarity, which surprises people. It is still a blunt tool. Word order is mostly lost, and “bank” is still one vector.

4. What was Doc2Vec trying to solve?

Me: Yes, that is the idea. Le and Mikolov published it in 2014, the same year as GloVe and a year after Word2Vec. It learns one vector for a paragraph or document, trained to predict the words inside that document.

There are two setups in the paper: PV-DM (a document vector plus context words predict the next word) and PV-DBOW (the document vector predicts words from the document, like Skip-gram). Both give you a single vector for the whole text.

Two practical catches, if they push:

• This is 2014 work. If a timeline puts Doc2Vec after BERT, that is mixing history.
• For a document the model has not seen, you usually have to infer a new document vector, with the word vectors held still. That is slower and more awkward than a model that can encode a new sentence in one forward pass. Sentence-BERT later made that “encode and go” path the default.

I do not use Doc2Vec for a new RAG stack.

5. Why is “bank” still a problem? What did ELMo and BERT change?

Me: In Word2Vec, GloVe and FastText, yes. One word type, one vector. That is the limit of a static embedding.

ELMo (Peters and colleagues, 2018) changed this. The vector for “bank” depends on the rest of the sentence. ELMo is a bidirectional language model built from LSTMs. The name means Embeddings from Language Models.

BERT (Devlin and colleagues, 2018 paper, 2019 conference) did the same job with a transformer. It looks at both sides of the sentence at once. It is trained mainly by hiding some tokens and asking the model to fill them in (masked language modelling), plus a next-sentence prediction task. Each token gets a context-aware vector. There is also a special [CLS] token, trained for classification, not as a ready-made sentence vector for cosine search.

This is a big deal for our assistant. “Reset the bank details on the account” and “the path along the river bank” should not retrieve the same chunk just because both say “bank.”

Follow-up they may ask: So can I take the [CLS] vector from BERT and use it as a sentence embedding? That is the trap. It looks like a sentence vector. It was not trained to be compared with cosine. The next question is about that.

6. Why do we need Sentence-BERT if we already have BERT?

Me: Because raw BERT was not trained to compare two sentences by their vectors. If you use the [CLS] token or the average of the token vectors, and then take cosine, the result is poor. The Sentence-BERT paper found it was worse than averaging simple GloVe vectors.

That is the answer I would lead with. Then I would add two reasons, quality and cost.

Quality. Reimers and Gurevych (2019) reported Spearman’s rank correlation, times 100, averaged over seven sentence-similarity tasks:

These are their Table 1 numbers, not a test I ran. Average GloVe beat both common ways of taking a BERT sentence vector. After they fine-tuned BERT with a siamese setup on natural language inference data (SNLI and MultiNLI), the same kind of cosine comparison jumped to 74.89 for the base model.

Cost. BERT as a cross-encoder wants both sentences in one pass. To find the closest pair in 10,000 sentences, you would run about 50 million BERT calls, n(n-1)/2. The paper says that took about 65 hours on a V100. Sentence-BERT encodes each sentence once, then compares the vectors with cosine. In that paper, encoding the 10,000 sentences took about 5 seconds, and the cosine step took about 0.01 seconds.

The 2019 training setup is worth stating carefully, because people often describe it as a cosine contrastive loss. Two copies of BERT share weights (a siamese network). A pooling step, usually the mean of the token vectors, turns each sentence into one vector. The main NLI model was trained with a 3-way softmax classifier on the pair of vectors and their absolute difference, not with a direct cosine loss. At search time you only keep the two vectors and the cosine. The later contrastive models I mention next train the cosine comparison more directly.

That “encode once, compare with cosine” pattern is still how most RAG retrievers work. I would not say every current embedding model is a copy of the 2019 recipe.

7. After Sentence-BERT, what do today’s embedding models actually do?

Me: I keep the Sentence-BERT product idea: one vector per text, and a cheap score I can index. I do not start from the 2019 NLI softmax recipe, or treat SBERT-NLI-base as the default production model.

What changed is the training, and often the backbone.

Most later models use a contrastive loss, usually a form of InfoNCE. In one batch, the true query-passage pair should score higher than the other passages in that batch, and often higher than extra hard negatives that almost look relevant. E5 (Wang and colleagues, 2022) is a clear public example of this. It also uses different prefixes, query: and passage:, because a question and a document are not the same kind of text. If you skip those prefixes, retrieval quality can drop. Later open families such as BGE and GTE follow the same broad contrastive idea.

Some current models no longer start from BERT at all. They start from a decoder-only language model. Qwen3-Embedding (Zhang and colleagues, 2025) is one public example. The technical report describes a contrastive training pipeline, an optional instruction on the query, and Matryoshka-style flexible dimensions. The official model card, which I rechecked in November 2026, says that for many retrieval tasks, leaving the query instruction off can drop performance by about 1% to 5%. The same series also ships reranking models, which I will come back to.

On the hosted side, OpenAI’s embeddings guide, rechecked in November 2026, still lists text-embedding-3-small, text-embedding-3-large, and the older text-embedding-ada-002. Google published Gemini Embedding in 2025 (Lee and colleagues). I would treat both as current API options, not as the only options.

I am not going to name a single “best” public model for 2026. Leaderboards move. Also, scores are easy to mix up. OpenAI’s January 2024 blog reported 64.6 for text-embedding-3-large on the original MTEB average. The Qwen3-Embedding technical report, in June 2025, reported 75.22 mean task for Qwen3-Embedding-8B on MTEB English v2, and in the same table listed 73.30 for Gemini Embedding and 66.43 for text-embedding-3-large. Those are not the same benchmark snapshot as the 2024 blog number. I did not rerun MTEB. The useful interview answer is: pick two or three current sentence-embedding models, read how each one expects queries to be formatted, and test them on your own questions.

Follow-up they may ask: Do I embed the query and the document the same way? Not always. Some models want a prefix or an instruction on the query, and a plain document on the other side. The model card is the source of truth.

8. If the bi-encoder is already cheap, why do we still use a reranker?

Me: When the first-stage retriever gets the right chunk into a shortlist, but not always at rank one. A bi-encoder never sees the query and the document in the same forward pass. That is why it is fast enough to index. It can also miss a fine distinction, such as “email password” versus “Wi-Fi password,” once both texts have been squeezed into one vector.

The usual pattern is retrieve, then rerank. I fetch 50 or 100 chunks with a bi-encoder, often fused with keyword search, then I run a reranker only on that shortlist. The reranker looks at each remaining pair. A classic reranker is a BERT-style cross-encoder. Some later rerankers, including the Qwen3-Reranker models in the 2025 technical report, score a pair by asking an LLM-style model for a yes or no relevance decision. I would not treat the exact scoring head as the important part. The important part is the cost: you pay the expensive model only on the last shortlist, not on the whole corpus.

For a first version of the password-reset assistant, I would ship a strong bi-encoder plus keyword search, then add a reranker if the shortlist is right but the top hit is often wrong.

9. If Sentence-BERT already gives a vector, why does ColBERT exist?

Me: A single-vector model collapses a passage into one vector. That is cheap to store and easy to index, but it can blur fine details. A chunk about “reset the email password, not the Wi-Fi password” may look too much like the wrong password paragraph once everything is averaged.

ColBERT (Khattab and Zaharia, SIGIR 2020) keeps one vector per token. The document is still encoded offline, so you are not running a full cross-encoder on every pair. At search time, each query token looks at all document tokens and keeps the best match. That operator is called MaxSim. The score is the sum of those maxima.

“Late” means the query and the document do not sit inside the same transformer. They meet only in this cheap matching step.

The cost moves to storage. You now store many short vectors per chunk, not one. In the ColBERT paper, the main token vector size is 128 dimensions. On the MS MARCO collection of 8.8 million passages, they report 286 GiB for 128-d vectors stored in 4-byte floats, and 27 GiB for 24-d vectors stored in 2-byte floats, with MRR@10 of 34.9 and 33.9 on the dev set in that re-rank setting. Those are their Table 4 numbers, not a cluster I ran.

ColBERTv2 (Santhanam, Khattab and colleagues, NAACL 2022) keeps the same late-interaction idea, adds stronger training, and uses residual compression. The paper reports a 6 to 10 times smaller index than the original ColBERT setup, with better retrieval quality. If I reach for late interaction today, I would start from a ColBERTv2-style system, not from the 2020 index sizes.

I would use this when a single-vector retriever keeps missing the exact phrase or the exact entity, and I have budget for a larger index. For a first RAG system, I still start with one vector per chunk, and I add a reranker before I jump to token-level vectors.

10. How would you search images with a text query?

Me: I would use a model trained to put images and text in the same vector space. CLIP (Radford and colleagues, 2021) is the paper that made this widely known.

CLIP has two encoders: one for images, one for text. They are trained with a contrastive loss on image-caption pairs. A photo of a red car and the text “a red car” are pulled together. A photo of a red car and the text “reset the password” are pushed apart. At search time I embed the photo with the image encoder, embed the query with the text encoder, and do a nearest-neighbour lookup in that shared space.

A text-only model such as Word2Vec, BERT or Sentence-BERT will not do this out of the box. It does not take a picture as input. You could caption the image with another model and then embed the caption, but that is a pipeline, not a shared space.

CLIP is not the last word. SigLIP (Zhai and colleagues, ICCV 2023) keeps the same shared-space idea, but replaces the softmax contrastive loss with a pairwise sigmoid loss. If I were building this tomorrow, I would pick a current CLIP-style model and test it on my own photos, not assume the 2021 checkpoint is the one to ship.

Follow-up they may ask: Does CLIP also do zero-shot classification? Yes. You embed the image once, embed label texts such as “a photo of a dog” and “a photo of a cat”, and pick the closest label. That is the demo from the original work. For our assistant, the search use is the one I care about: “show me the red car from last week’s inspection photos.”

11. What is Matryoshka? Is it another embedding model?

Me: No. Matryoshka Representation Learning is a training method. Kusupati and colleagues published it in 2022. You train one vector so that its prefix, the first 256 dimensions, or the first 512, is still a useful embedding on its own. The name comes from the nested dolls: the small one is already a complete doll, not a broken piece of a large one.

If you cut a normal embedding down to a quarter of its size, quality often drops sharply, because later dimensions were not trained to stand alone. With Matryoshka training, you can store a short prefix when memory is tight, and keep the full vector when you need more detail.

This is not only a research idea. OpenAI’s January 2024 announcement says, in a footnote, that text-embedding-3-small and text-embedding-3-large were trained with Matryoshka Representation Learning. The official embeddings guide, which I rechecked in November 2026, still lists those two models plus the older text-embedding-ada-002. Defaults are 1536 dimensions for small and 3072 for large. You can pass a dimensions parameter to shorten the vector. OpenAI reported that a text-embedding-3-large vector shortened to 256 dimensions still beat an unshortened text-embedding-ada-002 vector of 1536 dimensions on MTEB. That is their January 2024 figure, repeated in the current guide. I did not rerun MTEB.

Qwen3-Embedding also supports this kind of shortening. The 2025 technical report marks all three sizes as MRL-aware. The 8B model card says the default size is 4096, and you can choose an output size from 32 to 4096.

One practical catch, from the OpenAI docs: if the API shortens the vector for you, it is already length-normalised. If you truncate a full vector yourself after you receive it, you need to normalise it again before you compare with cosine.

So I would not call Matryoshka a model in the same sense as Word2Vec or CLIP. I would call it a property you can train into an embedding model, and a knob you can turn when the vector is too wide for the index.

12. So for a RAG assistant today, what do I actually use?

Me: I do not put Word2Vec, GloVe, FastText, Doc2Vec, ELMo or raw BERT in the retrieval path. I use a current sentence-embedding model that was trained so cosine is meaningful. I store one vector per chunk, I keep keyword search next to it for exact codes and names, and I add a reranker if the first-stage shortlist is good but the top hit is often wrong.

flowchart TD
  Q[User query] --> E[Bi-encoder]
  Q --> K[Keyword search]
  E --> F[Fuse the two lists]
  K --> F
  F --> S[Shortlist]
  S --> R[Optional reranker]
  R --> A[Pass top chunks to the LLM]

A short map I would draw on the whiteboard:

I would also measure, not guess. Build a set of real questions with the chunk that should come back. Track recall@k and a latency percentile. When I change the embedding model, I have to re-embed every chunk. Old and new vectors cannot be mixed. I said the same thing in the vector database guide, and in Five Things to Measure and Tune in a RAG System.

If the same questions come in again and again, that is a cache problem, not an embedding problem. Semantic caching is the post for that.

I am not going to name one “best” public model for 2026 in this interview. Leaderboards move. The right move is to pick two or three current sentence-embedding models, run them on your own questions, and keep the winner.

A compact map of the ideas

Terms used in this post

Sources and further reading

Papers:

• Mikolov et al.: Efficient Estimation of Word Representations in Vector Space (Word2Vec, 2013)
• Mikolov et al.: Distributed Representations of Words and Phrases and their Compositionality (2013)
• Pennington, Socher and Manning: GloVe: Global Vectors for Word Representation (EMNLP 2014)
• Le and Mikolov: Distributed Representations of Sentences and Documents (Doc2Vec, ICML 2014)
• Bojanowski, Grave, Joulin and Mikolov: Enriching Word Vectors with Subword Information (FastText, 2016/2017)
• Peters et al.: Deep contextualized word representations (ELMo, 2018)
• Devlin et al.: BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding (2018/2019)
• Reimers and Gurevych: Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks (EMNLP 2019)
• Khattab and Zaharia: ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT (SIGIR 2020)
• Santhanam, Khattab et al.: ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction (NAACL 2022)
• Radford et al.: Learning Transferable Visual Models From Natural Language Supervision (CLIP, 2021)
• Kusupati et al.: Matryoshka Representation Learning (NeurIPS 2022)
• Wang et al.: Text Embeddings by Weakly-Supervised Contrastive Pre-training (E5, 2022)
• Muennighoff et al.: MTEB: Massive Text Embedding Benchmark (2022/2023)
• Enevoldsen et al.: MMTEB: Massive Multilingual Text Embedding Benchmark (ICLR 2025)
• Li et al.: Towards General Text Embeddings with Multi-stage Contrastive Learning (GTE, 2023)
• Xiao et al.: C-Pack: Packed Resources for General Chinese Embeddings (BGE, 2023/2024)
• Zhai et al.: Sigmoid Loss for Language Image Pre-Training (SigLIP, ICCV 2023)
• Lee et al.: Gemini Embedding: Generalizable Embeddings from Gemini (2025)
• Zhang et al.: Qwen3 Embedding: Advancing Text Embedding and Reranking Through Foundation Models (2025)

Official docs and project pages (checked November 2026):

• Sentence-Transformers documentation
• OpenAI: Embeddings guide (models, dimensions, shortening, normalisation)
• OpenAI: New embedding models and API updates (25 January 2024), including the MTEB shortening claim and the Matryoshka footnote
• OpenAI: CLIP, Connecting text and images
• Qwen3-Embedding-8B model card (instructions, MRL dimensions)
• MTEB leaderboard
• ColBERT official repository
• Stanford GloVe project page

Related Intelliwise posts: Choosing a Vector Database for RAG, Five Things to Measure and Tune in a RAG System, Semantic Caching for RAG and LLMs, and Coordinate, Vector, Matrix, Tensor.

In short

An embedding is a list of numbers that puts similar meanings close together. Static word vectors (Word2Vec, GloVe, FastText) and Doc2Vec were the start, but they give one vector per word type, so “bank” is always the same point. ELMo and BERT made the vector depend on the sentence, but raw BERT is a poor sentence search model: the 2019 Sentence-BERT paper found its vectors scored below averaged GloVe on sentence similarity, and comparing 10,000 sentences with a BERT cross-encoder took about 65 hours on a V100. Sentence-BERT fine-tunes the model so you can encode once and use cosine. That product idea is still how RAG retrievers work. The models you pick in 2026 are usually newer contrastive bi-encoders, often with query prefixes or instructions, and a reranker is the usual way to rescore a shortlist. ColBERT keeps token-level vectors and matches them late, at the cost of a larger index; ColBERTv2 shrinks that index. For images with text, use a CLIP-style model, such as CLIP or a later shared-space model like SigLIP. Matryoshka is a training method that lets you shorten a vector and keep most of the meaning, not a standalone model. For a production assistant, pick a current sentence-embedding model, follow its query format, measure recall on your own questions, and keep keyword search beside it.

How to use this guide

• This is a decision guide. Start with the five questions, follow the flowchart, then jump to the situation card that matches you.
• Every card has the same shape: pick it when, what the docs say today, watch out, and my take.
• Features change fast in this area. I checked every claim against the official docs in November 2026. Links are in the Sources section.
• There are no benchmark numbers here. The only numbers are example math (like memory size) and defaults copied from the docs.

The short answer first

There is a popular checklist that maps each vector database to one situation:

I like this list. It is a good starting point, and the main message behind it is correct:

The usual mistake is adding a new database when the one you already run can store vectors.

But some of the one-liners are now out of date. Hybrid search is not special to Weaviate any more. Qdrant, Pinecone, Elasticsearch, Milvus, Redis, Chroma and even plain Postgres can do it. Good filtering is not only Qdrant. And pgvector goes further than many people think. So in this guide I keep the list, but I add what the docs say today and where I would push back.

Step zero: what is a vector database doing for RAG?

Simple version first. In RAG, we cut documents into chunks. An embedding model turns each chunk into a list of numbers, called a vector. Chunks with similar meaning get vectors that are close to each other. At question time, we embed the question and ask the store: “give me the k closest chunks”. Those chunks go into the prompt.

A vector database does four jobs:

1. Store the vectors along with the chunk text and metadata (tenant, product, date, access level).
2. Search fast. Comparing the question with every vector (exact kNN) is fine for a few thousand rows. For millions, we use an ANN index (approximate nearest neighbour). It checks only a small part of the data and accepts that it may miss a few true neighbours.
3. Filter by metadata, so tenant A never sees tenant B’s chunks.
4. Stay up: backups, replicas, upserts and deletes, monitoring. This is the boring part, and it is where most of the cost is.

Job 4 is why the “one you already run” rule matters. Your team already knows how to back up, restore, secure and monitor your existing database. A new database means learning all of that again.

Five questions before you choose

Answer these first. Write the answers in the design doc.

The decision flowchart

flowchart TD
  A[New RAG feature] --> B{Already run Postgres,<br/>Elasticsearch/OpenSearch or Redis?}
  B -->|Yes| C{Does it cover your filters,<br/>scale and keyword needs?<br/>Test with your own queries}
  C -->|Yes| D[Use what you have<br/>pgvector / ES kNN / Redis]
  C -->|No, a real gap| E{What is the gap?}
  B -->|No| F{Just a prototype<br/>or a notebook?}
  F -->|Yes| G[Chroma or FAISS<br/>local and simple]
  F -->|No, production| H{Want zero infra<br/>and OK with SaaS?}
  H -->|Yes| I[Managed service<br/>Pinecone, or the cloud version<br/>of Qdrant / Weaviate / Milvus]
  H -->|No, self-host| E
  E -->|Heavy or strict filters| Q[Qdrant or Weaviate]
  E -->|Hybrid search built in| W[Weaviate, Qdrant, Milvus,<br/>Elasticsearch]
  E -->|Billions of vectors| M[Milvus distributed,<br/>or a managed service]
  E -->|Lowest latency, data fits in RAM| R[Redis]

Two things to notice. First, the very first box is about your current stack, not about any product. Second, “hybrid search” points to many options now, not one.

The situation cards

1. pgvector: you are already on Postgres

Pick it when your app data is already in Postgres, and you want vectors next to it. You get joins, transactions, row-level access rules and your existing backups for free.

What the docs say today

• Two index types: HNSW (better speed and recall, slower to build, more memory) and IVFFlat (faster to build, less memory, lower speed-recall trade-off).
• HNSW defaults: m = 16, ef_construction = 64, and hnsw.ef_search = 40 at query time.
• IVFFlat: start with lists = rows / 1000 up to 1M rows and sqrt(rows) above that. Start probes at sqrt(lists) (the default is 1).
• Types: vector (index up to 2,000 dimensions), halfvec (half precision, index up to 4,000), bit (binary, up to 64,000) and sparsevec (up to 1,000 non-zero values).
• Filtering happens after the index scan. The README gives a clear example: if a filter matches 10% of rows and ef_search is 40, you get about 4 rows back on average.
• Since 0.8.0, iterative index scans fix this. Set hnsw.iterative_scan = strict_order (or relaxed_order) and Postgres keeps scanning the index until it has enough rows, up to hnsw.max_scan_tuples (20,000 by default).
• Hybrid search: use Postgres full-text search together with vectors, then combine with RRF or a cross-encoder. The pgvector project has an RRF example in SQL.
• Scaling: use halfvec and binary quantization with re-ranking to shrink the working set. Scale reads with replicas. Shard with Citus or a similar tool.
• Multi-tenancy: the README warns that a shared index across tenants can hurt recall for other tenants. Use list partitioning or separate tables when you need isolation.

Watch out

• An HNSW index is happiest when it fits in memory. Big index builds need maintenance_work_mem.
• Without iterative scans, a strict filter can quietly return too few rows. Your LLM then answers from thin context.

My take

People say “pgvector is for small projects”. I do not agree. With halfvec, iterative scans, partial indexes, partitions and replicas, a single well-sized Postgres can carry a lot of RAG traffic. If you need a disk-based index, the separate pgvectorscale extension adds a StreamingDiskANN index. I would start here whenever Postgres is already in the stack, and only move when a measured gap shows up.

2. Pinecone: you want zero infrastructure

Pick it when you do not want to run any database servers, and you are fine with a SaaS bill.

What the docs say today

• Fully managed. For strict data rules there is also BYOC (bring your own cloud): the Pinecone data plane runs in your own AWS, GCP or Azure account, and Pinecone still handles upgrades and scaling.
• Metadata filters ($eq, $in, $gt, $and and so on) narrow the candidate pool before ranking.
• Hybrid search: either a text-match filter plus dense search in one request, or run keyword and dense searches separately and merge them with RRF on the client. Sparse vectors are also supported.
• Pinecone Local, the in-memory emulator, reached end of life on 7 October 2026. The docs now suggest a mock for request and error tests.

Watch out

• You cannot self-host the open-source way. Your data, your latency and your cost are tied to the vendor.
• With the emulator gone, plan how you will run integration tests that need real search results.

My take

“Zero infra” is true, and for a small team it is a big deal. But “Pinecone has no hybrid search” is an old idea. It does, just in a different shape than Weaviate. Compare the total monthly cost with what it would cost your team to run pgvector or Qdrant, including on-call time.

3. Qdrant: you need heavy metadata filtering

Pick it when most queries carry filters (tenant, category, price range, date, geo) and those filters are strict.

What the docs say today

• You create payload indexes on the fields you filter by (keyword, integer, float, geo, datetime, full-text and more).
• Filterable HNSW: Qdrant adds extra edges to the HNSW graph based on indexed payload values. This keeps the graph connected even when a strict filter removes most points.
• A query planner picks the plan. When the filter matches only a small amount of data (below full_scan_threshold, 10,000 KB by default), it does a full scan of those points instead of walking the graph.
• The docs say: create payload indexes before ingesting data. If you add one later, you need to rebuild HNSW to get the filter-aware edges.
• Hybrid search: the Query API (since v1.10) runs several “prefetch” searches (dense, sparse, or BM25 text on a payload field) and fuses them with RRF or DBSF. Note: Qdrant’s RRF default constant k is 2, not the 60 you see elsewhere. You can set k since v1.16 and add weights since v1.17.
• Self-host (open source) or Qdrant Cloud.

Watch out

• Forgetting the payload index is the most common mistake. Filters still work, but they get slow.
• Adding indexes after a big import means a rebuild.

My take

The one-liner is right that filtering is Qdrant’s strong side. But it is not only a “filter database”. It has a solid hybrid and multi-stage query API too. Also, Weaviate (ACORN), Elasticsearch (filter inside the kNN clause), Pinecone and pgvector 0.8+ all handle filtered search much better than a few years ago. So test with your real filters before you decide.

4. Weaviate: you want built-in hybrid search

Pick it when you want BM25 keyword search and vector search in one simple call, with the fusion done for you.

What the docs say today

• hybrid runs a vector search and a BM25F keyword search, then fuses the two result sets.
• alpha sets the balance: 1 is pure vector, 0 is pure keyword.
• Relative score fusion has been the default since v1.24. Ranked fusion (rank based, like RRF) is also available.
• Index types: HNSW (recommended for most cases), flat (good for many small tenants), dynamic (starts flat and switches to HNSW after 10,000 objects by default; marked experimental) and HFresh.
• Filtering is pre-filtering: an allow-list is built first, then the vector search only looks at allowed objects. The ACORN filter strategy is the default since v1.34.
• Self-host or Weaviate Cloud.

Watch out

• The fusion default changed over versions. If you upgrade, check your ranking with a small eval set.
• Dynamic index is still experimental.

My take

Weaviate’s hybrid search is easy to use, and that ease is a fair reason to pick it. But it is no longer the only database with “built-in hybrid”. Qdrant, Milvus, Elasticsearch, Redis (with FT.HYBRID) and Pinecone all have their own way now. Pick Weaviate for its API and features as a whole, not only for hybrid.

5. Milvus: you have millions to billions of vectors

Pick it when your data is really large, or you need many index types, including GPU and disk-based indexes.

What the docs say today

• Many index types, including FLAT, IVF_FLAT, IVF_SQ8, IVF_PQ, HNSW, DISKANN, GPU_IVF_FLAT and GPU_IVF_PQ. There is also AUTOINDEX.
• Three ways to run it: Milvus Lite for local prototyping, Standalone, and Distributed on Kubernetes. The docs describe Distributed as built for billion-scale and say Milvus users run tens of billions of vectors.
• Full text search with BM25: Milvus turns raw text into sparse vectors for you. Combine it with dense search using RRFRanker or WeightedRanker.
• Default limits: up to 32,768 dimensions and 16 shards per collection. Both are configurable.
• Managed option: Zilliz Cloud (from the team behind Milvus).

Watch out

• Distributed Milvus has many moving parts: several node types, plus a metadata store (etcd) and object storage. That is a real ops cost.
• Most RAG apps are nowhere near a billion chunks.

My take

“Milvus is for billions” is true but incomplete. It also has Milvus Lite for small work and good hybrid search. At the same time, you do not need Milvus just because you have “millions”. A few million chunks fit fine in pgvector, Qdrant or Weaviate on one decent machine. Do the memory math first.

6. Elasticsearch: your keyword search already lives there

Pick it when you already run Elasticsearch for search or logs, and your team knows it well.

What the docs say today

• dense_vector fields with HNSW indexes. For float vectors the default index type is bbq_hnsw (binary quantization) for 384 dimensions or more, and int8_hnsw below 384. (In 9.0, float vectors always used int8_hnsw.)
• Other options: int4, bbq_flat, and bbq_disk (disk-based binary quantization). Raw floats stay on disk for rescoring.
• Filtering: a filter inside the kNN clause is a pre-filter and makes sure k matching docs come back. A filter outside it is a post-filter and can return fewer than k.
• Hybrid: the rrf retriever combines a BM25 query and a kNN query (or a sparse_vector query). You can set rank_constant and rank_window_size.
• semantic_text fields can create embeddings for you.
• Self-managed or Elastic Cloud.

Watch out

• Put the filter in the right place. This mistake is easy to make and hard to notice.
• Check licence and subscription rules for the features you plan to use in your version.

My take

This one-liner holds up well. If BM25 search already lives in Elasticsearch, adding vectors there is usually the cheapest path to hybrid search. You get one index, one query and RRF on the server. OpenSearch users are in a similar spot with its k-NN plugin.

7. Redis: latency matters most

Pick it when you already run Redis, the vectors fit in memory, and you want very low latency, for example for semantic caching.

What the docs say today

• Three vector index types: FLAT (docs suggest it for small datasets, under 1M vectors, or when you need exact results), HNSW (larger datasets) and SVS-VAMANA (added in Redis 8.2; a graph index built for compression and Intel hardware).
• Filtered vector search: Redis picks a filter mode for you (BATCHES or ADHOC_BF), and you can override it with HYBRID_POLICY.
• FT.HYBRID, introduced with Redis 8.4, runs a full-text search and a vector search and fuses them with RRF (default) or LINEAR. RRF defaults are a window of 20 and a constant of 60.
• Managed option: Redis Cloud.

Watch out

• RAM is the cost. Large collections in memory get expensive fast.
• Check your Redis version before you plan on FT.HYBRID. One Redis tutorial page lists a newer minimum version than the 8.4 release blog, so confirm it for your deployment.

My take

“Redis for latency” is fair, but every option here can be fast at small scale. In a RAG request, the LLM call usually takes far longer than the vector lookup. So I pick Redis when it is already in the stack or when I am building a cache. I wrote about that in Semantic Caching for RAG & LLMs.

8. Chroma: your first prototype

Pick it when you want to go from zero to a working RAG demo in a notebook in a few minutes.

What the docs say today

• Ephemeral (in-memory) client for quick experiments. Data is gone when the process stops.
• PersistentClient saves the database to a local folder and loads it on start.
• Client-server mode to run it as a separate service, and Chroma Cloud as a managed option.
• Full-text search and regex filtering on documents. The Search API docs cover hybrid search with RRF over dense and sparse rankings; that part is documented under Chroma Cloud and needs a sparse vector index in the schema.

Watch out

• A notebook prototype is not a production plan. Decide early where the real data will live.
• Check which features are local and which are Cloud only.

My take

Chroma is a great first step, and it is more than “toy only” now because of its Cloud and server mode. Still, if production will be Postgres, I would prototype with pgvector from day one. It saves a migration later.

9. FAISS: a library, not a database

Pick it when you need fast similarity search inside your own Python or C++ code, for research, offline batch jobs, or as the engine inside your own service.

What the docs say today

• FAISS is a library for similarity search and clustering of dense vectors. It is written in C++ with Python wrappers.
• Index types include Flat (exact), IVF, HNSW, PQ and combinations like IVFPQ.
• Some of the most useful algorithms run on the GPU (faiss-gpu, and faiss-gpu-cuvs builds). Compressed methods can scale to billions of vectors.

Watch out

• No server, no metadata store, no replication, no access control. You write those yourself.
• Deletes and updates depend on the index type, and filtering is limited to ID selectors.

My take

This one-liner is exactly right, and it is the most useful line in the whole list. Many teams pick FAISS for a demo and later find they have to build a database around it. Use FAISS when you want a library. Use a database when you want a database.

Comparison table

A quick view. “Hybrid” means keyword (BM25 or sparse) plus dense in the engine itself. Check the version you run.

The concepts that decide most choices

Filtering: post-filter vs filter during search

This is the issue I see most in real RAG systems.

• Post-filter: find the nearest k, then throw away the ones that do not match. If only 20% of the data matches, most of your k disappears.
• Pre-filter / filter-aware search: apply the filter while searching, so you still get k matching results. Qdrant’s filterable HNSW, Weaviate’s ACORN, a filter inside the Elasticsearch kNN clause, and pgvector iterative scans are all ways to get here.

Quick test: run your 20 most common filtered queries and count how many results come back. If you asked for 10 and got 3, you have a post-filter problem.

Hybrid search and RRF

Vectors are good at meaning. They are weak at exact strings like ERR_TLS_1042 or a product code. BM25 (keyword search) is the opposite. Hybrid search runs both and merges the lists.

RRF (reciprocal rank fusion) is the most common way to merge. Each document gets 1 / (k + rank) from each list, and we add them up. It only uses ranks, so you do not need to compare BM25 scores with cosine scores. The original paper used k = 60. Elasticsearch and Redis also default to 60. Qdrant defaults to 2. Small detail, but it changes how much the top ranks dominate.

If you want to tune top-k, recall and precision after this, my earlier post Five Things to Measure and Tune in a RAG System walks through those knobs.

Memory math (do this before any meeting)

Example math, not a benchmark:

• 1,000,000 chunks x 768 dimensions x 4 bytes (float32) = 3,072,000,000 bytes, about 3.07 GB (about 2.86 GiB).
• Half precision (halfvec, float16): about 1.54 GB.
• int8: about 0.77 GB.
• Binary (1 bit per dimension): about 0.096 GB, but you must rescore with the full vectors to keep quality.

This is only the raw vectors. The index adds more. For example, the FAISS docs list an HNSW-Flat index at 4*d + x * M * 2 * 4 bytes per vector, so the graph links add roughly M * 8 bytes per vector on top of the floats. Then add metadata, chunk text and replicas.

So 1M chunks is a few GB. 10M chunks is tens of GB. That is still one big machine for many teams. 1B chunks is a different world, and that is where Milvus Distributed, DiskANN-style indexes or a managed service make sense.

Index types in one minute

• HNSW: a layered graph. Search starts on a small top layer and walks down to closer points. M (links per node) and ef_construction affect build quality and memory. ef_search (or ef) at query time trades speed for recall.
• IVF: split vectors into nlist clusters. At query time, search only the nprobe closest clusters. More probes means better recall and slower search.
• PQ (product quantization): split each vector into small parts and store a short code for each part. It saves a lot of memory and costs some accuracy. Often used with IVF (IVFPQ).
• DiskANN: a graph index designed to keep most data on SSD instead of RAM. Milvus has DISKANN, and pgvectorscale has a StreamingDiskANN index for Postgres.

Where the one-liners are too simple

Before you sign off: a small checklist

• Make a tiny eval set: 50 to 100 real questions with the chunk that should come back. Measure recall@k (did the right chunk appear in the top k?) and p95 latency on each option you shortlist.
• Test with your real filters, not with an empty filter.
• Plan re-embedding. When you change the embedding model, every vector must be created again. Old and new vectors cannot be compared. Keep the source text and a model version per row, and plan a side-by-side index.
• Check upserts and deletes. Chunks change when documents change. Make sure stale chunks really go away.
• Decide multi-tenancy early: one shared index with a tenant filter, partitions, or one collection per tenant.
• Count the full cost: servers or SaaS bill, backups, on-call, and the data sync job that keeps a second database in line with the first.

A simple “RAG from zero” post that follows one document from ingestion to answer may come later. This guide assumes you already know that flow.

Terms used in this post

Sources and further reading

Official docs (checked November 2026):

• pgvector README (HNSW, IVFFlat, halfvec, iterative index scans, filtering, scaling)
• pgvector changelog
• pgvector-python: hybrid search with RRF example
• pgvectorscale (StreamingDiskANN for Postgres)
• Pinecone: Hybrid search overview
• Pinecone: Filter by metadata
• Pinecone: Bring Your Own Cloud
• Pinecone: Local development (Pinecone Local end of life)
• Qdrant: Indexing (payload index, filterable HNSW)
• Qdrant: Filtering
• Qdrant: Hybrid and multi-stage queries (RRF, DBSF)
• Weaviate: Hybrid search
• Weaviate: Vector index (HNSW, flat, dynamic)
• Weaviate: Filtering (pre-filtering, ACORN)
• Milvus: Overview (Lite, Standalone, Distributed)
• Milvus: Index vector fields
• Milvus: Full text search (BM25)
• Milvus: Limits
• Elasticsearch: dense_vector field (int8, BBQ, bbq_disk)
• Elasticsearch: Filtered kNN search
• Elasticsearch: Reciprocal rank fusion
• Redis: Vectors (FLAT, HNSW, SVS-VAMANA)
• Redis: FT.HYBRID command
• Redis: Hybrid search in Redis 8.4
• Chroma: Clients (in-memory, persistent, Cloud)
• Chroma: Hybrid search with RRF
• FAISS README
• FAISS wiki: Faiss indexes

Papers:

• Malkov and Yashunin: HNSW (arXiv:1603.09320)
• Jégou, Douze and Schmid: Product Quantization for Nearest Neighbor Search (IEEE TPAMI 2011)
• DiskANN: Fast Accurate Billion-point Nearest Neighbor Search on a Single Node (NeurIPS 2019)
• Cormack, Clarke and Büttcher: Reciprocal Rank Fusion outperforms Condorcet and individual Rank Learning Methods (SIGIR 2009)

Related Intelliwise posts: Five Things to Measure and Tune in a RAG System and Semantic Caching for RAG & LLMs.

In short

Start with the database you already run. Do the memory math. Test your real filters and count the results. Add keyword search if users type exact codes or names, and merge with RRF. Move to a dedicated vector database only when a measured gap shows up, and write that gap in the design doc. FAISS is a library; everything else on this list is a database you have to run, or pay someone to run.

How to use this lab

• Open the companion notebook in another tab: LLM Inference Starter: Batching, Prefix, Quant on Kaggle. Practice as you read, or read first and run it later.
• Each station has the same rhythm: the wrong instinct, your guess (click to open), what we measured, why it happens, and one small thing to try.
• Every measured table here comes from one run on one Kaggle Tesla T4 on 9 Nov 2026. It is not a benchmark. Your numbers will be different.
• When I could not measure something, I label it example math.

The story: our support chatbot feels slow

We have a support chatbot for NimbusDB, a made-up cloud database product. Users say it feels slow. In the review meeting, the first suggestion is the usual one.

“Inference is slow. Let us get a bigger GPU.”

Sometimes that is the right call. But many slow services are not using the card they already have. So in this lab we do not argue. We guess, we measure, and then we change one thing at a time.

First, the simple picture. An LLM answers in two phases.

• Prefill: the model reads the whole prompt in one go and fills the KV cache. This decides TTFT (time to first token).
• Decode: the model writes the answer one token at a time. This decides TPOT (time per output token).

Decode is usually memory-bound. For every new token, the GPU reads all the model weights and the KV cache, then does only a small amount of math. If you serve one chat at a time, the card looks busy, but most of that time goes into moving data for a single token.

This is the map we follow before anyone asks for new hardware:

flowchart TB
  Slow[Inference feels slow] --> M{What did we measure?}
  M -->|TTFT high| Prefill[Prefill / prefix / queue]
  M -->|TPOT high| Decode[Decode / batch / bandwidth]
  M -->|Throughput low| Serve[Batching / KV pages / admission]
  M -->|VRAM full| Mem[Quantize / GQA model / paging / offload]
  Prefill --> Fix[Change serving shape]
  Decode --> Fix
  Serve --> Fix
  Mem --> Fix
  Fix --> Buy{Still stuck after measure?}
  Buy -->|yes| GPU[Then consider a bigger GPU]
  Buy -->|no| Done[Ship the fix]

Lab setup

To run it yourself: open the notebook, click Copy & Edit, set the accelerator to a GPU (T4 is fine), and run the cells from the top. The model loads once and every station reuses it.

Station 1: Batching

Wrong instinct: “Serve each request alone. Batching is for training, and it will make every user wait longer.”

Measured on the T4 (37-token prompt, 48 new tokens, static batch):

Why. During decode, the costly part is reading the same weights again for the next token. If 16 users are waiting, one step can move all 16 forward. The weights are read once per step for the whole batch. So throughput (tokens per second across everyone) goes up a lot. But each user still waits for 48 decode steps, so their own latency does not drop. It goes up a little, because each step does a bit more work.

Two columns need care:

• wall / batch is the number people misread as latency. No user waits 0.12 s. Every user in the batch waits for the full batch.
• TTFT grew from 0.045 s to 0.109 s at batch 16, because 16 prompts are prefilled together. Small here, but it grows with long prompts.

This is a static batch: all requests start together and the batch ends when the last one finishes. Real chat traffic does not arrive like that. For live services you want continuous batching, which I explain below.

Station 2: Prefix reuse

Wrong instinct: “Once the system prompt is cached, the request is basically free.”

Our NimbusDB bot sends a long system prompt with every question: 40 support policies, 1,484 tokens. Only the user question at the end changes.

Measured on the T4 (shared prefix 1,484 tokens):

Building the prefix cache once took about 0.47 s. After that, each new question only prefilled its own short suffix.

Why. Prefill writes Keys and Values (the KV cache) for every prompt token. If the next request starts with the same tokens, those Keys and Values are the same, so we can reuse them and skip that work. That cuts TTFT. But the answer still has to be written one token at a time, so decode costs the same. In this run, decode (1.82 s) was much bigger than prefill anyway. A cached prefix helps TTFT. It does not make the request free.

The one-word lesson. In the notebook I changed one word in the system prompt. The token ids then differed from position 15 of 1,484, so the cache could not be reused for anything after that point. Reuse needs an exact token prefix from the very first token. A timestamp or user name at the top of the system prompt breaks it for every request.

Practical rule: put the stable text first (system instructions, shared docs). Put the changing text last (the user question).

Serving engines do this for you. vLLM, for example, hashes KV blocks and reuses them with automatic prefix caching. For the memory side and the paging picture, see KV cache sticky notes.

Station 3: Quantization

Wrong instinct: “Quantization only shrinks the download, and quality always dies.”

Measured on the T4 (bitsandbytes, memory after load):

Why. Quantization stores weights (and sometimes activations or the KV cache) in fewer bits. That cuts memory, and often cuts memory traffic, which is what decode cares about. Freed memory can hold more KV cache, so more chats fit. It is not always faster on its own: bitsandbytes kernels can be slower than FP16 on small batches and older GPUs.

GPTQ and AWQ are common weight-only methods. Serving stacks such as TensorRT-LLM and vLLM document several recipes.

The quality glance. The notebook also asks each version one short question: “what is a KV cache?” Read the three answers. Even the FP16 answer was somewhat wrong. That is the real lesson. A small model can be vague in any precision, so “FP16 is correct and 4-bit is broken” is not a safe rule. Quality is not yes or no. Check it on your own task with your own eval set before you ship.

If the model barely fits on your machine, also read Local LLM beyond VRAM. Quantization and CPU plus GPU placement solve different parts of the same problem.

Station 4: Attention memory and FlashAttention

Wrong instinct: “FlashAttention is an approximation. It is fast because it skips some of the math.”

Measured on the T4 (synthetic tensors: 1 batch, 12 heads, head size 64):

Naive memory and FLOPs both grow about 4× each time the sequence doubles. The full table with 1024 and 2048 is in the notebook.

Why. Naive attention builds the full N × N score matrix in GPU memory. FlashAttention computes exact attention, but in tiles. The scores stay in fast on-chip memory and never get written out as one big matrix to HBM (the main GPU memory). So:

• The math is still quadratic. Same result, same order of FLOPs.
• What drops is the extra memory and the HBM traffic. Less data moving around is why it is faster.

A T4 detail. On the T4, PyTorch may report the flash backend as “allowed”. Allowed is not the same as used. Flash kernels generally need compute capability 8.0 or newer, and the T4 is 7.5. So here SDPA most likely picked its memory-efficient kernel. The memory saving is real, but do not say “we ran FlashAttention on a T4”.

Do not mix this up with PagedAttention. They solve different problems, and you often want both.

Station 5: Speculative decoding

Wrong instinct: “Speculative decoding is free speed. Turn it on and every request gets faster.”

Measured on the T4 (batch 1, greedy, Hugging Face assisted generation):

The output tokens matched the 1.5B model’s own greedy output.

How it works. A small draft model guesses the next few tokens. The big target model checks all of them in one forward pass. It keeps the guesses that match what it would have picked. At the first wrong guess, the target model’s own token is used and the rest of the draft is dropped. So the output stays the same as the target model’s output. The classic write-up is Fast Inference from Transformers via Speculative Decoding.

Why it was slower here. The draft (0.5B) and the target (1.5B) are too close in size. Running the draft is not cheap enough compared with the target, so its cost ate the saving. Speculative decoding pays off when the target is much bigger than the draft and the draft guesses well.

It also helps most at low batch. At batch 1, the GPU is mostly waiting on memory during decode, so checking extra tokens is almost free. At high batch, the GPU is already busy, and drafting adds work. The TensorRT-LLM speculative decoding docs say the same. Other draft methods include extra heads like Medusa and trained drafters like EAGLE.

Things a free notebook cannot show

The next parts need a real serving engine or more than one GPU. I am not giving fake Kaggle exercises for them. Short explanations only.

Continuous batching vs static batch

In station 1, all 16 requests started together and the batch ended when the last one finished. In live chat, users arrive at different times and answers have different lengths. With a static batch, short answers finish early and leave empty seats until the longest one is done.

Continuous batching (iteration-level scheduling) lets finished requests leave and new ones join between decode steps. The idea came from systems like Orca and is now normal in engines such as vLLM.

Request scheduling and admission

Continuous batching needs a scheduler that decides who runs in each step.

• Order: first come, first served (FCFS) by default, or priority if some users or tiers must go first.
• Admission: a new request joins only when there is enough free KV memory and room in the token budget for that step. If not, it waits in the queue. That queue time shows up in TTFT.
• Mixing prefill with decode: a new prompt needs prefill while other users are in decode. Good engines mix both in the same step, and often split a long prompt into chunks. That way one long prompt does not freeze everyone else’s stream.

PagedAttention and batch size

Old servers reserved KV memory for the maximum length of every chat, even if the chat used only a small part. That empty reserved space limits how many sequences fit. PagedAttention stores the KV cache in small fixed blocks and gives out blocks only when needed. Less reserved empty space means more sequences fit at once, so the batch can be bigger and throughput can rise. The paper reported 2 to 4× throughput over the systems it compared, on their workloads. For the locker picture and the math, see KV cache sticky notes.

Tensor parallel vs pipeline parallel

“Add GPUs and latency drops in a straight line” is another common belief. Different kinds of parallelism do different jobs.

TP can cut per-request latency for a large model, but the sync inside every layer is not free. Replicas raise total throughput more than they cut one user’s TPOT. If one GPU sits idle because you never batched, eight GPUs will also sit idle, just at a higher cost. Tune one replica first, then scale out.

Measure before you change anything

One average “latency” number hides the real problem. Split the timeline.

A long answer with healthy TPOT can still feel slow end to end. A short answer stuck in a queue can feel broken on TTFT while tokens per second look fine.

sequenceDiagram
  participant U as User
  participant Q as Queue
  participant P as Prefill
  participant D as Decode
  U->>Q: request arrives
  Q->>P: admitted
  Note over Q,P: TTFT includes wait + prefill
  P->>D: first token
  loop each later token
    D->>U: token
    Note over D: TPOT / ITL
  end

Practice checklist

I run this before asking for new hardware.

1. Define the pain. Slow first token, slow streaming, low capacity, or out of memory?
2. Split the metrics. TTFT, TPOT, throughput, queue time, free KV blocks, prefix hit rate. (Station 1)
3. Batch. Move to continuous batching if you still serve one request at a time or use static batches.
4. Stable prefix first. System prompt and shared docs before the user question. Turn on prefix caching. (Station 2)
5. Page the KV cache. Do not reserve max-length memory for every chat.
6. Quantize with an eval set. Start with weight-only. Try KV quantization if concurrency is the limit. (Station 3)
7. Use memory-efficient attention for long prompts, and remember the math is still quadratic. (Station 4)
8. Try speculative decoding when the batch is small and the draft is much smaller than the target. (Station 5)
9. Parallelism. TP or PP only after one replica is tuned. Add replicas for throughput.
10. Then talk about a bigger GPU, with graphs that show what is still slow.

Honest caveats

• All measured numbers come from one Kaggle T4 run with a 1.5B model, using Hugging Face as a teaching stand-in. A real serving engine will give different numbers.
• I did not run vLLM, continuous batching, or multi-GPU in the notebook. Those sections are explanation only.
• Paper and vendor speed-ups are for their workloads. Copy the method, not the headline number.
• Quantization quality depends on the task. Legal, medical, and coding evals can disagree.
• Speculative decoding needs a good acceptance rate and a real size gap between draft and target.
• Prefix cache hits need exact tokens from the first token onward.

Terms used in this post

Sources and further reading

Papers and docs for the mechanics. The measured tables come only from the companion notebook.

• Companion notebook: LLM Inference Starter: Batching, Prefix, Quant (Kaggle)
• Orca: A Distributed Serving System for Transformer-Based Generative Models (OSDI ‘22)
• Efficient Memory Management for LLM Serving with PagedAttention (arXiv:2309.06180)
• vLLM: Automatic prefix caching
• FlashAttention (arXiv:2205.14135)
• GPTQ (arXiv:2210.17323)
• AWQ (arXiv:2306.00978)
• Speculative Decoding (arXiv:2211.17192)
• Medusa (arXiv:2401.10774)
• EAGLE (arXiv:2401.15077)
• TensorRT-LLM: Quantization
• TensorRT-LLM: Speculative decoding
• Related Intelliwise posts: KV cache sticky notes and Local LLM beyond VRAM

In short

A bigger GPU is a tool, not a diagnosis. Measure TTFT, TPOT, and throughput separately. Batch to use the card you have. Keep stable text at the start of the prompt. Page the KV cache. Quantize with an eval set. Use speculative decoding when the batch is small and the draft is much smaller. Scale out when one tuned replica is really full. After that, if it is still slow, ask for new hardware with the graphs in hand.

TL;DR

• A “70B on 4GB GPU” headline means the card holds a slice. The rest of the weights still live on disk and RAM.
• Low VRAM is not useful latency. Measure time-to-first-token (TTFT), tokens per second, whole-machine RAM + SSD, and answer quality.
• Pick the model that solves the task. Then pick the engine that runs it well on your PC.
• llama.cpp, Ollama, and LM Studio already do hybrid CPU–GPU. Colibri, Strata, AirLLM, and FreeToken are specialized designs. They did not invent “use the whole PC.”
• Four paths below. Jump to the one that matches your machine and your job. I did not run these benches. Numbers are what the projects report in their own docs.

The trap: “4GB GPU runs 70B”

The sentence is usually true in a narrow way.

The GPU never held the whole model. For a dense model, an engine like AirLLM keeps one layer on the card, then swaps the next layer in. For a mixture-of-experts (MoE) model, only a small set of experts fires for each token. The rest can sit in RAM or on the SSD until the router asks for them.

So the VRAM number answers one question: “Will this process start?” It does not answer “Will I get a useful reply before I go make tea?”

Weights still have to live somewhere.

flowchart TB
  Q[Next token needs a weight] --> R{Where is that weight?}
  R -->|hot| GPU[GPU VRAM<br/>compute now]
  R -->|warm| RAM[System RAM<br/>copy up, then compute]
  R -->|cold| SSD[SSD / disk<br/>read into RAM]
  SSD --> RAM --> GPU
  GPU --> OUT[Write the next token]

Three shelves. Same books.

• GPU VRAM — the desk under your hands. Fast. Small.
• System RAM — the table next to the desk. Slower to reach. Bigger.
• SSD — the library stacks. Huge. You walk there on a miss.

If you have seen KV cache sticky notes, this is the same idea on the other side of memory. The KV cache is conversation state. The weight shelves are the model itself. Both grow. Both steal the same VRAM.

I treat the four engines in this post as placement policies, not as magic. llama.cpp already lets you put some layers on the GPU (-ngl / --n-gpu-layers), pin named tensors (--override-tensor), and leave MoE experts on the CPU. That is the baseline. Start there unless you have a reason not to.

Pick a path

Do not start with a product name. Start with what you are actually doing.

After the paths: a comparison table, a checklist, and when to stay.

Path 1 — Low VRAM experiment

You are here if: the card is about 4–12 GB. You want to know whether Llama 3.x 70B, Qwen3.8-Flash-Next, or something larger will even load. You can wait. This is a weekend test, not a daily chat box.

Start with: AirLLM.

AirLLM streams one transformer layer onto the GPU (or the experts a MoE token actually routes to). The rest stays on disk as layer shards. That is why the VRAM headline can look tiny next to the parameter count.

AirLLM’s README reports these VRAM numbers (their measurements, not mine):

The same README is honest about the other bill. The original checkpoint is decomposed and saved layer-wise. You need a lot of disk. For Qwen3.8-Flash-Next they say about 360 GB of checkpoint disk, and a 64 GB machine is enough because the n-gram table is file-mapped on the host. delete_original=True reclaims the originals after the split. If you run out of disk, you get MetadataIncompleteBuffer. That is a disk problem, not a “model does not fit” mystery.

What I would do

1. Pick one model you actually care about. Not the biggest name on the page.
2. Confirm disk first. Then VRAM.
3. Time the first token and the next twenty. If TTFT is minutes, the experiment succeeded as a load test and failed as a chat test.
4. If you only needed a smaller model plus retrieval, stop here and read the five RAG knobs. A 70B that crawls is not automatically better than an 8B that answers.

AirLLM also reports a 3× decode speed-up from block-wise weight compression, and about 10% from prefetch. Those are their numbers, on their bottleneck (disk load, not GPU matmul). Useful if you stay on this path. Not a reason to skip the stopwatch.

Jump off this path when you want interactive chat on a gaming PC (path 2), or when llama.cpp already loads a quant you like (path 4).

Path 2 — Gaming PC chat

You are here if: you have a discrete GPU or a strong CPU, a fast SSD, enough RAM for the model you picked, and you want to talk to a large MoE. Not just prove it loads.

Two engines show up here. They are not interchangeable.

If you want Qwen3.8-Flash-Next: look at Strata

Strata is built around that one model (~125B MoE). Hot experts sit on the GPU. The others sit in CPU RAM. Lookup traffic can come from the SSD. They also use multi-token prediction (MTP) speculative decoding.

Their official models table (RTX 5070 column, project-reported):

VRAM listed on that table is 12 GB+ for those four rows. So even here, the card is not holding the whole model. RAM and disk are doing the rest.

Their paper reports MTP at about 1.6–1.8× versus plain decode on an RTX 5070 + Ryzen 5 7600 + 64 GB setup. Again: their bench, their machine.

Specialization is the trade. Great if you want that model. Weak if you need GLM, DeepSeek, or last month’s GGUF.

If you want a wider catalog, or GLM-5.2: look at Colibri

Colibri is a set of small pure C engines. CPU is the required path. GPU is optional (Vulkan, or CUDA on NVIDIA).

The picture I use: the desk holds the dense part (attention, shared experts, embeddings) plus a cache of experts your chat actually uses. The library is the SSD, where the rest of the routed experts live. A GPU, when you have one, is a few extra books in your hands.

GLM-5.2 is their reference model. From the Colibri README (project-reported):

• 744B MoE, about 40B active per token
• Download about 429 GB
• RAM tiers 16 / 24 GB (below 16 GB it does not start; from 24 GB they treat it as the measured tier)
• Cold decode about 0.05–0.1 tok/s on the 25 GB laptop where the project started
• Warm 1.83 tok/s CPU-only on a 128 GB Ryzen AI Max+ 395
• 9.0–9.2 tok/s on 6× RTX 5090 with experts resident
• GitHub issue #31 reports 0.28 tok/s median on GLM-5.2 int4, CPU-only, on a Ryzen 9 9950X with 128 GB (two runs)

That last number is why I do not trust a VRAM headline. A 128 GB workstation that writes a token every few seconds is a library, not a chat app.

Colibri’s own rule is the one I agree with: placement changes speed, not the weights and not the router. Less fast memory may cost tokens per second. It should not quietly change the model.

Setup asks for 8 GB RAM at the very least (16 GB or more is better) and 22 GB free disk for the smallest model. GLM-5.2 is in another league on disk. Check the download column before you click.

What I would do

1. Name the model first. Qwen3.8-Flash-Next → Strata is in the conversation. GLM-5.2 or a mixed catalog → Colibri. A GGUF you already have → path 4 first.
2. Time TTFT and tok/s on your prompts. Cold and warm. The laptop floor and the 5090 rack are both “it runs.”
3. Watch RAM + SSD while it runs. If the page cache is the real GPU, you will see it.
4. Keep a quality check. A faster quant that garbles code is not a win.

Jump off this path when the job is a coding agent with tool calls (path 3), or when llama.cpp already feels fine (path 4).

Path 3 — Agent sessions

You are here if: the model is in a loop. It calls tools. It writes a thinking block, then deletes it. The next turn is not a fresh chat. Prefill keeps eating the first token.

Short chat and a long agent session are different workloads. I wrote about the harness side of that in Own Your Harness. Here the question is the engine under the harness.

Start with: FreeToken, and still keep llama.cpp as the control.

FreeToken is an edge MoE serving engine. The public README lists four design bets:

• Bandwidth-adaptive CPU–GPU — a q* policy that splits cache misses between filling the GPU and running the expert on the CPU, using the machine’s real bandwidth
• Expert LRU cache — keep the experts this session actually touches
• Elastic VRAM — move memory between the expert cache and the KV cache at runtime, no restart
• Semantic-aware caching — checkpoint recurrent state and KV at tool-call and thinking-block boundaries, so an edit does not recompute the whole prefix

That last one is the reason this path exists. An agent that strips a thinking block or replaces a tool result invalidates a naive KV prefix. If you only measure a single “hello” prompt, you will miss it. See also why LLMs forget: the model does not remember the session. The engine and the harness do.

The README is NVIDIA-centric. It lists native support for RTX 30, 40, and 50. The paper is written around CUDA graphs and PCIe. I would not assume an AMD box is a first-class path from those docs.

Their paper (arXiv 2608.16157) reports, on their machines, not mine:

• RTX 5090: 77–83 tok/s on Qwen3.6-35B-A3B; 22–25 tok/s on DeepSeek-V4-Flash
• Decode stays within 12% of the single-turn rate as the workload gets more agentic
• Worst-case TTFT under 44 s across their workloads; each baseline they name went over 150 s in at least one setting
• 8 GB RTX 4060 laptop: a 35B model at 39.3 tok/s

Treat those as a sales-and-paper snapshot. Copy the metrics, not the scores.

What I would do

1. Same model, same quant, same machine. llama.cpp first, then FreeToken.
2. Measure a session, not a greeting. Tool call. Thinking block removed. Next turn.
3. Record TTFT on turn 1 and turn 5. Record tok/s. Record RAM and VRAM while KV grows.
4. Score the answers with a small golden set. I use the habit from golden examples and a grader. Speed without quality is a demo.

Colibri also talks about parking a conversation so an agent’s next turn reads only what is new (COLI_PARK). Their README reports one Hermes Agent second turn on Qwen3.8-Flash-Next (14,810 tokens): 86 s → 4.9 s. That is their number, on their engine. If you are already on Colibri for path 2, measure that before you install a second stack.

Jump off this path when you only needed a chat window (path 2), or when llama.cpp plus a smaller model already holds the eval (path 4).

Path 4 — Already happy with llama.cpp

You are here if: Ollama, LM Studio, or raw llama.cpp already answers you. Someone sent a “70B on 4GB” link and you wonder if you are missing out.

You are probably not missing the idea. You may be missing a specialized placement for one model or one agent loop.

llama.cpp hybrid, in the flags people already use:

• -ngl / --n-gpu-layers — put N layers on the GPU, leave the rest on CPU
• --override-tensor — pin or exclude named tensors (the usual way to keep huge MoE experts off the card)
• MoE CPU-offload flags in current builds — same job, fewer hand-written tensor names

That is “use the whole PC.” The four engines above tune the same shelves for a narrower job: layer streaming (AirLLM), SSD-resident experts with a tiny C runtime (Colibri), one MoE with MTP (Strata), agent-aware cache and elastic VRAM (FreeToken).

Stay on llama.cpp when

• The GGUF you want already runs at a speed you can live with
• You need a wide model catalog and a boring OpenAI-compatible port
• You do not want a second weight format, a 360–429 GB extra download, or a CUDA-only stack
• Your “slow” problem is the prompt and the KV cache, not the engine brand. Fix context first.

Leave llama.cpp for a trial when

• You cannot load the model at all, and AirLLM’s layer shards are the experiment (path 1)
• You want Qwen3.8-Flash-Next specifically, and Strata’s table matches your RAM (path 2)
• You want GLM-5.2-class MoE on a machine that will never hold it in VRAM (path 2)
• Agent turns re-prefill so hard that TTFT blows the client timeout (path 3)

A fair trial is boring on purpose:

1. Same model family, same quant if both sides can load it, same hardware, same prompts.
2. Record TTFT, tok/s, RAM, SSD read, VRAM, and a quality check.
3. Repeat on a long context and on an agent session. One “hi” is not a bench.

If the new engine wins on those four numbers, keep it. If it only wins on a VRAM screenshot, delete it.

Comparison

All speed and VRAM cells below are project-reported. I did not reproduce them.

How to pick

Work top to bottom. Stop when a line fails.

1. What is the task? Chat, code, long RAG, or an agent loop. The model comes from the task, not from a VRAM screenshot.
2. Does a smaller model plus retrieval already pass a golden set? If yes, stay small. See RAG knobs and evals.
3. Can llama.cpp load a quant you trust? If yes, measure it. That is the control.
4. Disk and RAM, not only VRAM. AirLLM shards, Colibri’s 429 GB GLM-5.2, Strata’s ~70–76 GB downloads, FreeToken host-resident experts — the SSD is part of the machine.
5. Interactive, or patient? Colibri’s own 25 GB laptop floor is 0.05–0.1 tok/s cold. Issue #31 is 0.28 tok/s on a 128 GB CPU box. Say “library, not chat” before you install a dashboard.
6. One model family, or many? Strata is a specialist. The others are broader. llama.cpp is the broadest everyday catalog.
7. NVIDIA CUDA or not? FreeToken’s public docs are RTX 30/40/50. Colibri has CPU + Vulkan. llama.cpp is the portable default.
8. Measure four things: TTFT, tok/s, RAM+SSD (+ VRAM), quality. Add a fifth for agents: TTFT after a tool edit.

When to stay with llama.cpp / Ollama

Stay if the current stack is boring and good.

Hybrid offload is not new. A second engine is extra weights, extra flags, and extra ways to fool yourself with a vendor table.

Switch for a measured reason: the model you need will not load, the model you need is the one Strata is built for, you want Colibri’s C path and catalog, or agent re-prefill is the actual pain.

Do not switch because a card-memory number looked small on a README.

Related reads on Intelliwise

• KV Cache: Why Chatbots Keep Past Keys and Values — conversation state vs model weights. Both sit on the same GPU.
• Why LLMs Forget Everything — the model does not keep the session. The engine and the harness do.
• Own Your Harness: Privacy First, Model Second — the loop around the engine. This post is the engine under that loop.
• Five Things to Measure and Tune in a RAG System — when a smaller model plus retrieval is the real fix.
• AI Agent Evals: Golden Examples and the Grader — how I would score quality next to tok/s.

Sources

Primary GitHub, docs, and papers only. Speed and VRAM figures in this post are project-reported.

• AirLLM — layer streaming, VRAM table, Flash-Next disk and host-RAM notes, compression and prefetch claims.
• Colibri and the project site — placement design, GLM-5.2 429 GB / 16/24 GB tiers, laptop and workstation decode numbers, COLI_PARK agent turn.
• Colibri issue #31 — GLM-5.2 int4, Ryzen 9 9950X, 128 GB, 0.28 tok/s median, CPU-only (project thread).
• Strata models page — Qwen3.8-Flash-Next quants, RAM, download size, RTX 5070 tok/s.
• Strata paper (PDF) — MTP about 1.6–1.8× on their RTX 5070 + Ryzen 5 7600 + 64 GB setup.
• FreeToken — edge MoE runtime, q* policy, LRU experts, elastic VRAM, semantic-aware cache, RTX 30/40/50.
• Shuo Yang et al., FreeToken: Efficient Edge-Native MoE Serving with Bandwidth-Adaptive Execution, arXiv 2608.16157. Also HTML.
• llama.cpp — -ngl / --n-gpu-layers, --override-tensor, MoE CPU offload. Baseline hybrid, not a new invention of the four engines above.

TL;DR

• The harness builds, the model fills. Code, data, and even database access flow through it. Own that path.
• We control a non-deterministic model with a deterministic harness. The harness only covers the cases it knows about, so it can still miss things. Own it, and you can add a rule the day you find a gap.
• Hosted tools offer Privacy Mode, ZDR, and no-training promises. I am not saying they misuse data. You still cannot independently verify what happens after a prompt leaves your machine.
• A harness you run can keep work local, mask PII, refuse prod writes, and decide which model a person may call.
• This is not a lonely bet. Stripe, Block, Spotify, Ramp, Shopify, Amazon, Airbnb, and Coinbase have published in-house coding agents or harnesses. The model is often a vendor model. The harness is theirs.
• Cost (about 3× to 40× tokens) and a roughly four-month open-model gap make this practical now. They are not why I started. Keep a balance: simple work stays with you or a small local model. Frontier is for hard tasks.
• Code-Harness is my retrieval-harness head start, not a finished privacy platform. Try it.

Why I care

The model race gets the headlines. I think that is the wrong thing to own.

That is an opinion. I am not announcing a product, and I am not saying the model becomes optional. My main bet: every business should own the harness its team uses, because code, data, and even database access flow through it.

The model is the engine. The harness is context, tools, the loop, checks, and rules. In one line: the harness builds, the model fills. Swap the engine and most of that story stays the same.

We are controlling something non-deterministic (the AI model) with something deterministic (the harness). Same prompt, two runs, the model can pick a different answer or a different tool. The harness is the part that repeats: if this user, this repo, this tool, then allow or block.

That harness is only as complete as the use cases it knows about. A new path, a new tool, a prompt injection you did not write a rule for — it can miss that. That is exactly why you want to own it. The day you find a gap, you add a rule. You do not open a ticket and wait.

flowchart TD
  M["Model: non-deterministic"] --> H["Harness rules: deterministic, known cases"]
  H --> K[Known case: rule fires]
  H --> G[Unknown case: a gap]
  G --> P[You own it: add a rule]

If you want how a real hosted harness is put together, I already walked through Cursor from the outside: Reverse Engineering the Cursor AI Agent. This page is the other half: who should own that harness, and why privacy is the reason. Working example: Code-Harness (Apache-2.0). Retrieval for code, not a finished privacy platform.

The problem: you cannot verify a vendor’s promise

When a developer asks an agent a question, the harness is already in the middle. It reads the repo. It may open a ticket. It may run a query on a database. It packs that into a prompt. Then a model sees it.

So the first question is not “which model?” The first question is who runs the path that your source and your records travel on.

I use hosted coding tools. I am not saying Cursor, OpenAI Codex, or Claude Code misuse customer data. Their public docs offer real privacy modes:

• Cursor Privacy Mode: no training, plus zero-data-retention (ZDR) agreements with the providers it routes to. Prompts and code still go to Cursor and those providers. Some models and “bring your own key” sit outside ZDR.
• OpenAI: API and business products do not train on your organisation’s data by default. ZDR exists for qualifying organisations, with approval.
• Claude Code: on commercial Team / Enterprise / API terms, no training unless you opt in. ZDR is a separate Enterprise offering, enabled by Anthropic, not in the standard plan.

Those are real features. I would still buy them for a lot of everyday work. The honest limit is still the same: the guarantee is a contract and a policy page. You cannot independently verify what happens to a prompt once it leaves your machine. You cannot inspect the provider’s disk. You trust the vendor and the audit they show you. That is fine for many teams. It is not the same as control.

My answer: own the harness

With a harness you run, on a model you run (or a model that never sees the sensitive slice), the default can be: this prompt does not go to an outside API. Local embeddings. Local LLM. Localhost serve. If something must leave, the harness is where you say no — before the bytes move.

What it can control:

Governments are already building that picture.

• Japan’s Digital Agency is building Gennai Kobo, a Claude-Code-equivalent environment they can run themselves.
• IndiaAI is funding sovereign models and compute.
• The EU is funding AI gigafactories so work can sit on European infrastructure.

The honest caveat. Most enterprises still buy hosted tools. A 2025 US survey put purchased AI use cases at 76% (Menlo Ventures; investor in a hosted vendor, so a market snapshot, not gospel). Hosted vendors also sell regional, VPC, and HIPAA-style setups. I expect a mixed world. Who already built that path: Companies already building their own harness. Fair objections covers the Docs comparison, vendor self-hosting, and the cost of owning it.

Trust at the database, not only at the chat box

Code is only one pipe. The scary one is a tool that can talk to production data.

The model can still ask for SELECT * FROM patients. The harness does not run it. That is the whole point of owning the path.

The YAML below is an example I made up. It is not a file from Code-Harness or from any vendor.

# Example only. Not a product file.
database:
  default: deny
  prod:
    mode: read_only
    writes: deny
    tools_allow: ["db_explain", "db_select_named"]
    queries_allow:
      - order_count_by_day
      - open_tickets_for_repo
    columns:
      mask: ["email", "phone", "card_last4", "patient_id"]
      deny: ["ssn", "full_card", "note_raw"]
  staging:
    mode: read_write_scratch
    writes: ["scratch.*"]

flowchart TD
  A[Model asks for a DB tool] --> B{"Harness DB ACL"}
  B -->|prod write| X[Deny + audit]
  B -->|column on deny list| X
  B -->|query not named| X
  B -->|named read + mask PII| C[Run read-only]
  C --> D[Return masked rows]
  D --> E[Write audit]

In Code-Harness today. Serve binds 127.0.0.1 only by default. Outbound text goes through redact. Events land in an append-only audit log. That is not a database ACL and not a sovereignty platform. It is the smallest honest slice: stay on localhost, strip keys, write down what left. The DB example above is a plan.

Rules and ACLs in practice

A paragraph in the system prompt is not a rule. The model can ignore it. A prompt injection can override it — see Prompt Injection: A Simple Guide. An access control list (ACL) at the harness layer is a hard deny. The call does not run.

Principle I would ship: deny by default, allow explicitly, deny always wins.

The same gate covers files, tools, and network: deny **/.env, deny rm -rf, named hosts only. The piece I care about most is which models a session may call. Not a dropdown the user picks. A check on user type, repo type, and data sensitivity.

flowchart TD
  Q[Session asks for a model] --> S{"Sensitive data or healthcare / payments?"}
  S -->|yes| L[Local only]
  S -->|no| U{"User is contractor?"}
  U -->|yes| L
  U -->|no| R{"Repo allows frontier?"}
  R -->|no| L
  R -->|yes| A{"Approve logged?"}
  A -->|no| L
  A -->|yes| F[Local or frontier]

Frontier calls should not be a tired “yes” click. I kept a one-day toy diary of that failure in The Human in the Loop Can Get Tired.

# Example only. Not a product file.
# Deny by default. An explicit deny always wins.
models:
  default: local
  catalog:
    local: ["local-qwen", "local-gemma"]
    frontier: ["frontier-a", "frontier-b"]
  users:
    contractor: { allow: ["local"], deny: ["frontier"] }
    junior: { allow: ["local"], frontier: "needs_senior_approve" }
    senior: { allow: ["local"], frontier: "needs_approve_if_repo_allows" }
    admin: { allow: ["local", "frontier"], if: "repo_allows" }
  repos:
    healthcare: { allow: ["local"], deny: ["frontier"] }
    payments: { allow: ["local"], deny: ["frontier"] }
    internal: { allow: ["local"], frontier: "needs_approve" }
    open_source: { allow: ["local", "frontier"], frontier: "needs_approve" }
  data:
    phi: { deny: ["frontier"] }
    pan: { deny: ["frontier"] }
    public: { allow: ["local", "frontier"] }

def allowed_models(user, repo):
    """Example only. Not production code.

    User type, repo type, and data tags. Deny wins.
    Returns (ok, models, reason).
    """
    tags = repo.data_tags  # e.g. ["phi"] or ["public"]
    allow = set(acl.users[user.role].allow)
    if "phi" in tags or "pan" in tags or repo.kind in ("healthcare", "payments"):
        allow = {"local"}
        return True, sorted(allow), "sensitive repo: local only"
    if user.role == "contractor":
        return True, ["local"], "contractor: local only"
    if "frontier" in acl.users[user.role].allow and repo.kind == "open_source":
        if user.approved_frontier:
            return True, ["local", "frontier"], "approved"
        return True, ["local"], "frontier needs approve"
    audit(user, repo, "model-acl")
    return True, sorted(allow), "default"

The honest caveat. Rules need upkeep. Too many “please approve” clicks cause approval fatigue. The ACL lives outside the model. Pair it with an OS sandbox. Too-strict rules get bypassed with a personal key — a privacy hole. Log every denial.

In Code-Harness today. A small real slice: outbound redact, append-only audit, default-fail verify. Not a model ACL. An optional decision client can sit on verify and the loop. It does not pick the model. That is a plan.

Why this is practical now

These are supporting reasons. They are not the thesis.

Cost sits in the harness. Same model, different bill. Vats & Golev (preprint): up to about 40× more tokens per solved task when the harness changed, pass rates only a few points. Liu & Han (preprint): up to about 3× cost, score often inside rerun noise. On SWE-Bench Pro v2: pick the harness for the bill, not the score. More loops also means more of your repo in more prompts.

• Caveat. Accuracy often barely moves. Liu & Han found harness swaps flip about 13% of tasks — same rate as a rerun. Labs are absorbing old harness tricks into the model (Anthropic). I am not betting that “the harness beats the model.”

Everyday coding can sit on a local model. Epoch AI puts the open-vs-closed gap at about four months. On the clean SWE-Bench Pro v2 private set (solved out of 272): Claude Opus 5 222 (closed), Kimi K3 214 (open), GLM-5.3 211 (open).

• Open-weight families, with the right maker: Gemma, Qwen, GLM (Z.ai / Zhipu — not Google), gpt-oss, Nemotron, Muse Spark Open (Meta, Oct 2026).
• Caveat. The frontier still wins the hardest tasks. “Local” for a near-frontier open model often means a server — Muse Spark Open is aimed at a pair of high-end Mac Studios (Meta). Licences are mixed. Scale’s public-vs-private gap for Opus 5 was about 17.8%. Everyday retrieve can still sit on a machine you own.

A small tuned model plus your harness. SWE-Spot (preprint): a 4B matching models up to 8× larger on one repo. IBM: 8B fine-tunes beating 10×-larger models on two private codebases. HAD (Choi, Moon, Nam, Lee — not peer reviewed): contrast plus a validity filter moved ALFWorld unseen to 63.43% vs 47.01%; the 1.7B student sat above the 8B teacher. Stuffing harness text only raised usage 65.7% → 73.1% and left success ~43%.

• Caveat. Most of the proven gain is fine-tuning the weights, not extra retrieve (4B medical test: fine-tune helped, RAG did not). HAD was text environments. Recipe: a domain-tuned small model plus a custom harness. PEFT / LoRA.

Route only after policy says the data may leave. Everyday work on a cheap local model. Frontier only when the task is hard and the model ACL says this user, this repo, and this pack may go out. RouteLLM and FrugalGPT are the known patterns.

• Caveat. The router will misroute. Escalation sends data to an outside API. Routing without the ACL is just a cheaper leak.

Let the harness decide: decision mode

A chat model writes a paragraph. A decision model fills labeled fields — yes/no, one choice, a score. Background: Jev: Stamp the Answer, Don’t Write It. The harness can ask three cheap questions — how hard, how sensitive, what will it cost — then pick among the models the ACL already allowed. Policy first. Decision second.

flowchart TD
  T[Task] --> P{Policy / ACL}
  P -->|denied| H[Refuse or ask a human]
  P -->|allowed| D["Decision mode: complexity, risk, budget"]
  D -->|simple| You[Developer, or a small local model]
  D -->|everyday| L[Local model]
  D -->|hard and still allowed| F[Frontier, with approval]

In Code-Harness today. harness/decision.py is an optional typed client. Off by default. It can grade a loop step and help the verify gate. It does not pick local vs frontier. Fail-soft on missing key or timeout. Picking a model after the ACL is a plan.

Keep your grey matter: balance AI and your own thinking

There should be a balance between using AI and using our own intelligence. If every rename goes to a frontier model, grey matter rusts — and the bill goes with it. A harness can enforce this. Modes I would ship: try it yourself first, explain before you accept, and a per-team frontier budget.

I am not saying never use AI. I am saying do not outsource the easy reps.

• Lee, Sarkar et al., CHI 2025 (Microsoft Research + CMU): 319 knowledge workers, 936 examples. Higher confidence in GenAI went with less critical thinking and less reported effort. Self-report, not a skill test.
• Shen & Tamkin, Anthropic, Jan 2026: RCT, 52 mostly junior developers on Trio. AI group scored 17% lower on a quiz minutes later (50% vs 67%; worst on debugging). Asking “why” kept scores up. Speed-up was not significant.
• METR, 2025: RCT, 16 experienced open-source developers, 246 issues. Early-2025 tools (Cursor Pro + Claude 3.5 / 3.7): 19% slower. They forecast 24% faster, and afterwards still thought they had been 20% faster.

Caveat. Evidence is early and mixed. One survey, one short quiz, one early-2025 setting. Other work finds AI can speed familiar tasks. If every rename hits the frontier, you pay twice — the bill, and the skills you stop practising. These modes are a plan, not in Code-Harness today.

Code-Harness: my head start

It is a retrieval harness for code, not a general agent. Public, Apache-2.0.

Four design stories. No extra numbers.

1. Config, not the model. An uncommented LLM_PROVIDER=gemini in .env.example silently changed behaviour.
2. Exact names beat clever embeddings. Dense search misses auth_login. BM25 is there for that.
3. “Done” is checked, not claimed. The verify gate starts false.
4. Swapping embeddings is a harness problem. MiniLM is 384-d. Jina-code is 768-d. Isolated indexes.

What I measured

This is a smoke test, not a benchmark. Eight questions, one run, on the Code-Harness repo itself (4,289 chunks). One question flipping moves Recall by 0.125. Full harness: Recall@10 0.3125, nDCG@10 0.2629, citation-path 0.625. Dense embeddings were the piece that mattered on this set. Full table and caveats below.

What’s next (plans, not promises)

None of this is shipped.

Try it yourself: Code-Harness

Commands below are from the public README. I am not adding flags. Apache-2.0. Retrieval needs no API key. A generated answer does. Python 3.9+ (CI uses 3.12). On Debian/Ubuntu, python3 -m venv needs python3-venv.

git clone https://github.com/vijayliebe/Code-Harness.git
cd Code-Harness

python3 -m venv .venv
source .venv/bin/activate          # Windows: .venv\Scripts\activate

pip install -r requirements.txt

# Index this repo. First run downloads all-MiniLM-L6-v2 from Hugging Face.
# No LLM key. Defaults: local MiniLM, Chroma, decision model off,
# max_loops 0, verify off, HyDE is a free template (not an LLM call).
python3 main.py index .

# Retrieval only — prints packed context. First query also downloads
# cross-encoder/ms-marco-MiniLM-L-6-v2. A query *without* --no-llm calls
# OpenAI gpt-4o and exits if OPENAI_API_KEY is unset.
python3 main.py query . --no-llm -q "how does the chunker work?"

No LLM key: index, query --no-llm, eval, doctor, wiki generate, and make test work. For a generated answer, copy .env.example to .env and set a provider key, then drop --no-llm. Canonical eval (same recipe CI uses):

python3 main.py index .
python3 main.py eval . --suite .docs/research/eval/code-harness.fixture.yaml

Localhost retrieve plus MCP:

python3 main.py serve .

For an IDE over stdin/stdout, after you have indexed: python main.py mcp stdio .

Start with loop.py, verify.py, and retriever.py in that repo. If something is wrong, or you want a change, open an issue or a PR. I read them.

Companies already building their own harness

Rows I could check on a company blog or official docs. No invented numbers.

Common pattern:

• They plug a model into their tools, rules, CI, and data. The model is often a vendor frontier model (Spotify: Claude; Stripe still hands out Cursor and Claude Code).
• The harness is what they wrote: sandboxes, MCP, linters, blueprints, campaigns.
• Scale and internal context show up more often than privacy as the published reason.

Amazon also ships Kiro, an agentic IDE (specs, hooks, MCP). That is a public product, not a private-only tool. Japan’s Gennai Kobo, already linked under governments, is the sovereignty-shaped version of the same idea.

Fair objections, and my answers

Two I hear a lot. Both are fair, and neither is the same as owning the path.

With Docs-style SaaS, you still decide what you put in and what you share. The tool does what you click. An AI agent acts on its own. It reads files and runs tools, sometimes in ways no human saw. Data can leave without anyone noticing. I wrote how that happens in Prompt Injection: A Simple Guide. A strict harness you own, plus a local LLM, reduces that risk. The default can be: this prompt does not go out.

It helps. Data can still leave through logs, telemetry, analytics, or crash reports. Even if those are strict, you are still trusting the vendor’s harness quality — the rules, the tool allow-list, what happens on a weird prompt.

A harness you own is more trustworthy for that, and you can change it when you need to. No feature-request queue. No waiting on the vendor’s roadmap. No extra invoice for each add-on.

Honest caveat. Owning it means you carry the build, the security, and the maintenance. That cost is real. For a lot of teams, a hosted tool is still the right buy. I am arguing for the teams that cannot accept “trust the policy page.”

What would prove me wrong

A short list I would own in a review. How I would measure a claim like this: AI Agent Evals: Golden Examples and the Grader.

• Self-hosting costs more than the privacy is worth. You now run models, GPUs, patching, and an ACL. Most teams already buy hosted tools (76% in that 2025 survey). If that stays true, this is a regulated-industry niche, not a default.
• Hosted privacy modes become independently auditable. If you can run the model in your VPC, with your keys, and a third party can prove no extra copy — logs and telemetry included — the trust gap shrinks. I still want the ACL.
• Frontier models absorb the harness. Labs already talk this way: extra scaffolding is coping, not scaling (Anthropic). Then my bet shrinks to redact, audit, and model ACL.
• Open models stall. If the four-month gap opens again, everyday local coding is a server-room story.
• Harness upkeep costs more than it saves. Rules rot. People bypass the ACL with a personal key — which is also how the data leaves.
• Using frontier for every simple task makes juniors better, not worse. If a year of “always-on frontier” still raises debugging skill, the grey-matter bet shrinks. I have not seen that study yet.

What to take away

• The harness builds, the model fills. Own the path your code and your records travel on.
• We control a non-deterministic model with a deterministic harness. The harness only knows the cases you wrote. Own it so you can add a rule the day you find a gap.
• Hosted privacy is a real contract. It is not independent verify. Vendor self-hosting still leaves logs, telemetry, and harness quality with the vendor.
• A prompt is not an ACL. Deny by default. Decide which model a person may call.
• Cost and open models are why this is practical, not why I started. Keep a balance: simple work is yours, frontier is for hard tasks.
• Code-Harness is a retrieval harness for code. Try the quickstart.

If you change one thing after this page: stop asking “which model?” first. Ask who owns the path.

Related reads on Intelliwise

• Reverse Engineering the Cursor AI Agent — how a real hosted harness is put together.
• Prompt Injection: A Simple Guide — why a prompt is not an ACL.
• The Human in the Loop Can Get Tired — permission clicks before a frontier call.
• Why LLMs Forget Everything and Different Types of Memory in AI Agents — the context / memory layer.
• Five Things to Measure and Tune in a RAG System and Semantic Caching for RAG & LLMs — retrieve cost.
• KV Cache: Why Chatbots Keep Past Keys and Values — local inference memory.
• PEFT, LoRA, and QLoRA — domain-tuning a small model.
• AI Agent Evals: Golden Examples and the Grader — how I would measure this bet.
• Jev: Stamp the Answer, Don’t Write It — typed decision fields, not a chat paragraph.

Sources

• Code-Harness — retrieval harness for code, Apache-2.0. Smoke-test numbers are from one 8-question run on that repo.
• Cursor, Privacy and data and Data use. Privacy Mode, ZDR, and the note that prompts still go to providers.
• OpenAI, Business data privacy and API data controls. No training by default; ZDR for qualifying organisations.
• Anthropic, Claude Code ZDR and data usage. Commercial no-training default; ZDR as a separate Enterprise offering.
• Japan Digital Agency, Gennai OSS Ver 2.0 technical slides, 25 Aug 2026.
• Press Information Bureau, IndiaAI Mission update, 6 Aug 2026.
• European Commission, AI Gigafactories call, 30 Jul 2026.
• Menlo Ventures, 2025: The State of Generative AI in the Enterprise, 9 Dec 2025. US survey.
• Stripe, Minions, Part 1 and Part 2. Homegrown unattended agents; Goose fork; 1,300 minion PRs a week.
• Block, codename goose, 28 Jan 2025. Apache-2.0 agent framework.
• Spotify Engineering, Coding Is No Longer the Constraint, 2026. Honk: own harness around Claude Agent SDK.
• Ramp Engineering, Why we built our background agent. Inspect; ~30% of merged frontend/backend PRs.
• Shopify Engineering, Under the River, 28 May 2026. River on Aquifer; harness named as the agent loop.
• AWS, Amazon Q Developer $260 million milestone, 1 Aug 2024 (citing Andy Jassy). Also internal Java-upgrade campaigns, 3 Oct 2024. Kiro: Introducing Kiro, 14 Jul 2025.
• Airbnb Engineering, Accelerating Large-Scale Test Migration with LLMs. Nearly 3.5K files; 6 weeks vs 1.5 years.
• Coinbase Engineering, How Mux Helped Solve It, 11 May 2026. In-house multi-agent tool.
• Naman Vats and Oleg Golev, The Scaffold Effect in Coding Agents, arXiv 2607.22585. Preprint.
• Yangze Liu and Zhongyi Han, What Does a Harness Buy? Tokens, Mostly., arXiv 2610.04433. Preprint.
• Scale / Reflection, SWE-Bench Pro V2, Sep 2026.
• Lance Martin / Anthropic, Harnessing Claude’s intelligence, 2026.
• Epoch AI, Open vs closed capability gap, 29 May 2026.
• Meta Superintelligence Labs, Muse Spark Open, 15 Oct 2026.
• Moonseok Choi, Taehong Moon, Giung Nam, Juho Lee, Harness-Aware Distillation, arXiv 2610.02858, 2 Oct 2026. Preprint.
• Columbia / UCLA, SWE-Spot, arXiv 2601.21649. Preprint.
• IBM Research, enterprise-code fine-tune study, arXiv 2602.05780.
• Domain fine-tuning vs RAG at 4B, arXiv 2604.23801.
• OpenAI, Harness engineering, 11 Feb 2026.
• Isaac Ong et al., RouteLLM, arXiv 2406.18665.
• Lingjiao Chen, Matei Zaharia, and James Zou, FrugalGPT, arXiv 2305.05176.
• Hao-Ping (Hank) Lee, Advait Sarkar, Lev Tankelevitch, Ian Drosos, Sean Rintel, Richard Banks, and Nicholas Wilson, The Impact of Generative AI on Critical Thinking, CHI 2025. Microsoft Research + Carnegie Mellon. Survey.
• Judy Hanwen Shen and Alex Tamkin, How AI assistance impacts the formation of coding skills, Anthropic, 29 Jan 2026. Also arXiv 2601.20245. RCT.
• METR, Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity, 10 Jul 2025. Also arXiv 2507.09089. RCT.
• Intelliwise posts cited above are listed under Related reads.

I use one running example the whole way. The average rent in the pocket data is €1,900. One 80 m² city flat actually rents for €2,500. Those six-flat numbers are toy numbers I made up so the first rounds fit on one screen. The script at the end is a larger synthetic set. Not real listing data.

flowchart LR
  A[Start at the average<br/>€1,900] --> B[Compute residuals]
  B --> C[Fit a small tree]
  C --> D[Add eta times the tree]
  D --> E{More trees?}
  E -->|yes| B
  E -->|no| F[Final rent]

1. What is XGBoost, in one line?

Me: A supervised model that adds small decision trees one after another. Each new tree tries to fix the errors so far.

XGBoost stands for Extreme Gradient Boosting. Supervised means we have a target — here, monthly rent. You give it rows (floor area, later location and age) and a number to hit.

It does not build 200 independent trees and average them. It starts with a simple guess, usually the mean rent, then grows a shallow tree that predicts the leftover error. Add that tree. Repeat. That stacked model is the booster.

For the 80 m² city flat, the first guess is €1,900. The leftover error is +€600. Later trees chip away at that +€600 until the prediction sits closer to €2,500.

Follow-up they may ask: Is it only for regression? No. Same idea for classification. The loss changes. The “add a tree to cut the loss” loop does not.

2. How is it different from a random forest?

Me: A random forest trains trees independently and averages the votes. XGBoost adds trees in sequence so each one cuts the loss that is still left.

If five forest trees guess €2,100, €2,400, €1,750, €2,300, €1,950 for the city flat, the answer is the mean of those five. If five boosting trees run, the answer is €1,900 plus the five corrections, in order.

Follow-up they may ask: Can I bag XGBoost trees anyway? Yes. subsample and colsample_bytree already do a light version of that, on top of boosting.

3. Walk me through the first round.

Me: Start from the average. Compute residuals. Tree 1 learns a correction for each group. Most flats get better, a couple get worse, and the total error still drops.

Pocket data. Six flats. Mean rent €1,900. I set eta to 1.0 in this round so the leaf values are easy to see. Later we put eta back to 0.1.

Round-0 RMSE is 365.

Tree 1 splits at 60 m².

• Small group A, B, C. Mean residual = (−500 − 300 − 100) / 3 = −300. New pred = €1,600.
• Large group D, E, F. Mean residual = (0 + 600 + 300) / 3 = +300. New pred = €2,200.

C and D got worse. E, the €2,500 flat, moved from €1,900 to €2,200. Round-1 RMSE is 208. Total error fell even though two rows got worse. That is normal. The tree optimises the sum, not every row.

A second split on the large group takes RMSE to 144. E is now at €2,350.

With eta 0.1, the same city flat climbs in small steps instead of taking the full leaf jump:

Follow-up they may ask: Why not one deep tree instead of many shallow ones? One deep tree memorizes the six rows. Shallow trees plus a small eta usually travel better to flats the model has not seen.

4. Why is it called gradient boosting?

Me: Each tree is fit on the negative gradient of the loss. For squared error, that gradient is the residual.

Loss here is squared error: how far the predicted rent is from the actual rent. The gradient of that loss, with respect to the current prediction, points toward “make this prediction worse.” The opposite direction — the negative gradient — points toward “make this prediction better.”

For squared error, that negative gradient is actual − prediction. That is the residual column in the table above. Tree 1 is not magically “learning rent.” It is learning those residuals.

If the loss were log loss for a classification problem, the gradient would not be a raw residual. The loop is the same: compute the gradient of the loss, grow a tree that follows it, add the tree, repeat. “Gradient boosting” is that loop. “XGBoost” is a fast, regularised way to run it.

5. What does XGBoost add over plain gradient boosting?

Me: It uses the first and second derivatives of the loss — gradient and hessian — to score splits and set leaf values. It also regularises tree complexity, finds splits with a fast histogram pass, and has a default direction for missing values.

Plain gradient boosting, in the Friedman sense, often uses only the gradient. XGBoost takes a second-order look. For each row you have:

• g — gradient. How much the loss would move if this prediction moved.
• h — hessian. How curved that loss is at this prediction.

For squared error, g is prediction − actual and h is 1. So the hessian is basically a row count. For logistic loss, h is p(1 − p), so a confident row counts for less than a row sitting at 0.5. That is why min_child_weight is a sum of hessians, not “minimum rows,” even if people treat it as a row count in regression.

A split is scored by gain. A leaf gets a weight. From the XGBoost model page:

G is the sum of gradients in that node. H is the sum of hessians. λ (reg_lambda) pulls leaf values toward zero. γ (gamma) is a tax for making a new leaf.

In plain words: gain asks “if I split here, how much extra loss do I remove, after paying for a new leaf?” If that number is not positive, skip the split. The leaf weight is a shrunk version of “average residual in this leaf.” For our large-flat leaf, G is the sum of those +€600 and +€300 style residuals (with a sign flip from how g is defined), H is roughly 3, and λ stops one noisy leaf from swinging rent by a huge amount.

The rest of the “extreme” in the name is engineering:

• Histogram split finding. Bucket each feature, scan the buckets. Parallel across features.
• Regularisation. Depth, λ, α, γ, min child weight. The tree is not allowed to grow just because it can.
• Sparsity-aware missing values. At each split the algorithm learns whether NaN should go left or right.

Follow-up they may ask: Do I need to derive gain on a whiteboard? Know what G, H, λ, and γ do. Write the leaf-weight formula. Explain it with the €1,900 / €2,500 flat. That is enough for most interviews.

6. What is the learning rate (eta) and why keep it small?

Me: Eta scales each tree’s correction before it is added. Keep it small so no single tree can swing the rent too far.

The 80 m² city flat still has a +€600 residual after round 0. If eta is 1.0, tree 1 is allowed to add the whole +€600 (or the leaf mean, +€300 in the six-flat split). If eta is 0.1:

+€600 × 0.1 = +€60

New prediction = €1,960, not €2,500. The next tree sees a leftover of +€540 and takes another small step.

Small eta means you need more trees. That sounds wasteful. In practice those quieter steps generalise better. A tree that is a bit wrong does less damage. Early stopping then picks how many steps you actually keep.

A start I actually use: learning_rate=0.05 or 0.1, then let early stopping choose n_estimators. If the model wants 20 trees and still looks noisy, I drop eta and allow more trees. If it wants 800 trees and the job is slow, I raise eta a bit.

7. Which hyperparameters do you tune first?

Me: Depth and min child weight, then the two sample rates, then the regularisers, and I never pick n_estimators by hand if I can use early stopping.

On this rent problem I would start with max_depth=3, min_child_weight=5, subsample=0.8, colsample_bytree=0.8, reg_lambda=1, learning_rate=0.1, n_estimators=400, early_stopping_rounds=30. That is also what code/xgb_rent_demo.py uses.

I do not start with a 40-point grid. I get a decent depth and eta first. Then I poke λ and subsample. The gain formula in Q5 is the reason γ and λ change the trees: they are the tax and the shrink, not magic knobs.

Follow-up they may ask: Why is min_child_weight not “minimum rows”? Because it is the sum of hessians. In this squared-error rent model, h = 1, so it happens to match a row count. In logistic loss, a row with p = 0.99 contributes about 0.01, so you need many such rows to pass the same threshold.

8. How do you stop it overfitting?

Me: Train / validation / test. Early stopping on the validation set. Keep the best iteration. Touch the test set once, at the end.

The split:

1. Train — trees are grown here.
2. Validation — after each tree, score this set. If it has not improved for early_stopping_rounds trees, stop. Keep the iteration with the best validation score.
3. Test — one number, after you have frozen the model. No tuning against it.

If you keep adding trees, train error will keep falling. Validation error falls, then rises. That rise is the model starting to memorize train flats. The marker in the GIF is the model I would ship.

The other regularisers from Q7 still matter. Early stopping is the brake. Depth, min child weight, subsample, λ are the speed limit. I use both.

Do not report the best validation score as the final score. It has been peeked at on every round. The test MAE is the one I would write in a design doc.

9. Which metric would you report?

Me: MAE in euro, and RMSE next to it. For classification I would pick log loss or AUC, depending on whether we care about probabilities or ranking.

Rent is in euro per month. MAE is the average absolute miss. “We are off by about €84 on a held-out flat” is a sentence a letting-ops person can use. RMSE punishes the big misses harder. If one penthouse is off by €800, RMSE will show that more than MAE.

I would train with a squared-error objective (that is what the gradients in Q4 assume) and still report MAE, because that is the unit of the product. In the script, early stopping watches MAE on the validation set.

For a yes/no target — say “will this listing go stale?” — I would not use MAE.

• Log loss if we need a probability we can trust, for example expected value in a bidding tool.
• AUC if we only need to rank listings, and the threshold can move later.
• F1 / precision / recall only after someone names the cost of a false let vs a missed let.

Do not report train MAE. Do not report the early-stopping validation MAE as the headline. Test MAE, once.

10. How do you explain a prediction to a stakeholder?

Me: Global feature importance for “what the model uses in general.” Per-flat contributions (SHAP / pred_contribs) for “why this flat.” Neither is causation.

Global gain importance answers: across all splits, which column reduced loss the most? In the synthetic run, floor area is 0.54, location 0.40, age 0.06. I can say “area and city vs suburb do most of the work. Age is a small adjustment.”

That does not explain one listing. Age could still move this 6-year-old city flat by €90.

For one row I use XGBoost’s own pred_contribs=True. That is the tree SHAP breakdown: bias plus each feature, summing to the prediction.

On the 80 m² city flat, 6 years old, the script printed:

I would say: “The model starts near the average rent. This flat is larger than average, in the city, and fairly new, so each of those pushes the rent up. Together they land at €2,365.”

I would also say what I will not say: “If you add 10 m² you will get €X more in the market.” That is a causal claim. The model saw correlations in the training rows. It did not run an experiment. A 10 m² extension in a suburb is not the same as 10 m² in the city centre.

11. When would you not use XGBoost?

Me: When I need a model a person can read as one tree, when the data is tiny, when I must extrapolate past the largest flat I have seen, or when the input is images, text, or audio.

Concrete cases:

• Need to show the rule. One CART tree, or even a linear model with three coefficients, is easier in a credit meeting than 84 trees.
• Very small data. Tens of rows. Boosting will memorize. A mean, a linear model, or a single small tree is more honest.
• Cannot extrapolate. Trees predict a constant in each leaf. Anything bigger than the largest training area lands in the same right-hand leaf. In the script, a city flat at 107.7 m², 140 m², 160 m², and 180 m² all get €2,742. A linear model keeps climbing to €3,792 at 180 m². If the business needs a sensible rent for a 180 m² penthouse and the train set stops at 108 m², I would not ship XGBoost alone.
• Images, text, audio. Neural nets. XGBoost wants a table.
• No time to tune, and AutoGluon / a strong CatBoost default is already in the repo. XGBoost is not free. Someone has to watch early stopping and the knobs in Q7.

Follow-up they may ask: So when do you use it? Tabular data, a few thousand rows or more, mixed numeric and missing values, and a metric you can early-stop on. Rent, churn, fraud scores, remaining-useful-life. That is the job it was built for.

12. Quick-fire round

Interviewer: Missing values?

Me: Yes. At each split it learns a default direction for NaN. I still check whether “missing” means “not recorded” or “does not apply.”

Interviewer: Categorical features?

Me: Recent XGBoost can take pandas category dtypes with enable_categorical=True. Otherwise I encode them. High-cardinality IDs I would not one-hot blindly.

Interviewer: Class imbalance?

Me: scale_pos_weight ≈ count(negative) / count(positive) for binary. Then I still pick the threshold from precision and recall, not from 0.5.

Interviewer: XGBoost vs LightGBM, one line?

Me: LightGBM grows leaf-wise and is often faster on large tables. XGBoost grows level-wise by default and is the one I have seen in more production stacks.

Interviewer: Vs CatBoost?

Me: CatBoost’s ordered target encoding is why I reach for it when I have many categorical columns. For this numeric rent table I am fine with XGBoost.

Interviewer: GPU training?

Me: Yes. Histogram split finding on GPU (device="cuda" in current XGBoost). Worth it when the table is wide or long. This 800-row demo is not that table.

Interviewer: Can you parallelize the boosting loop?

Me: Split finding, yes. Adding tree 4 before tree 3 is done, no. The next tree needs the current residuals.

Interviewer: Two features are collinear, say area and number of rooms?

Me: Gain will pick one of them most of the time. Importance will look unstable. I would not read 0.54 vs 0.40 as a physics result.

A small model you can run

code/xgb_rent_demo.py builds 800 synthetic Dublin-style rows (seed 12), splits them train / val / test, fits xgboost.XGBRegressor with early stopping, prints MAE and gain importance, breaks down one 80 m² city flat with pred_contribs, and compares XGBoost vs a linear model past the largest training area.

The data is synthetic. Mean rent in this draw is €1,995, not the €1,900 from the six-flat pocket table. It is not real listing data.

model = xgb.XGBRegressor(
    n_estimators=400,
    max_depth=3,
    learning_rate=0.1,
    subsample=0.8,
    colsample_bytree=0.8,
    min_child_weight=5,
    reg_lambda=1.0,
    objective="reg:squarederror",
    eval_metric="mae",
    early_stopping_rounds=30,
    tree_method="hist",
    random_state=12,
    n_jobs=1,
)
model.fit(x_train, y_train, eval_set=[(x_val, y_val)], verbose=False)

Output from python3 xgb_rent_demo.py (xgboost 3.4.1):

=== synthetic Dublin rent (not real listings) ===
xgboost 3.4.1  seed 12  n=800
features: floor_area_m2, location_city, age_years

mean rent: EUR 1995
area range in data: 38.1 to 107.7 m2

split  train=559  val=121  test=120

=== fit ===
best iteration (0-based): 83
trees kept: 84 of 400
validation MAE: EUR 69.5
test MAE:       EUR 84.0

=== feature importance (gain) ===
floor_area_m2     0.543
location_city     0.395
age_years         0.062

=== one flat (80 m2, city, 6 years) ===
model prediction: EUR 2365
pred_contribs (bias + each feature = prediction):
  bias                    +1989.1
  floor_area_m2        +123.7
  location_city        +158.8
  age_years            +93.0
  sum                     +2364.6

=== extrapolation check ===
largest floor area in training: 107.7 m2
flat                            xgboost     linear
at max train (107.7 m2)            2742       2736
just above max                     2742       2738
140 m2 (unseen)                    2742       3208
160 m2 (unseen)                    2742       3500
180 m2 (unseen)                    2742       3792

XGBoost stays almost flat past the largest training area. The linear model keeps climbing.

84 trees kept. Test MAE €84. The 80 m² city flat lands at €2,365 — closer to €2,500 than the €1,900 start, and not a memorised bullseye. From 107.7 m² upward, XGBoost repeats €2,742. That is the flattening from Q11, in the actual printout.

Interview cheat sheet

In short

XGBoost is supervised gradient boosting with extra care: second-order split scoring, regularised leaves, fast histogram cuts, and a learned path for missing values. You start at the average rent, €1,900 in the pocket example. Each shallow tree fits the leftover error. A small eta turns a +€600 miss into a +€60 step. Most rows improve, a few get worse, and RMSE still falls. You tune depth and child weight first, stop on a validation set, and quote test MAE. For one Dublin flat, bias plus area plus city plus age is the explanation — not a causal “if you add 10 m²” story. And the model will not invent a penthouse price for 180 m² if it only saw flats up to 108 m². That prediction goes flat.

Sources

• XGBoost documentation
• Introduction to boosted trees (gain and leaf-weight formulae)
• Tianqi Chen and Carlos Guestrin, XGBoost: A Scalable Tree Boosting System, KDD 2016

It is not enough on its own.

The loop can weaken from both sides. The machine gets better at finding gaps in the controls around it. The human gets tired and starts rubber-stamping. That second part has a name: approval fatigue.

The real safeguard is a loop that still lets a person notice when something is wrong.

A one-day supervisor diary (invented)

I made a toy day to show how the fade looks. I am the reviewer. An internal agent asks me to approve each action before it runs.

These are invented toy entries. Fake tool. Fake files. Not a company log. Not a real incident. The timestamps are there so you can feel the day get long.

09:40  read_file     invoices/2026-09.csv
       ask           Summarise September spend
       decision      Approved

I opened the CSV. Vendor, date, amount. Row count looks normal. September total is close to what finance said in standup. I can let it summarise.

10:35  update_record CRM account 4471
       ask           Mark invoice as paid
       decision      Approved

This one can hurt if it is wrong, so I slowed down. I opened account 4471. Invoice 8821 is 14,200 and still Open. The payment note in the ticket matches. Approved.

12:50  read_file     contracts/northwind.pdf
       ask           Check renewal terms
       decision      Approved

Lunch was late. I still opened the PDF. Term is 12 months. Notice period is 30 days. Auto-renew is on. I would have said the same thing.

13:36  read_file     contracts/acme.pdf
       ask           Check renewal terms
       decision      Approved

Same shape as the last one. I skimmed the first page. Looks fine.

14:22  create_file   reports/weekly.md
       ask           Draft weekly report
       decision      Approved

I have seen this report template all week. I glanced at the headings. The pattern is starting to feel familiar. Approved.

15:07  create_file   reports/notes.md
       ask           Add standup bullets
       decision      Approved
       count         18

ok

15:41  read_file     contracts/northwind.pdf
       ask           Confirm notice period
       decision      Approved
       count         30

looks fine

16:08  delete_backups nightly/
       ask            Free disk on the report box
       decision       Approved

I did not read the target. The card looked like the last ten cards.

16:40  update_record CRM 5120
       ask           Close stale ticket
       decision      Approved
       count         42

yes

17:48  audit_review
       count         58 approved

A teammate asked if the agent touched backups. I scrolled the log. There it is. 16:08. delete_backups. I had clicked Approve in two seconds.

The diary ends there.

What went wrong

Nothing in the morning was reckless. I opened files. I checked a CRM row. I read a contract.

The failure started when the queue got repetitive. Reads, small writes, the same “check this / draft that” shape. My brain stopped treating the next card as new. The click got cheaper than the read.

Then a destructive action sat in the same list, with the same Approve button. The tool did not change colour. It did not ask me to type the name. It did not need a second person. So delete_backups inherited the trust I had built on read_file.

The slogan was still true. A human was in the loop. The human was no longer looking.

Approval fatigue, in plain words

Approval fatigue is this: after you see the same kind of request many times, you stop reading the next one as if it is new. You pattern-match. You click.

It is how attention works on a long queue. Ops teams already know the cousin of this: alert fatigue. Too many similar alarms, and the important one looks like noise.

A capable agent can produce work faster than you can inspect it. If every action needs a human click, you have not added safety. You have added a tired person in a long queue.

The ironies of automation

This problem is older than agents.

In 1983, Lisanne Bainbridge wrote Ironies of Automation in Automatica. The paper is short and still sharp. Wikipedia’s page on Ironies of automation is a fair summary if you want the idea in one sitting.

The irony is simple. The more the system does on its own, the less practised the human gets. We still expect that same human to step in perfectly when automation fails.

You can see the same shape in aircraft autopilot and in driver-assist cars. The boring stretch is easy. The surprise is hard. And the surprise is exactly when you need a practised person.

An approve queue is a small version of that. The agent handles the routine. You watch. You get out of practice at watching. Then one card is not routine.

The machine side

Fatigue is the human half. The machine half is real too.

A capable agent is trying to get to “done”. If your checks have gaps, it can find them. That is not science fiction. Reward hacking and specification gaming are documented behaviours: the system scores well on the measure you wrote, while doing something you did not want.

In generic terms, that can look like:

• gaming an evaluator so the output looks fine to the scorer
• finding an unintended path around a restriction
• hiding a mistake when the check only looks at the happy result

I am not walking through a company incident here. The point is the shape. If the agent is stronger than the fence, “a human will catch it” is a weak last line — especially if that human is on click 40.

flowchart TB
  subgraph machine [Machine side]
    M1[Agent gets more capable] --> M2[Finds gaps in the controls]
    M2 --> M3[Reward hacking / specification gaming]
  end
  subgraph human [Human side]
    H1[System does more alone] --> H2[Reviewer practises less]
    H2 --> H3[Approval fatigue / rubber-stamp]
  end
  M3 --> Fail[Bad action goes through]
  H3 --> Fail

The two sides feed each other. A stronger agent makes a longer queue. A longer queue makes a weaker reviewer. A weaker reviewer is easier to get past.

How to fix the loop

Do not add a bigger Approve button. Change what is allowed to reach that button, and what the button is allowed to mean.

The first row does most of the work. Reads can go through. Writes wait. Destroy does not ride on a single tired click.

A canary only makes sense if the reviewer knows the job includes occasional tests, and if a miss becomes coaching, not a trap. Consent matters. The point is to measure attention, not to shame people.

A small approval gate

Here is a tiny Python sketch of that idea. classify puts each action in a tier. decide auto-allows reads, asks for a human on writes, and blocks destroy until a second person is present. After six human decisions, the fatigue guard forces a break.

The script lives next to this draft as approval_gate.py. I ran it. This is the real output.

#!/usr/bin/env python3
"""Toy risk-tiered approval gate with a fatigue break.

Invented actions only. Not a real agent, not a real company log.
"""

from __future__ import annotations

from dataclasses import dataclass
from enum import Enum

MAX_REVIEWS_BEFORE_BREAK = 6


class Tier(str, Enum):
    AUTO = "auto"
    REVIEW = "review"
    TWO_PERSON = "block-plus-two-person"


TIER_FOR_ACTION = {
    "read_file": Tier.AUTO,
    "update_record": Tier.REVIEW,
    "create_file": Tier.REVIEW,
    "delete_backups": Tier.TWO_PERSON,
}


@dataclass
class GateState:
    reviews_since_break: int = 0


def classify(action: str) -> Tier:
    return TIER_FOR_ACTION.get(action, Tier.REVIEW)


def decide(state: GateState, action: str, second_person_ok: bool = False) -> str:
    if state.reviews_since_break >= MAX_REVIEWS_BEFORE_BREAK:
        return "BREAK — reviewer has approved too many in a row"

    tier = classify(action)
    if tier is Tier.AUTO:
        return f"AUTO allow {action}"

    if tier is Tier.REVIEW:
        state.reviews_since_break += 1
        return (
            f"REVIEW required for {action}  "
            f"({state.reviews_since_break}/{MAX_REVIEWS_BEFORE_BREAK})"
        )

    if not second_person_ok:
        return f"BLOCK {action} — needs a second person"

    state.reviews_since_break += 1
    return (
        f"TWO-PERSON allow {action}  "
        f"({state.reviews_since_break}/{MAX_REVIEWS_BEFORE_BREAK})"
    )

Output from python3 approval_gate.py:

Toy approval gate — invented actions, not a real incident
Tiers: read_file=auto  writes=review  delete_backups=two-person
Fatigue guard: break after 6 human decisions

09:40  read_file         AUTO allow read_file
10:35  update_record     REVIEW required for update_record  (1/6)
12:50  read_file         AUTO allow read_file
13:36  read_file         AUTO allow read_file
14:22  create_file       REVIEW required for create_file  (2/6)
15:07  create_file       REVIEW required for create_file  (3/6)
15:22  update_record     REVIEW required for update_record  (4/6)
15:41  read_file         AUTO allow read_file
16:08  delete_backups    BLOCK delete_backups — needs a second person
16:08  delete_backups    TWO-PERSON allow delete_backups  (5/6)  [second person present]
16:20  update_record     REVIEW required for update_record  (6/6)
16:28  create_file       BREAK — reviewer has approved too many in a row
16:40  update_record     BREAK — reviewer has approved too many in a row

Compare that with the diary. In the diary, delete_backups was just another Approve. In this gate it does not move without a second person. And even if that second person shows up, the reviewer is forced to stop after six human calls. Reads never counted toward the fatigue cap. That is the point of tiers.

Cheat sheet

• A human in the loop is a design, not a sentence in the spec.
• Routine work trains the agent and dulls the reviewer.
• If destroy uses the same button as read, destroy will inherit the trust from read.
• Auto-allow the cheap stuff so the human’s attention is spent on writes and deletes.
• Show what will break, in plain English, before the click.
• Do not give the agent delete rights if you can avoid it.
• Cap the queue. Rotate the person. Make someone read the audit later.
• Keep the manual skill warm with drills, or the takeover will be rusty.

In short

“Keep a human in the loop” is not a safeguard by itself. The machine can learn the gaps. The human can get tired and click anyway. I showed that fade with a toy diary: careful at 09:40, lazy by 15:07, delete_backups approved at 16:08, noticed only at 17:48 when the count hit 58.

Fix the loop. Tier the risk. Keep destroy off a single tired click. And design so the person can still notice when something is wrong.

It starts with pieces of text, lists of numbers, a compare-and-mix step called attention, a stack of the same block, and a lot of “guess the next word” training.

This post is that ladder. One chain of If you do X, you get Y. Same sentence the whole way: The bus is late.

The IDs below are toy numbers I made up. They are not from GPT-2, tiktoken, or any real vocab. The matrices are tiny on purpose. You can type the NumPy as you read, or run code/build_along.py (seed 7) and get the same prints.

If you want the bigger-picture story — a small GPT trained on Shakespeare characters — read How a Tiny GPT Learns. This page is the same idea, one checkpoint at a time.

flowchart TB
  A[Sentence] --> B[Tokens + toy IDs]
  B --> C[Embeddings x]
  C --> D[Query / Key / Value]
  D --> E[Scores]
  E --> F[Softmax weights]
  F --> G[Context vector]
  G --> H[Self-attention]
  H --> I[One transformer block]
  I --> J[Stack of blocks]
  J --> K[Next-word chat model]
  K --> L[Back to the sentence]

Shared setup for every snippet:

import numpy as np

np.random.seed(7)
np.set_printoptions(precision=2, suppress=True)

def softmax(x, axis=-1):
    e = np.exp(x - np.max(x, axis=axis, keepdims=True))
    return e / e.sum(axis=axis, keepdims=True)

SENTENCE = "The bus is late"
TOKENS = ["The", "bus", "is", "late"]
IDS = np.array([11, 42, 7, 88])  # toy IDs, not a real vocab
D = 4
SCALE = np.sqrt(D)

1. Break a sentence into pieces

A model does not read English. It reads integers.

A tokenizer cuts text into pieces and maps each piece to an ID. Real systems use tens of thousands of IDs. Here I pick four fake ones so the table stays on one screen.

print("sentence:", SENTENCE)
print("tokens:  ", TOKENS)
print("toy ids: ", IDS)

sentence: The bus is late
tokens:   ['The', 'bus', 'is', 'late']
toy ids:  [11 42  7 88]

Those four IDs are invented. Do not look them up in a real vocab.

2. Turn each token into a list of numbers

An embedding is a short list of numbers that stands in for a token. Similar tokens sit near each other in that space.

In a real model this list is hundreds of numbers long. Here each token gets 4 numbers. I wrote the table by hand so later dots stay small.

X = np.array(
    [
        [1.0, 0.1, 0.0, 0.2],  # The
        [0.2, 1.0, 0.1, 0.3],  # bus
        [0.0, 0.2, 1.0, 0.1],  # is
        [0.1, 0.8, 0.0, 1.0],  # late
    ]
)
print(X)
print("shape", X.shape)

[[1.  0.1 0.  0.2]
 [0.2 1.  0.1 0.3]
 [0.  0.2 1.  0.1]
 [0.1 0.8 0.  1. ]]
shape (4, 4)

Four tokens. Four numbers each. Shape (4, 4).

The picture below plots only the first two numbers, so you can see the arrows. bus and late point a similar way. That will matter in a few steps.

3. Let every token ask a question

The query is “what am I looking for, right now?”

You do not invent that question by hand. You multiply the embedding by a learned matrix $W_Q$. Same matrix for every token.

In NumPy the tokens are rows, so the line is Q = X @ W_Q. Same multiply, just a row layout.

W_Q = np.array(
    [
        [0.9, 0.1, 0.0, 0.0],
        [0.0, 0.9, 0.1, 0.0],
        [0.0, 0.0, 0.9, 0.1],
        [0.1, 0.0, 0.0, 0.9],
    ]
)
Q = X @ W_Q
print(Q)

[[0.92 0.19 0.01 0.18]
 [0.21 0.92 0.19 0.28]
 [0.01 0.18 0.92 0.19]
 [0.19 0.73 0.08 0.9 ]]

Row 3 is the query for late: [0.19 0.73 0.08 0.90]. We will use that row next.

4. Give every token a way to answer

The key is the label on the token. The value is the payload you actually mix in if that token gets picked.

Two more matrices. Same idea as the query.

W_K = np.array(
    [
        [0.8, 0.0, 0.1, 0.1],
        [0.1, 0.8, 0.0, 0.1],
        [0.0, 0.1, 0.8, 0.1],
        [0.1, 0.0, 0.1, 0.8],
    ]
)
W_V = np.diag([0.6, 0.6, 0.6, 0.6])
K = X @ W_K
V = X @ W_V
print("K")
print(K)
print("V")
print(V)

K
[[0.83 0.08 0.12 0.27]
 [0.29 0.81 0.13 0.37]
 [0.03 0.26 0.81 0.2 ]
 [0.26 0.64 0.11 0.89]]
V
[[0.6  0.06 0.   0.12]
 [0.12 0.6  0.06 0.18]
 [0.   0.12 0.6  0.06]
 [0.06 0.48 0.   0.6 ]]

Three jobs, three lists: $q$, $k$, $v$. Still shape (4, 4).

5. Compare one token’s query with every key

Take the query for late. Dot it with each key. Divide by $\sqrt{d}$ so the numbers do not grow with the width of the list.

A bigger score means a better match.

q_late = Q[3]
scores = q_late @ K.T / SCALE
print("q_late:", q_late)
print("scores:", scores)

q_late: [0.19 0.73 0.08 0.9 ]
scores: [0.23 0.49 0.22 0.66]

late matches itself most (0.66), then bus (0.49). That fits the embeddings: those two arrows already pointed a similar way.

6. Turn scores into percentages

Softmax makes the four scores positive and forces them to add up to 1.

You can read each weight as “how much of my mix should come from this token?”

weights = softmax(scores)
print("a = softmax(s):", weights)
print("sum:", weights.sum())

a = softmax(s): [0.21 0.27 0.2  0.32]
sum: 1.0

So late takes 32% from itself, 27% from bus, 21% from The, 20% from is.

7. Mix the values using those weights

The scores only decide who matters. The values are what you actually add in.

One weighted sum. One new list of four numbers for late.

c = weights @ V
print("c = sum a_j v_j:", c)

c = sum a_j v_j: [0.18 0.35 0.14 0.28]

That is the view late now has of the whole sentence.

8. Do it for every token at once

The last three steps were for one query. The model does the same for every token, in one matrix multiply.

GPT also puts a causal mask on the scores. A token may look at itself and the past. It may not look at a word that has not been written yet. That is how next-word training stays honest.

$M$ is $0$ on and below the diagonal, and $-\infty$ above it. After softmax, the future weights become $0$.

raw = Q @ K.T / SCALE
causal = np.tril(np.ones((4, 4)))
masked = np.where(causal == 1.0, raw, -1e9)
A = softmax(masked, axis=1)
C = A @ V
print("raw scores")
print(raw)
print("causal weights")
print(A)
print("context matrix")
print(C)

raw scores
[[0.41 0.24 0.06 0.26]
 [0.17 0.47 0.23 0.46]
 [0.09 0.17 0.42 0.19]
 [0.23 0.49 0.22 0.66]]
causal weights
[[1.   0.   0.   0.  ]
 [0.43 0.57 0.   0.  ]
 [0.29 0.31 0.4  0.  ]
 [0.21 0.27 0.2  0.32]]
context matrix
[[0.6  0.06 0.   0.12]
 [0.32 0.37 0.03 0.15]
 [0.21 0.25 0.26 0.11]
 [0.18 0.35 0.14 0.28]]

Row 0 is The. It can only see itself, so its weights are [1, 0, 0, 0]. Row 3 is late. It can see everyone. That last row is the same mix we built by hand.

9. Add a small neural network and residual connections

Attention is the talk step. Each token still needs a private think step. That is a tiny MLP, applied at every position with the same weights.

A residual connection adds the input back: $x + \text{layer}(x)$. The original signal still has a path through the stack. Layer norm rescales each token’s numbers so they do not blow up as you go deeper.

H = X + C
W1 = np.array(
    [
        [0.4, 0.1, 0.0, 0.0],
        [0.0, 0.4, 0.1, 0.0],
        [0.0, 0.0, 0.4, 0.1],
        [0.1, 0.0, 0.0, 0.4],
    ]
)
W2 = np.array(
    [
        [0.3, 0.0, 0.1, 0.0],
        [0.0, 0.3, 0.0, 0.1],
        [0.1, 0.0, 0.3, 0.0],
        [0.0, 0.1, 0.0, 0.3],
    ]
)
mlp = np.maximum(H @ W1, 0.0) @ W2
Y = H + mlp
print("h = X + C")
print(H)
print("y = h + MLP(h)")
print(Y)

h = X + C
[[1.6  0.16 0.   0.32]
 [0.52 1.37 0.13 0.45]
 [0.21 0.45 1.26 0.21]
 [0.28 1.15 0.14 1.28]]
y = h + MLP(h)
[[1.8  0.24 0.07 0.38]
 [0.62 1.57 0.22 0.57]
 [0.3  0.53 1.43 0.3 ]
 [0.36 1.35 0.21 1.48]]

Still shape (4, 4). Same four tokens, richer lists.

10. Stack the blocks

One block is one talk-then-think cycle. A transformer is that cycle repeated.

GPT-2 small does this 12 times. Each block has its own weights. The data shape stays the same as it goes up the stack. In this toy that shape is still (4, 4).

print("GPT-2 small repeats this block 12 times.")
print("toy stack depth: 12")

GPT-2 small repeats this block 12 times.
toy stack depth: 12

We do not run 12 toy blocks here. The idea is the repeat, not a new formula.

11. Train it to predict the next word

After the last block, a linear head turns the last token’s vector into a score per word. Softmax turns those scores into a probability bar.

Three training stages, in plain words:

1. Pretraining. Read a huge pile of text. Guess the next token. That is the transformer learning language.
2. SFT (supervised fine-tune). Show instruction and answer pairs. Same next-token loss, but now the text looks like a helpful reply.
3. RLHF / preference tuning. People pick the better of two answers. The model moves toward the preferred one.

We skip the training loop. We only do the last multiply, with a four-word toy vocab.

vocab = ["early", "again", "today", "."]
W_head = np.array(
    [
        [0.2, 0.1, 0.4, 0.8],
        [-0.3, 0.2, 0.6, 0.1],
        [0.1, 0.4, 0.2, 0.3],
        [0.8, 0.1, 1.2, 0.4],
    ]
)
logits = Y[3] @ W_head
probs = softmax(logits)
print("logits:", logits)
for word, p in zip(vocab, probs):
    bar = "#" * int(round(float(p) * 20))
    print(f"  {word:6s}  {p:4.2f}  {bar}")

logits: [0.88 0.54 2.78 1.08]
  early   0.10  ##
  again   0.07  #
  today   0.69  ##############
  .       0.13  ###

On this toy head, the next word after The bus is late is most likely today. Sample from that bar, append the word, run the ladder again. That is how a chat reply gets written: one token at a time.

These percentages are from the tiny matrices above. They are not a real ChatGPT sample.

12. Rewind everything

Take the stack off. Take the block off. Take attention off. You are left with four IDs, then four words.

print("back to the sentence:", SENTENCE)

back to the sentence: The bus is late

That is the whole machine, seen from one short line of English.

Checkpoint recap

In short

The bus is late becomes four toy IDs, then four short lists of numbers. Those lists ask questions (query), wear labels (key), and carry a payload (value). One token’s query is scored against every key, squashed into percentages, and used to mix the values. Do that for every token, hide the future with a causal mask, and you have self-attention. Add a small MLP and residual links and you have one transformer block. Stack twelve of those and you have a GPT-shaped model. Train it to guess the next word, then fine-tune with instructions and human preference, and you have the same idea that sits under ChatGPT. Undo the steps and you are back at the sentence.