TL;DR. Naive RAG has been superseded. “Chop up the text, embed it, dump the top hits into the context window” gives mediocre results; today’s standard is multi-stage. The myth that “the large context window makes RAG obsolete” is also wrong: one-million-token windows are real, but they lose content in the middle and cost many times more per query. Good knowledge bases in 2026 need hybrid search and reranking as a minimum. That is the biggest lever for an internal AI system that gives reliable answers.
What this is about
Most AI use cases in mid-sized companies sooner or later come down to the same thing: an internal knowledge base that reliably answers questions from contracts, manuals, tickets and emails. The technique behind it is called RAG (retrieval-augmented generation). Instead of letting the model guess, you retrieve relevant documents and pass them along as context. The principle is simple. The implementation has evolved a lot over the last two years. If you still build the textbook RAG of 2023, you leave quality on the table.
What “naive RAG” is, and why it is not enough
The first generation worked like this: cut documents into chunks, embed each chunk as a vector, find the most similar chunks for a question and put them in the context window. That gives usable demos and disappointing production. Two weaknesses are typical. Pure vector similarity misses exact terms such as part numbers, proper names or error codes. And the retrieved chunks are often similar, but not the most relevant.
The 2026 standard: multi-stage retrieval
Production-ready knowledge bases today combine several techniques:
- Hybrid search. Semantic vector search plus classic keyword search (BM25), with the results merged. That way the system finds related meanings and exact terms.
- Reranking. A second, specialised model re-sorts the candidates by actual relevance before they go into the context window. This step has the best ratio of effort to effect.
- Enriching context. Anthropic’s “Contextual Retrieval” prepends a short generated note to each chunk, saying which document and section it comes from. In Anthropic’s measurement this cuts the retrieval failure rate by 49%, and with reranking by 67%.
- GraphRAG. For highly connected knowledge (who belongs to what, what depends on what), a knowledge graph instead of plain text chunks helps answer questions that connect several sources.
Retrieval failure rate, top 20 results
Reranking and retrieval in 2026: the real state of the art
The reranker landscape is large but well measured. In Agentset’s public leaderboard (as of February 15, 2026), ZeroEntropy’s zerank-2 and Cohere Rerank 4 Pro are almost level at the top, with Voyage rerank-2.5 close behind. For running on your own hardware and under strict data protection requirements, open models such as bge-reranker-v2-m3 and Jina Reranker v2 have become established. For context: these are vendor and community leaderboards, not peer-reviewed studies. Measuring on your own corpus remains a must.
Late-interaction models such as ColBERT form a class of their own. Instead of squeezing a document into a single vector, they store one vector per token and compare at a fine grain. That was a research topic for a long time. Now it has arrived in production: the vector database Qdrant supports several vectors per document natively, ColBERTv2 with PLAID has shown that it scales, and ECIR 2026 dedicated a workshop of its own to the topic.
Things have moved before reranking, too. With late chunking, the whole document is embedded first and then split, so every chunk keeps the document’s context. Jina offers this directly in its embedding models. Matryoshka embeddings allow shorter vectors, so less storage in the index, with little loss of quality. And multimodal models such as Cohere embed-v4 embed text and images in the same space and increasingly replace separate OCR pipelines.
GraphRAG: when the graph pays off, and when it does not
GraphRAG long had a cost problem. During indexing, a language model extracts entities and relationships from every document. That is powerful but expensive. This is where Microsoft’s LazyGraphRAG comes in: it shifts the language model’s work from indexing to query time. According to Microsoft Research, indexing then costs as much as ordinary vector RAG, about 0.1% of full GraphRAG. The technique is available through Microsoft Discovery. The open GraphRAG package (currently version 3.3.0 from October 2026) contains building blocks of it, but does not list a dedicated LazyGraphRAG mode. Alternatives include LightRAG and Neo4j’s Graphiti.
That largely removes the cost argument against the graph. The architecture question remains:
- The graph pays off when questions have to connect several sources through relationships (who supplies whom, what depends on what), when “global” questions about the whole corpus need answers, such as trends and summaries, or when traceable chains of relationships are required, for example for compliance.
- The graph is too much for classic lookups (“What does contract X say about notice periods?”), for small corpora and for fast-changing data, because the graph has to be updated with every change.
Rule of thumb: build and measure the baseline of hybrid search and reranking first. The graph is the next step when answers fail precisely at linking knowledge across several documents.
The myth of the giant context window
A popular claim goes: “Models with a million tokens of context no longer need RAG, you just throw everything in.” Reality is more nuanced. Large context windows are real and useful; the current Claude models handle a million tokens at the normal price. But models suffer from the “lost in the middle” effect: content at the beginning and end of the context is used more reliably than content in the middle. If you pour everything in, you get worse hits and pay for every token.
A simple calculation with Claude Sonnet 5.5 at $2 per million input tokens shows how large the cost difference is:
Cost per query, our own calculation
Without a cache, targeted retrieval is about 60 times cheaper. With a cache the gap shrinks to about three times, but the quality question remains. The production pattern is therefore still “retrieve first, then reason”: fetch the right chunks first, then let the model work on them. The large context window does not replace retrieval. It complements it, for instance when a single long document has to be read in full.
New: retrieval as the agent’s tool
The latest development combines RAG with agents. Instead of retrieving once up front, search is provided as a tool that the agent calls several times as needed: ask, refine, look up, until the answer stands. This is called agentic RAG. For complex questions with several steps, it is a clear jump in quality over a single retrieval of the top hits.
Agentic retrieval: the real leap
This shift now has a research field of its own: deep research. A survey of autonomous research agents describes the core pattern. The agent breaks a question into sub-questions, formulates searches, evaluates the hits, reformulates and retrieves again until the evidence is solid. Search methods and rerankers turn from a hard-wired step into a toolbox the agent picks from depending on the question. This is measured with dedicated benchmarks such as DeepResearch Bench (report faithfulness, citation accuracy) or BrowseComp (research on the open web).
The infrastructure for this is standardised. Through the Model Context Protocol, search systems are connected as tools that every agent calls the same way. The knowledge base becomes one building block among several; more on that in our post on MCP as industry infrastructure. The distinction: RAG answers “What have we indexed?”, a live tool answers “What is the current state?”. Good agent systems need both. The new bottleneck has been described too: dozens of tools clog the agent’s context with their descriptions. Research responds with retrieval over the tools themselves, for example ScaleMCP. Retrieval is therefore no longer a stage in a pipeline but a capability of the system.
Frequently asked questions about RAG in 2026
Is RAG dead?
No, only the naive RAG of 2023 has died. In production the question has shifted: not whether to retrieve, but how, multi-stage, with hybrid search, reranking and increasingly agentic. Knowledge bases over private, current documents simply do not work without retrieval.
RAG or a large context window: which is more economical?
For recurring queries against a large corpus, targeted retrieval. In our calculation, a query with the whole one-million-token corpus costs $2.00, with RAG $0.03. Even with a cache, the full context stays about three times as expensive. The full context window pays off for small, fixed corpora or when a single document has to be read in full.
When do I need GraphRAG?
When answers have to link knowledge from several sources through relationships, or when “global” questions about the whole corpus are asked. For plain lookups the graph is too much. Since LazyGraphRAG, the old cost argument hardly holds; according to Microsoft Research, indexing is on par with vector RAG.
Which embedding model should you choose in 2026?
Established candidates include Gemini Embedding, the Qwen3 embedding models for many languages and Cohere embed-v4 for corpora with images. But leaderboards such as MTEB shift every month and do not transfer one to one to your data. What matters is a small test on your own corpus with real questions from your users.
Do I need a reranker?
In production, almost always. The usual setup is hybrid search plus reranking of the best 50 to 100 candidates. In Anthropic’s measurement, this step alone cuts the failure rate from 2.9 to 1.9%. Hosted, Cohere Rerank 4, zerank-2 or Voyage rerank-2.5 are common choices; on your own hardware, bge-reranker-v2-m3.
What this means for a project
| Maturity | Setup |
|---|---|
| Textbook (2023) | Vector search, top hits, straight into the context window |
| Solid baseline (2026) | Hybrid search (vector and keyword), reranking, enriched context |
| Advanced | Agentic RAG with repeated retrieval and, for connected knowledge, GraphRAG |
The good news for mid-sized companies: the leap from “nice but unreliable” to “trustworthy in operation” is no longer rocket science. It lies in the architecture, not in a more expensive model. If you want to run an internal knowledge base seriously, plan for hybrid search and reranking as a minimum.
A word of caution: exactly this knowledge base is the largest attack surface for indirect prompt injection, that is, outside text the model treats as an instruction. What goes into the knowledge base should therefore be checked. If you are planning an internal knowledge base, we are happy to talk.
