Retrieval-Augmented Generation: Choosing the Right RAG Pattern

Retrieval-Augmented Generation (RAG) has become the default architecture for connecting large language models to knowledge they were never trained on: internal documentation, product catalogs, legal archives, live operational data. The idea is simple — retrieve relevant context at query time and inject it into the prompt — but the design space around that idea is now large enough that “we use RAG” tells you almost nothing about a system.

This article walks through the main RAG patterns in use today, what each one actually fixes, and — most importantly — how to choose between them. The recurring theme: most RAG failures are retrieval failures, and most teams add complexity in the wrong place because they never measured where their pipeline breaks.

Why RAG at all?

Before choosing a pattern, it is worth restating what RAG buys you compared to the alternatives:

• Knowledge freshness. The model answers from documents indexed minutes ago, not from a training snapshot.
• Private data without training. Fine-tuning teaches style and skills far better than it teaches facts; RAG grounds answers in your data without touching model weights.
• Traceability. Because the answer is generated from retrieved passages, you can cite sources — essential in regulated or customer-facing contexts.
• Cost control. Indexing documents is orders of magnitude cheaper than training, and the knowledge base can be updated incrementally.

The trade-off is that you now own a search engine, and the quality of your system is bounded by the quality of that search engine. Every pattern below is, at its core, a different answer to the question “how do I make retrieval good enough for this workload?”

The anatomy of a RAG pipeline

All variants share the same skeleton:

flowchart LR
  subgraph OFFLINE ["Offline — ingestion"]
    D["Documents"] --> P["Parse & clean"]
    P --> C["Chunking"]
    C --> E["Embedding"]
    E --> I[("Index<br/>vector / keyword")]
  end
  subgraph ONLINE ["Online — query time"]
    Q["User query"] --> T["Query transformation<br/>(optional)"]
    T --> R["Retrieval"]
    R --> RR["Reranking<br/>(optional)"]
    RR --> PA["Prompt assembly"]
    PA --> G["LLM generation"]
    G --> A["Answer"]
  end
  I -.-> R

Every pattern is a set of decisions about these boxes: how you chunk, how you retrieve, whether the query is transformed, whether the loop runs once or several times, and who decides — the pipeline or the model itself.

Pattern 1: Naive RAG

The baseline: split documents into fixed-size chunks, embed them, store them in a vector database. At query time, embed the question, fetch the top-k nearest chunks, paste them into the prompt, generate.

What it is good at. Single-hop factual questions over a homogeneous, well-written corpus — “what does the expense policy say about hotel limits?” For an internal FAQ or a documentation assistant, naive RAG frequently reaches 80–90% of the achievable quality at a fraction of the complexity.

Where it breaks, predictably:

• Vocabulary mismatch. Dense embeddings can miss exact identifiers — error codes, part numbers, function names — that keyword search would catch trivially.
• Bad chunk boundaries. Fixed-size splitting cuts tables, procedures and arguments in half; the retrieved fragment lacks the context needed to be useful.
• Multi-hop questions. “Which of our suppliers are affected by the new regulation?” requires combining facts from documents that are not similar to the question — nearest-neighbor search cannot see that.
• Ambiguous or conversational queries. “What about the second option?” embeds to noise without conversation context.

The right reaction to these failures is not to jump to the most sophisticated architecture — it is to identify which failure you have and apply the targeted fix below.

Pattern 2: Advanced RAG — fixing retrieval quality

This family keeps the single-pass structure but upgrades individual stages. These are the highest-return improvements in practice, and they compose.

Hybrid search

Run dense (vector) and sparse (BM25/keyword) retrieval in parallel and merge the results, typically with Reciprocal Rank Fusion. Sparse retrieval catches exact terms, dense retrieval catches paraphrases. For technical corpora full of identifiers — codebases, API docs, product SKUs, legal references — hybrid search is close to mandatory, and most vector databases now support it natively.

Reranking

First-stage retrieval optimizes for speed over millions of chunks; it is deliberately crude. A reranker — usually a cross-encoder that scores the query together with each candidate — re-orders the top 50–100 candidates and keeps the best few. This is routinely the single largest precision win available in a RAG stack, at the cost of 50–300 ms of added latency. If you deploy one advanced component, deploy this one.

Query transformation

Fix the query before it hits the index:

• Rewriting turns conversational input (“and for Germany?”) into a self-contained question using chat history.
• Multi-query / decomposition splits a compound question into sub-questions retrieved independently — a lightweight answer to multi-hop needs.
• HyDE (Hypothetical Document Embeddings) asks the LLM to draft a hypothetical answer and retrieves with its embedding, bridging the stylistic gap between short questions and long documents.

Smarter chunking

Chunking deserves more attention than it gets, because no downstream component can recover information destroyed at ingestion:

• Structure-aware splitting — respect headings, paragraphs, code blocks and tables instead of counting characters.
• Small-to-big (parent-document) retrieval — index small, precise chunks for matching, but feed the LLM the surrounding section. This decouples “what is easy to find” from “what is useful to read” and is one of the best cost/benefit tricks in the catalog.
• Contextual enrichment — prepend each chunk with its document title and section path (or an LLM-generated one-line summary) before embedding, so the chunk carries its own context.

Pattern 3: Agentic RAG — the model drives retrieval

The previous patterns are fixed pipelines: every query follows the same path. Agentic RAG inverts control — the LLM decides whether, where and how many times to retrieve, using retrieval as a tool.

Typical building blocks:

• Routing. A classifier or the model itself sends the query to the right source: vector index, SQL database, web search, or no retrieval at all for small talk. Essential as soon as you have more than one heterogeneous source.
• Iterative retrieval. The agent retrieves, reads, notices a gap (“I found the supplier list, now I need the regulation’s scope”), reformulates and retrieves again — solving multi-hop questions properly instead of hoping one query covers them.
• Self-correction (Self-RAG / CRAG). After retrieval, the model grades the evidence: relevant? sufficient? If not, it re-queries, falls back to another source, or explicitly answers “I don’t know” instead of hallucinating over weak context.

The cost is real. Each reflection loop adds an LLM round-trip: latency multiplies (often 3–10× a single-pass pipeline), spend multiplies, and behavior becomes harder to test and reproduce. Agentic RAG is the right tool for complex, high-value questions — analyst workflows, research assistants, multi-source investigations — and the wrong tool for a high-traffic FAQ endpoint where a tuned single-pass pipeline answers in under a second.

Pattern 4: Graph RAG — when relationships are the data

Some questions are not about finding a passage but about traversing relationships: “which projects depend on a library affected by CVE-2026-1234?”, “summarize the main themes across this year’s incident reports”. Chunk-level similarity search structurally cannot answer these — the answer lives in no single chunk.

Graph RAG builds a knowledge graph (entities and relations, usually extracted by an LLM at ingestion) alongside or instead of the vector index. Retrieval becomes graph traversal from matched entities, and hierarchical community summaries can answer corpus-wide “global” questions that top-k retrieval will never see.

The price is the heaviest ingestion pipeline of any pattern — entity extraction over the whole corpus is expensive to build and to keep synchronized. Choose it when your questions are genuinely relational or corpus-global and that value justifies the maintenance; skip it when users ask local, factual questions, which is most of the time.

Choosing: a decision framework

The single most important rule: start with the simplest pipeline, build an evaluation set, and let measured failures — not architecture enthusiasm — justify each addition. Roughly 80% of production RAG problems are retrieval problems, and most of those are solved in the “advanced” tier without any agentic machinery.

flowchart TD
  S["Start: naive RAG<br/>+ evaluation set"] --> M{"Measure:<br/>where does it fail?"}
  M -->|"Retrieval misses<br/>terms or chunks"| ADV["Advanced RAG<br/>hybrid search, reranking,<br/>chunking, query transformation"]
  M -->|"Multi-hop or<br/>multi-source questions"| AG["Agentic RAG<br/>routing, iterative retrieval,<br/>self-correction"]
  M -->|"Relational or<br/>corpus-wide questions"| GR["Graph RAG<br/>knowledge graph,<br/>community summaries"]
  M -->|"Good enough"| SHIP["Ship it"]
  ADV --> M
  AG --> M
  GR --> M

A symptom-driven map:

And weigh the operational axes explicitly before adopting a pattern:

• Latency budget. A sub-second product feature can afford hybrid search and a fast reranker; it cannot afford three agent loops.
• Cost per query at your traffic. A reranker adds cents per thousand queries; an agentic loop adds LLM calls per query. Multiply by your QPS before deciding.
• Corpus dynamics. Rapidly changing data favors simple ingestion (naive/advanced); heavy ingestion pipelines like Graph RAG lag behind reality.
• Failure tolerance. Customer-facing and regulated use cases need citations, evidence grading and an explicit “I don’t know” path far more than they need cleverer retrieval.

Evaluate before you architect

None of the choices above can be made honestly without measurement. The minimum viable evaluation setup:

1. A golden set of 50–200 real user questions with expected answers and expected source passages. Real questions — not questions invented while reading the documents.
2. Retrieval metrics computed on their own: recall@k and MRR against expected passages. If recall@20 is 60%, no prompt engineering will save you — fix retrieval first.
3. Generation metrics: faithfulness (is every claim supported by the retrieved context?) and answer relevance, typically scored by an LLM judge, as popularized by frameworks such as RAGAS.
4. Regression runs on every change — chunking size, embedding model, reranker, prompt. RAG stacks are coupled systems; improvements in one stage regularly degrade another, and only a fixed benchmark makes that visible.

Production concerns that outlive the pattern choice

Whichever pattern you select, the issues that dominate real deployments are rarely the retrieval algorithm:

• Access control. Retrieval must filter by the caller’s permissions at query time — an index that ignores ACLs is a data-leak generator with excellent semantic search.
• Freshness. Prefer incremental, event-driven indexing over nightly full rebuilds; stale answers erode trust faster than mediocre ones.
• Observability. Log the query, the transformed query, retrieved chunk IDs and scores, and the final prompt. When a user reports a wrong answer, this trace is the difference between a five-minute diagnosis and guesswork.
• Caching. Semantic caching of frequent questions and embedding caches for unchanged chunks cut both cost and latency substantially.

Conclusion

RAG is not one architecture but a spectrum. Naive RAG is a legitimate production pattern for simple factual workloads, not just a prototype. The advanced tier — hybrid search, reranking, query transformation, small-to-big chunking — is where most systems should land, because it fixes the retrieval failures that actually dominate. Agentic RAG earns its latency and cost only for complex, multi-source, high-value questions, and Graph RAG only when your questions are about relationships rather than passages.

The discipline that separates RAG systems that work from those that merely demo well is unglamorous: build the evaluation set first, measure where retrieval fails, and add exactly the pattern that failure calls for — nothing more.