LLMs are frozen at their training cutoff and know nothing about your private data. RAG fixes that: it retrieves relevant chunks from your knowledge base at query time and augments the prompt with them, so the model answers from real, current, cited sources instead of guessing.
Retrieval-Augmented Generation is a pattern that connects a language model to an external knowledge store. Instead of relying only on what the model memorized during training, you retrieve the most relevant passages for a question and paste them into the prompt as context. The model then generates an answer grounded in that retrieved evidence.
Documents are split into chunks, embedded into vectors, and stored in a vector database. Done once, ahead of time.
At query time, the question is embedded and the closest chunks are fetched by semantic similarity.
Retrieved chunks are stitched into the prompt; the model answers using them, ideally with citations.
| Piece | What it does | Common choices |
|---|---|---|
| Chunker | Splits documents into retrievable units | Fixed-size, recursive, semantic, by-heading |
| Embedding model | Turns text into vectors capturing meaning | text-embedding-3, open models |
| Vector store | Indexes vectors for fast similarity search | pgvector, Pinecone, Weaviate, FAISS |
| Retriever | Finds top-k relevant chunks per query | Dense, keyword (BM25), hybrid |
| Reranker | Re-scores candidates for precision | Cross-encoders, LLM rerankers |
Fine-tuning bakes knowledge into weights - it's slow, expensive, and stale the moment your data changes. RAG keeps knowledge outside the model, where it's cheap to update and easy to control.
Grounding answers in retrieved text sharply cuts made-up facts and lets you cite sources for verification.
Update the knowledge base and answers change instantly - no retraining. Add, edit, or delete a document and it's reflected on the next query.
Your data never enters model weights. You can filter retrieval by user permissions so people only see what they're allowed to.
No training runs. You pay for embeddings once and a bit of extra context per query - a fraction of the cost and effort.
Same idea, four ways to picture it, so it clicks whoever you are.
RAG is an open-book exam. The model doesn't have to memorize everything; it looks up the relevant pages first, then writes the answer with the book open in front of it.
It's like putting a search index in front of a function. Instead of hardcoding knowledge, you query the store for the rows that matter, then pass them in as arguments to the model call.
Think of a research assistant who pulls the exact files from the cabinet before your meeting, so your briefing quotes real documents instead of vague memory.
A chef who checks the recipe card before cooking, rather than guessing from memory. The skill is the same; the facts come from the card.
RAG has two pipelines: an offline ingestion pipeline that builds the index, and an online query pipeline that runs per request.
Load → clean → chunk → embed → upsert into the vector store. Metadata (source, section, permissions, timestamp) rides along with each chunk.
Embed the question → search top-k → (optionally) rerank → assemble a grounded prompt → generate → return answer + citations.
flowchart LR
subgraph Ingest["📦 Offline ingestion"]
D["📄 Documents"] --> CH["✂️ Chunk"]
CH --> EM1["🔢 Embed"]
EM1 --> VDB[("🗄️ Vector DB")]
end
subgraph Query["⚡ Online query"]
Q["❓ User question"] --> EM2["🔢 Embed query"]
EM2 --> SR["🔎 Similarity search"]
VDB --> SR
SR --> RR["📊 Rerank"]
RR --> PR["🧩 Build prompt + context"]
PR --> LLM["🧠 LLM"]
LLM --> ANS["✅ Grounded answer + citations"]
end
q_vec = embed(question)
hits = vector_db.search(q_vec, top_k=20, filter=user_acl)
top = rerank(question, hits)[:5]
context = "\n\n".join(h.text + f" [source: {h.source}]" for h in top)
answer = llm.generate(
system="Answer ONLY from the context. Cite sources. Say 'I don't know' if absent.",
user=f"Context:\n{context}\n\nQuestion: {question}"
)Notice the system prompt constrains the model to the retrieved context and tells it to abstain - this is what turns retrieval into trustworthy, grounded generation.
Three views: offline ingestion, the online query path, and an advanced flow with reranking and abstention.
sequenceDiagram
autonumber
participant Src as 📄 Source docs
participant Pipe as ⚙️ Ingest pipeline
participant Emb as 🔢 Embedding model
participant VDB as 🗄️ Vector DB
Src->>Pipe: Load raw files (PDF, HTML, DB)
Pipe->>Pipe: Clean & normalize text
Pipe->>Pipe: Split into chunks (+ metadata)
loop for each chunk
Pipe->>Emb: embed(chunk)
Emb-->>Pipe: vector[1536]
end
Pipe->>VDB: upsert(vectors + text + metadata)
VDB-->>Pipe: index updated
sequenceDiagram
autonumber
participant U as 👤 User
participant App as 🖥️ RAG app
participant Emb as 🔢 Embedding model
participant VDB as 🗄️ Vector DB
participant LLM as 🧠 LLM
U->>App: Ask a question
App->>Emb: embed(question)
Emb-->>App: query vector
App->>VDB: search(query vector, top_k, filter=ACL)
VDB-->>App: top-k chunks + sources
App->>App: Assemble prompt (context + question)
App->>LLM: generate(grounded prompt)
LLM-->>App: answer with citations
App-->>U: Answer + linked sources
sequenceDiagram
autonumber
participant U as 👤 User
participant App as 🖥️ RAG app
participant VDB as 🗄️ Vector DB
participant RR as 📊 Reranker
participant LLM as 🧠 LLM
U->>App: Question
App->>VDB: hybrid search (dense + BM25), top_k=20
VDB-->>App: 20 candidates
App->>RR: rerank(question, candidates)
RR-->>App: ordered by relevance
alt top score below threshold
App-->>U: "I don't have enough information"
else relevant context found
App->>LLM: generate(top 5 chunks + question)
LLM-->>App: grounded answer + citations
App-->>U: Answer + sources
end
From an empty knowledge base to a cited, grounded answer - the whole journey in order.
Gather the documents - wikis, PDFs, tickets, databases - and strip boilerplate, headers, and noise so only useful text remains.
Break documents into chunks sized for retrieval (often 200–800 tokens) with slight overlap, attaching metadata like source and section.
Run each chunk through an embedding model to get a vector that captures its meaning in high-dimensional space.
Upsert vectors plus their text and metadata into a vector store with an ANN index for fast similarity search.
A request arrives. The question is embedded with the same model used for ingestion so the vectors live in the same space.
Search the vector store for the nearest neighbors, filtered by the user's access permissions. Hybrid search adds keyword matching.
A cross-encoder or LLM reranker re-scores the candidates so the most on-point passages rise to the top before they hit the prompt.
If the best score is too low, the system says "I don't know" rather than hallucinate - a crucial trust step.
Stitch the selected chunks into the prompt with instructions to answer only from context and cite each source.
The LLM produces an answer using the retrieved context, attaching citations that point back to the exact sources.
The user gets a grounded, verifiable answer with clickable citations. Feedback and logs feed evals to improve retrieval over time.
Most RAG systems fail on retrieval, not on the model. These are the usual culprits.
Chunks too large bury the answer in noise; too small and they lose context. Splitting mid-sentence or mid-table destroys meaning. Chunk by structure, not by blind character count.
Embedding queries and documents with different models, or changing the model without re-indexing, puts vectors in incompatible spaces and retrieval collapses.
Relying on raw vector similarity alone often surfaces "related but wrong" chunks. Skipping a reranker is the most common cause of plausible-but-off answers.
Cramming 30 chunks into the prompt raises cost and triggers "lost in the middle," where the model ignores the middle of a long context. Retrieve broad, but send few.
If the system never says "I don't know," low-relevance retrievals get answered anyway, and the model confidently hallucinates. Always gate on a relevance threshold.
Forgetting to re-embed changed docs serves outdated answers; skipping access filters leaks data across users. The index is a living, permissioned asset.
Evaluate retrieval and generation separately, so you know which half to fix.
| Metric | What it tells you | Good sign |
|---|---|---|
| Recall@k | Did the relevant chunk make it into the top-k retrieved? | High: the answer is reachable |
| Context precision | What fraction of retrieved chunks are actually relevant? | High: little noise in the prompt |
| Faithfulness / groundedness | Is the answer supported by the retrieved context? | High: few hallucinations |
| Answer relevance | Does the answer actually address the question? | High: on-topic responses |
| Citation accuracy | Do the cited sources really back the claims? | High: trustworthy references |
| Latency & cost / query | End-to-end time and token spend per request | Within your SLA and budget |
Three representative patterns showing RAG in production-style use.
A SaaS company wants an assistant that answers customer questions from its docs without inventing features.
A legal team needs to ask questions across thousands of contracts and get exact clauses, not paraphrases.
An engineering org unifies wikis, runbooks, Slack, and code docs into one assistant.
RAG is evolving from a single retrieve-then-read step into richer, more adaptive systems.
Agents that plan multi-step retrieval - decomposing questions, searching iteratively, and deciding when they have enough evidence.
Combining vector search with knowledge graphs to answer questions that require connecting facts across many documents.
Measuring recall, precision, and faithfulness continuously - treating retrieval quality as the metric that matters most.
Bigger context windows won't kill RAG; they complement it - RAG controls cost, freshness, and access at any scale.
Systems that decide whether to retrieve at all, and how much, based on the question's difficulty and confidence.
Retrieving over images, tables, audio, and video - not just text - for grounded answers across every data type.