Large language models have a memory problem. They know what they learned during training. Nothing else. Ask about your company’s internal documents, yesterday’s news, or anything that happened after their training cutoff, and they either guess or admit ignorance.
RAG fixes this. Sort of.
Retrieval-Augmented Generation is exactly what the name suggests: before generating an answer, the system retrieves relevant information from external sources, then uses that information to produce a more accurate response. The model doesn’t need to have memorized your company handbook if it can look it up first.
The Core Idea in Plain Terms
Think about how you answer questions when you’re not sure. You don’t just guess. You look something up first, then form your answer based on what you found. RAG works the same way, except the “looking up” happens automatically before the AI responds.
Here’s what happens when you ask a RAG system a question:
- Your question gets converted into a numerical representation (an embedding)
- The system searches a database for content with similar representations
- The most relevant chunks of text get pulled and added to your question
- The language model receives both your question and the retrieved context
- It generates an answer based on that combined input
As user ozr explained on Hacker News: “It’s right there in the name - first you Retrieve relevant information (often a vector lookup) then you use it to Augment the prompt, then you Generate an answer.”
The language model never “learns” your data. It just sees it at the moment of answering, like being handed a relevant page from a textbook right before an exam question.
Why This Matters
Language models hallucinate. They generate plausible-sounding but incorrect information because they’re pattern-matching, not fact-checking. Training data might be outdated, incomplete, or simply wrong about your specific situation.
RAG attacks this problem directly. By grounding the model in retrieved documents, you give it actual sources to work from rather than relying purely on statistical patterns from training. The model can only hallucinate so much when you’ve literally handed it the correct answer to pull from.
This addresses three problems at once:
Currency: Models have training cutoffs. Information changes. RAG lets you connect a model to continuously updated sources without retraining.
Specificity: A model trained on the general internet knows nothing about your company’s internal processes, products, or terminology. RAG lets you point it at your specific knowledge base.
Verifiability: When answers come from retrieved sources, you can trace them back. The model isn’t just making things up from statistical correlations.
As hirako2000 put it on Hacker News: “RAG’s point is to remove the limit LLMs alone have which is that they are limited to the training data as source of information.”
How the Retrieval Actually Works
The retrieval step is where things get technically interesting. Most RAG systems use vector search, also called semantic search. Your documents get processed into numerical vectors (embeddings) that capture their meaning. When a question comes in, it gets the same treatment. The system then finds documents whose vectors are mathematically closest to the question’s vector.
This works better than keyword matching because it captures meaning, not just words. A question about “employee vacation policy” can match a document titled “PTO Guidelines” even though they share no exact words.
But vector search isn’t the only option. As AI researcher Simon Willison noted on Hacker News: “You don’t have to use vector search to implement RAG. You can use other search mechanisms instead or as well.”
Some practitioners prefer traditional text search (BM25) for certain use cases. One commenter on a recent Hacker News thread about local RAG observed: “In a lot of the cases bm25 has been the best approach used in many of the projects we deployed.” Others combine approaches, using both keyword and semantic search to catch different types of relevance.
The choice depends on your data. Dense technical documents with specific terminology might work better with keyword search. Conversational content with varied phrasing might need semantic matching. For code specifically, one developer noted: “Don’t use a vector database for code, embeddings are slow and bad for code. Code likes bm25+trigram.”
What RAG Cannot Do
Misunderstanding RAG’s limitations causes most failures. The technique is powerful but narrowly scoped.
RAG does not make language models smarter. It gives them access to information they wouldn’t otherwise have. But if the model can’t reason well about that information, or if the retrieval pulls the wrong documents, you still get bad answers.
One Hacker News user offered a memorable analogy: “Using RAG feels like asking an acquaintance to write a book report by giving them semi-randomly cut out paragraphs from the book.”
That’s the fundamental limitation: RAG retrieves chunks, not understanding. If your question requires synthesizing information across an entire document, or connecting insights from multiple sources in complex ways, chunk-based retrieval might not give the model what it needs.
Another commenter in the same thread explained the constraint more precisely: “Simple RAG works well when questions are highly correlated with specific chunks of documents. It does not allow an LLM to synthesize an entire corpus to an answer (e.g. a book report).”
RAG also doesn’t prevent hallucination entirely. It reduces hallucination risk by providing factual grounding, but models can still:
- Misinterpret retrieved documents
- Generate confident answers from ambiguous sources
- Fill gaps between retrieved chunks with fabricated details
- Combine accurate quotes into inaccurate conclusions
The technology works best when questions have clear, localized answers within your document collection. It struggles with questions requiring holistic understanding or synthesis across large bodies of text.
RAG vs Fine-Tuning
A common question: when should you use RAG versus fine-tuning a model on your data?
Fine-tuning adjusts the model’s weights based on your data, effectively teaching it new patterns. RAG keeps the model unchanged but provides external information at query time.
These serve different purposes entirely.
Fine-tuning changes how the model writes and reasons. It’s useful for teaching a model your company’s tone, your industry’s terminology, or your preferred output format. It does not reliably add new facts. Models struggle to distinguish fine-tuned information from their base training.
As one practitioner put it: “Fine tuning is not good for really adding/removing facts but is great for changing the form of the output.”
RAG adds information access. It lets models answer questions about content they never saw during training. The model’s style stays the same, but its knowledge expands.
For most business applications, you want both working together: a model fine-tuned for your communication style, connected via RAG to your current documentation. Neither alone solves the complete problem.
When RAG Makes Sense
RAG works well for:
Customer support systems: Connect a chatbot to your help documentation. Questions get matched to relevant articles, and the model generates natural-language answers grounded in your actual support content.
Internal knowledge bases: Let employees ask questions about company policies, procedures, and documentation without digging through folder structures or wikis.
Product documentation: Users can ask natural language questions about how products work and get answers pulled from actual documentation.
Legal or regulatory compliance: When answers must be traceable to specific source documents, RAG provides the paper trail that pure model outputs cannot.
The common thread: situations where you need the model to work with specific, bounded, verifiable information rather than general knowledge.
When to Consider Alternatives
RAG isn’t always the right choice.
Long-context models: Models with very large context windows (100K+ tokens) can sometimes hold entire document collections directly. If your knowledge base is small enough, you might not need retrieval at all. As one practitioner observed: “85% of the time we don’t need the vectordb” after discovering that simpler approaches solved their use case.
Structured data queries: If your information lives in databases with clean schemas, traditional query systems might serve better than vector search. RAG excels at unstructured text, not tabular data.
Real-time information: RAG assumes your knowledge base is already built. For truly live data (stock prices, weather, current news), you need API integrations, not document retrieval.
Complex reasoning tasks: Questions requiring multi-step reasoning across many sources challenge RAG’s chunk-based approach. Some problems need agentic systems that can search, reason, and iterate rather than retrieve-once-and-generate.
Implementation Realities
Building production RAG systems involves more decisions than tutorials suggest. Chunk size matters: too small and you lose context; too large and you waste the model’s attention on irrelevant content. Retrieval quality varies dramatically based on your embedding model choice and search parameters.
Many practitioners have found that simpler approaches often beat complex ones. The tooling landscape remains unsettled. Some developers find commercial RAG platforms helpful. Others prefer building from simple components. “SQLite works shockingly well,” one Hacker News commenter noted about their production system.
Another observation from the practitioner community: “A little BM25 can get you quite a way with an LLM.” The implication is clear. Start simple. Measure results. Add complexity only when measurements prove you need it.
What works depends heavily on your specific data, questions, and accuracy requirements. The companies getting real value from RAG usually started simple, measured results carefully, and iterated based on actual performance rather than theoretical best practices.
The Bigger Picture
RAG represents a specific architectural choice: keep the language model general but connect it to specialized knowledge at query time. This trades some capability for flexibility. Your information stays in sources you control, updates happen without retraining, and answers can be verified against retrieved documents.
As minimaxir wrote on Hacker News: “RAG is the one paradigm of modern AI that’s completely uncontroversial (hallucinations aside) and will persist even if there’s an AI-industry crash.”
The core idea of augmenting generation with retrieval solves a real problem that doesn’t go away with better base models.
What’s less certain is whether current implementations capture the full potential. Today’s RAG systems mostly do simple similarity search followed by single-shot generation. More sophisticated approaches exist: ones that search iteratively, verify retrieved information, or synthesize across many sources. These remain areas of active development.
The technique has proven its value for narrow, fact-based queries against structured knowledge bases. Whether it scales to messier, more ambitious applications remains an open question. The companies deploying RAG successfully have usually been honest about its limitations, using it where it genuinely helps rather than forcing it into problems that require different solutions.
For most organizations exploring AI, RAG is worth understanding. Not because it solves everything. Because knowing what it does, and what it doesn’t, helps you make better choices about where AI can actually help your work.
