Retrieval-Augmented Generation: Making LLMs useful for enterprise data
The hallucination problem
Large language models are powerful generalists. Ask GPT-4 about quantum computing or Shakespeare, and it will give you a reasonable answer. Ask it about your company's refund policy or last quarter's sales figures, and it will confidently fabricate something that sounds plausible but is completely wrong.
This is not a bug — it is a fundamental limitation of how LLMs work. They generate text based on patterns in their training data, not from a database of facts. Retrieval-Augmented Generation (RAG) addresses this by connecting the model to your actual data.
How RAG works
RAG adds a retrieval step before generation. When a user asks a question, the system first searches a knowledge base for relevant documents, then passes those documents to the LLM as context. The model generates an answer grounded in the retrieved information, not its training data.
The basic pipeline looks like this:
- Ingestion: Documents are chunked, embedded (converted to vector representations), and stored in a vector database.
- Retrieval: When a query arrives, it is embedded using the same model, and the most similar document chunks are retrieved.
- Generation: The retrieved chunks are injected into the LLM prompt as context, and the model generates a response.
Building our production RAG pipeline
We built a RAG system to help our consulting team quickly find relevant case studies, technical specifications, and project documentation. The pipeline processes over 50,000 documents across 15 categories.
Chunking strategy matters more than you think. Naive fixed-size chunking (e.g., 512 tokens per chunk) breaks sentences and context. We settled on a semantic chunking approach that splits at paragraph boundaries and groups related paragraphs together.
Embedding model selection is critical. We tested OpenAI ada-002, Cohere embed-v3, and open-source E5-large. For our use case, Cohere embed-v3 gave the best retrieval quality, but E5-large was close behind at a fraction of the cost.
Hybrid search improves recall. We combine vector similarity search with BM25 keyword search. The vector search handles semantic queries ("how did you handle scaling issues?"), while keyword search handles precise lookups ("project Phoenix Q3 2024").
Evaluation is everything
The hardest part of building a RAG system is not the pipeline — it is knowing whether it works. We built an evaluation dataset of 200 question-answer pairs and measure retrieval recall, answer faithfulness, and answer relevance.
Without systematic evaluation, you are flying blind. Every change to chunking strategy, embedding model, or retrieval parameters needs to be validated against the evaluation set.