Building a RAG System on Internal Data
Retrieval-Augmented Generation (RAG) connects a language model to your own documents so it answers questions using your actual knowledge, not generic training data. A working system requires four components: a document ingestion pipeline, a chunking and embedding strategy, a vector store, and a retrieval layer that feeds context into your LLM at query time. Done right, it takes one to three weeks for a focused team.
Most AI pilots fail the same way. A team gets access to GPT-4 or Claude, builds a quick demo on public information, and then hits a wall when someone asks, "Can it answer questions about our internal processes?" The model confidently makes things up. The demo dies.
The problem is not the model. It is the missing connection between the model and your actual knowledge. That gap is exactly what Retrieval-Augmented Generation solves.
RAG is not a product you buy. It is an architecture you build. A well-designed RAG system can let your support team query five years of tickets in plain English, give your sales team instant answers grounded in current pricing docs, or help engineers navigate a sprawling internal wiki without digging through Confluence for twenty minutes. The value is real, but so is the complexity. This guide covers how to build it properly.
What RAG Actually Does (And Why It Is Not Just a Vector Search)
The mental model most people start with is too simple: "I upload my documents, and the AI reads them." That is not how it works.
A RAG system does not give the LLM a library card. It gives the LLM a briefing document. At query time, the system retrieves the most relevant passages from your data, injects them into the model's context window, and the model generates a response grounded in those passages. The model never reads your entire document library. It reads a curated slice, chosen by your retrieval logic, for each specific question.
This distinction matters for system design. Your retrieval quality is your answer quality. A mediocre embedding with poor chunking will produce a mediocre answer even if you are using the best available LLM. Most RAG failures happen at the retrieval layer, not the generation layer.
Step One: Audit and Prepare Your Internal Data
Before you write a single line of code, spend time understanding what you are actually working with.
Internal data is messy in predictable ways. PDFs with inconsistent formatting. SharePoint folders that nobody has organized since 2019. Confluence pages where half the content is outdated but nobody deleted it. Google Drive links that lead to documents owned by people who left the company.
Do a practical audit first. Identify two or three high-value knowledge sources where the content is relatively current and trusted. A well-maintained internal wiki, a structured product documentation repository, and a clean collection of support resolutions are all good starting points. Do not try to ingest everything at once.
Data quality issues to watch for:
- Duplicate documents with conflicting information
- Version drift, where older versions of policies or procedures live alongside current ones
- Mixed formats, including PDFs, Word docs, HTML, and plain text, each requiring different parsers
- Confidentiality tiers, where some documents should only be retrievable by certain roles
Access control is often the piece that gets ignored in the early prototype and becomes a major problem in production. If your HR documents and your engineering runbooks live in the same vector store with no access filtering, you will eventually surface the wrong content to the wrong person. Design your permission model before you ingest anything sensitive.
Step Two: Chunking Strategy
Chunking is how you break your documents into retrievable units. It sounds mechanical. It is actually one of the most consequential design decisions in the whole system.
Chunk too large and your retrieved passages will contain a lot of irrelevant context, diluting the signal. Chunk too small and you lose the surrounding context that makes a passage meaningful. A 200-token chunk from the middle of a process document might not contain enough information to be useful on its own.
A few approaches that work well in practice:
Fixed-size chunking with overlap. Split documents into chunks of roughly 300 to 500 tokens with a 50 to 100 token overlap between chunks. Simple to implement, works well for dense technical documentation. The overlap ensures that sentences crossing chunk boundaries do not lose context.
Semantic chunking. Split at natural semantic boundaries, typically paragraph or section breaks. This preserves logical units better but requires more preprocessing logic. Worth the effort for narrative documents like policies or FAQs.
Hierarchical chunking. Store both a summary chunk and the full detailed chunk for each section. Use the summary for initial retrieval, then retrieve the full section when the summary matches. This is a more advanced pattern, used by teams at companies like Cohere and in frameworks like LlamaIndex's node postprocessor.
For most first builds, fixed-size chunking with overlap is the right starting point. Get it working, then optimize based on where retrieval is failing.
Step Three: Embeddings and Vector Storage
Embeddings convert your text chunks into numerical vectors that capture semantic meaning. Similar text ends up with similar vectors, which is what makes similarity search possible.
For embedding models, you have three practical options in 2026: use OpenAI's text-embedding-3-large, use a hosted embedding model from Cohere or Voyage AI, or run an open-source model like nomic-embed-text locally if data sovereignty is a concern. The performance differences between the top commercial options are modest. The more important decision is consistency: whatever model you use to embed your documents, you must use the same model to embed your queries at retrieval time.
For vector storage, the options have matured significantly. Pinecone and Weaviate are the most common choices for teams that want a managed solution. pgvector, which adds vector search to PostgreSQL, is a strong option if you want to keep your data stack simpler and your organization already runs Postgres. Chroma is popular for local development and smaller-scale deployments.
For an internal knowledge base serving a few hundred employees, pgvector is often the right call. You get SQL-based access control, familiar tooling, and one fewer external dependency. RAG systems are increasingly becoming a core part of broader AI adoption initiatives; integrating RAG into your business tech stack requires careful coordination with your existing data infrastructure and governance policies.
Step Four: Retrieval Logic
Basic RAG uses cosine similarity to find the top-k most semantically similar chunks to a query. That works. It is also where most first-generation RAG systems plateau.
A few retrieval patterns that meaningfully improve answer quality:
Hybrid search. Combine semantic similarity with keyword search using BM25 or full-text search. Semantic search handles conceptual questions well but can miss exact matches for product names, error codes, or specific identifiers. Hybrid search with reciprocal rank fusion typically outperforms either method alone.
Re-ranking. After retrieving the top 20 or 30 candidates, run them through a cross-encoder re-ranker to reorder by relevance before passing to the LLM. Cohere's Rerank API and open-source models like ms-marco-MiniLM work well here. The cost is a small latency increase; the benefit is meaningfully better context selection.
Query expansion and decomposition. For complex multi-part questions, generate two or three reformulations of the query before retrieval, then merge the retrieved sets. This catches cases where the user phrased the question differently than the document used to describe the answer.
None of these are required for version one. All of them matter by version two. For teams building AI agents for internal knowledge management, retrieval logic becomes even more critical, as agents often need to compose multiple retrieval steps to answer complex questions about your internal operations.
Step Five: The Generation Layer and Prompting
Once you have retrieved relevant chunks, you assemble them into a prompt for the LLM. The structure matters.
A clean RAG prompt includes a system instruction that tells the model to answer only from the provided context, the retrieved chunks clearly labeled, the user's original question, and an explicit instruction for what to do when the answer is not in the provided context. That last part is important. You want the model to say "I don't see that information in the available documents" rather than invent an answer.
For grounding and citation, instruct the model to reference which document or section an answer came from. This dramatically increases user trust and makes it easier to debug retrieval failures when they happen.
Model selection at the generation layer matters less than most people expect. GPT-4o, Claude 3.5 Sonnet, and Gemini 1.5 Pro all perform well at RAG generation tasks. The differences show up at the edges: very long context windows, complex multi-step reasoning, or highly technical domains. For most internal knowledge base use cases, the limiting factor is retrieval quality, not generation quality.
Step Six: Evaluation and Iteration
A RAG system that is not measured is a RAG system that will quietly degrade over time.
Building an evaluation pipeline is not optional. You need a baseline set of questions with known correct answers drawn from your actual documents. Run your system against them regularly. Measure retrieval precision (did the right chunks get retrieved?), answer accuracy (did the model answer correctly?), and hallucination rate (did the model generate claims not grounded in the retrieved context?).
Tools like RAGAS, an open-source RAG evaluation framework, give you a structured way to track these metrics. Pair automated evaluation with regular spot-checks from domain experts on your team. Automated metrics catch regressions; human review catches subtle quality issues that numbers miss.
Document staleness is also a real operational concern. If your internal policies change and the vector store still contains the old version, users will get outdated answers confidently delivered. Build a refresh cadence into your ingestion pipeline from day one.
If your organization is still figuring out where RAG fits within a broader AI adoption plan, the free Voyant AI Readiness Assessment can help map your current capabilities and identify the highest-value starting points before you commit engineering resources.