Every enterprise wants a chatbot that answers questions from their internal knowledge. Almost every first attempt disappoints. The demo dazzled; the production system hallucinated, cited the wrong document, or confidently made things up.
The instinct is to blame the model and reach for a bigger one. That's almost always the wrong move.
Retrieval is where RAG lives or dies
RAG has two halves: retrieval and generation. Teams obsess over the generation half — which model, which prompt — and neglect the retrieval half. But if you hand the model the wrong context, no amount of model quality will save you. Garbage in, confident garbage out.
Invest your effort here:
Chunking
Naive fixed-size chunking splits ideas in half and destroys context. Chunk along semantic boundaries — sections, headings, logical units — and preserve enough surrounding context that a retrieved chunk stands on its own.
Hybrid search
Pure vector search misses exact matches (product codes, error strings, names). Pure keyword search misses semantic paraphrases. Combine both, then rerank the merged results. This single change fixes more RAG systems than any model upgrade.
Reranking
Retrieve broadly, then use a cross-encoder reranker to pick the genuinely relevant few. Feeding the model ten precise chunks beats feeding it fifty noisy ones.
Grounding and citations
Constrain the model to answer only from retrieved context, and make it cite its sources. If it can't find an answer, it should say so — not invent one. Citations also give users a way to verify, which is what builds trust.
The takeaway
When your RAG system disappoints, resist the urge to swap the model. Open up the retrieval pipeline instead. That's almost always where the problem — and the fix — lives.