Retrieval is almost always the culprit, not the model — how to diagnose and fix it.
When a retrieval-augmented system answers badly, the instinct is to change the model or rewrite the prompt. The problem is nearly always that the right passage was never retrieved.
Diagnose retrieval separately
Before touching the prompt, take a failing question and check whether the correct source passage appears in the retrieved set at all. If it does not, no amount of prompt engineering can fix the answer. If it does and the answer is still wrong, then the generation step is at fault. These are different bugs with different fixes.
The usual retrieval failures
Chunks too large, so the relevant sentence is diluted by surrounding text. Chunks too small, so the context needed to interpret them is missing. Pure vector search missing exact identifiers, product codes and names, which keyword search finds easily. And no reranking, so a marginally relevant passage outranks the right one.
Hybrid search and reranking
Combining keyword and vector search covers each other's weaknesses, and a reranking model applied to the top candidates reliably improves what reaches the model. Both are usually larger wins than upgrading the generation model.
Keep a set of questions with their known correct source passages and measure retrieval hit rate as its own metric. You cannot improve what you have not separated.
Want this for your business?
Let's talk about how we can help you build and grow.
