RAG Evaluation Harness
Seven retrieval configurations benchmarked against a hand-verified eval set over a corpus of computer vision papers. No configuration beat plain dense retrieval on recall@1 — every added component improved recall@5 while making the top result worse. 14 findings, 10 of them negative.
CASE STUDY
Project breakdown.
Select a section to explore the project one part at a time.
Problem
“The system is wrong” is not a diagnosis.
Retrieval-augmented generation is almost always evaluated end to end: ask questions, score answers, report one number. That number tells you the system failed. It does not tell you where, and the two places it can fail need opposite fixes.
- Retrieval failed. The answer was never in the context. Fix chunking, embeddings, or k.
- Generation failed. The answer was in the context and the model ignored it, or blended it with what it already knew. Fix the prompt or the model.
Measured separately, this system is lopsided in a way no aggregate score would reveal. Retrieval puts the correct chunk first 25% of the time. Generation is faithful to whatever context it receives 97.7% of the time. A single end-to-end figure would have looked respectable and pointed at nothing.
What this project asks
- Does adding retrieval machinery improve retrieval? Sparse indexes, rank fusion, and cross-encoder rerankers are standard advice. Measured against a plain dense baseline on the same corpus and the same questions, does each one actually earn its place?
- Does a larger embedding model retrieve better? Benchmark leaderboards say yes. Does that hold on domain text the model was never trained on?
- Can faithfulness scoring catch a hallucination? If every individual claim in an answer traces to the retrieved context, is the answer necessarily grounded?
- Is the evaluation set itself trustworthy? Every metric here depends on hand-assigned gold chunks. What happens when those labels are verified rather than assumed?
The corpus is ten arXiv computer vision papers — chosen because the eval questions are hand-written and their correctness has to be judged rather than assumed. Everything runs locally on Ollama and ChromaDB, with no API keys and no paid services.