Practical Insights on RAG System Evaluation and Operations

A structural audit of a real-world production RAG (Retrieval-Augmented Generation)-based customer support system yielded the following key findings.

1. The Futility of Heuristic Evaluation Existing heuristic evaluation methods based on keywords or source references showed almost no correlation with response quality. In contrast, the LLM-as-judge approach using Claude Haiku 4.5 proved far more reliable by identifying hallucinations and providing justifications.

2. Mistaking Retrieval Failures for Model Failures When an agent responded that it had no information, the issue was often not a model knowledge problem but a problem at the retrieval stage. For example, cases were found where Chroma's cosine similarity threshold was too strict, causing relevant documents to fail to be retrieved. Before tuning the generation stage, one must first verify that the context window actually contains the relevant data.

3. The Relationship Between Model Selection and Cost/Quality Comparative testing of various models found that Gemma 4 26B recorded higher quality scores than the existing model (Gemini Flash Lite Preview) while being 75% cheaper, making it the most efficient choice.

4. The Impact of Grounding Constraints A system prompt constraint such as "only state facts found within the retrieved documents" improves accuracy, but there is a trade-off in that it simultaneously lowers the model's helpfulness score.