Separate retrieval failures from answer failures. For each test question, record the expected source, retrieved passages, generated answer, citation support, and whether a reviewer would approve the result. Report critical failures separately from aggregate scores.
Build a Representative Evaluation Set
Collect realistic questions with expected supporting passages, task type, and unacceptable mistakes. Include absent answers, conflicting documents, exceptions, and questions requiring exact identifiers.
Measure Retrieval Before Scoring Answers
Record which relevant source was expected and which passages were actually returned. Classify missing evidence separately from a generation error.
Example: The retriever returns the correct reimbursement exception, but the answer omits it. That is different from never retrieving the exception.
Assess Support, Relevance, and Completeness
Have reviewers check whether claims follow the cited evidence, address the question, and include material qualifications. Inspect automated scores alongside sampled human judgments; do not use one score as a correctness guarantee.
Set Release Criteria by Consequence
Agree on acceptable errors for the task and report critical failures separately from average scores. Rerun the same set after changing documents, chunking, retrieval, prompts, or the model; add newly observed failures.
The mix of questions affects what an evaluation score means. A test set dominated by straightforward lookups can obscure problems with exceptions, conflicting versions, or questions that have no supported answer. Group results by these task types before combining them. In a hypothetical policy assistant, strong performance on routine definitions would not resolve repeated failures on an exception that changes the user’s decision. The release discussion should identify that gap directly and decide whether to narrow the supported scope, improve the system, or require additional review.
Discuss the implementation scope with Sprinklenet. A useful starting point: a task-specific RAG evaluation set and release criteria.
References
The recommendations above are Sprinklenet’s practical guidance. Technical context: Microsoft RAG Evaluators, NIST Generative AI Profile.
