Building AI Systems That Can Cite Their Sources
A citation should let the reader inspect the passage that supports a claim.
Knowledge Spaces
Applications
Governance
Delivery
Advisory
Industries
Government and Teaming
Company
Contact
Contact
Scope a Six-Week Pilot
Separate retrieval failures from answer failures. For each test question, record the expected source, retrieved passages, generated answer, citation support, and whether a reviewer would approve the result. Report critical failures separately from aggregate scores.
Collect realistic questions with expected supporting passages, task type, and unacceptable mistakes. Include absent answers, conflicting documents, exceptions, and questions requiring exact identifiers.
Record which relevant source was expected and which passages were actually returned. Classify missing evidence separately from a generation error.
Example: The retriever returns the correct reimbursement exception, but the answer omits it. That is different from never retrieving the exception.
Have reviewers check whether claims follow the cited evidence, address the question, and include material qualifications. Inspect automated scores alongside sampled human judgments; do not use one score as a correctness guarantee.
Agree on acceptable errors for the task and report critical failures separately from average scores. Rerun the same set after changing documents, chunking, retrieval, prompts, or the model; add newly observed failures.
The mix of questions affects what an evaluation score means. A test set dominated by straightforward lookups can obscure problems with exceptions, conflicting versions, or questions that have no supported answer. Group results by these task types before combining them. In a hypothetical policy assistant, strong performance on routine definitions would not resolve repeated failures on an exception that changes the user’s decision. The release discussion should identify that gap directly and decide whether to narrow the supported scope, improve the system, or require additional review.
Discuss the implementation scope with Sprinklenet. A useful starting point: a task-specific RAG evaluation set and release criteria.
The recommendations above are Sprinklenet’s practical guidance. Technical context: Microsoft RAG Evaluators, NIST Generative AI Profile.
Most weeks I send one short read on enterprise AI: what’s working in real deployments, what’s not, and what I’d do in your seat.
Jamie Thompson, Founder and CEO