RAG Evaluation Harness

A small evaluation harness for a personal knowledge-base assistant. The goal is not a polished chat UI — it’s a repeatable loop that answers whether retrieval, grounding, and refusal behavior are getting better or worse as the pipeline changes underneath.

Shape

The working model stays deliberately narrow:

sources -> parsing -> chunking -> embeddings -> index -> retrieval -> answer -> review

Every run leaves behind enough to debug a failure without re-running it:

1. The input question and the source area it should have drawn from.
2. Retrieved chunks with scores and metadata — the whole ranked set, not top-1.
3. The generated answer and its cited spans.
4. A reviewer label for recall, grounding, usefulness, and refusal quality.

That record is the unit of work; the eval set and any regression gate are just views over a pile of these traces. The reasoning behind that stance is in Evidence-First LLM Systems.

Log

2026-07 — Frozen question set

Split evaluation into two index-independent checks — reaching the expected source area and grounding the cited span — so a re-index can move which chunks surface without invalidating the set. Wrote up the approach in A Retrieval Eval That Survives Index Changes.

2026-06 — First loop

Stood up the manual review loop over a fixed question set. Open questions still driving the work:

1. Should chunking preserve document hierarchy first, or optimize for compact semantic spans?
2. Which labels are sufficient for manual review without slowing every run?
3. How should citation quality be scored when the answer is useful but the cited span is too broad?
4. Which regressions should hard-block a retrieval change, and which are just a review queue?

Notes

The evaluation loop begins manually and stays that way until a metric earns automation. A small set of fixed questions — including ones whose correct answer is a refusal — exercises recall, citation quality, usefulness, and fabrication resistance before any automated score becomes meaningful.