A Retrieval Eval That Survives Index Changes

The first eval I wrote for a personal knowledge-base assistant was worthless, and it took a re-index to notice. Recall looked great, then I swapped the embedding model and every number moved — not because retrieval got worse, but because the “expected” chunks had been chosen by looking at what the old index returned. The eval measured agreement with a snapshot, not correctness.

This note is the fix: a small question set built to outlive the index it runs against. It’s part of the ongoing RAG Evaluation Harness.

The Anchor Is the Source, Not the Chunk

The mistake is anchoring an expected answer to a chunk ID or an embedding neighborhood. Both are properties of the current index. Anchor instead to something index-independent:

• The question, in the user’s words.
• The source area that should answer it — a document, a section, a claim — identified by stable metadata, not by chunk offset.
• A short rationale for why that source is authoritative.

Now a re-index can change which chunks surface without changing whether the right source area was reached.

question        "How is the retrieval score defined?"
source_area     doc: retrieval-notes  §: scoring
expects         a span stating cosine over unit-normalized embeddings
disallows       any answer citing the deprecated dot-product note

Score Reaching, Then Grounding

With source areas as the anchor, evaluation splits into two cheap checks that don’t need a golden answer string:

1. Reaching — did any retrieved chunk fall inside the expected source area? This is robust to chunking and embedding changes because it’s defined on the source, not the vector.
2. Grounding — does the generated answer’s cited span actually support the claim? Checked independently of whether the phrasing matches a reference.

Separating them tells you where a regression lives. Reaching drops → retrieval problem. Reaching holds but grounding drops → generation or citation problem. One number can’t distinguish those; two can.

Freeze Before You Tune

The discipline that makes the set trustworthy is boring: write the questions and their source areas before touching retrieval parameters, and never edit an entry to make a failing run pass. A failing entry is a finding. If it’s wrong, fix it in a reviewed change with a note — not silently mid-tuning.

What I’d Add Next

• A handful of questions whose correct answer is a refusal (no source covers them) — the set should punish confident fabrication.
• Near-duplicate source areas, to see whether the ranker prefers the authoritative one over a stale copy.
• A per-run diff of reaching/grounding against the last green run, so a review only looks at what moved.

None of this needs a framework. A directory of question files and a script that emits a trace per run is enough to stop lying to yourself about recall.