ADR-012: Fixtures, evaluation gates, and delivery standards

Status: accepted · Date: 2026-09-10

Context

Model quality is empirical and expensive. A full corpus run can conceal a bad parser or prompt for hours.

Decision

Progress from synthetic fixtures to stratified pages, one book, then five books. Split evaluation by source. Keep private source-derived data out of Git. Gate the pilot on ≥90% text fidelity, ≥95% claim support, Recall@10 ≥0.85, zero known incorrect accepted merges, complete provenance, reproducible graph rebuild, and seven recorded practice days. Report absent measurements as not_measured.

Code is typed, modular, dependency-injected, and tested at observable boundaries. Comments explain intent/invariants/units; tests never execute extracted commands. The implementation backlog is dependency ordered and stops after each fixture-backed review.

Consequences

The system ships slower but failures are attributable and reports are honest. Targets remain targets until a run records denominator, split, revision, and result. Revisit gates only through a documented decision.

Alternatives considered

Running the entire corpus first was rejected because a parser or prompt defect would be expensive to diagnose. Splitting by neighboring chunks was rejected because retrieval metrics would leak near-duplicate text. Treating model-card benchmarks as Deadwire measurements was rejected because they use different data and protocols. Tests that assert private implementation details were rejected in favor of observable contracts.

Implementation detail and example

The progression is synthetic fixtures, a stratified source-page sample, one complete book, the five-book pilot, and only then a larger corpus. Reports include run ID, commit, package/container versions, model revisions, fixture version, source split, denominator, failures, runtime, peak memory, retries, and cache reuse. Public charts show file counts separately from unique books and display not measured until a real run exists; illustrative charts say synthetic example.

Configuration and failure behavior

The pilot gates are ≥90% text fidelity on a labeled sample with its denominator, ≥95% reviewed claim support, Recall@10 ≥0.85 on held-out queries, zero known incorrect accepted merges, complete provenance, reproducible graph rebuild, five selected books with accounted failures, one approved path, and seven recorded practice days. A failed gate blocks promotion and creates a backlog item; it does not justify changing the metric or fabricating concepts.

Verification and revisit trigger

Run unit, contract, property, documentation, Compose, SVG, schema, cache, and recovery checks before each stage review. Compare lexical and dense retrieval baselines, report OCR command-token fidelity separately, and keep private source-derived evaluation material outside Git. Revisit a gate only through a new ADR containing rationale, affected denominators, and an unchanged historical comparison.