Evaluation and reporting

Keep public synthetic fixtures in tests/fixtures; keep source-derived evaluation pages outside Git. Split by source document, never adjacent chunks, so retrieval does not leak near-duplicates.

Progression

Run synthetic fixtures, a stratified page sample, one complete book, then five books. Each report records run ID, commit, Python/container versions, model revisions, fixture version, sample sizes, failures, and cache hit rates.

Metric	Definition	Pilot gate
OCR fidelity	max(0, 1 - corpus CER); denominator is all reference characters	≥0.90
OCR CER / WER	summed edit distance / summed reference characters or whitespace-delimited words	report separately
Command fidelity	exact token/flag preservation on labeled commands	report separately
Schema validity	valid proposals / all proposals	≥0.95
Claim support	reviewed supported claims / reviewed claims	≥0.95
Recall@10	macro mean of relevant IDs in first ten / all relevant gold IDs, per query	≥0.85
nDCG@10	graded relevance with rank discount	report baseline and candidate
Merge precision	correct accepted merges / accepted merges; undefined when none accepted	descriptive
Incorrect accepted merges	count of reviewed accepted merges found incorrect	0, with review complete
Provenance completeness	claims with valid source refs / claims	1.00
Queue completion	completed planned units / due units	observe for seven days

The README’s chart is generated from evaluation JSON. Before a run exists, show “not measured”; zero is not a measurement. Synthetic charts must be labeled synthetic example. Charts must include thresholds, sample sizes, direction of improvement, and limitations.

deadwire report evaluation.json report.svg renders the dependency-free SVG report used by the handoff. The JSON must include a run identifier, model revision, and metric objects with names, values (or null), thresholds, sample sizes, and improvement direction. The renderer annotates missing values as not measured and never turns a target into an observed result.

Do not treat deduplication rate as a target. Track false merges separately. Learner improvement requires repeated practice observations over 30 days and cannot be inferred from extraction metrics.

Calculate measurements from observations

deadwire evaluate now computes OCR character/word error rates, OCR fidelity, and lexical/dense/candidate retrieval Recall@10 and nDCG@10. It reads labeled observations and recorded rankings; it does not run an embedding or generation model, invent relevance labels, or approve claims.

From the repository root, reproduce the public synthetic demonstration:

deadwire evaluate examples/evaluation.synthetic.json \
  --output /tmp/deadwire-evaluation.synthetic.json
deadwire report /tmp/deadwire-evaluation.synthetic.json \
  docs/assets/evaluation-synthetic.svg

Synthetic example only. This uses two fabricated OCR observations and one query with manually supplied rankings. A perfect candidate result here proves neither corpus retrieval quality nor improvement from a real model. The graph demonstrates reporting behavior; no Deadwire book was evaluated for these values. Amber markers are targets, green bar lengths are the synthetic observations, and each row names its sample size and improvement direction. All fraction bars use a 0–1 scale; error rates above one retain their numeric value and explicitly label the capped bar.

For a real run, keep the observation JSON outside Git. Preserve run_id, model_revision, dataset_id, synthetic, split, source_ids, and development_source_ids. Set synthetic: false only for actual labeled source material and recorded outputs. Every OCR case and retrieval query must identify one declared evaluation source. Held-out source IDs may not overlap development IDs. This checks the declared IDs; a reviewer must still verify document identity and ensure renamed or duplicated books did not leak across the split.

Each OCR case records a unique case_id, source_id, exact reference, and observed output. Preserve spaces, indentation, case, and Unicode code points. Each retrieval case records a unique query_id, source_id, a relevance map from document ID to integer grade 0–3, and three ordered ID arrays named lexical, dense, and candidate. All three systems are scored against the same queries and labels. Duplicate ranked IDs and queries without any positively relevant gold document are rejected. Empty result arrays are allowed and score zero when a gold set exists.

Denominators and interpretation

• CER: sum character Levenshtein distances across cases, divide by total reference characters. A long page carries more weight than a short page.
• WER: the same computation after splitting on whitespace; no case folding or punctuation normalization occurs. WER can miss indentation damage, so keep CER and future command-token checks separate.
• Fidelity: max(0, 1 - CER). Insertions can make CER/WER exceed one. Only fidelity is clamped, and the report retains raw edit counts and denominators.
• Recall@10: per-query count of positively relevant retrieved IDs in ranks 1–10 divided by all positively relevant IDs in that query's gold set, then average equally across queries. Unjudged results count as irrelevant.
• nDCG@10: use gain 2^grade - 1, discount each rank by log2(rank + 1), divide by the score of the ideal gold ordering, then macro-average queries. This exposes ordering differences that recall alone cannot show.

Absent sample families produce null, never an invented zero or perfect score. Aggregate output contains no reference/output text, and includes an evidence_sha256 over the validated, canonically serialized input. Preserve the original private input with the report to reproduce the measurement. The digest is not a proof that human labels are correct. Model quality comparisons also need representative held-out sources and uncertainty estimates; the tiny public fixture establishes software behavior only.

Release evidence validation

The readiness report rejects synthetic/development runs, missing run/model/dataset identifiers or input digest, non-finite values, booleans masquerading as scores, ambiguous duplicate metric names, and missing/zero sample sizes for fraction metrics. Fraction measurements must be in 0–1 and meet their fixed thresholds. The Incorrect accepted merges observation requires integer value: 0 and review_complete: true; sample_size: 0 is allowed when no merges were accepted. This avoids imposing a merge quota.

The evaluator currently does not compute human claim support, command-token fidelity, merge decisions, provenance resolution, runtime/memory, confidence intervals, or student outcomes. Those gates remain unmeasured until their evidence is collected. Human counts and approval flags supplied to the readiness CLI are declarations that must be audited against actual records; a passing summary alone never proves the course ready for students.