ADR-008: Structured knowledge and review-only merging

Status: accepted · Date: 2026-09-10

Context

Books repeat concepts, use version-specific language, and sometimes disagree. Similarity can find candidates but cannot decide whether two concepts are identical.

Decision

Extract atomic proposals with definitions, domains, prerequisites, related/contrasting concepts, claims, commands, difficulty, importance, and source references. Use lexical+dense reciprocal-rank fusion, rerank 30 candidates, and generate deterministic merge proposals. Human approval is mandatory for semantic merges in MVP.

Mechanics and tests

Preserve aliases, source deltas, contradictions, and redirects. Validate claims and commands independently. Tests include near-duplicates that must remain separate, exact duplicates that may merge, contradictions that must persist, invalid JSON, repair, and no-auto-merge behavior.

Consequences

Review volume is real work, but false merges are more damaging than missed merges. Deduplication rate is descriptive rather than a target. Revisit thresholds only with a labeled merge set and false-merge analysis.

Alternatives considered

Free-form Markdown extraction was rejected because downstream review and graph sync need typed fields. Exact-name deduplication was rejected because the same technique has aliases and different versions. Automatic semantic merging was rejected because related, contradictory, and version-specific claims are not identity. A vector database was deferred because exact cosine search is sufficient for the five-book pilot.

Implementation detail and example

A proposal records an immutable candidate ID, editable title, definition, domain, prerequisites, related and contrasting concepts, atomic claims, commands, difficulty, importance, and source references. Retrieval first combines lexical and dense results with reciprocal-rank fusion, then reranks the top 30 using the cross-encoder. Similarity creates a review candidate such as “unconstrained delegation” versus “resource-based constrained delegation”; it never merges them automatically.

Configuration and failure behavior

Chunk limits are tokenizer-specific and token counts are stored to prevent silent embedding truncation. Claims without valid source references are rejected or quarantined. Contradictory claims retain both source references and an explicit contradiction edge. Approved merges preserve aliases and redirects; a rejected proposal remains available for audit. A failed model parse never creates a graph node or canonical note.

Verification and revisit trigger

Use source-separated held-out queries to measure Recall@10 and nDCG@10 against lexical and dense baselines. The merge fixture set includes exact duplicates, near-duplicates that must remain distinct, contradictions, version changes, invalid JSON, one repair, and no-auto-merge assertions. Revisit thresholds only after a labeled review set demonstrates precision and recall, with zero known incorrect accepted merges in the pilot.

Draft originality quarantine

Before a staging writer persists a model response, it normalizes whitespace and checks for a 24-word contiguous span already present in the supplied source excerpt. A match raises a quarantine failure and leaves no lesson file; this conservative detector is a copyright/safety gate, not a plagiarism score. Human review may inspect the quarantined response privately, while published lessons must be original synthesis with only bounded command tokens and provenance references.

Retrieval implementation

The pilot retrieval primitive in src/deadwire/retrieval.py tokenizes lexical matches, computes exact cosine similarity for precomputed vectors, and combines ranked IDs with reciprocal-rank fusion (k=60, top 30). rerank_candidates accepts an injected cross-encoder-compatible scorer and preserves lexical/dense ranks alongside the rerank score. src/deadwire/semantic.py supplies lazy, local SentenceTransformer adapters for BAAI/bge-small-en-v1.5 embeddings and BAAI/bge-reranker-v2-m3 reranking, with optional immutable revisions, trust_remote_code=False, MPS selection, and explicit CPU fallback. The LM Studio client also supports bounded embedding batches and parallel requests while preserving order. The adapters are not loaded by tests and do not download models implicitly. It records lexical and dense ranks on every hit so a reviewer can explain why a chunk was returned. Missing embeddings are allowed for a lexical-only cold start; adding a vector database is deferred until measured scale requires it.