ADR-007: Incremental ingestion and exact provenance

Status: accepted · Date: 2026-09-10

Context

The collection contains 303 PDFs, 2 EPUBs, Markdown, text, and unsupported media. OCR can damage commands; an LLM cleanup can erase the original.

Decision

Inventory supported formats, verify iCloud stability, hash bytes, and process only invalidated stages. Prefer embedded PDFKit text; render deficient pages at 300 DPI within a pixel cap and run accurate Vision OCR. EPUB/Markdown/text use direct parsers. Keep raw extraction and normalized proposals separate.

Mechanics and tests

Record method, page/spine/line location, regions, confidence, source revision, extractor version, and artifact hashes. Use resumable stage states and one writer lock. Fixtures cover scans, columns, Unicode, commands, corrupt/password files, placeholders, duplicate hashes, renames, and changed revisions.

Consequences

The first run is slower but future additions are cheap and auditable. OCR confidence guides review; it never marks a command safe. Video and audio are inventoried as deferred, never accidentally parsed.

Alternatives considered

OCR-first processing was rejected because embedded PDF text is faster and usually preserves commands better. Replacing raw extraction with LLM-normalized text was rejected because normalization can silently alter flags, paths, or identifiers. Treating a file path as identity was rejected because renames and duplicate copies are common. Executing extracted examples to “validate” them was rejected on safety grounds.

Implementation detail and example

PDF extraction prefers embedded text; pages below the text-quality threshold are rendered at 300 DPI with a documented pixel cap and sent to Vision’s accurate recognition mode. References use PDF source revision plus page index, displayed label, and region; EPUB uses spine item and element/span; Markdown and text use line/span coordinates. Raw text, proposed normalization, and command verification are separate fields.

Configuration and failure behavior

Before processing, inventory checks file availability and stable size/mtime. A placeholder or unreadable iCloud file is recorded with its status and never passed as an empty document. Stage records include input hash, schema and extractor versions, prompt/model settings, attempts, and failure state. A retry is bounded; a partial artifact is quarantined and a rerun of a completed stage is idempotent.

Verification and revisit trigger

Fixtures cover clean text, scans, columns, Unicode, commands, corrupt/password-protected PDFs, duplicate hashes, renames, changed revisions, interrupted imports, and OCR confidence. Compare OCR character/word error and exact command-token fidelity against labeled references. Revisit DPI, language correction, or pixel caps only after a fixture-backed comparison shows improved fidelity without identifier corruption.

Distillation consumes deterministic chunks rather than a whole-book prefix. split_text targets 4,000 characters with 200 characters of overlap near paragraph boundaries; each output is named <source-hash>-<chunk-index>.md and references the source path, chunk index, and source hash. A rerun skips only the same chunk destination, so adding a page or changing bytes creates the appropriate new work without re-OCRing unchanged revisions.