ADR-006: Local models and cache identity

Status: accepted · Date: 2026-09-10

Context

Full security text is sensitive and may be copyrighted. The host can run local inference. Extraction, embedding, reranking, and synthesis are different tasks and must not share an unexplained score.

Decision

Use the host Codex CLI with --model gpt-5.6-luna for distillation drafts, always with --sandbox read-only --ephemeral and an output file outside the source tree until reviewed. Use LM Studio's local OpenAI-compatible /embeddings endpoint for batched and optionally parallel vectors; it is not a chat or synthesis provider. Hugging Face supplies model artifacts, cards, licenses, and immutable revisions, not hosted inference. Set trust_remote_code=False, use local caches outside iCloud, and record MPS/CPU fallback.

Mechanics and tests

Cache keys include source hash, schema/prompt hash, extractor version, model revision, tokenizer settings, and relevant config. Model changes invalidate only dependent inference artifacts. Structured output gets one repair attempt, then quarantine. Tests assert cache reuse and selective invalidation with fake adapters; a held-out evaluation is required before promotion.

Consequences

Local inference protects privacy but consumes time and RAM. BGE-small may truncate long text, so chunk limits and token counts are recorded. A larger model is an evaluated replacement, never an invisible upgrade.

Alternatives considered

An uncontrolled cloud fallback was rejected because source text is private and may be copyrighted. The owner-selected host Codex CLI path is explicit and recorded. A single general model for every task was rejected because extraction, embedding, reranking, and tutoring have different error surfaces. An unpinned Hugging Face download was rejected because a moving revision makes a rerun incomparable. A separate clustering model was rejected for deduplication; similarity proposes candidates, while evidence and review decide identity.

Implementation detail and example

The initial role map is host Codex CLI gpt-5.6-luna for original lesson drafts, LM Studio /embeddings for batched vectors, BAAI/bge-small-en-v1.5 as the reference embedding model, and BAAI/bge-reranker-v2-m3 for top-30 cross-encoder reranking. No LM Studio chat model is part of the distillation path. Hugging Face is used for model cards, licenses, downloadable artifacts, and immutable revisions, never hosted inference. trust_remote_code=False and host caches outside iCloud are mandatory.

Configuration and failure behavior

Extraction begins with a 16K context budget, bounded section chunks, and a 4K output budget. Codex requests record model ID, CLI version, prompt hash, and token limits. Embedding requests record batch size, worker count, model ID, and vector dimensions. Codex is read-only and ephemeral; an unavailable model is a recorded failure, never a provider substitution. Invalid structured output gets one repair attempt, then quarantine. A model change invalidates dependent inference artifacts but does not re-OCR unchanged bytes.

Verification and revisit trigger

Synthetic fixtures test deterministic cache keys, selective invalidation, bounded truncation, malformed output, and CPU fallback. A model promotion requires held-out retrieval, schema validity, claim support, and command-token results with denominators. Revisit the model map only when a pinned replacement beats the baseline on the same source-separated evaluation set and its license/privacy review is recorded.