ADR-011: Recovery, backups, and privacy-aware operation

Status: accepted · Date: 2026-09-10

Context

Long OCR/model jobs can fail, iCloud can expose placeholders, and logs can accidentally leak source text or secrets.

Decision

Use explicit stage states, bounded retries, partial-artifact quarantine, one writer lock, health checks, and resumable recovery. Bind services to loopback, prepare owner-only host secrets into Docker-internal read-only volumes, rotate bounded logs, and redact text/prompts/tokens. Back up canonical iCloud files with iCloud history and Docker volumes with stopped-service archives.

Tests and consequences

Recovery drills interrupt each stage and assert no duplicates. Restore drills compare graph rebuild output. Privacy tests scan logs for fixture text and secrets. The owner remains responsible for source licensing and lab authorization.

Alternatives considered

Retrying indefinitely was rejected because a poisoned input or unavailable iCloud file would consume the queue forever. Logging full prompts was rejected because prompts contain private excerpts. Backing up only Docker volumes was rejected because canonical notes and learning evidence live in iCloud. Silent provider failover was rejected because it would make model provenance and privacy policy ambiguous. Codex CLI use follows the owner’s configured account and data policy; LM Studio embeddings remain local.

Implementation detail and example

Each stage records pending, running, completed, failed, or quarantined, with attempt count, input hash, versions, duration, and redacted error. One active writer owns mutations; a second process exits with a clear lock error. A backup captures stopped Compose volumes and a redacted manifest; restore starts services, verifies checksums, and rebuilds projections from canonical notes.

Configuration and failure behavior

Services bind to loopback, host secrets remain owner-only and are mounted from Docker-internal read-only volumes, logs rotate at a bounded size, and source text, prompts, tokens, and credentials are redacted. A retry uses bounded exponential backoff and stops at the configured maximum. An interrupted import resumes from the last completed stage; a partial artifact is quarantined. A failed restore or checksum mismatch blocks publication and preserves the prior backup.

Verification and revisit trigger

Run interruption drills at every stage, replay completed stages to prove idempotency, restore a fixture backup, and scan logs for fixture secrets and source phrases. Revisit operations only after a new deployment target supplies equivalent isolation, backup, recovery, and privacy evidence.