docs/architecture.md
Rama main · réplica de lectura del repositorio original.
# Architecture
kei is a Rust CLI that transfers iCloud Photos media and metadata to local
storage. This guide identifies the owner of each major decision and the
boundaries that protect user data.
Use it as a starting map. Read the owning module and its direct callers before
changing behavior.
When statements overlap, the Owners table defines ownership, the Safety
contract catalog defines required conditions, and source code wins over both.
## Navigation
This index lists the highest-traffic sections, not every heading. Link labels
match their headings, so searching a label finds the section.
- [Owners](#owners): find the decision owner and its boundary.
- [Design rules](#design-rules) and [Wrong-layer traps](#wrong-layer-traps): check layer placement before editing.
- [Command dispatch](#command-dispatch) and [Container startup](#container-startup): find entry points.
- [Sync and provider checkpoints](#sync-and-provider-checkpoints) and [Full and incremental enumeration](#full-and-incremental-enumeration): trace source progress.
- [Download and publication](#download-and-publication), [Durable pending retry](#durable-pending-retry), and [Import-existing](#import-existing): trace local work.
- [Account ownership and legacy adoption](#account-ownership-and-legacy-adoption), [Transactional source catalog and replay](#transactional-source-catalog-and-replay), and [Current provider work admission](#current-provider-work-admission): trace durable source ownership.
- [Selection generation shadow](#selection-generation-shadow), [Private selection and destination replay](#private-selection-and-destination-replay), and [Observed incremental shadow capture](#observed-incremental-shadow-capture): trace selection and destination projections.
- [Data-safety invariants](#data-safety-invariants), [Safety contract catalog](#safety-contract-catalog), and [Secrets and diagnostics](#secrets-and-diagnostics): find the required safety conditions.
- [Change-impact traversal](#change-impact-traversal) and [Change-impact checklist](#change-impact-checklist): find related owners and consumers before editing.
- [Tests](#tests): find proof entry points and the test guide.
- [Managed edited-primary layouts](#managed-edited-primary-layouts) and [Support evidence boundary](#support-evidence-boundary): trace specialized flows.
- [Maintaining this guide](#maintaining-this-guide): update this file with the change.
## Design rules
- User media and metadata must not be lost, corrupted, truncated, overwritten,
or silently discarded.
- Local file and metadata rewrites are opt-in.
- Provider-specific parsing and identity rules stay in the iCloud adapter.
- Sync policy stays in the sync and download orchestration layers.
- Path rendering does not decide whether an asset should sync.
- SQLite transitions and provider checkpoints must survive interruption.
- Prefer the smallest complete implementation in the owning module.
### Wrong-layer traps
- `src/commands/service.rs` plans provider libraries and collection passes.
Platform service installation and execution belong in `src/service/`.
- `src/selection.rs` parses selectors. Live collection lookup and pass planning
belong in `src/commands/service.rs`; path rendering cannot resolve them.
- `src/sync_loop/precheck.rs` handles the database change pre-check. Zone
checkpoint decisions belong in `src/sync_cycle.rs`, not in the Photos adapter
or file-transfer code.
- Facades such as `src/download/mod.rs`, `src/download/file.rs`, and
`src/state/db.rs` expose entry points. Read the child owner in the Owners
table before adding behavior or looking for its tests. `src/state/db.rs` also
owns connection lifecycle and blocking-pool dispatch; keep that work there.
- Reports, health, metrics, and support diagnostics consume observations.
They do not supply provider completeness or checkpoint authority.
## Owners
| Area | Owner | Boundary |
|------|-------|----------|
| Process startup and dispatch | `src/lib.rs` | Starts the runtime, resolves bootstrap paths, configures logging, dispatches commands, and maps exit codes. |
| CLI shape | `src/cli.rs` | Defines clap arguments and parsing. It does not execute command behavior. |
| Runtime configuration facade | `src/config.rs` | Preserves configuration types, entry points, and visibility. |
| Configuration input | `src/config/input.rs` | Owns the TOML schema and file loading. |
| Runtime policy types | `src/config/runtime.rs` | Owns resolved configuration types, media selection, and date-bound semantics. |
| Configuration resolution | `src/config/resolve.rs` | Resolves TOML, environment, and command inputs into runtime policy, including shared sync/import path fields. |
| Configuration paths | `src/config/paths.rs` | Resolves bootstrap, data, and credential paths and validates download directories. |
| Folder-template validation | `src/config/templates.rs` | Owns default category templates and validates token placement. |
| Configuration persistence | `src/config/persistence.rs` | Projects runtime policy to TOML and writes first-run configuration. |
| Selection grammar | `src/selection.rs` | Parses album, smart-folder, library, exclusion, and unfiled selectors. |
| Sync/watch facade and runner | `src/sync_loop.rs`, `src/sync_loop/runner.rs` | Preserves command-facing entry points; the runner composes startup, sync cycles, and watch control. |
| Sync session recovery | `src/sync_loop/session.rs` | Coordinates authentication, 2FA recovery, library initialization, and idle session-lock handoffs through the shared auth and command owners. |
| Database change pre-check | `src/sync_loop/precheck.rs` | Owns scoped database tokens, selected-zone checks, and inclusion of pending local work. It does not replace the per-zone checkpoint gate. |
| Sync plan refresh | `src/sync_loop/planning.rs` | Resolves and refreshes library pass plans through the command owner; reports shared-library notices and path warnings. |
| Local drift checks | `src/sync_loop/reconcile.rs` | Runs bounded local drift recovery and periodic read-only catalog diagnostics. |
| Watch policy | `src/sync_loop/watch.rs` | Owns watch cadence, metadata follow-up timing, and one-shot recovery rules. |
| Cycle ledger reporting | `src/sync_loop/reporting.rs` | Records cycle-run summaries and combines refresh-tail outcomes for reporting. |
| One sync cycle | `src/sync_cycle.rs` | Chooses source enumeration, reconciles config drift, dispatches each library, and advances or preserves provider checkpoints. |
| Library and pass planning | `src/commands/service.rs` | Resolves libraries, collection scope, album plans, smart folders, unfiled passes, and cross-zone hydration. |
| iCloud Photos adapter | `src/icloud/photos/` | Owns CloudKit records, queries, change streams, provider identity, albums, smart folders, and metadata decoding. |
| Album facade and counts | `src/icloud/photos/album.rs`, `src/icloud/photos/album/counts.rs` | Preserves album configuration and entry points; counts owns count queries and same-library batching. |
| Targeted provider lookup | `src/icloud/photos/album/lookup.rs` | Resolves requested records and merges identity evidence conservatively. |
| Provider enumeration | `src/icloud/photos/album/planning.rs`, `src/icloud/photos/album/enumeration.rs` | Planning selects rank ranges and page sizes; enumeration starts fetchers and delivers completion tokens. |
| Provider page fetching | `src/icloud/photos/album/fetch.rs` | Owns query construction, empty-page probes, record pairing, deduplication, and asset emission. |
| Enumeration completion | `src/icloud/photos/album/completion.rs` | Collects fetcher completion evidence and requires unanimous tokens before returning a checkpoint candidate. |
| Album hydration | `src/icloud/photos/album/hydration.rs` | Matches named-album members across source zones and hydrates durable asset or master identities. |
| Provider change streams | `src/icloud/photos/album/changes.rs` | Owns sequential change scans, delta buffering, and last-good-token delivery. |
| Download facade and dispatch | `src/download/mod.rs`, `src/download/orchestration/dispatch.rs` | Preserves download entry points and composes full, incremental, targeted-backfill, and durable-retry work. |
| Download models and configuration | `src/download/orchestration/models.rs`, `src/download/orchestration/config.rs` | Owns controls, results, per-zone checkpoint evidence, reporting projections, checkpoint reasons, coverage fingerprints, and configuration hashes. |
| Download context and selection | `src/download/orchestration/context.rs`, `src/download/orchestration/selection.rs` | Loads library-scoped state and identity evidence, derives pass configurations, and checks incremental routing eligibility. |
| Full enumeration | `src/download/orchestration/full.rs` | Owns bounded enumeration, album snapshots, pass counts, and token evidence. |
| Incremental enumeration | `src/download/orchestration/incremental.rs`, `src/download/orchestration/delta.rs` | Runs separate streaming and collecting strategies with one shared delta-state owner for provider-event bookkeeping and routing facts. |
| Download recovery | `src/download/orchestration/recovery.rs`, `src/download/orchestration/url_refresh.rs` | Runs durable pending recovery and refreshes exact download tasks with current provider URLs. |
| Local catalog maintenance | `src/download/orchestration/reconciliation.rs`, `src/download/orchestration/cleanup.rs`, `src/download/orchestration/maintenance.rs` | Reconciles catalog paths, removes durably owned stale temporary files, and repairs metadata-capture revisions. |
| Asset planning | `src/download/planner.rs` | Applies filters, derives tasks, records dispatched pending work, and persists membership and identity mappings. |
| Filter facade and configuration | `src/download/filter.rs`, `src/download/filter/config.rs` | Preserves filter entry points and shares path and filter settings between sync and import. |
| Asset eligibility | `src/download/filter/eligibility.rs` | Owns content, date, and filename filters and media classification, without path or filesystem decisions. |
| Rendition selection | `src/download/filter/versions.rs` | Owns RAW alignment, primary and companion selection, and selected-rendition metadata keys. |
| Task metadata | `src/download/filter/metadata.rs` | Builds metadata payloads with preloaded album and people groupings. |
| Expected asset paths | `src/download/filter/expected_paths.rs` | Derives paths shared by sync and import before collision resolution; uses the rendition-selection owner. |
| Collision planning | `src/download/filter/collisions.rs` | Owns normalized path claims and existing-file collision resolution. |
| Download task derivation | `src/download/filter/tasks.rs` | Composes eligibility, expected paths, collision results, and metadata into download tasks. Path rendering stays in `src/download/paths.rs`. |
| Download pipeline facade | `src/download/pipeline.rs`, `src/download/pipeline/streaming.rs` | Preserves entry points and composes run modes, producer execution, consumer execution, and pass finalization. |
| Local-file adoption | `src/download/pipeline/adoption.rs` | Validates current and pending local-file evidence before adoption or an on-disk skip. |
| Streaming producer | `src/download/pipeline/producer.rs` | Plans assets, forecasts disk space, records skip counts, and dispatches pending tasks. |
| Streaming consumer | `src/download/pipeline/consumer.rs` | Runs bounded workers and accumulates transfer results and deferred state writes. |
| Single-task execution | `src/download/pipeline/task.rs` | Coordinates one transfer, metadata completion, temporary-file ownership, and worker error classification. |
| Download pass and outcome | `src/download/pipeline/pass.rs`, `src/download/pipeline/outcome.rs` | Executes explicit task passes, finalizes streaming state, retries failed tasks, and aggregates sync outcomes. |
| Download progress and summary | `src/download/pipeline/progress.rs` | Formats durations and sync summaries and reports rate-limit pressure. |
| File facade and transfer | `src/download/file.rs`, `src/download/file/transfer.rs` | Preserves entry points and owns HTTP retry, resume, stream writes, and temporary-file cleanup. |
| File validation and fingerprints | `src/download/file/validation.rs`, `src/download/file/fingerprint.rs` | Validates response lengths and media, checks local size evidence, and captures same-read hashes and snapshots. |
| File publication and replacement | `src/download/file/publication.rs`, `src/download/file/replacement.rs` | Handles no-overwrite collisions and conditional replacement with displaced-file verification and restoration. |
| Confined local copies | `src/download/file/reconciliation.rs` | Retains verified files and parent-directory capabilities through local copy and state finalization. |
| File platform primitives | `src/download/file/platform.rs` | Owns platform rename, exchange, hard-link, and directory durability operations. |
| Linux replacement recovery | `src/download/file/replacement_recovery.rs` | Journals conditional replacement when atomic exchange is unsupported, and recovers interrupted publication without overwriting concurrent edits. |
| Staging identity | `src/download/file/staging.rs` | Coordinates destination guards, versioned staging names, and cleanup leases so retries and collisions keep independent staging files. |
| Managed layout file handover | `src/download/file/primary_layout.rs` | Snapshots layout fingerprints, copies verified members, publishes, recovers prepared links, and retires aliases for a confined managed layout. Transition policy stays in `src/download/orchestration/primary_layout.rs`. |
| State finalization | `src/download/finalize.rs` | Persists downloaded or failed outcomes and retries deferred state writes. |
| Sparse identity retries | `src/download/orchestration/delta/sparse_identity.rs`, `src/state/db/sparse_identity.rs` | Selects bounded source retries and persists generation-fenced evidence; the cycle owner retains checkpoint authority. |
| Durable retry resolution | `src/download/retry.rs` | Revalidates pending provider identity and builds exact retry tasks. |
| Path rendering | `src/download/paths.rs` | Expands folder templates, normalizes names, and handles collision suffixes. |
| Metadata facade and values | `src/download/metadata.rs`, `src/download/metadata/values.rs`, `src/download/metadata/xmp_fields.rs` | Preserves writer entry points and shared values, with one owner for managed XMP properties, namespace initialization, and field encoding. |
| Metadata evidence | `src/download/metadata/probe.rs`, `src/download/metadata/source_gps.rs` | Probes existing EXIF/XMP fields and performs bounded source EXIF GPS decoding for write planning. |
| Embedded writer dispatch | `src/download/metadata/embedded.rs`, `src/download/metadata/formats.rs` | Validates approved input and selects a format writer using content and extension detection. |
| Format-specific metadata preparation | `src/download/metadata/heif_writer.rs`, `src/download/metadata/xmp_writer.rs`, `src/download/metadata/native_writer.rs` | Prepares HEIF-family, XMP Toolkit, or native-only EXIF output. Publication stays in the prepared-file owner. |
| Embedded replacement publication | `src/download/metadata/prepared.rs` | Owns exclusive temporary files, fingerprints, and stable-input publication through the file replacement primitives. |
| Sidecar ownership and publication | `src/download/metadata/sidecar.rs` | Prepares sidecars, validates ownership, and conditionally publishes writes, including reconciled sidecars. |
| HEIF facade and file-backed reads | `src/download/heif.rs`, `src/download/heif/file.rs` | Preserves HEIF entry points and typed errors; locates native Exif with bounded file reads without loading media payloads. |
| HEIF layout | `src/download/heif/boxes.rs`, `src/download/heif/items.rs` | Owns box boundaries, item tables, extent resolution, and item-location encoding. |
| HEIF metadata relationships | `src/download/heif/relationships.rs` | Resolves primary-image metadata ownership and encodes item references. Ambiguous ownership fails closed. |
| HEIF native Exif | `src/download/heif/exif.rs` | Extracts native Exif and repairs capture timestamps, offsets, and paired subseconds within existing TIFF storage. Unsafe or insufficient storage leaves repair pending; native values are verified before publication. |
| HEIF XMP | `src/download/heif/stream.rs`, `src/download/heif/xmp_write.rs` | Reads the selected packet and plans bounded item-map replacements without re-encoding image payloads. The in-memory `xmp_read` remains a test and fuzz oracle. |
| HEIF streaming | `src/download/heif/stream.rs` | Reads bounded control data and metadata packets, streams unchanged file ranges, and compares protected payloads with fixed buffers. |
| HEIF preservation checks | `src/download/heif/stream.rs`, `src/download/heif/preservation.rs` | Streams production checks of non-XMP payloads and opaque metadata; the in-memory validator remains a test and fuzz oracle. |
| Metadata rewrite facade and immediate execution | `src/download/metadata_rewrite.rs`, `src/download/metadata_rewrite/immediate.rs` | Preserves rewrite entry points and coordinates download-time metadata writes. The pipeline retains media publication and state finalization. |
| Metadata write planning | `src/download/metadata_rewrite/planning.rs` | Owns opt-in flags and embedded field decisions shared by immediate and queued execution. |
| Metadata write execution | `src/download/metadata_rewrite/embedded.rs`, `src/download/metadata_rewrite/sidecar.rs` | Coordinates fingerprint-guarded embedded writes and source-aware sidecar writes through the metadata owners. |
| Queued metadata retries | `src/download/metadata_rewrite/queued.rs` | Tags durable rewrite work, executes bounded retry pages, and retires completed markers. |
| Rewrite evidence and groupings | `src/download/metadata_rewrite/capture.rs`, `src/download/metadata_rewrite/grouping.rs` | Verifies capture-repair fingerprints and timestamps; loads current library-scoped groupings for queued work. |
| Account namespace | `src/account.rs` | Hashes exact configured login and provider realm without alias inference. |
| Account database boundary | `src/state/db/account.rs`, `src/commands/migrate_state.rs` | Validates an independent owner before state consumption, creates authenticated state, and explicitly adopts a preserved legacy snapshot. |
| SQLite facade and connection lifecycle | `src/state/mod.rs`, `src/state/db.rs` | Preserves state entry points, opens connections, and dispatches blocking SQLite work. Child owners retain their transaction boundaries. |
| Observed provider pages | `src/icloud/photos/inbox.rs`, `src/state/db/provider_inbox.rs` | Binds validated incremental observations to the account, retains source identities and original bytes atomically with a shadow receipt, and applies bounded backpressure. It does not authorize provider checkpoints. |
| Provider source catalog | `src/icloud/photos/projection.rs`, `src/state/db/provider_catalog.rs` | Validates replay provenance, derives versioned source facts, and atomically indexes records, references and unresolved evidence. Its receipt does not authorize queues or checkpoints. |
| Selection shadow manifests | `src/state/db/provider_selection.rs`, `src/download/orchestration/queue_projection.rs` | Retains account-bound current planning evidence and per-pass destinations, then validates lossless replay. This shadow receipt has no selection, publication or checkpoint authority. |
| Provider work admission | `src/icloud/photos/album/work.rs`, `src/download/orchestration/queue_projection.rs`, `src/state/db/provider_work.rs` | Confirms retained identities against current provider records, applies current task planning, and atomically admits guarded existing queue obligations and retained conflict debt. Checkpoint authority stays with the existing cycle. |
| SQLite schema | `src/state/schema.rs` | Owns schema versions and migrations. |
| State-store contracts | `src/state/db/contracts.rs` | Defines store roles and records exchanged with callers. |
| Asset state transitions | `src/state/db/assets.rs` | Owns asset lifecycle, download finalization, retry eligibility, and provider source-state transitions. |
| Shared SQLite writes and row decoding | `src/state/db/asset_writes.rs`, `src/state/db/rows.rs` | Shares asset writes within caller-owned transactions, column projections, date codecs, and row decoding. |
| Metadata capture retries | `src/state/db/metadata_capture_retry.rs` | Stores generation-fenced ambiguity evidence and bounded retry deadlines. The cycle owner retains checkpoint authority. |
| Durable provider identity | `src/state/db/identity.rs` | Stores library-scoped asset/master mappings and legacy state ownership. Provider record parsing stays in the iCloud adapter. |
| Durable membership | `src/state/db/membership.rs` | Stores album snapshots, provider relations, and grouping projections. |
| Durable metadata work | `src/state/db/metadata.rs` | Stores capture revisions, path-specific rewrite work, and repair receipts. It does not write media files. |
| Durable checkpoints | `src/state/db/checkpoints.rs` | Stores provider checkpoints, scoped database tokens, and enumeration progress. Sync policy decides when checkpoint proof permits a commit. |
| Temporary-file ownership | `src/state/db/temp_files.rs` | Stores losslessly encoded paths and durable temporary-file ownership. Filesystem cleanup stays in download orchestration. |
| Durable reconciliation | `src/state/db/reconciliation.rs` | Stores destination reservations and reads catalog evidence for reconciliation. |
| Durable import adoption | `src/state/db/import.rs` | Atomically adopts imported files into state and reads imported records. Matching policy stays in the import command. |
| Durable layout journals | `src/state/db/primary_layout.rs` | Stores preservation-backed layout bindings, operations, members, path claims, and immutable historical receipts. |
| Prior publication recovery | `src/state/db/pending_publication.rs` | Recovers a rendition's unchanged prior current publication and retains conditional recovery evidence. |
| Legacy preservation evidence | `src/state/db/legacy_preservation.rs` | Stores immutable legacy evidence and generation-fenced current-coverage receipts. It never assigns identity, writes files, or infers provider completion. |
| State reports | `src/state/db/reports.rs` | Reads status, verification and manifest data, and records sync-run history. |
| Import-existing | `src/commands/import.rs` | Matches existing files to expected iCloud paths and adopts verified files into state. |
| Service integration | `src/service/` | Owns install, uninstall, status, service execution, and platform renderers. |
| Support evidence | `src/support/`, `src/state/db/support.rs` | Allowlisted bounded runtime history and strictly offline immutable collection; see [support export](support-export.md). |
| Operator surfaces | `src/commands/status.rs`, `src/commands/doctor.rs`, `src/commands/manifest.rs` | Read local state for status, redacted diagnostics, and catalog export. |
| Reports and monitoring | `src/cycle_reporter.rs`, `src/report.rs`, `src/health.rs`, `src/metrics.rs`, `src/notifications.rs` | Converts cycle facts into reports, health, metrics, and notifications. |
Album tests use `icloud::photos::album::<owner>::tests::<test_name>` instead
of `icloud::photos::album::tests::<test_name>`. The owners are `lookup`,
`counts`, `planning`, `enumeration`, `fetch`, `completion`, `hydration`, and
`changes`. All 88 original tests retain their names, attributes, and assertions,
including the ignored live lookup test. Shared fixtures stay in
`src/icloud/photos/album/test_support.rs`. Album tracing keeps the
`kei::icloud::photos::album` target.
Album-owner dependencies are one-way: hydration uses enumeration, lookup and
change scans; enumeration uses planning, fetching, and completion; fetching uses
planning types and completion evidence. Lookup, counts, planning, completion,
and changes do not call sibling owners. The facade retains stream routing and
public entry points. No album child owns durable checkpoint policy.
Targeted lookup rejections retain fixed `master_status` and `child_status`
response categories in bounded support history. These distinguish omitted,
not-requested, returned, tombstoned and provider-error members without storing
raw codes, messages or identifiers. `returned` describes response presence,
not owner, identity, resource or checkpoint proof. Lookup resolution and
checkpoint policy do not consume these diagnostic categories.
`child_reference_status` separately describes a missing, malformed, sparse,
matching or different reference, or an omitted, unavailable, invalid or
unrequested child. These fixed labels expose no reference values and do not
establish a usable pair.
Best-effort album identity hydration has task-scoped anonymous progress in
`icloud/photos/hydration_progress.rs`. Every 30 seconds and on return or drop,
it reports fixed phase labels, source ordinal/role (`owner` then `fallback`),
aggregate received-page and
record counts, request attempts/retries, and current-wait elapsed time. Counts
are observations, not completeness or checkpoint evidence. The scope does not
change request timeouts, retry policy, source priority, or identity acceptance;
provider identifiers, cursors, request bodies and raw errors never enter its
state. Other provider operations have no active diagnostic scope.
File tests use `download::file::<owner>::tests::<test_name>` instead of
`download::file::tests::<test_name>`. The owners are `transfer`, `validation`,
`fingerprint`, `publication`, `replacement`, and `reconciliation`. HTTP tests
retain their nested `wiremock_tests` module under `transfer::tests`. Test names
and assertions are unchanged; the Windows partial-replacement test remains
Windows-only. File tracing keeps the `kei::download::file` target.
File-owner dependencies are one-way: transfer calls validation and publication;
publication calls replacement, platform primitives, and fingerprinting;
replacement calls platform primitives and fingerprinting; reconciliation calls
platform primitives and fingerprinting; validation uses fingerprinting for
local-size evidence. Platform primitives do not decide replacement or retry
policy. In particular, Windows partial-exchange recovery stays in replacement.
Windows native move and backup-replacement calls share an absolute, lossless
UTF-16 path adapter. It resolves ordinary Win32 separators and components
without filesystem traversal, then supplies extended drive or UNC paths.
Existing verbatim namespaces and native units remain intact; embedded NUL is
rejected. Stored paths, confinement, no-overwrite flags, ownership checks and
replacement recovery do not change. This avoids MAX_PATH failures in nested
managed-primary staging without requiring a registry or manifest change.
Pipeline tests use `download::pipeline::<owner>::tests::<test_name>` instead
of `download::pipeline::tests::<test_name>`. The owners are `adoption`,
`consumer`, `outcome`, `pass`, `producer`, `progress`, `streaming`, and `task`.
Test names and assertions are unchanged. Shared test fixtures stay in
`src/download/pipeline/test_support.rs`.
XMP Toolkit preparation checks movie box boundaries with bounded file reads before
updating the temporary copy. Nested QuickTime `meta` boxes without a supported
FullBox prefix are rejected because the toolkit can corrupt their existing child
boxes. The optional four-byte zero terminator is accepted only at the end of a
`udta` child list. Rejection preserves the original media and leaves embedded
rewrite work pending through the existing retry owner.
## Main flows
### Container startup
`docker/entrypoint.sh` owns command collision routing and PUID/PGID privilege
drop. The explicit command list mirrors visible clap commands before checking
external executables. Without PUID/PGID, it executes as root without ownership
repair. With both numeric IDs, it recursively repairs UID and GID on /config
and /photos, including symlinks themselves without following their targets.
Traversal or repair failures warn before privilege drop. Every restart scans
all inodes; KEI_CHOWN_PHOTOS=0 skips /photos repair when host permissions are
already managed, while /config remains repaired.
For `support-export`, the wrapper queries the owning Rust CLI parser before
volume initialization. The private query ignores captured CLI environment
bindings and returns an exit status without runtime, config-file reads or
credential resolution. Export launches and their help routes skip both
volume traversal and ownership repair, including incomplete export arguments,
then drop to the requested numeric UID:GID. Option values cannot select this
exception. Status 0 selects export, and only status 1 permits ordinary volume
initialization. An unexpected query exit, including a crash or signal, aborts
before traversal. Normal launches retain their existing ownership initialization.
The Docker HTTP healthcheck defaults to port 9090. A deployment using a custom
`[server].port` must set `KEI_HEALTHCHECK_PORT` to that same port, including
when overriding the command with a custom config path. Docker does not parse
TOML separately. Disabled HTTP servers require a deployment healthcheck override.
See `docker-compose.yml` for runtime examples.
### Command dispatch
```text
src/main.rs
-> kei::main_inner
-> src/lib.rs::run
-> src/cli.rs
-> command owner, service owner, or src/sync_loop.rs
```
`run` composes private phases in `src/lib.rs`: `load_startup_config`,
`resolve_startup_output`, `initialize_logging`, and `dispatch_command`.
Config loading retains the Docker fallback and passes parse errors to
`doctor` without blocking its diagnostic report. Output resolution keeps the
friendly-mode request separate from the mode allowed by the environment.
The log-writer guard stays in `run` until dispatch returns. Startup injects
the captured environment password before dispatch. Command dispatch retains
the setup wizard's one-shot sync path and the service owner's entry point.
The CLI requires a subcommand. `kei sync` enters the sync path. Commands such
as `status`, `doctor`, and `manifest` read local state without entering the
normal iCloud sync loop.
`main_inner` records whether stdin is interactive before it starts the async
runtime. Command owners use that single input mode for password and 2FA
prompts. A foreground command returns `TwoFactorRequired` to the exit
classifier, which reports auth exit code 3 and the `login get-code` and
`login submit-code` recovery flow. Only the sync owner may convert that error
into a durable wait, and only when the resolved configuration is in watch or
service mode.
Interactive login and `login get-code` may retry once from clean local auth
state when Apple rejects verification-code delivery with HTTP 403 and the
attempt loaded a persisted cookie jar or session. The failed session removes
only its cookie jar, session, and validation cache while it still holds the
per-account lock, then the shared auth owner creates one clean replacement
session. Password material and SQLite state remain unchanged. `reset session`
exposes the same locked cleanup explicitly. Cleanup advances a generation in
the existing lock file. A watch process that released its lock for idle sleep
detects a changed generation when it wakes and stops before its old in-memory
session can recreate reset credentials. Reauthentication handoffs carry their
pre-release generation into replacement-session creation, so a reset in that
gap also wins. The session guard explicitly unlocks on drop, including failed
generation validation and rejected principal replacements. Closing only its own
descriptor would leave a forked child's copy holding the shared lock until
exec or exit. An explicitly released guard does not unlock again on drop;
generation checks and independent-process exclusion remain in force. Blocking
generation writes retain the shared guard until completion even if their caller
is canceled, so cleanup cannot unlock while a generation write is pending.
### Account ownership and legacy adoption
Configured login and realm determine collision-resistant names for state,
auth artifacts and encrypted credentials. Account-owner format version 1 is
independent of the application schema version and must be checked before its
migrations. Production connections require an owner expectation; unbound
connection helpers are restricted to tests. All state commands, including
read-only export and reset, enforce this boundary. State reset checks ownership
before confirmation and requires `--yes` when the process input mode cannot
prompt. An absent database remains a successful no-op. Sync and import require a
usable authenticated provider identity and pin it independently of filenames.
Watch validation, replacement sessions and initialization retries retain that
principal pin. Changed or missing authenticated identity stops before further
provider or state work; it cannot become a recoverable quiet-watch warning.
Legacy discovery fails closed rather than creating fresh state over retained
debt. Explicit `migrate-state` requires verified operator ownership
confirmation and a fresh login under the account lock. SQLite backup retains
rows, unknown tables and committed WAL content, then binding, schema migration
and integrity validation occur in a private stage. Same-directory publication
refuses replacement and fsyncs the file and parent. The legacy source and
companions remain. Conflicting scoped provenance is rejected, not rewritten.
Legacy session and encrypted credential files are preserved and ignored;
keyring identity remains unchanged. See [account migration](account-state-migration.md)
for operator steps and compatibility limits.
### Observed incremental shadow capture
After opening an authenticated, account-owned database, sync attaches capture to
its Photos service and libraries. Normal incremental `changes/zone` responses
are bounded and validated before `DeltaRecordBuffer` pairing or selection. The
adapter rejects ambiguous JSON keys and unusable source identity or scope for
the whole page. Original response bytes retain unknown fields, tombstones and
unresolved relationships, including numeric lexemes that a typed JSON re-encode
could alter. Cookies and authentication headers are not captured; resource URLs
remain transient payload, never durable identity.
Schema 29 adds `provider_shadow_pages`, ordinal source identities in
`provider_shadow_records`, and a per-scope last-observed pointer in
`provider_shadow_receipts`. An immediate SQLite transaction rechecks the
independent account owner and commits all three together. Provenance includes the
account namespace and authenticated provider fingerprint, versioned realm,
CloudKit container and environment, private/shared database, validated zone and
returned owner when present, request cursor and successor. Unknown zone metadata
is retained in the payload without changing the scope key. Exact page replay is
idempotent; a changed payload remains a separate observation.
Original decoded response bytes are limited to 16 MiB per page. The inbox allows
512 MiB of charged payload and provenance bytes, including source identities.
This is a logical budget, not a physical SQLite/WAL file quota. An oversized
page, full inbox, failed write or unusable identity stops that page before
emission or successor acceptance. Such a refusal propagates through orchestration;
an uncaptured rank inventory cannot replace it and advance the checkpoint.
Existing typed invalid-token fallback and authentication handling remain intact.
Exact already-captured replay remains possible at capacity. No observations are automatically pruned. SQLite WAL `NORMAL`
remains unchanged; a successful commit does not promise that every acknowledged
observation survives power loss. Network filesystems are not newly qualified.
Schema 36 adds a private-only canonical transport index. The provider adapter
recomputes evidence from the retained page after account validation. It
recognizes exactly two direct descriptor paths: a complete `CPLMaster`
`fields.resOriginalRes` `ASSETID`, and a complete `CPLAsset` or `CPLMaster`
`fields.resComputeStateData` `ASSETID`. Each must have a nonblank `recordName`
and `recordChangeTag`, a numeric `size`, nonblank `fileChecksum`,
`referenceChecksum` and `wrappingKey`, and a validated download URL. Only the
whole validated `downloadURL` token is masked. Record type, tag, descriptor
bytes and unknown bytes stay exact, and nested or arbitrary `downloadURL`
fields are not eligible. Each candidate is bound to the envelope that produced
its byte path: a changes candidate is resolved only against `zones[0].records`
and a lookup candidate only against the paired root `records`, so a lookalike
record at the same ordinal in another envelope can never lend eligibility or
mask an unknown-envelope URL. This is a provider-versioned asset identity, not a
verified content hash. The private compute path is conditional coverage: it
does not claim that observed reporter records satisfy this descriptor shape.
A scope that lacks the realm, database, zone name, or exact nonblank owner is a
raw observation with no coalescing proof, not a rejected capture. An invalid
private page, malformed scope, or mismatched index entry still fails validation.
The canonical key is an unambiguous hash of account namespace,
provider fingerprint, exact scope, request cursor, successor, `moreComing`,
classifier version and masked body digest, so different cursors or scopes remain
independent classes. A hit reloads the representative through the checked page
loader, recomputes its key and masked bytes, and requires exact metadata and
source-identity equality. Unsupported scopes stay ordinary raw observations;
an invalid private page or a mismatched index entry fails validation. A replay
always reprojects the durable source, so an interruption after capture but before
projection recovers on the next equivalent observation.
The same schema step compacts finite legacy duplicates before ordinary admission.
It streams one body at a time inside the migration savepoint and requires a
validated raw hash, provider-owned evidence, exact account/provider/scope/cursor/
successor/`moreComing` binding, complete masked-byte equality, and a fresh pure
projection that equals every stored record, reference, debt and charge. Only then
may an old body become empty. The old row keeps its original ID, ordinal source
identities, historical observation hash and catalogue/work/selection references;
a typed alias stores the canonical page, classifier version, masked digest,
canonical certificate and an immutable provenance binding over the original ID
and hash, canonical ID and hash, class key, version and ordinal identities.
Known work receipts, work retries/scans and selection sources must still match
that old identity, source hash and independent confirmation bytes before the
body may be dropped; an unsupported or mismatched dependency pins the page as
raw. Loads expose `Raw` versus `LegacyAlias` provenance and verify the
representative hash, certificate, provenance binding and index on every access.
Alias chains and cycles are refused. The historical hash is never presented as a hash
of representative bytes, and no original hash is fabricated. Unknown formats,
unsupported projector versions or failed projection proof stay raw; inability to
prove the optional projection (including its byte budget) makes a page
ineligible for compaction rather than failing migration, while SQLite and
storage errors still abort. Migration
charges retained provenance plus alias and index overhead, does not raise budgets
or prune catalogue rows, and rolls every alias and body drop back with the schema
version on any failure. Replaying a discarded URL variant validates the alias and
returns the canonical ID; it creates no new alias.
A shadow receipt is only the last observed page, and replay can move it back. It
is not a coverage frontier, materialization receipt or verified filesystem
progress. Existing `CheckpointEvidence`, sparse and legacy generation proofs,
recovery debt and filesystem guards retain checkpoint authority. Capturing a
tombstone cannot authorize local media deletion. Existing download and metadata
queues still consume the current stream. Capture alone cannot admit queue work;
the bounded retained-source admission in
[Current provider work admission](#current-provider-work-admission) is a
separately validated owner.
Capture covers only incremental pages observed by existing sync in selected
scopes. Rank inventories, bootstrap, lookup and deletion-validation scans are
not represented as complete zone capture or historical completeness. Rank EOF
plus a delta bridge is not an atomic snapshot or proof of absence. New future
observations cannot retire old expired-epoch debt. Complete queue projection,
capture-owned cursors, durable catalog selection and separate progress remain
later stages; the bounded current-generation slice in
[Current provider work admission](#current-provider-work-admission) does not
complete them. Historical-version retention, unrecoverable-debt acknowledgment,
epoch recovery and compaction still require explicit policy and support proof.
The additive migration preserves existing rows and uses the migration owner's
savepoint. Re-entry validates the table columns, primary keys, replay key and
SQLite-assigned page identity required by source and receipt links. A conflicting
unknown table fails without partial schema or version changes.
Older binaries supporting only schema 30 must refuse the current database.
### Transactional source catalog and replay
Schema 30 adds `provider_catalog_records`, literal source references in
`provider_catalog_references`, unresolved indexing evidence in
`provider_catalog_debt`, and versioned per-page receipts in
`provider_catalog_pages`. Records refer to the original captured page and source
ordinal. All payload versions remain separate observations; replay order and
opaque provider cursors do not establish a current version. Original inbox bytes
retain fields that the catalog does not interpret.
Startup replays pending captured pages before planning libraries, one page at a
time with cancellation between transactions. Live capture indexes its committed
page before lossy pairing. The provider adapter revalidates account-bound scope,
original hash, page metadata and source identities, then derives facts on the
blocking pool. Read transactions provide a coherent source snapshot; the immediate
write transaction rechecks ownership and reloads the source before inserting all
records, reference facts, unresolved evidence and the final receipt together. A
failed projection retains the captured page and existing checkpoint for restart.
It cannot trigger an uncaptured rank fallback.
Unknown kinds, typeless tombstones, malformed references and incomplete asset or
container links retain unresolved evidence. A literal target name is not proof of
that target's identity or materialization. No tombstone authorizes local deletion.
Receipt version 1 means source facts were indexed, including unresolved evidence;
it does not mean selected, downloaded, rewritten, verified, historically complete
or acknowledged recovery debt. Existing queues also accept the bounded current
work admission in
[Current provider work admission](#current-provider-work-admission); source
indexing alone cannot authorize it.
Derived page facts have a 16 MiB logical charge limit and a separate 512 MiB
catalog index budget. These are operational backpressure limits, not physical
SQLite/WAL quotas or a retention promise. A full index holds the checkpoint and
retains pending source observations; exact completed-page replay validates facts
and remains possible at capacity. Startup has no fixed latency guarantee. Nothing
is automatically pruned or discarded. WAL `NORMAL`, filesystem qualifications,
existing checkpoint evidence, publication receipts and metadata obligations are
unchanged. Historical intermediate retention and unrecoverable-debt acknowledgment
still require explicit policy decisions before dependent stages.
### Current provider work admission
Schema 31 adds `provider_work_receipts`, generation-bound
`provider_work_obligations`, and the `provider_work_scan` scheduling position.
Before normal dispatch, up to 64 retained, indexed CPLAsset observations per
cycle can trigger scoped child lookup followed by a complete current child/master
lookup. Original bounded confirmation bytes and source page/ordinal provenance
are retained. Duplicate keys, duplicate or omitted records, provider errors,
wrong owners, wrong zones and changed child/master relationships cannot confirm
work. Provider confirmation is an observation, not an atomic provider snapshot.
This slice supports one private library-wide pass with an explicit default
owner, without recent caps, album exclusions, retry-only selection or metadata
backfill-only execution. Default `PrimarySync` selection lazily discovers the
private `/zones/list` response using the pinned authenticated session. Discovery
requires bounded original JSON with unique keys, an explicit zones array, unique
zone names, no zone errors and no unsupported continuation. Only a live exact
`PrimarySync` entry with a nonblank string owner replaces the constructor's
ownerless primary library after that exact selected scope returns a recognized
indexing state. `RUNNING` retains provisional readiness and requires durable
reconciliation; owner qualification alone never certifies a checkpoint.
The complete discovered scope is preserved, including explicit owners
other than `_defaultOwner`; lookup and inventory references must still match it.
The complete validated private zone descriptors are cached separately from
initialized libraries. Primary-only lookup initializes only the selected primary;
unrelated private indexing failures do not block it. Private and shared library
maps publish only after all requested library initialization succeeds. A failed
full-map initialization preserves an already qualified primary and retries
initialization using the validated private descriptors. Shared endpoint evidence cannot
replace private primary ownership. Missing, deleted, blank or malformed primary
owners preserve the ownerless constructor; no owner is inferred from a name.
The current-work admission slice still requires `_defaultOwner` and is unchanged.
Default selection needs only the cached private listing and selected primary's
indexing success. Normal sync planning and import still request all libraries;
their existing full-map initialization contract is unchanged.
The discovered primary flows through default library selection, `get_library`,
`all_libraries`, `PhotoLibrary::all`, and the existing sole Unfiled admission
checks. Private-discovered non-primary photo zones (`SharedSync-*`) retain their
existing explicit-owner route. Attaching capture before or after discovery
preserves the owned DB context. Retained primary source pages with explicit
returned ownership already use the qualified scope and can confirm current work;
ownerless source pages remain separate retained observations. No source scope is
rewritten or history acknowledged. Shared-endpoint zones, ownerless zones and
different owners remain outside this slice. Other selections retain observations
and use their existing pipeline without an admission receipt. Current
media/date/filename filters and RAW, companion and path planning use the existing
owners. The work configuration fingerprint binds their frozen evidence separately
from enumeration and checkpoint hashes. Hidden or deleted current records retain
deferred evidence without new media work.
An immediate transaction independently validates authenticated account ownership,
source bytes, indexed identity, complete scope and current confirmation. Frozen
resource checksums, sizes and metadata must match the retained confirmation.
New rows, asset/master mappings, grouping/retry markers, work obligations,
deferred conflict evidence and the final receipt commit together. Any conflicting
existing generation, metadata, mapping or legacy identity defers unchanged.
Identical existing queue rows retain their retry and publication metadata.
Admitted-receipt replay verifies frozen obligations and never reconstructs an old
generation over a queue that has since advanced. Receipt state describes work
admission, not download completion, metadata publication or verified media.
Future confirmations that differ from a retained receipt only in complete
eligible Asset `downloadURL` tokens reuse that receipt's original raw bytes,
hash, scope, source and master as the charged representative. The current
authenticated confirmation and planned records are still validated first, and
the retained URL is never used for current download work. This requires exact
masked-byte equality, the same work source/config/scope/master and historical
source hash, and an existing receipt whose own hash still matches its bytes.
Non-URL bytes, unknown or incomplete descriptor shapes, and genuine checksum,
tag, metadata or size changes remain distinct raw confirmations. Reuse adds no
receipt row and does not clear conflict/debt, resurrect admitted work or merge
sources or configurations. Retained receipt confirmations stay historical,
finite evidence tied to their admitted obligation; reuse reads the existing
bytes and never rewrites or duplicates that stored confirmation set. Equivalent
replay therefore grows no further, while genuinely novel confirmations are
still admitted subject to the existing budget.
Matching unfinished projected generations fence shared asset admission, identity
mapping and metadata refresh, including independent path publication receipts.
Content and metadata fences permit safe collision-leaf replanning by the existing
path owners; the original receipt still retains its admission-time filename.
Completed generations do not indefinitely pin the mutable canonical queue.
Schema 32 adds source/config retry deadlines in `provider_work_retries`.
Deferred admission commits its attempt and deadline with the receipt and scan
bookkeeping; admission clears that retry schedule in the same transaction.
The first delay is one hour, doubles on repeated deferral and stops at 24 hours.
A new source or current configuration has its own immediately eligible work.
Each bounded admission selects the oldest due unadmitted source independently
of the historical scan position. A due early source can be retried while new
observations continue to arrive; a future retry does not consume the 64-attempt
budget or block fresh tail work. The SQL query can scan a mostly admitted
source prefix; the attempt limit bounds provider/planner work, not SQLite VM
steps or retained-catalog scan latency. Retry schedules never acknowledge debt
or promote a cursor.
The watch precheck includes due retained work for the same eligible selected
passes and current work fingerprint as admission. Matching unfinished
pending/failed download obligations also wake normal dispatch after restart.
The existing targeted recovery pass owns their publication. Metadata-only
markers keep the existing option-aware metadata precheck owner. Local wake-up
adds selected zones without inventing a provider database successor; provider
and zone checkpoints retain their existing gates. No due local work permits the
normal no-change shortcut. Deferred sources remain retained and become due at
their deadlines; an idle cycle does not imply materialization or completeness.
Work plans have a 16 MiB logical charge limit and work evidence has a separate
512 MiB logical budget. Nothing is automatically pruned; backpressure retains
source observations, unfinished work and the current checkpoint. These are not
physical database/WAL quotas or fixed startup latency promises.
Current versions under current configuration are the approved materialization
scope. Historical observations and all unfinished debt remain retained. Admission
of a current version does not prove historical completeness, acknowledge old
unrecoverable debt, authorize local tombstone deletion or select a historical
retention policy. Existing CheckpointEvidence, sparse/legacy proofs, opt-in
metadata writes, publication guards and WAL NORMAL qualification remain in force.
### Selection generation shadow
Schema 33 adds account-bound `provider_selection_generations`, source links in
`provider_selection_sources`, per-pass `provider_selection_decisions`, and
rendition/path obligations in `provider_selection_destinations`. The existing
private library-wide retained-work path captures its planning result before
queue admission and verifies strict provider-owned equivalence on replay.
Every non-confirmation manifest field stays exact; an eligible paired current
confirmation may differ only in validated private `ASSETID` URL tokens. The
retained representative keeps its original bytes and hash, while the current
response and URLs remain the downstream planning input. A changed decision,
identity, version, metadata or resource fact stays a distinct generation.
Original current child/master confirmation bytes retain unknown fields and
numeric lexemes.
Current confirmation is separate from the historical trigger observation;
source order, page IDs and opaque cursors cannot establish a current version.
Manifest format 1 explicitly records `confirmed_sources_only` coverage. It does
not represent a complete library inventory, a global recent frontier, album
membership coverage or absence. Selected, excluded and deferred decisions remain
separate from admitted queue work and verified media. Destinations include the
pass, child, logical rendition and path so one rendition can have distinct
copies without changing the canonical asset identity. Native path encoding is
lossless, including non-UTF-8 roots; relative task destinations are anchored
without changing the task path. This stage captures only
the existing admission profile. Multi-pass selection and activation require the
next qualified stage.
Capture and replay independently check authenticated account ownership, complete
private source scope, indexed original source bytes and identity, current paired
confirmation, resource and metadata fingerprints, and all normalized manifest
rows. One immediate transaction commits a manifest and its source, decision and
destination rows. Failure retains source observations and the existing cursor;
no queue admission can substitute for missing shadow evidence. Exact replay is
idempotent at capacity. A manifest is bounded to 16 MiB of serialized evidence
and this store has a separate 512 MiB logical charge budget including normalized
row payload. These are backpressure limits, not physical SQLite/WAL quotas.
Nothing is pruned, retired or acknowledged. Existing work receipts, unfinished
generation fences, metadata queues, filesystem publication guards and checkpoint
proofs remain authoritative. WAL NORMAL and filesystem support are unchanged.
This shadow storage is the first PR of the selection phase, not completion of
replay-driven multi-pass materialization.
### Private selection and destination replay
Schema 34 adds active generations in `provider_active_generations`, original
validated rank pages and record indexes in `provider_selection_rank_pages` and
`provider_selection_rank_records`, bounded decisions in
`provider_active_decisions`, source links in `provider_active_sources`, and exact
physical obligations in `provider_active_destinations`. These tables supplement
the schema-33 shadow store and the existing source inbox and catalog.
`icloud/photos/album/selection_capture.rs` validates complete bounded query
responses before the existing pairing and filtering stream consumes them.
`download/orchestration/generation.rs` composes existing selection policy,
current confirmation, transactional queue projection and destination replay.
`state/db/provider_generations.rs` owns stored declaration, provenance,
admission, frozen metadata intent and completion validation.
Activation requires an authenticated account-owned database and one exact
private provider scope with an explicit default owner. Every selected pass must
be a supported private named album or Unfiled query using the same concrete
store and scope. Mixed, shared, smart-folder and unsupported profiles keep the
whole existing dispatcher. Retry-only, metadata backfill, read-only execution
and capture-local timestamp replacement keep their existing owners. Ordinary
explicit metadata refresh with preservation can use active destinations.
A generation binds account and provider ownership, full zone identity, selected
passes and queries, source dependencies and selection, rendition, path and
metadata configuration. Lossless native destination paths are independently
anchored. A rank page records its exact request and original response bytes,
including unknown fields and tombstones. Rank pagination remains an observation
window. EOF plus a delta bridge does not establish a snapshot or prove absence.
Current paired confirmation determines resource identity; observed order and
opaque rank or source tokens do not select versions. Ambiguous or incompatible
facts remain deferred. Existing global/per-filter recent, ties, exclusions,
RAW, companion and naming owners decide eligibility.
An immediate projection transaction commits the validated decision, source
links, per-pass/rendition/path obligations, existing queue records, mappings and
grouping changes together. Unfinished old versions and frozen grouping/writer
intent fence incompatible newer work. Independent healthy work can proceed.
Current query coverage is sealed only from that generation's own evidence;
replaying a newer inventory cannot seal an interrupted old window. Configuration
or source changes create a distinct generation or require existing validated
reseed. They cannot acknowledge omitted eligibility or old unresolved debt.
Destination media completion requires an actual confined file fingerprint and
independently verified selected source checksum. An exact active physical
receipt can be reused across compatible passes after current selection and
file verification; another path retains its own obligation. Opaque provider
signatures remain independent from source and local SHA-256. Fresh or legacy
fallback publication requires a separately verified per-path source receipt;
validated native reuse relies on its own lossless generation/destination receipt,
including operation-owned prepared metadata output. Imported generic
local checksums cannot establish source provenance. Metadata completion is
separate and uses the existing option-aware rewrite owner with the exact native
path, frozen grouping and writer flags. Disabled flags cannot finish older
writer debt or enable writes implicitly. Before an embedded replacement, the
active receipt stores the measured prepared output tied to the exact intent,
input and source. Reopen can recognize only that operation-owned output;
arbitrary damage cannot become publication proof. Filesystem publication and
receipt finalization retain their existing compare-and-swap guards.
Legacy TEXT paths remain compatibility projections. A non-UTF-8 receipt cannot
update a checksum or clear markers through a lossy text path. Both legacy
metadata queues retain ambiguous native-path markers without dispatching that
text path to a sibling file; exact native work uses its active receipt. These
markers are retained evidence, not an acknowledgment of another physical path.
XMPFiles retains its existing UTF-8 filename requirement. Native-capable HEIF
and sidecar owners use native paths without granting additional rewrite options.
Startup and watch can wake unfinished selected work even when the canonical
asset row already has one downloaded destination. An unchanged completed
coverage window reuses exact selection and publication evidence. Historical
scheduling visits one due old root per cycle, at most 64 decisions and 16 MiB of
serialized manifests, with a persisted continuation and retry deadlines. Within
that transaction, each unique original rank page is strictly decoded and its
complete ordered record index audited once. Compact page proofs are discarded
when the operation ends; new reads use a stable read transaction. The existing
512 MiB evidence quota bounds unique serialized input, without retaining all
parsed pages. Original-fact reconstruction separately limits retained serialized
page inputs to 64 MiB; unavailable bounded evidence remains deferred debt.
The current generation receives a bounded priority metadata lane; remaining writers
share the existing physical writer budget and persisted general continuation.
HTTP 401, 403 and 421 current-confirmation failures stop additional hydration in
the selected run and preserve the existing bounded reauthentication route.
Non-authentication proof failures retain their own deferred debt. Active cleanup
reconstructs original failed tasks' child/master and provider-rendition binding
from their validated declarations, then uses the existing exact URL-refresh
owner. A refreshed URL cannot replace frozen selection, metadata, path or
publication authorization. A transient per-invocation registry composes failed
historical and current runs; it is not durable identity or a separate memory quota.
The active store has a separate 512 MiB logical evidence charge budget and
16 MiB page/manifest limits. Capacity refusal rolls back the whole observation or
projection; no partial page can authorize progress. No records or debt are
pruned. Logical charges do not bound physical SQLite files or WAL usage. Seals
and retained source audits can still read substantial history; no fixed startup
latency or constant total decode-work guarantee is advertised.
Captured observations, admitted decisions, verified media and verified metadata
are separate durable facts. CLI download counts retain the existing task
semantics and do not claim unique-file counts or full-zone completion. Active
progress cannot replace `CheckpointEvidence`, legacy/sparse generation proofs,
filesystem guards or source checkpoint policy. Unfinished old selection debt
vetoes the source successor, including when later configuration uses the legacy
dispatcher. Local wake-up cannot invent a provider cursor. Tombstones do not
authorize local media deletion. WAL `NORMAL`, filesystem support and power-loss
limits remain unchanged. Capture-owned cursor promotion, historical intermediate
retention, unrecoverable-debt acknowledgment, expired-epoch replacement,
full-scan absence and compaction require separate decisions and qualification.
### Sync and provider checkpoints
```text
sync_loop::run_sync
-> resolve configuration, credentials, libraries, and pass plans
-> optional scoped changes/database pre-check
-> sync_cycle::run_cycle
-> download::download_photos_with_sync for each active library
-> sync_cycle source checkpoint decision
-> cycle reporting and watch control
```
Photo readiness is explicit provider evidence. `PhotoLibrary` accepts `FINISHED`
or `RUNNING`; missing and unknown states fail closed. Sync checks readiness before
watch shortcuts and again at the checkpoint boundary. Before downloading from
`RUNNING`, the cycle persists `indexing_reconciliation_required:<zone>` in the
existing metadata table. Presence is the obligation, including unknown future
payloads; no schema migration is needed. The previous trusted zone and database
checkpoints remain unchanged. Cross-zone album sources also contribute readiness.
While indexing, the download dispatcher uses full queries without incremental
replay, retained-work admission, metadata repair, retry retirement, or album
snapshot completion. Ordinary media still uses the existing verified publication
and durable finalization path. Managed primary layouts wait for `FINISHED`.
Watch cycles with a hold bypass the database no-change shortcut and rebuild
collection and album plans so omitted albums can reappear after indexing.
A retained hold forces full reconciliation after `FINISHED`. A scope refreshed
while readiness is `RUNNING` remains stale even if the cycle's later probe says
`FINISHED`; the next wake rebuilds it after readiness. Retry-only inventory can
download known failed work but cannot discharge recovery of previously omitted
assets. The held full-query path continues to suppress deletion-capable
maintenance, album snapshot completion, and the retained-token bridge even when
its initial probe reports `FINISHED`.
Complete natural EOF, successful local work, a fresh FINISHED scope, and a final
FINISHED probe qualify that inventory. One fenced transaction removes
`indexing_reconciliation_required:<zone>` and writes
`indexing_replay_required:<zone>` without moving the trusted source cursor or
database cursor. The later trusted cycle replays the old cursor and applies
source deletions. Replay presence, including an unknown future payload, bypasses
the watch database shortcut across reopen. Only normal checkpoint promotion
removes replay debt, including aggregate config-drift promotion. Rejection of
the retained cursor preserves the existing checkpoint gates.
Bounded coverage, interruption, retry-only selection, failed transfers or
metadata, stale scope, missing proof, state-write failure, and readiness
regression cannot qualify an inventory or advance its cursor. Provisional
inventory cannot replace a comparable inventory anchor.
Managed primary configuration, recorded bindings and pending operations retain
the RUNNING guard, including a later suffix configuration. Once FINISHED, the
held phase uses the same full-enumeration owner and token/EOF evidence, but
only consumes validated provider identities. It does not invoke local task
planning, journal recovery, publication, metadata drains, retry retirement or
snapshot pruning. A fresh FINISHED plan, unbounded selected scope, valid token
and final FINISHED observations from every source qualify that inventory.
The fenced transition additionally writes
`indexing_materialization_required:<zone>`. Presence, including unknown payloads,
forces a fresh full managed materialization through the existing journal and
publication owners before retained-cursor replay. It bypasses quiet watch
shortcuts and refreshes plans across restarts. An empty change feed therefore
cannot omit assets that inventory-only qualification never queued.
Managed materialization debt survives retry-only/bounded runs, failed transfers,
metadata failure, interruption, regression and checkpoint-write failure. When an
old cursor exists, both full publication and its incremental bridge must qualify
before one checkpoint transaction removes materialization and replay debt.
Existing local-byte, collision, sidecar, ownership and interrupted-journal guards
still control every managed write. Readiness probes are observations, not an
atomic snapshot guarantee from Apple.
Inventory-drop diagnostics use a complete count measurement from the initial
full-query invocation, captured before activity and delta-bridge composition.
The download owner records natural completion, usable counts, and canonical
resolved pass/query/filter coverage. The cycle compares only equivalent complete
scopes after the existing success, interruption, cancellation and stale-plan
vetoes. Configured bounds, missing counts, targeted subsets and retries cannot
replace a complete anchor. A complete selected smart-folder-only query can qualify;
source replay preserves that query's measurement without adding delta activity.
Mixed refresh subsets cannot supply a complete anchor.
Public `api_total_at_start` remains additive activity telemetry.
Diagnostic anchors use `inventory_anchor_v1:` metadata keys scoped by enumeration
hash, zone and coverage digest, with version and exact canonical coverage in the
payload. Legacy `inventory_anchor:` rows remain untouched and are never certified
retroactively. Malformed or unsupported evidence is ignored; the next eligible
inventory seeds a fresh baseline without an upgrade-time drop warning. Inventory
evidence never authorizes source checkpoints or proves retained deletion history.
The same accepted drop facts feed JSON reports and the durable cycle ledger.
CloudKit HTTP 401, 403, and 421 failures during enumeration or metadata
hydration return a session-expired outcome. The cycle preserves its provider
checkpoint and pending capture work. It does not treat these failures as a
reason to fall back to full enumeration or retry incomplete pass tokens.
Incremental producers pass session failures to the download pipeline before
stream completion. The pipeline records the interrupted run before status
commands read its outcome.
Before bounded mid-cycle reauthentication, the watch loop removes the cached
validation result and resets the HTTP connection pool. A recovered session
replays the retained checkpoint. Ambiguous provider identities remain pending;
authentication recovery does not authorize choosing a child or discarding work.
Incremental asset-identity lookups, paired-asset lookups, and selected-album
relation hydration in `src/download/orchestration/delta.rs` cancel provider reads
on shutdown. Started state writes finish outside these cancellable waits.
`support_hydration_wait_v1` reports a fixed phase and started, waiting, returned,
or cancelled outcome at INFO, with a waiting observation every 30 seconds.
Returned means the read returned, including errors; it does not certify replay,
identity, or a checkpoint. Interrupted hydration retains the existing checkpoint
and pending configuration. These waits have no new overall time limit and do
not establish a universal shutdown deadline.
Selected relation targets use bounded child lookups followed by complete-pair
validation before a full-zone fallback scan in each configured source.
Omitted, deleted, changed-pair or foreign-scope results remain unresolved;
lookup absence does not authorize deletion. Authentication and request failures
propagate to the checkpoint hold. Sources retain their existing priority.
Within one library sync, relation targets from album passes share a read only
when the request context, endpoint, full zone/owner and ordered fallback sources
match. Targets are combined before reading; no absence cache survives an error,
interruption or cycle boundary. A remaining unresolved target still blocks the
checkpoint through the existing relation gate. A fallback scan can still take
multiple pages and has no new overall deadline.
Legacy-master hydration scans all current visible and hidden children before
metadata-capture repair or pending-download recovery selects an owner. Hidden
children remain identity candidates; soft- and hard-deleted children do not.
Without a saved owner, multiple matching children remain unresolved, even
across visibility states.
This does not change download selection or permit checkpoint advancement
without the existing completion proof.
Hidden children can supply identity evidence without a valid `assetDate`.
Recovery counts all matching children before validating a selected child's
capture date. Missing, null, or out-of-range dates cannot authorize an owner
claim, metadata refresh, pending-file adoption, or download-policy exclusion.
Invalid selected dates retain repair or retry work and hold the checkpoint.
The display/path epoch fallback is not recovery evidence.
A new legacy-owner claim requires consistent, present per-rendition `added_at`
evidence matching the provider's `addedDate`, in addition to the existing
rendition checks. Missing provider dates do not use the epoch fallback as
identity evidence. Matching dates do not rank otherwise ambiguous children.
Whole-second legacy dates match within that second because older writers
discarded milliseconds. Fractional stored dates require an exact match.
Different stored rendition dates, including mixed whole-second and fractional
rows, remain unresolved at the atomic claim gate. Direct pending lookups also
require matching rendition evidence before they can persist a new owner.
Another historically mapped child blocks a new claim even when it is now
missing or has its own catalogue row. The state owner checks family history
and mixed rendition dates inside the claim transaction. Normal enumeration
and pending recovery use the same restrictions; read-only retry planning does
not persist a claim. Unresolved pending ownership holds the checkpoint.
Metadata-capture retry evidence includes each rendition's added date, at
millisecond precision. A changed date or an older fingerprint without dates
makes retained work eligible again. Conflicting or missing date evidence uses
the existing bounded retry queue. This does not delete or retire legacy rows,
replace saved owners, or restore dates overwritten by an earlier repair.
Recovery of those rows requires separately preserved historical evidence.
Ambiguous metadata-capture repairs retain a schema-27 retry row keyed by
library, catalogue asset ID, and target capture revision. An unchanged failure
retries after one hour, doubling up to 24 hours. Changed catalogue identity or
rendition evidence is eligible immediately. A durable generation rejects stale
attempts; its counter survives retry retirement. The state reader excludes
deferred candidates before applying the existing 500-identity batch limit.
Unresolved and deferred counts are separate from failed repair attempts.
Deferred work keeps the capture revision pending and its checkpoint blocked,
even when a cycle makes no repair attempt. The cycle checks each library
before committing inventory anchors or provider checkpoints, including
`--refresh-metadata` runs that bypass automatic capture repair. Idle health
and other-library success cannot turn retained ambiguity into a successful
backup. Completion retires retry rows only after the existing repair or
source-deletion rules remove the stale work. Retry scheduling does not select a child or authorize
metadata rewrites, deletion, or checkpoint advancement.
The download owners store these holds in `CheckpointEvidence`. Report counter
changes and result composition cannot clear them.
Unresolved asset-only delta hydration is incomplete work, even when no media
transfer fails. The producer records `unresolved_asset_identity:<zone> = 1` in
the existing metadata table before stream completion. The cycle owner clears
that zone's marker atomically with a proven incremental checkpoint, including
an inventory followed by a successful delta bridge. An inventory alone cannot
clear it. Another zone's success and process restart retain the marker.
Status aggregates all markers; cycle reporting does not advance health's last
success while any remain. Selected zones with markers bypass watch-mode
no-change shortcuts. Idle health also checks markers in unselected zones.
Intentional bounded checkpoint holds alone are not failures. The marker keys
remain in metadata; JSON report version 3 is unchanged.
The Photos adapter emits aggregate, fixed-label identity lookup diagnostics.
They distinguish omitted records, unexpected types, decode failures, invalid
master references, and record-level provider errors. Reference-zone context is
classified without logging provider identifiers or response bodies. These
observations do not authorize identity guesses or cross-zone retries.
`exact_lookup_rejection_v1` describes a rejected lookup request with fixed
scope/pairing/presence labels and an expected-owner category. It applies the
existing scope matcher; the older reference-zone JSON-equality classification
alone does not establish rejection. Missing or malformed master references and
record errors are distinguished from an explicit different-master reference.
No owner or record value is emitted.
A paired lookup can retain an inconclusive changed-master hint from a live,
scoped child, including when the old master explicitly returns `NOT_FOUND` or
`UNKNOWN_ITEM`. An old-master tombstone or any other record error does not
authorize a hint. Incremental asset hydration makes at most one additional paired
lookup in that same scope during a normal download run. Dry-run and filename-only
modes retain their prior held behavior. Only a complete current pair may update the mapping
through the existing state owner. A missing or deleted redirected master,
foreign scope, another relationship change, cancellation or failed write holds
the checkpoint. Other lookup consumers keep the hint unresolved. Sparse share
links do not authorize this same-scope retry. Selected sparse-target recovery
uses the separate strict qualification described below.
Unpaired asset deltas retain typed sparse-share evidence from
`isSparsePrivateRecord` and the `linkedShare*` fields. The marker accepts exact
`INT64` and `NUMBER_INT64` type spellings, or an absent type, with integer
value `1`; this does not broaden deletion-field validation. Targeted record
lookups request these fields too; ordinary media enumeration keeps its existing field
projection. A structurally valid link remains unverified. Malformed links and
valid links without a usable master remain unresolved. Link identifiers are
opaque and their debug output is redacted. Lookup changes do not overwrite the
original delta evidence. A link alone does not authorize a target lookup.
Selected sparse-target recovery is owned by
`src/download/sparse_target.rs`. The cycle supplies only current selected,
unrestricted full-library plans; it does not search an unselected library or
infer membership for album, rank, bounded, managed-primary or provisional
selections. A fresh source lookup must agree with its retained link. The Photos
adapter qualifies target owner/zone and the exact child/master pair before the
normal target download pipeline can materialize missing selected renditions.
Ordinary downloadable validation remains strict; no deletion exception is added.
A target receipt is useful only with exact current identity, metadata, source
provenance and independent protected-path-safe files. Trusted Legacy File
acceptance does not satisfy this source proof. The source checkpoint transaction
revalidates the source generation, target catalogue/configuration snapshot and
held target files. Partial downloads retain the source obligation. Existing
bounded source retry selection remains authoritative; no general retry budget,
checkpoint reset or debt-backed advancement is introduced. Recovery decisions
use fixed allowlisted labels in retained support history; a target-ready or
prepared historical file is partial work, not a complete backup.
Schema 26 adds `unresolved_sparse_identities`, keyed by library and source
record in the account-scoped database. `src/state/db/sparse_identity.rs` stores
the first observed link, latest delta link, separate lookup evidence, attempt
time, and retry deadline. The Photos adapter owns the versioned link encoding.
Observation and the existing unresolved marker commit in one transaction.
The `sparse_identity_generation` metadata counter assigns monotonically
increasing generations, including when a cleared source reappears.
`src/download/orchestration/delta/sparse_identity.rs` owns retry selection for both incremental
paths. A matching valid sparse lookup starts a one-hour delay. Repeated
matching results double the delay to a 24-hour maximum. Each library execution
selects at most 100 due, identified sparse source-only lookups, ordered by
retry deadline or last attempt, then generation and source key. Exact
source/master mappings bypass suppression and do not consume that budget. Malformed or changed
incoming evidence cannot inherit a cached negative result. Unclassified
sources still need an initial lookup before sparse retry policy can apply.
A deferred source remains unresolved and blocks the checkpoint. Retained
sources absent from replay re-enter normal hydration and media planning.
Authoritative source-only deletion lookups retain the completed delta token in
`last_outcome` as `["source_deleted_v1", token]`. This is provider evidence,
not a persisted claim that local processing completed. On restart or the next
batch, the same complete delta snapshot and unchanged source evidence can
reuse that result. The ordinary source-state transition runs again before a
current receipt is issued. A missing delta token, changed link, or authoritative
master mapping requires normal recovery.
For a different token, the Photos adapter scans the complete raw zone delta
since each saved deletion checkpoint. One scan validates a whole batch.
Any source change, including restoration with the same link, invalidates its
cached deletion. Sources absent from a complete scan retain their deletion
evidence at the current boundary with fresh generation fences. Missing tokens,
invalid pages, incomplete scans, and cancellation supply no reuse evidence.
Normal source lookups can still establish fresh results after a failed scan.
Validation does not advance the sync checkpoint or complete local processing.
This lets more than 100 deleted sources complete over bounded lookup batches
while unrelated zone records change, without relaxing checkpoint or state-write
guards. Existing schema-26 outcome labels remain readable.
Sparse observation and retry writes acquire an immediate SQLite transaction
before reading generation or stale-attempt evidence. Independent WAL writers
therefore serialize before those reads, preserving generation fences without
a deferred snapshot upgrade.
Hydration, explicitly soft-deleted source `CPLAsset` deltas, and exact-source
hard-deletion tombstones supply generation-fenced receipts, not permission to
advance a checkpoint. Incremental results carry these receipts in
`CheckpointEvidence`, not `SyncStats`. Inventory and delta-bridge composition
retain the receipts in zone-local evidence. The cycle owner still requires
normal processing and checkpoint proof. The checkpoint transaction validates every retained source receipt
before clearing its row and zone marker. Missing, changed, or stale receipts
roll back the transition. Inventory alone, interruption, failed state writes,
and another library's success cannot clear the obligation. Configuration
reconciliation uses the same fence when publishing its staged checkpoints.
Status reports unresolved and deferred counts without provider identifiers.
A linked record name, missing/deleted shared zone, or history of removing a
Shared Photo Library is not a source deletion or master identity. No automatic
cross-zone recovery or checkpoint relaxation is added.
A complete inventory followed by an explicitly rejected retained cursor creates
`retained_checkpoint_hold:<zone>`. The versioned receipt contains a cursor
fingerprint, requested enumeration/preservation policy hashes, an attempt count,
and the next attempt deadline; it contains no provider identity or cursor value. Unchanged
held zones defer provider, inventory and legacy preparation work for one hour.
The initial rejection and two automatic retries form a finite budget of three
attempts per boundary and policy. Exhausted holds perform no further provider
work. Due attempts reserve their count and next deadline durably before requests,
so interrupted or incomplete attempts consume a reservation. A failed reservation
performs no provider work. Version-one receipts without a count retain one spent
attempt. Malformed or unsupported receipts remain an actionable hold with no
provider work. Genuine policy or active cursor changes permit bounded re-evaluation. Original assets, owner/date facts, legacy
generations, transfer retries and sparse/deletion evidence remain intact.
A fresh inventory and its counts cannot establish lost historical deletion
facts or resolve this hold. Permanently rejected boundaries remain actionable
holds; this protocol does not rebootstrap them. A successful validated replay
of the retained boundary removes the hold in the same transaction that stages
or activates its successor. Transaction-time metadata comparisons fence both
hold staging and checkpoint promotion against changes to the planned cursor
and enumeration configuration, including the exact staged candidate read during
aggregate promotion. Existing legacy/sparse generation validation
still applies. Global configuration activation remains atomic across selected
zones. Zone-local hold aggregation keeps status and backup safety truthful even
when another zone succeeds.
Ordinary incremental producers carry validated changes/zone completion separately
from query enumeration. Smart-folder-only passes also replay the source zone;
that delta has no media destinations, and the selected folder query still owns
media admission. Its query token never substitutes for the source successor, and
a missing query token retains the selection-proof hold and same-cycle query
recovery policy. Mixed unfiled and smart-folder refreshes retain the actual
delta successor and every identity, partial-scan, cancellation and durable-write
veto. A successful smart query does not certify retained-boundary replay. Typed
provider cursor rejection is distinct from malformed or missing successor data.
The per-zone provider checkpoint and the scoped database pre-check token have
different gates:
- A zone checkpoint may advance after a transfer failure when the exact retry
work is durable and enumeration/token proof is complete.
- An exhausted album grouping write preserves the zone checkpoint. Replay
retries the relationship before it skips media that is already downloaded,
and the relationship insert is idempotent.
- The broader database pre-check token advances only after a clean aggregate
cycle for the exact account, selection, filter, config, and selected-zone
scope.
Eligibility-config drift preserves the active checkpoint while a complete
inventory and delta bridge build a replacement. Path-config drift preserves
provider checkpoints while local catalog paths are reconciled.
Schema 25 stores local reconciliation destination choices in
`reconciliation_paths`. The planner reserves current and historical catalog
paths and previous choices across all libraries, with library-qualified asset
and rendition ownership. Unselected renditions keep their paths reserved when
live resolution changes. Before it copies any media, it commits the new choices
in one SQLite transaction. A failed
reservation write prevents publication. Retries reuse the reserved leaf even
when provider lookup order or availability changes. The recorded source path
stays unchanged until file and metadata validation and state finalization pass.
Each choice is keyed by the provider checksum and byte size. A changed rendition
gets a separate destination; retries of the same content reuse its choice.
Migration retains schema-24 choices with unknown content as occupied paths. It
does not infer their content from a catalog row that may already describe a newer
version. Reconciliation can create a new sibling for these legacy choices.
Choices remain reserved after completion and config drift because old copies
remain on disk. This ledger does not authorize overwrite or deletion.
Full downloads, incremental downloads, and pending retries load this ledger
and the catalog paths before planning. They reuse their own reserved
destinations and cannot download into or adopt a path owned by another asset
or rendition. When the ledger is active, downloads commit new destinations
before publication. Pending retries commit the final choice after recorded-path
overrides. A failed reservation write prevents download dispatch. After a
failed finalization, retries can adopt a verified reserved sibling even when
the catalog still records the previous source. Dry runs and filename listings
do not commit choices. Recorded retry paths follow the same ownership checks. Explicit truncated-file repair can still use its own path
when the existing fingerprint and repair authorization pass.
Durable pending retries can record a destination in another directory after a
recorded-path override. Only pending retry planning replays that exact choice
when its library, asset, rendition, checksum, size and path ownership match and
the destination is beneath the current download root without parent traversal.
It preserves the root spelling and validates the saved destination key. Ordinary
downloads and local reconciliation retain the requested-directory replay check.
Retries stop planning another pass once every pending rendition for that asset
is matched or safely adopted; outstanding companions still require planning.
This does not migrate or discard reservations, alter selection completeness, or
authorize promotion of a pending path configuration with smart-folder selection.
Cross-parent pending adoption requires an independently durable local SHA-256
for the exact destination and current provider generation. It checks every saved
alias before allowing a pass to use ordinary adoption. MMCS is a generation
signature, not a local SHA-256; size and media headers cannot supply this proof.
The file owner retains no-follow directory capabilities and the regular-file
handle, revalidates identity and bytes before the immediate state write, and
retains those capabilities through that write. Failed adoption writes retain
debt and later attempts must reopen and reverify; no hash-only deferred adoption
write is queued. Both ordinary and selection enumeration inspect all matching
aliases before metadata actions, skip adoption, task adoption or touched-state
promotion. A refusal holds that asset's debt while other enumerated assets may
continue; it cannot enter the size-only fallback.
Fresh downloads to an affected reserved pending destination retain the original
publication inode and current-root no-follow namespace through immediate and
bounded deferred finalization. Each attempt revalidates both identity and the
publication fingerprint. Changes observed at revalidation retain the media and
pending debt instead of committing stale verified provenance. Retained ownership
and revalidation do not atomically exclude arbitrary Unix changes after the last
validation while a database write awaits execution. Cleanup re-enumeration and
collecting incremental dispatch preserve the affected task's proof requirement;
cleanup also requires the original failed generation's checksum and size.
Equivalent root spelling is rebased component-wise using platform ownership keys;
this does not relax filesystem no-follow traversal or mutate the saved ledger.
**Recovery limitation:** an existing cross-parent reserved file without a trusted
matching local hash remains untouched and pending. This includes a crash after
publication but before any durable receipt commits. These bytes cannot be
automatically adopted, replaced or discarded; this slice does not fetch fresh
provider bytes to establish an independent comparison.
Download planning also loads current-content publication receipts from
`asset_metadata_paths`. An additional album copy can satisfy a download only
in that pass's destination and filename family. The planner rejects foreign
catalog ownership, opens the file without following links, and checks its
local SHA-256. Old provider generations cannot satisfy current downloads.
These checks do not move the catalog's current path or change retry receipts.
Without reconciliation reservations, an unsafe historical path disables optional
receipt reuse for that pass instead of blocking ordinary downloads. Reserved
paths retain strict validation. Live Photo planning applies filters before file
checks and does not hash the still when same-size current companions satisfy
the existing skip rules. New companions use a verified still filename. Existing
companions with older numbered still stems remain usable after hash verification;
Kei does not rename or delete them.
Historical smart-folder membership cannot complete local path reconciliation. It reports separate conditional evidence when the fresh smart query
is its only remaining dependency: album snapshots and catalog identity must be
complete, with no deferred missing file or failed local work. The cycle owner
can discharge that dependency only after successful current smart enumeration
and durable work. During pending path drift, recent-count and lower-date-bounded
selections drain ordered provider queries to EOF while forwarding only their
configured selection to the existing download planner. The global recent
frontier also drains to EOF and must report the same usable token as every
selected pass. This costs a complete query on path drift, with bounded retained
frontier IDs and streaming tail filtering. Cancellation stops the raw drain.
Fresh query evidence certifies selected local work separately from source
checkpoint coverage. Any excluded tail, including equal-date global-frontier
IDs, still holds zone and database checkpoints. Stale plans, interruption,
unresolved identity, failed enumeration, local work or state transactions retain
the pending path hash. Every selected library must finish. Atomic promotion
compares the active path hash and exact pending target before publishing the new
hash and removing its marker. Eligibility inventory-and-bridge promotion remains
separate. Unchanged cycles skip catalog reconciliation and the exhaustive path
proof; ordinary bounded smart-folder refreshes may still query the provider.
Local path reconciliation retains an unowned legacy master with multiple
historical children as unresolved work, even when targeted lookup returns one
usable sibling. It does not reserve that master's path or relax preservation
eligibility. Independently owned current children can still reconcile, and later
complete inventory can prepare preservation and release the checkpoint hold.
A historical child media receipt at a protected original is not a current claim
when the exact mapped child and rendition have a separate ready, source-backed
current file. A missing current file remains unsupported by this historical
exception; historical rows do not authorize initial materialization. The state owner snapshots the family dependencies and rederives
that exception inside preservation preparation. The file owner retains confined
handles, verifies their bytes and namespace immediately before commit, and
refuses cancellation or drift. Historical receipts, originals and owner facts
remain unchanged. Explicit local-byte acceptance does not satisfy this exception.
Ordinary adoption and skip decisions cannot reuse a protected original merely
because a child has a historical receipt there.
Local path reconciliation also loads these publication receipts. A verified
copy in the selected pass satisfies that rendition even when the catalogue's
current path belongs to another album. Reconciliation preserves a verified
numbered still filename when planning its companion, so changing Live Photo
selection or album order does not create another copy of an owned pair.
Local path reconciliation opens source and destination leaf entries without
following symlinks. It hashes the opened file and rechecks its identity before
accepting a destination, including entries that appear during publication.
Temporary copies use new unique names, verified bytes, and no-overwrite
publication. On Unix, they start with owner-only permissions, so a failed copy
remains private. A completed copy receives the source permissions.
An unsafe or replaced entry preserves the previous catalogue path
and leaves reconciliation incomplete. A blocking reconciliation failure skips
that library's normal source/download pass, so adoption cannot bypass the
rejection. Its provider checkpoint and the aggregate database pre-check token
remain unchanged. Ambiguous temporary entries remain for
inspection.
Reconciliation retains directory capabilities through state finalization.
Configured roots, recorded sources, destinations, and temporary siblings must
not contain `..` components. Reconciliation rejects these paths before lexical
normalization or filesystem writes, because a linked preceding component can
change which directory `..` selects. Ordinary relative roots remain supported.
Descendant directories are opened without following symlinks. Source reads,
temporary creation, validation, and publication use these retained handles;
namespace replacement leaves reconciliation incomplete. Windows directory
handles deny rename and delete sharing while the receipt is live. Unix checks
that the retained parent still occupies the planned namespace before accepting
the result. Publication and temporary-file checks remain descriptor-relative.
When the configured root changes, the recorded old source is opened beneath
the common lexical ancestor of the old path and new root, with no-follow
traversal below that anchor. This allows root moves without trusting linked
source directories. Destination writes stay beneath the new configured root.
Relative paths are normalized inside the filesystem owner, not in durable
catalogue keys.
A reconciled media copy receives the same capture mtime as a normal download.
If the retained source and destination paths resolve to the same pathname,
reconciliation only updates the catalogue path spelling. It preserves media
mtime and existing or absent sidecars, then validates media before finalization.
This permits changes between relative and absolute download roots.
Before changing timestamps, reconciliation rejects a distinct destination that
shares the source file's identity, including a hard link. The source mtime and
catalogue path stay unchanged until the conflicting entry is resolved.
When XMP sidecars are enabled, an existing source packet is validated and
copied byte-for-byte, including custom properties and ownership markers. Path
migration does not upgrade or regenerate an existing packet. If no source
sidecar exists, the normal metadata planner generates one from the current
payload and source GPS facts. A source read failure stops before publication;
it cannot leave an incomplete packet that blocks the next attempt. Generated
packets do not infer native accuracy provenance from a local checksum.
Reconciliation reserves recorded and planned paths by asset and rendition. Keys
use the filesystem owner's checked absolute paths, so equivalent relative and
absolute roots share ownership. These keys do not change task or catalogue path
spellings. Invalid reservation paths leave reconciliation incomplete and block
file finalization. A collision with another asset or rendition selects a stable
identity-suffixed path. Each rendition can reuse its own reservation across
passes and retries, including after a
partial state-write failure. Existing files alone do not select another name;
the confined copy owner checks their bytes and rejects conflicts.
Sidecar publication uses the same retained directory capabilities and refuses
conflicting destination bytes. Reconciliation records the new catalogue path
only after capture mtime, sidecar work, and final input checks succeed. It
retains the old files and keeps failed work pending. Reconciliation planning
includes existing media destinations so a retry after a metadata or state-write
failure can finish without downloading or creating a second media copy.
Reconciliation retries the selected destination and leaves conflicting media
or sidecars untouched; it does not allocate another collision filename on each
attempt. Ordinary download collision naming and on-disk skip rules are unchanged.
Ordinary download planning also treats a state-protected legacy original or exact
sidecar as occupied when the directory cache cannot see that spelling, and selects
an independent identity suffix instead of targeting the protected identity.
Legacy preservation inventory failures retain the existing reason and budgets.
`legacy_inventory_failure_v2` adds fixed phase/subreason labels, whether a valid
terminal marker was observed, and the failing child reference's known family
context and soft-deletion flag. A missing master reference leaves family
membership unknown. Context describes the failed validation, not counts for
all records. EOF observation is not successful hydration or checkpoint proof.
These fields exclude identifiers, tokens, URLs, paths and provider error text.
### Full and incremental enumeration
Debug builds can set `KEI_REQUEST_DUMP_DIR` to capture Photos session
bodies as `000001.req.<epoch-ms>.json` and `000001.res.<epoch-ms>.json`.
The counter pairs requests and responses; timestamps are captured separately.
Bodies are unredacted and private; request URLs and headers are not captured.
Use private local storage and redact bodies manually before sharing. This dump
is not a redacted doctor bundle.
HTTP error bodies retain the existing size bound and may contain non-JSON text.
Successful responses are serialized from parsed JSON without pretty-printing.
Writes are best-effort, with no run subfolder or cleanup.
Release builds without debug assertions do not include this diagnostic code.
A signed URL expiry stops the current full-download batch. The pipeline retains
failed tasks, including queued tasks cancelled by that expiry, and makes one
cleanup attempt through targeted asset and master lookups. User shutdown stops
cleanup lookup and download dispatch. It does not enumerate albums again to
refresh those URLs or rebuild unfiled exclusions. Cleanup updates only the URL
on an already-selected task, preserving its destination and publication
authorization. The library, child identity, rendition, checksum, and size must
still match. Changed resources, missing lookups, a second expiry, and cancelled
tasks retain durable retry work. Successful
cleanup does not convert incomplete enumeration into checkpoint proof.
`exact_refresh_outcomes_v1` separates requested/refreshed/remaining task keys
from planned unique child identities, observed child results, planned paired
identities and observed paired results. These are identity counts, not HTTP
request counts. Planned counts may include work cancelled before a result.
Fixed child/task rejection categories distinguish selected-master, rendition,
checksum and size mismatches. Task rejection counts deduplicate durable task
keys; categories are observations, not permission to change selection or retry.
Authentication failures during refresh stop cleanup and return a session-expired
outcome. Refresh rate-limit observations contribute to the cycle count even
when provider retries are exhausted. The `expired_url_refresh_failed` diagnostic
reports failure, authentication, and rate-limit counts without provider details.
Collecting incremental preflight and expiry recovery use the same bounded
child/master lookup owner. The authenticated session and exact requested zone
scope each lookup. Explicit zone/owner evidence and child/master references
must agree; incremental recovery also pins the originally selected master and
provider rendition. The selection owner translates virtual RAW/JPEG task keys
back to provider keys once, before refresh; later provider policy changes cannot
retarget the selected resource.
Refresh changes only the URL when child, master, rendition, provider checksum
and size still agree. Unmatched resources retain durable failure work; expiry
and lookup absence never authorize deletion. No zone enumeration, path
replanning, or cross-instance mapping import is used for URL refresh.
Explicit-task passes isolate CDN expiry to the failed resource so healthy
peers finish, including the single cleanup retry. Authentication thresholds,
user shutdown and fatal state-write failures still stop dispatch. Interrupted
and queued unfinished tasks remain in the returned failure debt and are
persisted before checkpoint decisions. Only successful exact tasks retire retry
debt. A collecting pass makes at most one expiry refresh and retry after its
optional aged-URL preflight. Failed resources stay durable for the next cycle.
Refresh logs record phase start/completion, counts and elapsed time. Refreshed
resources carry their lookup-observation time into worker dispatch for a DEBUG
age measurement. The first enumeration observation is labelled separately;
it does not measure a refreshed URL's age. Neither observation establishes
provider issuance time or expiry, which remain unknown.
Full enumeration streams records/query results and gathers a provider token
from every active pass. Natural stream completion and usable, unanimous pass
tokens are the authoritative proof. Count probes and pagination differences
are diagnostics. Recoverable pass-token gaps can be retried in the same cycle.
Download orchestration normalizes each child to its `CPLAsset.recordName`
before streaming or collecting paths plan it, so page boundaries and sibling
order cannot change its state ID. It retains a legacy master-keyed state ID
only when the provider checksum matches and durable identity history permits
that child to adopt it. Before a download pass adopts a legacy master-keyed
row, it atomically records that child as the durable owner. Later sibling
mappings cannot change the owner. Print-only and dry-run paths select from
existing state without claiming an owner. Cleanup retries carry the child
record name and reapply the state ID selected by the first pass before they
rebuild download tasks.
The iCloud adapter validates a complete changes/zone page before pairing records,
invoking hydration callbacks, or accepting its successor. A page must contain one
zone matching the requested name and any requested owner, an explicit records
array without record errors, and a nonblank token. Continuing pages must use a
new cursor. An empty terminal page may return the request cursor. Rejection
emits no records from that page; earlier accepted pages remain replayable from
the durable checkpoint. These checks also protect raw hydration and source
change validation. Source change validation retains its stricter record identity
and cancellation requirements. Diagnostics omit provider cursors and identifiers.
Incremental enumeration consumes changes/zone events. It persists provider
identity mappings before applying created, soft-deleted, hard-deleted, or
hidden transitions. An asset-only `CPLAsset` creation hydrates its paired
master through `masterRef`, the durable mapping, or a targeted identity lookup
before routing; an inconclusive lookup preserves the prior zone checkpoint.
Album snapshots and smart folders may require targeted refresh work before or
alongside the incremental stream.
Both strategies use `IncrementalDeltaState` in `orchestration/delta.rs` for
identity mappings, source-state transitions, album and relation bookkeeping,
event counts, and completion tokens. The streaming strategy emits created
assets through a bounded channel and defers album, relation, and unpaired-asset
work. The collecting strategy observes all events before state writes, hydrates
missing identities, and applies relations before routing created assets.
These orders preserve each strategy's token-safety and selected-album behavior.
Shutdown races provider-event and successor waits in both strategies; the owned
receiver is dropped before the successor wait to release provider backpressure.
Receiver closure cancels pending changes/zone requests and retry delays, but does
not interrupt an already-started durable shadow capture. Cancellation during
either consumer wait supplies no replay-completion proof.
Downloadable photo records require non-blank `CPLMaster` and `CPLAsset` record names
and a usable `assetDate` before they enter filtering or path planning. Full
enumeration reports a malformed record as incomplete. Incremental enumeration
marks its zone token unsafe. Both routes preserve the prior checkpoint so an
unchanged provider record remains retryable, and neither route counts the
record as a policy, filename, or date skip.
Recent and date-bounded runs may advance only when the producer proves the
bound did not truncate the stream. Incremental count-form recent selection
uses the full selection owner after complete strict delta replay. Global uses
the library-wide ranked window before filters; per-filter uses each pass's
window. Delta arrival order and asset/pass-pair counts never define that window.
`orchestration/recent.rs` stores a fixed-size `recent_selection_recovery:<zone>`
receipt in the account-owned database. It binds the resolved provider/pass
scope, exclusions, selection and download configuration, and the source cursor
hash. Missing, malformed, mismatched or pending receipts require current
inventory even when the next delta has no changes. The pending receipt is
written before selection. A completed receipt is usable only after its actual
changes/zone successor becomes the requested cursor, so a failed checkpoint
commit cannot skip recovery. Cap increases and removal consume this obligation
without a manual token reset. Watch prechecks include this local recovery even
when the provider reports no changed zones. Enumeration safety hash version 3
includes count and recent scope, forcing one conservative inventory/bridge for
legacy cursors, including prior cap removal, before trusting quiet streams.
Selection rank tokens prove only the existing selection EOF contract. The
combined result returns the delta successor and carries separate completed
replay evidence for configuration bridges; a pure full fallback has no such
evidence. Existing identity, state-write, interruption, sparse/legacy and
bounded-inventory vetoes remain authoritative. Unresolved delta identity does
not starve independently valid recent media selected by current inventory;
its source veto, failed-write counters and pending recovery receipt remain in
force. Current queue/publication guards remain authoritative for each job. A
truncated count window keeps
its recovery obligation and prior cursor; repeated bounded inventory remains a
known cost until a larger catalog selection generation stage qualifies it.
Typed provider session errors retain the existing bounded reauthentication
route across current selection; other selection failures do not trigger fallback.
This receipt neither claims an atomic rank snapshot nor proves absence or
permits local deletion.
### Download and publication
```text
PhotoAsset
-> planner::TaskPlanner
-> pipeline::run_download_pass
-> file::download_file
-> optional metadata_rewrite
-> verified .part publication
-> finalize downloaded or failed state
```
Only producer-dispatched work becomes pending through `upsert_seen`. Filtered
or skipped assets must not be left as retryable work unless a dedicated state
transition owns that result.
Full enumeration divides the configured download-worker limit across concurrent
album passes. Each pass gets the same allocation, rounded down. The remainder
stays unused so replacement passes cannot exceed the total limit. Passes keep
their existing cancellation, publication, and state-finalization paths.
Asset dates used for filtering, path rendering, file metadata, and sidecars are
resolved from Apple's UTC instant plus the asset's `timeZoneOffset` when that
offset is usable. Missing or invalid offsets retain host-local rendering,
preserving the previous path and metadata behaviour. Date-only lower bounds
use a conservative UTC enumeration bound and then apply the exact resolved
calendar-date filter, so enumeration safety does not depend on the backup host
timezone.
Capture-local path rendering does not change the download config hash, so a
sync that sees no other drift stays incremental and never re-enumerates an
already-downloaded asset. Date-only created bounds change the eligibility hash
but not the path hash, so expanding a date window runs the required inventory
without treating existing media as path drift. Matching legacy hashes that
mixed date eligibility into path state migrate without reconciliation. When a
legacy hash is ambiguous, reconciliation uses the same durable path-family and
file-integrity proof as normal sync to discard the planned collision task
before copying or changing state, so an asset cannot collide with its own
recorded file.
For an asset carrying a usable offset, a path derived under host-local
rendering is not a current derived path, so a full sweep forwards that asset
and downloads it into its capture-local folder. The earlier copy stays where
it is: kei never deletes local media, and no command removes a file that no
longer matches a derived path. `import-existing` adopts through the same
derivation. Assets without a usable offset retain host-local paths and remain
compatible with icloudpd's date-folder layout.
Existing embedded and sidecar timestamps are repaired only through the
explicit `sync --refresh-metadata` flow, which is bounded by the same
no-overwrite probe gate as any other embedded write. An offset tag names the
zone of one specific timestamp, so the embed path attaches it only to a
timestamp it writes in the same pass, or to an existing one the probe proves
already renders the capture-local instant. When writing a timestamp into a
file that has orphaned datetime offsets, the writer clears those offsets before
installing the timestamp and its resolved offset. Attaching an offset to a
wall clock left by host-local rendering would assert an instant the asset never
had. `sync --refresh-metadata --repair-capture-timestamps` explicitly relaxes
the no-overwrite gate for state-recorded downloaded files. It requires embedded
datetime output, a usable Apple offset, and bytes that still match the recorded
checksum. Rows without that provenance, and files whose embedded format is not
supported by the active build, remain pending. The writer replaces the timestamp
and its offsets together through the stable-input publication path. For HEIF
media, an existing native Exif timestamp is patched in place with its paired
offset before XMP is written; ambiguous, shared, or unsupported native layouts
remain pending. Normal downloads and ordinary metadata refreshes still preserve
an existing timestamp. Capture repair is separate durable debt, so an ordinary
metadata drain cannot retire it. If ordinary embedded metadata would change the
same media while capture debt is pending, that embed waits for the explicit
repair; verified no-write and sidecar-only work may complete independently.
### Durable pending retry
Failed and pending rows are not recovered by replaying the entire provider
inventory. The retry owner:
1. Removes only work already proven source-deleted.
2. Resolves current provider records in targeted batches.
3. Uses durable asset/master mappings and checksum/size evidence for legacy
identities. A persisted legacy owner resolves matching siblings without
changing the selected child.
4. Adopts an existing matching local file when safe.
5. Marks current filter exclusions as policy-excluded.
6. Persists unknown or transient verification state.
7. Queues exact unresolved asset/version/path tasks.
Unknown identity is not permission to delete or forget work.
A pending rendition can retain a valid prior publication after a later download
attempt failed. Ordinary sync and targeted retry recovery may reuse only its
current catalog path when prior
publication time, unchanged provider checksum and size, a matching path receipt,
equal recorded local/download hashes, the current filename family and positive
catalog/reservation ownership all agree. Recovery runs before fresh plan choices
are saved; recovered renditions and failed recovery writes stay out of transfer
and reservation dispatch. A healthy rendition about to receive a fresh choice
uses the same current-file proof so that a later failure can still recover it.
Existing durable choices keep their replay policy; saved different destinations
still retain debt.
The file is hashed beneath the current root without following links. A retained
file and namespace, plus the process's
destination guard, remain live through immediate conditional state finalization.
The state owner rechecks the row, receipt, identity, protected legacy evidence,
reservations and provider generations in one transaction. Recovery preserves the
original path, publication timestamp, all receipts, source provenance and metadata
debt; it does not mint provider progress from local bytes. Unknown identity or a
saved different destination remains pending. Cancellation before commit or a
failed state write cannot dispatch a fallback copy from that recovery attempt.
The destination guard coordinates participating writers in this process; it is
not an interprocess or network-filesystem lease. Managed selection keeps its
separate destination and source-provenance finalization.
Policy-excluded rows stay outside the actionable pending reader. After a
successful source pass, targeted revalidation checks their durable provider
identities only for explicit deletion. Present, omitted, malformed, and
transient responses retain the policy-excluded rows.
A provider checksum change makes the previous local path historical. Retry
adoption requires durable proof that the path holds the current provider
version and that its filename matches the recorded task filename.
### Import-existing
`src/commands/import.rs` shares configuration, selection, pass planning, and
path derivation with normal sync. Before matching or consulting prior imports,
it uses the download context's identity selector to install the same child or
guarded legacy master state ID as sync. It prepares selected rendition rows
and paths for every selected pass before adoption, then checks the complete
candidate census. Incomplete enumeration, fetcher panic, or cancellation before
completion leaves all adoption rows, mappings, and legacy owners unwritten.
The preflight uses memory proportional to selected renditions, all durable
catalogue paths and current receipts, and cached directory entries. Metadata
and receipt slices are shared; raw provider JSON is not retained. There is no
fixed memory cap or disk spool. Catalogue paths include historical and omitted
assets, so an unselected owner still prevents adoption of its file. Equivalent
relative and absolute paths use the same confined ownership key.
Import uses sync's collision-family owner for full-ID and ordinal filenames.
Different-size candidates remain distinguishable. An ambiguous short name,
sanitized full-ID collision, or multiple ordinal siblings is refused unless an
exact current-generation durable receipt and verified local SHA-256 identify
the file. Provider MMCS checksums are compared to provider checksums, never to
local SHA-256. It optionally compares remote prefix
bytes, hashes the local candidate, and calls `ImportStateStore::import_adopt`.
Adoption commits the provider child/master mapping and any compatible legacy
owner in the same transaction as the imported row. Failed adoption rolls back
both. Dry-run selects identity without persisting mappings or ownership.
An unclaimed legacy row completes adoption before it can skip later rehashing.
Adoption checks durable destination reservations in the same transaction as
catalog updates. A different asset, rendition, or provider content generation
cannot adopt a reserved path. Unknown legacy content remains occupied.
A refused adoption logs a warning and does not count as a match.
Size/mtime snapshots may skip later rehashing only while the current provider
checksum, path, size, mtime, and stored local checksum still match. Ambiguous
paths always require a local hash verification before that shortcut.
## Data-safety invariants
### File landing
The media publication sequence is:
```text
write or resume .part
-> validate response and content length
-> validate expected size
-> validate SHA-256 checksum
-> validate content type and sniffed bytes
-> apply configured pre-publish metadata
-> publish without replacing an existing final path
-> fsync the parent directory
-> finalize SQLite state
```
The transfer owner opens an existing resume file through a retained parent
handle with no-follow access and requires a regular file. It reads the Range
offset from that file handle and uses the same handle for append writes.
Resume probes reject symlinks. Identity checks reject a changed leaf before
body writes, content validation, and publication. A final identity or byte check
must pass before downloaded state can finalize. The task retains the file
through metadata work and publication; an opt-in metadata replacement must
match the writer's output fingerprint before it is accepted.
The durable temporary-path claim alone does not authorize an append.
Staging names use fixed-length SHA-256 digests of the lossless absolute
destination and provider checksum generation, encoded as
`kei-v1-<destination>-<generation><temp_suffix>`. MMCS checksums select a
provider generation; they do not verify downloaded bytes. Different
destinations have independent staging even when their bytes match.
`file::staging` acquires an in-memory destination guard before claiming or
inspecting a part. The task retains it through retries, resume, metadata,
publication and claim retirement. Different generations and suffixes for one
destination share that guard. Cancelled waiters cannot retire a peer's claim.
The weak registry upgrades and prunes entries under one short mutex; active
owned guards and waiting futures keep the shared destination mutex alive.
Completed and cancelled guards retire dead entries. No permanent lock files
are created. This coordination covers workers in one kei process, including
separate databases used inside that process. It does not coordinate independent
processes or hosts; existing cross-process limits remain, including network
filesystems whose advisory locking is disabled or unsupported.
Cleanup of versioned staging names takes the same destination guard through
claim retirement. A busy guard preserves both bytes and the claim. Legacy
checksum-only files are never adopted into a new destination's staging
identity. Existing exact-path ownership claims can still authorize their
stale orphan cleanup under the same age and confinement rules. Unclaimed
legacy files remain untouched; this change performs no migration or discard.
The streaming consumer cancels and drains after its authentication threshold,
so detached filesystem or metadata operations complete before guard release.
A resumed request rejected with HTTP 416 gets at most one fresh request per
attempt. The retained part stays intact until an acceptable response can
restart the transfer; a second 416 terminates that attempt. Transfer retry
pauses and bandwidth waits observe shutdown before another request or chunk
write. A cancelled bandwidth reservation is refunded, and interruption reports
the retained byte count without publishing or finalizing the file.
Before returning a body-stream error or interruption, the transfer owner flushes
pending filesystem writes. Retry offset reads and temporary-claim retirement
therefore follow write completion. A flush failure is a disk error rather than
a resumable transport error. Successful transfers also sync the data before
validation and publication.
Schema v19 records the exact temporary path before a state-backed download can
write or resume it. Normal completion and graceful interruption retire that
claim. A process crash leaves the claim as durable cleanup authority. Later
cleanup inspects only claimed paths, rejects every symlink in the path, and
holds verified directory or file handles through removal so an ancestor swap
cannot redirect deletion. It retires the claim after deletion or when the path
is missing or no longer the claimed stale file. A suffix and file age alone
never authorize deletion.
`kei sync --repair-truncated` is the only media-replacement path. Pending
retry planning requires the durable reconcile truncation marker, confirms the
recorded file is still truncated, and fingerprints its bytes. After the new
download and configured metadata writes pass, the file owner replaces only
that exact fingerprint. It uses atomic exchange where supported and the
journaled fallback owned by `src/download/file/replacement_recovery.rs`
otherwise. A changed
target is restored or retained and the state row stays failed. All other
downloads keep no-overwrite publication.
When no-overwrite publication finds an existing destination, the file owner
compares it with the verified `.part` file. Identical bytes are deduplicated.
Different or unverifiable bytes return a typed collision and retain the
`.part` file. The pipeline records the task as failed before it can write a
sidecar or finalize downloaded state.
A file may be safe on disk while its state write is still a cycle failure. Do
not weaken deferred state-write handling or infer that a visible final file
means the database transition succeeded.
### Checkpoint advancement
`SyncResult.checkpoint` carries the per-zone completion, interruption, state
durability, identity, and token-block evidence. The source-checkpoint classifier
reads this evidence and the result's source token and outcome. It does not read
the reporting snapshot in `SyncResult.stats`.
Full enumeration obtains evidence directly from pipeline finalization. Attached
pending recovery and metadata-capture repair also supply execution evidence.
Same-zone composition combines this evidence once, then projects its counters
into the report. Recovery pass and record lists stay in the per-zone evidence;
cycle reports do not accumulate them. Report fields and reason values keep
their existing JSON representation.
Incremental result assemblers still use `SyncResult::from_execution` to capture
their execution statistics at the result boundary. This is the boundary of the
pilot, not a second authority after result assembly. Later composition must use
the captured evidence, never reconstruct it from a reporting snapshot. Query
token holds, inventory/delta bridges, and unresolved identities update the
evidence in their existing policy owners.
Preserve the zone checkpoint on:
- Dry-run
- Stale pass planning
- Session expiry or interruption
- Incomplete enumeration
- Non-durable state
- Missing, blank, mismatched, or otherwise blocked token proof
Transfer and metadata failures may use a newer checkpoint only when the
recovery work needed after that checkpoint is durable.
### Metadata
Provider metadata may be captured in SQLite without changing local media.
Embedding EXIF/XMP or writing sidecars requires explicit configuration.
The iCloud GPS decoder selects the first valid coordinate pair from
`locationV2Enc`, `locationEnc`, then plain latitude/longitude fields. Coordinates
must be finite and within inclusive latitude [-90, 90] and longitude [-180, 180]
bounds. Nonfinite optional altitude is omitted independently of the pair.
The iCloud adapter retains shared metadata once and compact resource facts for
each rendition, independently of download URL availability. Catalogue dimensions
describe the provider-declared resource dimensions, not probed track or displayed
dimensions. Live Photo motion
duration comes from the companion duration ratio; missing or invalid companion
facts remain unknown rather than inheriting still-image values. Planning,
adoption, and import project metadata for the selected resource. RAW preference
can swap logical original and alternative keys; download policy supplies both
resource candidates for those historical keys. Refresh resolves every rendition
against its current provider checksum inside the state transaction, not against
preloaded row identities or current RAW preference. Missing or conflicting matches
update shared library metadata but leave dimensions and duration unknown. Full
metadata snapshots and hashes are computed only when needed, rather than retained
for every possible rendition.
Refresh updates the downloaded rendition family atomically, checking
each prepared capture-repair receipt against its own rendition's incoming hash.
An individual upsert guards the affected rendition and provider checksum without
invalidating another rendition's prepared receipt.
Failed receipt validation or state writes preserve checkpoint failure evidence.
HEIF probes and embedded preparation share one process-wide blocking admission
lane, independent of download concurrency. This conservative admission bounds
simultaneous retained parser work; it is not needed for shared parser state and
also serializes file I/O. Concurrency may be widened only with measured resource
evidence.
Production keeps a compact view of the `pitm`, `iinf`, `iloc`, and `iref` controls.
Other `meta` children, including image-bearing `idat`, stay as file ranges.
The separate file-backed source-GPS locator also budgets its required control
tables rather than the complete `meta` body.
Headers are scanned without a growing top-level or opaque-box inventory.
Preparation patches changed controls and streams all unchanged ranges with
64 KiB buffers; preservation checks compare original file ranges with fixed
buffers. A large `meta` or `idat` payload alone is not a resource refusal.
The retained-control view, selected XMP packet, complete native Exif item, and
replacement XMP packet each have an 8 MiB materialization allowance. This reuses
the prior writer's 8 MiB whole-`meta` allowance for the data still materialized;
the selected Exif/XMP allowances are new operation restrictions, not format limits
or a claim about all valid assets. Complete native TIFF parsing/alias validation
and the XMP toolkit graph still require materialization. Larger operations return
an explicit resource-unsupported error naming the required bytes and allowance,
preserve the original, and retain truthful metadata retry work. The allowance is
not a total-process RSS bound: C++ graphs, parsed structures, source/output control
views, and fixed stack buffers also consume memory. Existing fixture sizes do not
establish a representative camera/iCloud metadata distribution.
Source fingerprints are checked before and after preparation, and guarded
publication rechecks both source and output. Errors, panics, and dropped prepared
work release admission and the existing temporary-file guard removes owned output.
Native-only builds identify unsupported HEIF before the generic JPEG/TIFF probe
or preparation can read the whole file; approved-input checks still use streaming
fingerprints before preparation reports unsupported.
HEIF-family embedded XMP updates use the byte-preserving item-map writer and
reject layouts that cannot be changed without re-encoding unknown item-graph
data. A file may hold one XMP item per image, so probe and write both resolve
the packet through its `cdsc` association with the primary image and refuse
item maps where several are equally plausible. An unassociated packet answers
for the primary only when it is the sole candidate. The HEIF probe reads both
XMP and the standalone Exif item associated with the primary image before
planning datetime or GPS writes. A new XMP item receives a `cdsc` reference
naming the primary image. When the primary is the first input of one `tmap`
and its sole Exif descriptor already names exactly the primary and that tone
map, and no existing XMP item already describes that tone map, the XMP receives
the same two targets so Apple keeps the HDR rendition. Missing, conflicting,
additional, or ambiguous relationship or ownership evidence fails closed.
External item data references and top-level boxes whose absolute offsets are
not adjusted are also rejected.
Before publication, validation confirms that every construction-method-0 item
other than the resolved XMP packet, and every opaque `meta` sub-box, remains
byte-identical, and that re-reading the rewritten file resolves the packet just
written, allowing for the space padding an in-place replacement leaves in the
reused extent. Every embedded writer exclusively creates a unique sibling and
replaces the source only while the displaced bytes and prepared replacement
still match their approved fingerprints. Atomic exchange is preferred.
Metadata-only retries also pass the checksum-gate fingerprint into the writer, closing the
interval between catalogue validation and the writer's read. Existing regular
files and links at candidate temporary paths remain untouched. Safe pre-exchange
failures remove only the uniquely owned prepared file. Concurrent edits preserve
the source and its catalogue checksum evidence, keep the durable rewrite marker,
and leave any ambiguously displaced entry at its reported sibling path. The
replacement file retains the source permissions and mtime.
On Linux, filesystems that reject `RENAME_EXCHANGE` use a journaled fallback.
An exclusively created, owner-only sibling directory holds a version-1 JSON
manifest with lossless filename bytes and both SHA-256 fingerprints. Its name
is `.kei-replace-` plus the SHA-256 hash of the destination filename. The
prepared file is hard-linked into that directory and synced before the
original is displaced. Installation and restoration use hard links that refuse
an existing destination. The final path can be absent between displacement and
installation. Directory syncs precede the durable commit marker. Normal
completion removes only byte-verified recovery entries.
Before download discovery or catalog path reconciliation, Linux walks the
download tree through retained directory handles and recovers these journals.
The configured root may be a symlink alias; recovery resolves that trusted
anchor while refusing links beneath it. Reconciliation also recovers each
recorded source media and sidecar before planning, including old roots after
a directory change. Missing parent namespaces fail verification rather than
being treated as missing journal entries.
Directory reads require `/proc/self/fd`. Journal links are rejected. Dry-run and
filename-only download modes do not recover journals. Queued metadata work also checks its media
and sidecar journals before testing existence or checksums. Recovery restores
uncommitted originals or completes cleanup of committed replacements. Unknown
versions, malformed manifests, changed bytes, and conflicting destinations
retain the journal and fail the operation. Journal file locks prevent local
writers from recovering an active transaction. Terminal cleanup holds the lock
through manifest unlink, then closes all manifest handles before directory
removal. Only `ENOTEMPTY` from removal of a reverified empty, manifest-less
journal is deferred with a warning; publication still verifies the target and
removes only its matching prepared inode and bytes. Unknown entries, changed
bytes or namespaces, other removal errors, and directory fsync failures remain
errors. Empty journal recovery applies the same bounded cleanup. Historical
prepared files are not swept by this path. NFS mounts with `nolock` do not
provide cross-host locking; do not run kei on the same destination from
multiple NFS clients. Recovery does not authorize provider checkpoint progress
or downloaded-state finalization by itself.
`METADATA_CAPTURE_REVISION` identifies the catalogue semantics produced by the
current binary. Schema v18 stores per-asset revisions and per-library active
and pending repair state. A normal sync hydrates stale downloaded rows in
bounded provider-lookup batches, independent of album, media, and date filters.
Only an identity that cannot be resolved from durable asset/master evidence
uses the bounded legacy hydration path. Unselected libraries keep separate
pending state and do not force work in selected libraries. The legacy missing-hash
fallback checks only the selected library. While revision repair has pending
work, it owns that backlog instead of forcing a second full enumeration.
Ambiguous provider children remain pending and block the affected checkpoint;
a matching rendition alone does not prove which child owns the metadata.
The `metadata_capture_ambiguity_counts_v1` diagnostic reports stored rendition,
matching child, and full-evidence matching child counts. It contains no IDs,
paths, checksums, or provider metadata. These counts do not authorize identity
selection, even when only one child matches every stored rendition.
Schema v28 can preserve an unclaimed legacy master receipt with at least two
historical child mappings without assigning it to a child. A complete current
inventory must contain at least one live child. A sole survivor must satisfy the
same independent current-child receipt and file checks as a multi-child family;
historical siblings missing from that inventory do not establish ownership or
authorize source deletion. Zero-current-child families and saved legacy owners
remain excluded. `download/legacy_preservation.rs` owns eligibility and
file validation; `state/db/legacy_preservation.rs` owns immutable evidence and
generation-fenced receipts. Migration creates empty tables and protection
triggers only. It does not classify or rewrite existing records. Schema-27
binaries reject a migrated database; read-only reports require the current
schema and never migrate it.
Preparation protects the original catalogue rows, capture revisions, retry
receipts, relationship history, media paths and sidecar presence. It preserves
the old checkpoint. A valid original path already claimed by another catalogue
asset defers that family without creating preservation receipts or changing its
ownership, retries or checkpoint. Independent families in the same preparation
scan can still progress; malformed evidence remains an error. Before preparation,
unclaimed masters with multiple historical children already protect every
catalogued original path and its exact `.xmp` sidecar across libraries. Pending
metadata debt does not remove that protection. Ordinary writers, import adoption,
publication proofs, layout handovers and replacement recovery share this path
census; deferred originals retain their rewrite markers. A complete
unfiltered zone inventory must cover hidden children, all pages, an explicit
completion marker and a nonblank final cursor.
Each scan is limited to 10,000 pages and 1,000,000 observed records. It retains
compact latest identity/reference/deletion evidence for unrelated records and
full latest records for candidate masters and children, then hydrates those
families after the final page. Historical child mappings cannot replace complete
family discovery. Later updates and tombstones replace earlier evidence, including
references that enter or leave a candidate family.
The byte guards allow at most 64 MiB for one serialized response page and 256 MiB
for retained record representations, identity storage and pagination cursors.
Cumulative transferred bytes are counted separately and do not consume the
retained-evidence budget. These counts are representation bounds, not process RSS;
one response is decoded before its page-size guard. A cycle considers at most 64
new candidates. Exhausting a limit retains the hold. Sparse, malformed,
cross-library or incomplete evidence cannot activate preservation. No diagnostic
count is ownership proof.
Failed preparation discovery records a versioned, library-scoped
`legacy_preservation_inventory_retry:` receipt in the metadata table. An unchanged
configuration and durable original-family evidence defer another preparation scan
for at most one hour across restart. Retry scheduling, capture cycle counters and
unchanged relation refresh timestamps do not reset that delay. Changed
configuration or original-family evidence permits
a new attempt; malformed, unsupported or out-of-range retry receipts never defer
work. Deferral supplies no provider evidence or checkpoint permission. Prepared
records still require fresh certification, and metadata-capture retry scheduling
remains independent. Successful discovery clears its failure delay.
Preparation and certification use separate complete scans; a preparation result
cannot certify work performed later. Failure warnings expose only a fixed stage
and category plus aggregate page, record, transferred-byte and retained-byte
counts. Provider error text, record identities, paths and URLs remain private.
Activation requires a normal completed current inventory and, when a retained
cursor exists, a successful delta bridge. Independently identified current
children need separate, finalized receipts and verified files for every selected
rendition. Qualification requires independently verified per-path source
provenance for the exact child, rendition, provider generation and catalogue
path, with a freshly verified current local SHA-256. Provider MMCS signatures
are generation identities and are never compared to local content hashes.
Historical download hashes and publication without durable source provenance
remain insufficient. A new unverified publication that changes local bytes
clears prior source provenance; verified metadata rewrites retain their source
receipt. Exact-byte relocation can reuse an unambiguous source receipt bound
to the same library, child, rendition and provider generation. Pending
rewrite, capture-repair or path-reconciliation debt keeps this conservative
qualification blocked. Retained reconciliation destination choices are naming
evidence, not pending work, and remain part of the dependency comparison. No metadata writer is enabled automatically. A replay
that creates debt holds the cursor; a later cycle may qualify after configured
current-child work completes.
Original media and sidecars are fingerprinted through confined paths. Symlinks,
pending temporary writes, replacement journals, shared paths and changed bytes
reject qualification. Multiple hard links also reject qualification by default. Preparation warnings retain the aggregate
`invalid_original_files` candidate count and expose `shared_file_links` as a subset,
counting each candidate once at its first rejected media or sidecar. Typed shared-link
errors also expose only the fixed reason `shared_file_links` at certification and
checkpoint stages. The TOML-only `[download].legacy_preservation_allow_hardlinks`
flag defaults to false. True waives only the hard-link-count check for original
and current-child media and sidecars during legacy qualification and revalidation.
Actual use emits one redacted aggregate warning per library cycle, deduplicated
by evidence path; enabling the flag without linked evidence emits no warning.
Other confined-path, byte, receipt, debt and checkpoint validation stays intact.
Preparation, certification and checkpoint validation consume the same flag.
Its value is bound into the separate legacy policy hash; ordinary download and
enumeration hashes remain unchanged. False retains the previous legacy hash.
The opt-in assumes external tools are coordinated so they do not change linked
contents, metadata, or kei-managed names during the operation. It does not prove
inode independence: distinct original and current paths can share an inode.
It adds no account-wide alias census or read-only enforcement. Existing
identical-file reuse can set timestamps through aliases even when metadata
outputs are disabled. Metadata repair, timestamp repair, pending-file handling
and replacement recovery remain unchanged. General hardlink support and stronger
preservation guarantees remain open under #884.
Shared-root replacement recovery checks protected paths from
all libraries before opening a journal, including unselected libraries. Missing
protected parents retain their path reservation through the nearest existing
ancestor; they do not abort unrelated recovery or clear preservation/checkpoint
holds. Other filesystem errors and existing protected journals still block
recovery. Current child files and sidecar presence are also bound
into the receipt. The preservation owner never moves, rewrites, adopts or deletes
original paths. With the hardlink opt-in, that path protection cannot prevent
changes made through another alias by an unchanged write path.
Normal downloads, metadata repair, local path reconciliation, imports and source
tombstones respect the protection before byte writes; SQL guards provide defense
in depth. `reconcile` reports drift in protected originals without converting
them into download retries. Verification and manifest export retain the original
identities, dates, paths and history; download status does not establish ownership.
Unprepared ambiguous originals use the same protection census, including supported
relative paths and their exact sidecars. A connection-local SQLite TEMP cache
indexes confined path keys; relevant identity and path changes invalidate it
transactionally, and external commits, schema changes and working-directory
changes require a fresh census. Ordinary writes and mapping timestamp refreshes
do not rebuild it. This cache does not change durable preservation evidence.
Source metadata recovery distinguishes protected legacy originals from healthy
managed primary-layout sources. Managed sources continue through their layout
owner; pending media or sidecar replacement journals still hold reconciliation.
The final SQLite transaction compares the original snapshot, current child and
family evidence, receipt generation, configuration metadata and prior cursor.
It appends the proof and commits the cursor together. Staged config-reconciliation
cursors use the same transaction. Invalid or stale proof, incomplete current work,
interruption and write failure retain the prior cursor. Existing current-provider
identity and checkpoint guards remain independent and are never cleared to permit
preservation. An unchanged activated receipt can support ordinary incremental
cycles without another full inventory. Changed family, configuration, file or
receipt evidence reactivates the hold while retaining every original snapshot and
prior proof. Reactivation does not authorize historical attribution or rewriting.
`status`, cycle reports and health JSON expose `unattributed_legacy_assets` and
`unattributed_legacy_pending` separately from download failures. Pending current
proof makes one-shot exit status, cycle reports and health incomplete, including
idle health for unselected libraries. Certified but unattributed history alone
does not fail current sync. Certification refusals include the fixed-label
`legacy_certification_refusal_v1` diagnostic and a bounded `subreason`; unknown
errors use `unclassified`, without logging their private source text. The first counts
legacy records still needing attribution; the second counts records awaiting
valid current-work preservation proof. Original capture revision and retry history
remain stored. Advancing current sync does not claim a complete historical backup;
backup safety remains false while attribution is unresolved.
A missing selected receipt also emits `legacy_receipt_prerequisite_v1` with
one fixed `prerequisite`: child readiness, absent rendition or scoped path receipt,
binding mismatch, or missing/invalid verified source. This is a read-only snapshot
after refusal, not acceptance evidence. A newly eligible snapshot reports
`receipt_state_changed`; failed or unsupported inspection reports
`diagnostic_unavailable`. Both retain the original `child_receipt_missing` hold.
The lookup cannot grant qualification, retry work, write state or inspect media.
It emits no child/library identifiers, paths, hashes or private database errors;
support history retains only the enumerated prerequisite. One refusal does not
establish how many families lack proof. Primary sparse-identity recovery remains
independent.
Automatic repair processes at most 500 stale assets per library in one sync
cycle. When a clean batch makes progress and work remains, watch and service
mode wait at most 60 seconds before the next cycle. A stalled or failed batch
uses the configured watch interval so persistent provider failures do not cause
rapid retries. A one-shot sync processes one batch. `kei status` reads durable
remaining counts, and `sync_report.json` reports the current cycle's refreshed,
failed, and remaining counts. Revision storage uses one row per distinct asset.
A synthetic one-million-asset database grew by 87.363 MiB, or 17.51 percent.
One library with one million stale assets needs 2,000 successful batches, which
adds about 33 hours of follow-up waits plus provider and cycle processing time.
The watch pre-check includes only libraries with pending capture work or
serviceable rewrite work, even when iCloud reports no provider changes. A
rewrite marker is serviceable only when a metadata writer is enabled. A capture
refresh preserves download status, paths, checksums, and media bytes. It stores
corrected catalogue metadata, the current revision, and any configured rewrite
marker before the library revision can become active. Lookup or state failures
leave the row stale, report the remaining work, and preserve the affected
provider checkpoint. A library promotes its active revision only when every
live downloaded row is current. Persisted rewrite markers may drain later
because they are durable recovery evidence.
Metadata failure markers must survive so a later run can retry metadata
without downloading the media again. Full enumeration, single-pass incremental,
and collecting incremental sync all commit changed catalogue metadata and its
configured rewrite marker together, before filtering, album routing, or
deciding whether unchanged media needs downloading. A failed commit preserves
the provider checkpoint. Queued rewrites drain once per cycle, after every
producer has finished, so a single writer owns each file. Read-only runs never
drain. Only that drain retires a marker on success, because it writes the file
from the same row it then clears. Each bounded rewrite batch loads current
album and people rows for its library-scoped asset IDs. A grouping-read failure
leaves the affected markers pending. A completed download writes the snapshot
it was planned from, so it records a marker on failure but never retires one.
The first task payload includes its concrete pass album, even when the cycle's
grouping preload is empty. Provider membership changes update the
`asset_albums` read model and metadata retry markers in one transaction. The
projection uses child identity or the recorded legacy state owner, never a
sibling's master reference alone. Individual membership changes use indexed
identity lookups; container refreshes enumerate only that container. Schema v22
adds a reverse index for legacy owners without changing their recorded
identities. Completed snapshots can remove missing memberships; interrupted
snapshots preserve the prior live memberships. External grouping sources remain
unchanged. The end-of-cycle writer uses the accumulated memberships for each
tracked media path. Retry markers do not authorize writes when metadata outputs
are disabled.
`asset_metadata_paths` records each successfully finalized media path, its
provider rendition checksum, local fingerprints, and metadata retry receipts.
Registration and downloaded-state finalization share one transaction. The
catalogue still owns sync selection and keeps one path per rendition. Metadata
rewrites also visit the additional recorded paths that match the current
provider rendition. Each completion checks the selected path and fingerprints,
then clears only that path's debt. A failed copy does not block successful
copies or lose its retry evidence. Capture-repair receipts remain path-specific.
Missing paths retain retry evidence; old provider renditions remain recorded
but do not receive current-rendition metadata.
Schema migration seeds only the path already recorded in each catalogue row.
It does not scan for, infer ownership of, or authorize writes to older copies
that were never recorded. Additional paths enter the registry through verified
download finalization or explicit import adoption.
XMP sidecars record the exact properties that kei writes in the kei namespace.
A later rewrite deletes a cleared property only when that marker proves kei
owned the prior value. Unmarked standard properties and unrelated third-party
namespaces remain unchanged. An existing sidecar must be readable and parseable,
and its bytes must still match the writer's initial read at publication. A
failed check preserves the sidecar and its durable rewrite marker. When the
complete merged packet already matches the existing bytes, a retained regular
file snapshot rechecks identity, parent confinement and exact bytes before
reporting success. This avoids republishing an unchanged packet. Changed packets
use a new temporary path so retained ambiguous bytes cannot block a later retry.
Source GPS facts for sidecars are read through file-backed parsers. JPEG APP1,
TIFF-based RAW, PNG `eXIf`, and HEIF Exif items use checked seeks and
fixed-size TIFF fields. When a HEIF carries several Exif items, `pitm` and
`cdsc` select the one describing the primary image. HEIF atom and item-location
counts are streamed without allocating from provider-controlled lengths.
CloudKit is authoritative for the currently decoded coordinates, altitude, and
capture timestamps. The location decoder currently maps only `lat`, `lon`, and
`alt` from `locationEnc`, so source EXIF supplies GPS receiver time, speed,
speed units, and horizontal positioning error. Native horizontal accuracy is
emitted only when both validated native coordinates equal the provider latitude
and longitude after decoding to decimal degrees. There is no geographic
proximity tolerance: rounding differences can omit accuracy, but nearby or
edited locations do not establish that accuracy describes the same fix.
Matching coordinates alone are insufficient after GPS embedding: kei may have
inserted them beside an older native accuracy value. Sidecar planning also
requires the current file to match SHA-256 evidence from a verified download,
captured before any kei metadata write. Normal downloads pass that baseline to
the sidecar phase, even when embedding is disabled. Download finalization
atomically stores it in the path's `asset_metadata_paths.source_checksum`.
Queued rewrites use only that source checksum, never `local_checksum` or
`download_checksum`. Those older columns also support size validation and can
be populated from already-rewritten bytes during checksum recovery.
Schema v23 adds the nullable source checksum without backfilling historical
rows. Missing evidence stays unknown across embedded writes, failed state
updates, restart, and later retries. Adoption and local reconciliation do not
establish new source evidence. Existing evidence remains scoped to its path
and provider rendition; a different rendition invalidates it. Metadata-only
finalization preserves source evidence without changing it. This conservative
rule can omit valid accuracy from older downloads or after unrelated embedded
edits. It does not delete native metadata or add an output-revision sweep.
Missing, invalid, or different coordinates omit accuracy. A supported rewrite
removes the obsolete value only if the existing property marker proves kei
ownership. Unowned values remain unchanged. This rule applies to new sidecars
and already-supported rewrites; automatic catch-up for untouched sidecars is
separate work in #799. GPS receiver time and speed keep their existing policy.
Source I/O failures still publish current CloudKit metadata, preserve prior
kei-owned source GPS fields as unknown, and retain the metadata retry marker.
Readable unsupported or malformed metadata permits a CloudKit-only sidecar.
Source media is never opened for writing on this path.
A drain only rewrites a file whose bytes still match the checksum on its row,
because the alternative is embedding into damage and then vouching for it. A
file that no longer matches keeps its marker and its recorded checksum, so
`kei verify --checksums` and `kei reconcile` continue to report it. When a
rewrite does change the media, the new hash is stored before the marker
retires, and the pre-rewrite hash is kept as the provider download checksum so
reconcile can still tell an intentional rewrite from a truncated file. A
retired marker therefore implies the row describes the bytes on disk.
Capture timestamp repair records the current provider metadata hash as durable
intent. Before publishing changed media, it records the exact prepared output
checksum and size. If publication succeeds but state finalization is
interrupted, a later explicit repair can recognize that exact output and finish
the checksum transition without trusting unrelated bytes. Input bytes matching
neither the recorded local checksum nor the prepared receipt remain drifted.
Provider metadata updates that would invalidate a prepared receipt remain
retryable until the published bytes are finalised in state. Source-deletion
transitions follow the same rule so a tombstone cannot hide the only proof of
published repair bytes.
Capture-instant changes are guarded across every version of the asset, even
when the metadata hash is unchanged. Addition-date-only updates preserve the
prepared receipt. An upsert replacing the receipt's own rendition with a new
provider checksum may invalidate that receipt; a sibling replacement may not.
Tasks rejected by the state upsert are not dispatched or marked failed, so the
existing receipt and catalogue evidence remain intact and the state-write
failure keeps the checkpoint ineligible.
Provider-version or non-metadata file replacement invalidates the repair state;
byte-identical re-adoption preserves it. When replacement or adoption occurs
during the explicit repair run, downloaded-state finalization creates fresh
pending repair debt for the installed checksum.
A rewrite whose result could not be measured records the checksum as unknown
rather than leaving the superseded value in place, because a known-stale hash
would make the next pass read kei's own rewrite as damage and refuse it. A row
that never recorded a checksum is rewritten by ordinary metadata refresh but
claims no provider download hash, so reconcile keeps reporting a file that is
short of its provider size. Capture timestamp repair instead leaves that row
pending because it cannot prove file provenance.
### State and serialization
Asset capture and addition dates use fractional Unix seconds in the existing
non-STRICT SQLite INTEGER columns. Readers accept legacy whole seconds and
round fractional seconds to milliseconds without scaling the whole timestamp.
Invalid stored dates fail decoding. Metadata refresh commits these dates with
the metadata and rewrite markers; operational timestamps remain whole seconds.
This representation requires no schema migration or historical backfill.
Schema, primary-key, sentinel, durable-key, and serialization changes are
cross-cutting. Search every reader and writer, migrations, fixtures, reports,
status output, and round-trip tests before changing them.
### Secrets and diagnostics
Passwords stay behind `SecretString` and password-source boundaries. Logging
uses a redacting writer as a backstop, but code must not send Apple IDs,
passwords, session cookies, bearer tokens, or unredacted provider identifiers
to logs or machine output. Preserve process hardening that limits credential
exposure through core dumps.
`password set` prompts only in interactive input mode. Headless callers must
use a password file or password command. The command resolves the secret and
passes it directly to the credential store without printing it.
## Safety contract catalog
Stable IDs connect safety rules to production owners and focused tests.
`scripts/check-contracts` rejects missing links in that chain.
| Contract | Owner | Required behavior |
|----------|-------|-------------------|
| `FILE_PUBLISH_NO_OVERWRITE` | `src/download/file.rs`, `src/download/pipeline.rs` | Publishing a completed `.part` file never replaces an existing final file unless `--repair-truncated` carries exact durable path and fingerprint authorization, or opt-in `photos.edited_naming="primary"` carries a journaled owned-slot handover with an independent exact preservation receipt. A no-replace collision succeeds only when the verified `.part` and destination bytes are identical. Different or unverifiable bytes retain retry evidence and cannot reach metadata writes or downloaded finalization. |
| `TEMP_FILE_DELETE_REQUIRES_DURABLE_OWNERSHIP` | `src/download/orchestration/cleanup.rs`, `src/download/pipeline.rs`, `src/fs_util.rs`, `src/state/db.rs` | Orphan cleanup deletes only an exact stale path claimed in durable state. It retains verified filesystem handles through removal and never follows a directory or file symlink. Normal completion and graceful interruption retire the claim. |
| `SYNC_TOKEN_ADVANCE_REQUIRES_CLEAN_CYCLE` | `src/sync_cycle.rs` | The database pre-check token advances only after a successful non-dry-run cycle with a current pass plan. |
| `SOURCE_CHECKPOINT_REQUIRES_DURABLE_RECOVERY` | `src/sync_cycle.rs`, `src/download/orchestration/` | A zone checkpoint advances only with complete token evidence and durable recovery for unfinished work. |
| `MALFORMED_REQUIRED_ASSET_FIELDS_BLOCK_CHECKPOINT` | `src/icloud/photos/asset.rs`, `src/icloud/photos/album/fetch.rs`, `src/icloud/photos/album/lookup.rs`, `src/download/orchestration/incremental.rs`, `src/sync_cycle.rs` | A live asset with a missing or invalid required identity or capture date blocks its zone checkpoint before filtering or path planning. |
| `UNKNOWN_PROVIDER_IDENTITY_REMAINS_PENDING` | `src/download/retry.rs` | Inconclusive provider identity retains the pending row and records verification evidence. |
| `POLICY_EXCLUDED_REQUIRES_EXPLICIT_SOURCE_DELETION` | `src/download/retry.rs`, `src/state/db.rs` | Policy-excluded rows become source-deleted only after targeted provider deletion evidence. Present or inconclusive responses retain them outside actionable pending work. |
| `METADATA_WRITES_REQUIRE_OPT_IN` | `src/download/metadata_rewrite.rs` | Media and sidecar metadata writes run only for explicitly enabled metadata flags. |
| `METADATA_EMBED_REWRITE_REQUIRES_STABLE_INPUT` | `src/download/metadata.rs`, `src/download/file.rs`, `src/download/metadata_rewrite.rs` | Every embedded metadata rewrite prepares a uniquely owned sibling and replaces the media only while both the destination and prepared bytes match their approved fingerprints. Failure preserves concurrent edits and durable retry evidence. |
| `HEIF_EMBED_REWRITE_REQUIRES_STABLE_INPUT` | `src/download/heif.rs`, `src/download/metadata.rs`, `src/download/file.rs`, `src/download/metadata_rewrite.rs` | A HEIF-family embedded rewrite accepts tone-map insertion only when `dimg` and primary Exif `cdsc` relationships prove the exact target and no existing XMP owns that tone map, prepares a uniquely owned sibling, and replaces the media only while both the destination and prepared bytes match their approved fingerprints. Failure preserves concurrent edits and durable retry evidence. |
| `XMP_SIDECAR_REWRITE_REQUIRES_STABLE_INPUT` | `src/download/metadata.rs`, `src/download/metadata_rewrite.rs` | An existing XMP sidecar is replaced only when it parses and its bytes still match the writer's initial read. Failure preserves the sidecar and durable rewrite marker. |
| `XMP_GPS_ACCURACY_REQUIRES_MATCHING_LOCATION` | `src/download/metadata.rs`, `src/download/metadata_rewrite.rs`, `src/state/db.rs` | Native horizontal accuracy requires valid native latitude and longitude equal to the exported provider coordinates, plus a matching verified-download source checksum. Recovered local or download checksums cannot establish native provenance. Readable missing, invalid, or mismatched evidence omits accuracy and clears only obsolete kei-owned values. Source I/O failure preserves unknown fields and durable retry evidence. |
| `METADATA_CAPTURE_REVISION_REPAIR_IS_DURABLE` | `src/download/orchestration/maintenance.rs`, `src/state/db.rs` | Revision repair updates catalogue metadata and configured rewrite evidence before promotion, stays library-scoped, and preserves the provider checkpoint on unresolved work. |
## Change-impact traversal
Use these search anchors after finding the owner. The related owners are an
impact set, not a claim that every run calls each one in sequence. Read direct
callers and follow shared values through their consumers. Each row names the
flow section that owns its behavior and safety details; this table is not a
second call graph.
This table covers the highest-traffic changes only. Use the Owners table to
find the owning module, then this table to find related consumers. Use the
Change-impact checklist to choose verification.
| Change | Search anchors and related owners | Flow or invariant |
|--------|-----------------------------------|-------------------|
| Collection selection | `Selection` in `src/selection.rs`; `collection_libraries`, `pass_scope_for_zone`, and `resolve_passes_for_scope` in `src/commands/service.rs`; sync callers in `src/sync_loop/planning.rs` and import callers in `src/commands/import.rs` | [Sync and provider checkpoints](#sync-and-provider-checkpoints), [Import-existing](#import-existing) |
| Provider progress | `check_changes_database` and `store_scoped_db_sync_token` in `src/sync_loop/precheck.rs`; cycle calls in `src/sync_loop/runner.rs`; `source_checkpoint_decision` in `src/sync_cycle.rs`; `commit_checkpoint_transition` in `src/state/db/checkpoints.rs` | [Checkpoint advancement](#checkpoint-advancement), [Sync and provider checkpoints](#sync-and-provider-checkpoints) |
| Enumeration proof | `download_photos_with_sync` in `src/download/orchestration/dispatch.rs`; full owner in `src/download/orchestration/full.rs`; incremental owner in `src/download/orchestration/incremental.rs`; `src/icloud/photos/album/enumeration.rs`, `src/icloud/photos/album/completion.rs`, and `src/icloud/photos/album/changes.rs` | [Full and incremental enumeration](#full-and-incremental-enumeration) |
| File landing and state | `TaskPlanner` in `src/download/planner.rs`; `src/download/pipeline/producer.rs`, `consumer.rs`, `task.rs`, and `pass.rs`; `download_file_with_mode` in `src/download/file/transfer.rs`; `publish_part_to_final` in `src/download/file/publication.rs`; `finalize_downloaded_with_proof` in `src/download/finalize.rs`; `mark_downloaded` in `src/state/db/assets.rs` | [File landing](#file-landing), [Managed edited-primary layouts](#managed-edited-primary-layouts) |
| Durable retry | `src/download/orchestration/recovery.rs`; `build_pending_retry_download_tasks` in `src/download/retry.rs`; targeted lookup in `src/icloud/photos/album/lookup.rs`; `DownloadStateStore` pending-state methods in `src/state/db/contracts.rs` | [Durable pending retry](#durable-pending-retry) |
| Paths and import parity | `src/config/templates.rs`; `DownloadConfig::with_pass` in `src/download/orchestration/config.rs`; `src/download/paths.rs`; `expected_paths_for` in `src/download/filter/expected_paths.rs`; callers in `src/commands/import.rs`; local reconciliation in `src/download/orchestration/reconciliation.rs` | [Import-existing](#import-existing), [File landing](#file-landing) |
| Metadata refresh and writes | `Config::runtime.refresh_metadata` reads in `src/sync_loop/runner.rs` and `src/sync_cycle.rs`; capture repair in `src/download/orchestration/maintenance.rs`; immediate and queued owners in `src/download/metadata_rewrite/`; format and publication owners in `src/download/metadata/` | [Metadata](#metadata) |
| Import adoption | `run_import_existing`, `import_assets`, and `StrictImportVerifier` in `src/commands/import.rs`; `ImportStateStore::import_adopt` and its SQLite implementation in `src/state/db/import.rs` | [Import-existing](#import-existing), [State and serialization](#state-and-serialization) |
| Service execution | dispatch in `src/lib.rs`; install, status, uninstall, and platform owners in `src/service/`; `src/service/run.rs` enters `src/sync_loop.rs::run_sync` | [Command dispatch](#command-dispatch), [Container startup](#container-startup) |
For schema, durable-key, sentinel, serialization, and shared-type changes,
follow [State and serialization](#state-and-serialization). Use
[the test guide](../tests/README.md) to choose focused proof; a diagnostic count
or a source-only assertion does not replace production-path state-transition
proof.
## Change-impact checklist
This table chooses verification for a change. It assumes the owner is already
known; use [Change-impact traversal](#change-impact-traversal) to find the code
to read first.
| Change | Check |
|--------|-------|
| CLI command or flag | `src/cli.rs`, dispatch in `src/lib.rs`, help output, Docker, services, Homebrew, docs, CLI tests |
| TOML or runtime config | defaults, setup output, CLI precedence, hashes, persisted examples, docs |
| Selection or pass scope | list/sync/import parity, shared libraries, unknown names, unfiled scope, membership snapshots |
| Provider checkpoint | full and incremental proof, interruption, config drift, retry durability, scoped DB pre-check |
| SQLite schema/query | migration from every supported version, all readers/writers, status/report/manifest output, real SQLite tests |
| Provider record parsing | missing/malformed fields, shared zones, identity mapping, metadata capture, fixtures |
| File or path behavior | `.part`, checksum, no-overwrite publish, fsync, import compatibility, collision handling |
| Metadata writes | opt-in gate, pre-publish mutation, sidecars, retry markers, feature combinations |
| Service behavior | Linux, macOS, Windows, container defaults, status, install/uninstall renderers |
| Credentials or logging | secret wrappers, source lifetime, redaction, core-dump hardening, diagnostics, error paths |
| Machine output | JSON/CSV shape, redaction, reports, health, metrics, downstream compatibility |
## Tests
- Unit tests live near their owner module.
- Cross-module and binary behavior lives under `tests/`.
- Strict changes-page recovery is also exercised through production `run_cycle`
with real earlier downloads, malformed later work, reopened SQLite and valid
replay in both incremental strategies. The proof preserves historical bytes
and retry/identity debt, then checks exactly-once transfers, unchanged download
receipts and two quiet cycles. See the strict changes/zone section of the
[test guide](../tests/README.md#strict-changeszone-pages).
- Live iCloud tests are ignored by default and run single-threaded.
- `tests/data/media-manifest.json` owns the bundled media inventory and size
budget. `download::orchestration::fixture_tests` exercises these bytes
through enumeration, planning, download, publication, and reopened SQLite.
Its selection, naming, metadata, and recovery modules replace content-dependent
live assertions. `tests/data/live-migration.md` records the mapping. General
live entry points share `tests/data/live-selection.toml` and reject an empty
eligible selection before the suite.
`scripts/fixtures/` owns maintainer-only generation and sanitization and
the optimized extracted-source-package check. Tests do not fetch media.
- Shell suites cover crash, concurrency, state-machine, and container behavior.
- Fuzz targets cover parser and metadata trust boundaries.
- `ci.yml` defers draft PR work and keeps the required `CI` marker failing until
ready-for-review validation succeeds. Ignored PR metadata has a separate
concurrency key and check name; base retargets and full-label transitions
revalidate. `scripts/ci/policy.py` owns path and native-service routing.
`ci.yml` requires selected native smoke once, or delegates it to qualification.
`checks.yml` owns shared quality and always-on safety-contract validation;
`qualification.yml` composes heavy checks and tested artifacts.
`scripts/ci/artifact.py` binds archive/container evidence to one commit.
`local_release.py` validates local live/host-service attestations.
Release publication depends on both local evidence and hosted qualification;
nightly runs cannot publish. See [CI qualification](ci.md).
- `scripts/ci/opencode_review.py` owns the data-only advisory review client. PR review requires a new standalone `/opencode review` comment from a human with API-verified repository write access, or explicit default-branch dispatch. Requests pin the current PR head; per-PR publication serialization and head markers suppress duplicates. `opencode-review.yml` executes trusted default-branch code and separates read-only inference from deterministic COMMENT publication. Issue-open intake pins main and issue content, runs independent product/code reviews, and publishes separate advisory comments with stale-state and dedup checks. Code validation is static inspection only. PR reviews submit the complete diff without local usage caps; model capacity and Go allowance govern inference. Transport/job safeguards and publication validation remain. GitHub 403/429 responses stop the operation without automatic retry. See [review security and setup](opencode-review.md).
- `justfile` owns local script and workflow lint commands. Protected CI runs
the same shellcheck, shfmt, ruff, and actionlint checks with pinned versions.
See [the test guide](../tests/README.md) for the current suites and commands.
## Maintaining this guide
Update this file in the same pull request when a change:
- Moves an owning decision to another module
- Adds a command or cross-cutting state transition
- Changes file publication or metadata mutation
- Changes provider checkpoint or retry evidence
- Changes schema, durable keys, or serialization
- Changes the best test for an invariant
- Adds, renames, or moves a major section, or changes its link label
- Changes a traversal anchor, owner module, or flow link
Keep the guide focused on stable ownership and safety. Source code remains the
final authority. Do not cite build output, local session files, retained
investigation notes, or release history as an ownership claim. Describe the
current owner instead.
## Managed edited-primary layouts
`download/orchestration/primary_layout.rs` owns opt-in `photos.edited_naming` transitions. The shared selected-rendition renderer assigns current/original-archive roles without changing rendition keys, RAW selection or URL deduplication. The layout owner confirms current provider resources through the Photos adapter, prepares all selected members, preserves displaced media and sidecars as independent exact copies, then conditionally publishes and retires verified obsolete aliases. Default suffix families retain ordinary append-only behavior.
Schema 35 adds independently addressable layout bindings, operations, members, path claims and immutable preservation receipts under `state/db/primary_layout.rs`. Scoped native paths, stable child and pass identity, source/config decisions and a checked binding generation pin each operation. Durable phases are planned, prepared, preserved, publishing and committed; cancellation retains its journal and private prepared files. Recovery runs after filesystem replacement recovery and before ordinary source dispatch. A started handover finishes its recorded generation before new source/config work may plan another operation. Config/path reconciliation performs this recovery before reading catalogue paths, including operations at a previous root or template. First-binding migration inventories each scoped per-path rendition receipt and its own recorded placement independently of current selection or the latest catalogue filename; exact local bytes and the complete path-owner census remain required before preservation or retirement.
Managed recovery stops at the first unresolved operation; later independent
operations in that recovery pass remain pending. Reconciliation retains its
checkpoint hold. A separately completed independent family does not establish
continuation past a blocked operation in the coordinator.
The claims fence ordinary transfer, finalization, adoption and queued metadata. Current-publication retry recovery leaves managed plans with the layout owner and rechecks slot generation and exact local-byte claims in its state transaction, including claims created after the reader snapshot. Metadata for selected renditions is prepared off the visible path; it commits with the new binding and matching provider selection receipts. Ordinary whole-asset metadata refresh resolves ownership across every binding and pending claim for the scoped provider child, including when the caller has no concrete pass key; it does not queue competing writes for managed paths. Explicit metadata refresh uses the preserving owner and records its intent in the operation; pending operations recover before unchanged-binding shortcuts. Retained-source reuse carries the SHA-verified fingerprint through candidate receipt capture and independent copying; a second snapshot must equal the trusted first snapshot. Local path reconciliation uses this same layout owner for managed families. Current/archived media and sidecars must match their pinned fingerprints. Unknown ownership or local changes hold completion and protect existing contents. Real album membership is persisted before managed reconciliation commits matching metadata; an unfiled path label is never recorded as album membership. Original archives retain their prior metadata and retry evidence outside the ordinary writer queue; prepared capture-repair evidence stays visible to source-deletion guards. Archive receipt and whole-asset ownership lookups use scoped library/child indexes. Historical catalogue receipts remain evidence, not permission to write a superseded slot.
Files in a Live Photo family are reserved and prepared together. Publication across separate files is recoverable rather than simultaneously atomic to external readers. Filesystems without the required atomic conditional publication/retirement operations leave durable work pending. See [edited primary usage](edited-primary.md) for storage, exclusion and downgrade guidance.
## Support evidence boundary
`support-export` dispatches before startup configuration construction, logging,
credentials or provider initialization. `src/support/collect.rs` reads bounded
local files; `src/state/db/support.rs` owns immutable SQLite inspection without
normal `open_owned` or schema migration. Unsupported schemas and nonempty WAL or
journal snapshots degrade explicitly. Export output uses exclusive creation.
Normal sync, import and service operations publish typed evidence from their
existing owners through a bounded optional worker. The worker groups only
allowlisted fields, persists account-scoped history with private atomic writes
and a single-writer lock, and never feeds decisions back to provider, checkpoint,
state or publication policy. Its tracing layer is independent of log verbosity.
Stored history is revalidated against the same contract before export. Unknown
fields, labels, arbitrary errors and provider identifiers cannot broaden it.
The Photos owner projects only fixed owner morphology and JSON types for rejected
inventory scopes. Private discovery records identify primary versus other zones
and validated primary eligibility. A primary-publication observation follows the
actual qualified-primary assignment; selected indexing alone does not imply it.
Owner categories are revalidated as strings, so a malformed numeric owner cannot
escape through an aggregate-count allowance. Missing older fields remain missing.
Full enumeration also emits `support_token_equivalence_v1` from
`full/token_diagnostics.rs`. Each self-contained snapshot relates initial and
same-cycle recovery token groups for at most 64 selected passes. Tokens are
compared only in a bounded invocation-local table (128 values, at most 4096 bytes
per value); only assigned ordinals reach the support boundary. The provider owner
supplies fixed query and scope-comparison categories. A passive stream wrapper
records drainage order without extra polling or tasks. Missing drainage is null;
this order is not token-arrival, HTTP-completion or media-publication order.
`support/token_equivalence.rs` revalidates nested rows at capture and export,
rejects malformed or duplicate rows, and preserves explicit omission markers.
Group numbers have meaning only within one snapshot, never across invocations,
cycles or accounts. The recovery snapshot repeats the initial evidence so history
retention cannot silently sever cross-round relationships. This evidence has no
consumer in enumeration, recovery or checkpoint policy.
`photos/album/preservation_proof/missing_reference.rs` owns observation-only
lookup at the first missing-parent inventory refusal. Four byte-sized typed
pre-compaction categories live alongside every latest inventory record and count toward its
retained-byte budget, without a separate identity/count sampling cap. Lookup
payload shape is separate from scope availability. Scope `matched` means the
child scope matched; reference scope is also required and checked for a
`live_reference` payload before family classification.
`PhotosSession::post_diagnostic` uses a dedicated 64 KiB transport with fixed errors,
no request dumps, error-body logging or retries. The caller owns cancellation and
a five-second deadline. Its response never enters inventory or proof construction;
only fixed categories enter support history and export.
Random process-local aliases are bounded and renumbered at collection time.
Counts are scoped observations and must not be added as unique asset counts.
SQL observations before transaction commit are explicitly identified as attempts.
`src/support/mod.rs::classify_support_error` owns diagnostic typed-error
projection. It exports fixed classes and numeric status codes from the cause
chain, never messages or provider payloads, and does not decide retry policy.
The inbox, catalogue, selection and work owners publish fixed storage counters
after successful transaction commit and at existing capacity refusal branches.
The history worker retains a fixed summary per cycle, not response rows.
Budgets reuse existing admission accounting and are labeled last observed,
pre-admission logical charges. Migration returns its compaction summary to the
schema owner, which publishes it only after the migration savepoint commits;
failed migration reports no committed reclaim. Optional charge observations
cannot fail migration. Retained reasons and counters contain no source identity,
URL or arbitrary error text, and remain subject to queue loss and bounded
history retention. Unknown observations stay explicitly unavailable.
History and export schema version 1 are separate from report version 3 and state
schema 36. Raw reports and databases are never packaged. Interrupted writes
preserve the previous history; read errors and limits appear as partial evidence.
See [the operator guide](support-export.md) for limits and unavailable evidence.
### Explicit missing-reference inventory diagnostic
The opt-in `doctor --live --json --missing-reference-inventory` command branches
before normal startup, logging, service hooks and ordinary doctor filesystem
checks. `inventory_diagnostic.rs` reads bounded configuration and copied cached
auth only. It does not open a catalogue or construct a download/state owner.
`photos/diagnostic.rs` adapts the existing PhotosSession interface to privacy-safe
streamed requests with shared limits and no redirect/retry/auth fallback.
The existing PhotosService qualifies the private PrimarySync scope before the
preservation scanner's raw latest-record helper supplies fixed missing-reference
observations. The diagnostic performs no family hydration and returns no cursor
or CompleteLegacyInventory proof. At most one later lookup is observational.
Ordinary preservation retains its candidate, scope, date and completeness gates.
See `docs/missing-reference-inventory.md` for bounds and supervision requirements.
### Explicit trusted local bytes for legacy migration
`commands/trusted_legacy.rs` owns the offline `reconcile trust-legacy-files plan/apply`
interface and account lock. `download/trusted_legacy.rs` owns bounded plans,
confined retained file handles and final byte validation.
`state/db/trusted_legacy.rs` owns typed `TrustedLegacyFile` acceptances in
versioned, content-addressed `metadata` keys. Apply installs prefix-scoped
immutability guards in the acceptance transaction; no-acceptance catalogues are
unchanged. Their account/provider identity,
current-child generation, metadata/path and protected-family snapshots are
rebuilt from current state at apply and receipt lookup. Apply commits all entries
atomically after final file and cancellation checks. No source checksum is
inferred or written.
Only legacy current-child coverage consumes this alternate provenance. Its fresh
inventory, readiness, ownership, file and checkpoint guards still run.
Nonempty library-scoped acceptance rows extend the legacy dependency fence;
provider evidence records `verified_source` or `trusted_legacy_file` explicitly.
No-acceptance dependency serialization is unchanged. Native source verification,
GPS and ordinary adoption retain their verified-source requirements. See
[Trusted Legacy Files](trusted-legacy-files.md) for the command boundary.