docs/managed-service-upgrades.md
Rama main · réplica de lectura del repositorio original.
<!-- authority:non-specification -->
# Managed Linux service upgrades
Managed mode keeps the public HTTP listener and accepted connections in a stable
supervisor while replacing its private worker. It is opt-in for a Linux or WSL
`systemd --user` release service. macOS, Windows, direct HTTP and stdio installs
keep their existing behavior.
Install a release that includes `exomem.service_manager`, then enable the mode:
```sh
bash scripts/install-service.sh --release --seamless --profile standard
```
The first enablement uses the existing stopped-transition proof and changes the
public listener owner once, so connected clients must reconnect for that
bootstrap. On an existing unit, the installed `service.env` is the authority:
the installer reads its vault, state root, OAuth issuer and other variables and
leaves its bytes unchanged. It refuses configuration changes during bootstrap.
A fresh managed install reads the requested dotenv file.
If first enablement renders the managed unit but fails to start or verify it,
the installer leaves the service stopped with its legacy transition receipt.
Correct the cause, then repeat the same bootstrap options with
`--resume-stopped-transition`:
```sh
bash scripts/install-service.sh --release --seamless --profile standard --resume-stopped-transition
```
Repeat any original `--service-root` and `--package-version` pins. This entry
is refused unless the exact installer receipt proves that the managed unit and
its prior listeners are stopped. If the manager already recorded an active
launcher release, resume pins to that exact version; a different explicit
`--package-version` is refused before installation. A pending manager worker
transition also blocks installer recovery. Once bootstrap succeeds, the receipt is
cleared and the installer cannot re-enter a managed unit. Later worker failures
use the private `--resume` command below, not this bootstrap option.
Later worker releases are staged in a new venv under `<service-root>/releases`
while the old worker serves. The launcher venv at `<service-root>/.venv` is not
replaced. Run:
```sh
bash scripts/upgrade.sh --profile standard
bash scripts/upgrade.sh --package-version 0.80.0 --cli-sync always
```
For a personal candidate that has not been published, the staging module also
accepts a local wheel directly. Supply the full source commit that produced the
wheel; staging copies the artifact into its owner-only release directory and
records the declared revision, artifact digest, original filename, and verified
installed interpreter/package identity in `provenance.json` beside it.
```sh
<operator-python> -m exomem.service_upgrade --runtime-dir <runtime-dir> \
--wheel /absolute/path/to/exomem-0.80.0-py3-none-any.whl \
--source-revision <40-hex-commit> --profile standard
```
`<operator-python>` must come from a tested checkout or newly installed
interpreter that contains this staging feature; the manager's existing launcher
interpreter stays unchanged and is used only to create the release environment.
`--wheel` and `--package-version` are mutually exclusive. A local wheel is a
candidate, not a public release; it uses the same managed handoff once staging
has verified it. `scripts/upgrade.sh` does not forward wheel sources yet.
The script discovers the managed unit, asks its private control socket to
switch to the staged version, verifies the reported active release, then uses
the existing CLI sync and managed-install manifest helpers. `--cli-sync auto`
updates an existing uv-managed CLI, `always` installs it if needed, and `never`
leaves it alone. Failed staging leaves the old worker serving and removes the
release directory it created, also when wheel verification fails.
Once the target is confirmed active, the operator also re-runs `install-hook`,
from the newly promoted release, for every local Claude Code profile that
already wires the retrieve nudge hook: the default `~/.claude` plus any
sibling `~/.claude-*` profile whose `settings.json` (resolved through a
symlink, if it is one) names `exomem-retrieve-nudge.sh`. A profile that never
wired the hook is left untouched. This never fails or rolls back the upgrade
itself -- a per-profile error, such as the installer's own refusal of a
group-writable config, is reported under `hook_refresh` in the upgrade result
and surfaced by `exomem doctor` (`upgrade.hook_refresh`), not raised. Set
`EXOMEM_DISABLE_UPGRADE_HOOK_REFRESH` to opt out.
For status or recovery, read the runtime directory from the rendered unit's
`--runtime-dir` argument and use the launcher's interpreter:
```sh
<launcher-python> -m exomem.service_upgrade --runtime-dir <runtime-dir> --status
<launcher-python> -m exomem.service_upgrade --runtime-dir <runtime-dir> --resume
```
The private control socket is owner-only. A failed transition after the old
worker stops leaves a recovery record and the public ingress unavailable;
`--resume` rolls forward the recorded target after process-exit proof, and
waits out the same cold-start budget for the replacement to report ready. Do
not start an older release over a possibly migrated state root. Supervisor or
host restarts interrupt public connections and require normal client
reconnection.
A handoff that is taking minutes is not the same thing as a failed one. Read
`--status` before reaching for `--resume`: a transition still in flight reports
`phase: upgrading` with a `transition` id and a `pending` record naming the
step it is on, and its outcome appears under `last_transition` when it
finishes. Resuming a transition that is still running is refused, but a repair
the replacement is doing in the background is the kind of work that outlasts
an impatient operator's first look.
Changing the stable supervisor, bind host, port or `service.env` is an offline
maintenance operation. Prepare the replacement package, environment file or
unit edit first. Complete at least one normal `scripts/upgrade.sh` worker
handoff before changing the launcher venv: the private status must show a ready
worker whose active interpreter differs from the launcher, with no pending
transition. If the launcher still owns the active worker, stage the same
currently serving release first with
`bash scripts/upgrade.sh --package-version VERSION` and the matching profile.
From the checkout, capture the current
MainPID and listener, stop the unit, and prove they and the entire service
cgroup are gone before touching the launcher venv or installed configuration:
```sh
set -euo pipefail
. scripts/_service-common.sh
unit="$(exomem_unit_file)"
exomem_service_is_managed "$unit"
service_id="$(exomem_service_id "$unit")"
launcher="$(exomem_service_python "$unit")"
runtime_dir="$(exomem_managed_runtime_dir "$unit" "$launcher")"
status_json="$("$launcher" -m exomem.service_upgrade --runtime-dir "$runtime_dir" --status)"
printf '%s\n' "$status_json" | "$launcher" -c '
import json
import sys
status = json.load(sys.stdin)
active = status.get("active")
if status.get("phase") != "ready" or status.get("pending") is not None or not isinstance(active, dict):
raise SystemExit("managed service is not ready for maintenance")
if active.get("python") == sys.argv[1]:
raise SystemExit("stage the current worker release with scripts/upgrade.sh before launcher maintenance")
' "$launcher"
port="$(exomem_service_port "$unit")"
main_pid="$(exomem_service_worker_pid "$service_id")"
listeners="$(exomem_listener_pids "$port")"
[[ "$main_pid" =~ ^[1-9][0-9]*$ && "$listeners" == "$main_pid" ]]
group="$(systemctl --user show "$service_id" --property ControlGroup --value)"
systemctl --user stop "$service_id"
exomem_assert_service_stopped "$main_pid" "$port" "$service_id"
"$launcher" - "$group" <<'PY'
import sys
from pathlib import Path
group = Path(sys.argv[1])
if not group.is_absolute() or group == Path("/") or ".." in group.parts:
raise SystemExit("invalid service cgroup; maintenance refused")
root = Path("/sys/fs/cgroup") / str(group).lstrip("/")
if root.exists():
for members in root.rglob("cgroup.procs"):
if members.read_text().strip():
raise SystemExit("service cgroup still has processes; maintenance refused")
PY
env_file="$(exomem_service_binding_path "$unit" "$launcher")"
cp -p "$env_file" "$env_file.before-maintenance"
cp -p "$unit" "$unit.before-maintenance"
```
Keep the unit stopped if any proof fails. The backups preserve `service.env`'s
owner-only permissions. Replace a reviewed `service.env` only now, using
`install -m 600 PATH_TO_REVIEWED_ENV "$env_file"`; use
`systemctl --user edit --full exomem.service` for
unit changes. For a supervisor package update, install into the stopped
launcher interpreter with `./.uvbin/uv pip install --refresh-package exomem --python "$launcher" "exomem==VERSION"`; worker releases continue to use
`scripts/upgrade.sh`. Validate the edited unit with
`systemd-analyze verify "$unit"`, then run `systemctl --user daemon-reload`
and `systemctl --user start exomem.service`. Check managed `--status`,
`/health/ready`, and that unauthenticated `/mcp` still returns 401 before
accepting the maintenance. Changing OAuth issuer or signing keys requires
clients to authorize again. If startup fails, leave the unit stopped and
inspect the journal and retained state before another change; do not silently
start an older release over migrated state.
## Rehearse a release on a copy
Rehearse each release on a copy of the live vault and state before you install
it. The rehearsal runs the same handoff on a throwaway managed service, so it
shows what the live switch will do. The checks judge what callers see, not
whether a repair runs. Install the release only when all four pass:
- `handoff.ready_after_ms` is under 120000 (two minutes).
- `/health/ready` stays ready for 45 minutes after the switch.
- Every full lexical repair in that window publishes.
- During each repair in that window, one MCP read and one MCP write succeed,
and neither returns `MUTATION_BUSY` or `VAULT_LOCK_TIMEOUT`.
A failed check holds the install. A repair that holds `/health/ready` at
`not_ready` fails, and so does a write refused to its caller. Run the same read
and write in the live install window; there, use a write that is safe to repeat
and that the vault accepts, such as recording an episode.
`ready_after_ms` counts from the old worker's stop, so it includes the state
migrator when `handoff.migration.state` is `ran`.
The copy needs disk for the vault and the state directory, plus one release
environment. The throwaway service needs memory for two workers during the
standby warm. Set its `MemoryMax` from the serving worker's resident size.
The first start on a copy always repairs the catalog once. The freshness
checkpoint covers each page's absolute path and change time, and a copy keeps
neither. Let the current release settle on the copy first. Only the switch that
follows tells you about the candidate.
The supervisor verifies its systemd identity, so the throwaway service runs as
a transient user unit. Keep its paths short: the control sockets under the
runtime directory have a 104-byte limit.
1. Create the scratch root: `install -d -m 700 /tmp/xr /tmp/xr/{managed,home,logs,lease,cwd,state}`.
2. Create the launcher with `uv venv /tmp/xr/launcher` and the serving version's package and profile extras.
3. When no import or bulk apply is running, copy the vault to `/tmp/xr/vault`.
4. Compute the copy's state key with the key command below.
5. Create the state leaf: `install -d -m 700 /tmp/xr/state/KEY`.
6. Copy the live vault state directory into the state leaf with the state copy command below.
7. Set `vault_identity` in `/tmp/xr/state/KEY/.state-migration.json` to `KEY` with a JSON editor.
8. Write `/tmp/xr/rehearsal.env` with mode 600 from the template below.
9. Pick two free loopback ports as `PORT` and `LOCAL_PORT`.
10. Unless `/tmp/xr/home/local-token` exists from this `rehearsal.env`, issue a local session with the issue command below.
11. Start the transient unit with the `systemd-run` command below.
12. Wait until `curl http://127.0.0.1:PORT/health/ready` answers 200 and the latest repair event is `lexical_repair_finished`.
13. Run `/tmp/xr/launcher/bin/python -m exomem.service_upgrade --runtime-dir /tmp/xr/managed --status` and confirm `phase: ready` with no pending transition.
14. In a second terminal, start the probe loop below.
15. Note the time, then switch with the `service_upgrade` command below.
16. Read `last_transition.handoff` in the printed result.
17. Poll `/health/ready` and read the log events below for 45 minutes after the noted time.
18. Count failed reads and failed writes with the check command below; expect 0 and 0.
19. Stop the unit: `systemctl --user stop exomem-rehearsal.service`.
20. To rehearse the next release from this one, keep `/tmp/xr`. Otherwise remove it: `rm -rf /tmp/xr`.
Expect all four checks to pass in the 45 minutes after the noted time.
```sh
/tmp/xr/launcher/bin/python -c 'from pathlib import Path; from exomem.state_paths import vault_state_key; print(vault_state_key(Path("/tmp/xr/vault")))'
```
The service writes its state while you copy it. The state copy command copies
each SQLite database through SQLite's backup API, so each copy is consistent. It
tells a database by its file header, skips SQLite's side files and rebuild
temporaries (both `*.rebuild-*` and `.graph-rebuild-*`), and skips files that
disappear during the copy. A rebuild temporary left by a crash can hold a hot
journal, and a read-only open of it fails:
```sh
/tmp/xr/launcher/bin/python -I - LIVE_STATE_DIR /tmp/xr/state/KEY <<'PY'
import contextlib, shutil, sqlite3, sys
from pathlib import Path
src, dst = Path(sys.argv[1]), Path(sys.argv[2])
for path in sorted(src.rglob("*")):
target = dst / path.relative_to(src)
try:
if path.is_symlink():
continue
if path.is_dir():
target.mkdir(mode=0o700, parents=True, exist_ok=True)
continue
if path.name.endswith(("-wal", "-shm", "-journal")) or "rebuild-" in path.name:
continue
with path.open("rb") as handle:
database = handle.read(16) == b"SQLite format 3\0"
if database:
live = sqlite3.connect(path.resolve().as_uri() + "?mode=ro", uri=True, timeout=120)
copy = sqlite3.connect(target)
with contextlib.closing(live), contextlib.closing(copy):
live.backup(copy)
continue
shutil.copy2(path, target)
except (FileNotFoundError, sqlite3.OperationalError):
if path.exists():
raise # a locked or unreadable database must stop the copy, not shorten it
target.unlink(missing_ok=True)
PY
```
Write the environment file from this template, not from a copy of the live
file. It gives the copy its own paths and leaves out every variable that reaches
a live or external system. Replace only the marked live values, so the timings
match, and leave out a marked line that the live file does not set. Use the live
model cache path for `HF_HOME`, or for `HF_HUB_CACHE` when the live file sets
that; the hub stays offline:
```sh
EXOMEM_VAULT_PATH=/tmp/xr/vault
EXOMEM_STATE_ROOT=/tmp/xr/state
EXOMEM_LOG_DIR=/tmp/xr/logs
EXOMEM_WRITER_LEASE_STATE_DIR=/tmp/xr/lease
HOME=/tmp/xr/home
XDG_STATE_HOME=/tmp/xr/home/.local/state
XDG_CONFIG_HOME=/tmp/xr/home/.config
XDG_DATA_HOME=/tmp/xr/home/.local/share
EXOMEM_REST_API_KEY=NEW_RANDOM_VALUE
EXOMEM_DISABLE_MEDIA_EXTRACTION=1
EXOMEM_ASR_PREWARM=0
EXOMEM_DISABLE_CLIP=1
HF_HUB_OFFLINE=1
HF_HOME=LIVE_VALUE
EXOMEM_PRELOAD_MODELS=LIVE_VALUE
EXOMEM_STANDBY_WARM_SECONDS=LIVE_VALUE
EXOMEM_COLD_START_SECONDS=LIVE_VALUE
EXOMEM_LOCAL_PORT=LOCAL_PORT
EXOMEM_GITHUB_USER_ID=LIVE_VALUE
EXOMEM_GITHUB_USERNAME=LIVE_VALUE
EXOMEM_JWT_SIGNING_KEY=NEW_RANDOM_VALUE
```
Generate each `NEW_RANDOM_VALUE` separately, for example with `openssl rand -hex 32`.
Keep `rehearsal.env` with a kept copy. A new signing key invalidates
`/tmp/xr/home/local-token`, so delete that file when you change the key.
When the live file sets no owner identity, use any numeric id and any login.
The last four lines let the copy's own local session call MCP tools on
`LOCAL_PORT`, so the read and write checks run as the owner. The signing key is
new, so no live token is valid on the copy.
The issue command runs in a subshell from the empty working directory, so a
`.env` file elsewhere cannot override the copy's signing key or storage:
```sh
( cd /tmp/xr/cwd && set -a && . /tmp/xr/rehearsal.env && /tmp/xr/launcher/bin/exomem auth issue-local --client rehearsal --output /tmp/xr/home/local-token )
```
The probe loop sends one MCP read and one MCP write every 30 seconds for 50
minutes, which covers the 45 minutes after a switch made within 5 minutes of
its start. It writes one line per call, `TIME read|write RESPONSE`, and each
write records a disposable episode in the copy. The `save_episode` tool
ships in 0.149.0 and later, so rehearse from 0.149.0 or a newer release.
Run the loop:
```sh
TOKEN=$(cat /tmp/xr/home/local-token); END=$(( $(date +%s) + 3000 )); : > /tmp/xr/calls.log
while [ "$(date +%s)" -lt "$END" ]; do
NOW=$(date -u +%H:%M:%S)
for CALL in "read {\"name\":\"search_memory\",\"arguments\":{\"query\":\"rehearsal probe\"}}" "write {\"name\":\"save_episode\",\"arguments\":{\"subject\":\"Rehearsal probe\",\"summary\":\"Disposable rehearsal write at $NOW UTC.\",\"worked_on\":[\"rehearsal probe\"]}}"; do
BODY=$(curl -s -m 120 -X POST "http://127.0.0.1:LOCAL_PORT/mcp" -H "authorization: Bearer $TOKEN" \
-H 'content-type: application/json' -H 'accept: application/json, text/event-stream' \
-d "{\"jsonrpc\":\"2.0\",\"id\":1,\"method\":\"tools/call\",\"params\":${CALL#* }}" | tr -d '\n')
printf '%s\n' "$NOW ${CALL%% *} ${BODY:-no-response}" >> /tmp/xr/calls.log
done
sleep 30
done
```
The check looks for success, not for each kind of failure. A read passes
when its line holds `"isError":false` and no `"success":false`. A write passes
when its line holds `"state":"committed"`. A refused call still returns a
normal `"result"`, so the absence of an error is not proof that a call
succeeded. A search refused while an index rebuilds is one example. The check
command prints the failed reads, then the failed writes; expect both to be 0:
```sh
awk '$2 == "read" && (!/"isError":false/ || /"success":false/)' /tmp/xr/calls.log | wc -l; awk '$2 == "write"' /tmp/xr/calls.log | grep -vc '"state":"committed"'
```
The worker also inherits the user manager's environment. Run `systemctl --user
show-environment` first, and add `-p UnsetEnvironment=NAME` to the unit for each
variable there that reaches a live or external system.
```sh
systemd-run --user --collect --unit=exomem-rehearsal.service -p Type=simple -p MemoryMax=MEMORY_LIMIT -p KillMode=control-group -p SendSIGKILL=yes -p TimeoutStopSec=30 -p EnvironmentFile=/tmp/xr/rehearsal.env -p WorkingDirectory=/tmp/xr/cwd /tmp/xr/launcher/bin/python -m exomem.service_manager serve --runtime-dir /tmp/xr/managed --worker-python /tmp/xr/launcher/bin/python --host 127.0.0.1 --port PORT --unit-name exomem-rehearsal.service
```
```sh
HOME=/tmp/xr/home UV_CACHE_DIR=LIVE_UV_CACHE EXOMEM_DISABLE_UPGRADE_HOOK_REFRESH=1 /tmp/xr/launcher/bin/python -m exomem.service_upgrade --runtime-dir /tmp/xr/managed --package-version VERSION --profile standard
```
`UV_CACHE_DIR` keeps staging on the existing package cache instead of the empty
home. For a candidate that is not published, pass `--wheel` and
`--source-revision` in place of `--package-version`, as for a live upgrade. The
candidate stages under `/tmp/xr/releases`, never under the live service root.
The log is JSON lines. Read these events and messages after the noted time:
- A `lexical_repair_started` event with `fields.kind` `full` starts a full
repair. Its `lexical_repair_finished` event must report `result` `published`.
`targeted` and `catch_up` are the usual delta repairs.
- `standby built a detached lexical catalogue` and `adopted retrieval catalog
healed=` show that promotion adopted the standby's catalog.
- `managed retrieval catalog delegated to background repair` or `maintained
retrieval catalog warm-up failed` explain a slow or failed adoption.
The rehearsal runs without other writers. During a live full repair, a bulk
writer outside the service makes every publication of that repair abort. Pause
bulk writers for the live install window.
The rehearsal does not cover a collection store schema change or the move of
file collections into the store. Both need the writer lease coordinator, and
the copy runs without one so that it cannot reach the live coordinator.
## Release retention
After the target is confirmed active, the operator removes the release
directories that the service no longer needs. It keeps the serving release, the
release that the promotion replaced, and every release that `active.json` or
`transition.json` in the runtime directory records. The launcher venv lives
outside `releases/` and is never removed. The operator removes only directories
that the service user owns, that are not symlinks, and whose names are 32
lowercase hexadecimal characters. It keeps every other entry. An unreadable or
malformed manager record stops retention before it removes anything. A removal
failure or an interrupt never fails the operator's result. An interrupted
`scripts/upgrade.sh` stops itself, but the promoted release keeps serving: check
`--status`, then run `scripts/upgrade.sh` again to finish the CLI sync.
The manager's records are the only proof of use. A process outside those
records is not protected, and a later upgrade can remove the release it runs.
Never run a canary, a hand-pinned unit or a script from a release interpreter.
Give each such process its own independent interpreter.
`scripts/upgrade.sh` prints one summary line from the `release_retention`
report in the upgrade result:
```text
Release retention: removed 3 freed_bytes 2147483648 failures 0 skipped no
```
`removed` counts the deleted directories, and `freed_bytes` is the allocated
space they released. `at_least` before that number means that a failed or
interrupted removal freed space it could not count. `failures` counts the
directories it could not remove; the next upgrade tries them again. `skipped`
names why retention stopped early, for example `interrupted` or an unreadable
record.
`scripts/upgrade.sh` runs the `exomem.service_upgrade` module installed in the
launcher venv. Retention therefore reaches a host only after an offline
launcher upgrade, described above. Until then, the summary line reads
`Release retention: this operator predates release retention`. To remove unused
releases before the launcher upgrade, run the operator from a newer interpreter.
It stages and promotes a worker release like `scripts/upgrade.sh`, but it skips
the CLI sync and the managed-install manifest helpers:
1. Install this release in an interpreter outside the service, such as a tested checkout's venv.
2. Read the runtime directory from the rendered unit's `--runtime-dir` argument.
3. Read the profile from `service_profile` in `~/.config/exomem/managed-install.json`.
4. From the checkout root, run `<operator-python> -m exomem.service_upgrade --runtime-dir <runtime-dir> --profile <profile>`.
5. Read the printed JSON result: it contains a `release_retention` report.
## The standby sequence
A worker release is no longer replaced cold. The supervisor first spawns the
candidate as a **standby** beside the worker that is still serving. The standby
binds its own private socket, **proves** the maintained lexical catalog current
read-only, warms the rebuildable in-memory caches, **adopts** the published graph
snapshot, builds the semantic corpus context, and loads models when preload is
allowed. It takes no writer
lease, publishes nothing, schedules no drain, media or watcher work, and owns no
descendants. In particular it never reconciles or requests repair of the lexical
catalog: that is a publication, and the worker still serving is this vault's
single repair owner. A catalog that is not current leaves `lexical` waiting until
that owner publishes one — which the warm budget covers and its expiry records.
The old worker serves throughout.
The exception is a catalog the candidate can never prove: one whose schema or
semantic identity is an earlier release's (a release that bumps the lexical
catalog schema, for instance). The serving worker keeps that catalog current for
its own code, so waiting would only burn the warm budget. The standby instead
builds its own release's catalog into a lexical rebuild temp beside the live one,
holding the temp's advisory lock for its whole life. The build takes no
publication barrier and changes no byte the serving worker owns; `lexical` is
ready once it exists, and a stopped standby cancels and removes it. At promotion,
once it owns the vault, the standby adopts that temp in place of the live catalog
and records `lexical_catalogue: adopted` (or `rebuild-after-promotion` when it
could not, leaving the ordinary repair to rebuild). The promoted worker's warm
then re-parses exactly the pages whose file signature changed since the build and
admits retrieval, without a whole-catalog rebuild. Adoption replaces a private
state file, so promotion also rebuilds the private-identity inventory once, in
the background, off the request thread. A catalog at the current schema that
merely lags is still the serving worker's to repair, as above.
Only when the standby reports cutover readiness does the supervisor pause
ingress, drain, stop the old worker and prove its descendants exited, run the
offline migrator if the target declares a state migration, promote the standby
over its private control surface, and resume. The unavailable window is the
drain plus the promotion, not a cold start.
Two budgets divide that sequence, at the stop. Everything up to and including
the migrator runs under the cutover budget, because until the old worker is
signalled the supervisor can still abandon the upgrade and give admission back
to a worker that is serving. Once it has stopped there is no such worker, so
the wait for the replacement to report ready — the promoted standby, or a cold
one-worker start — runs under the cold-start budget instead. Ending that wait
early buys nothing: it turns "unavailable for another minute" into "unavailable
until an operator resumes", and the resume waits on the same replacement. A
candidate that *exits* before readiness, or that answers with a different
release, still fails immediately; only a live candidate on the right release is
waited out.
`/health/ready` reports this as a `cutover` block beside the serving `status`:
```json
{
"status": "ready",
"cutover": {
"components": {
"lexical": "ready",
"graph_snapshot": "waiting",
"semantic_corpus": "waiting"
},
"cutover_ready": false,
"standby": true,
"carried_from_standby": []
}
}
```
Adoption is what makes the promoted worker's first governed write incremental:
it proves the inherited sidecar against disk and makes that checkpoint this
process's delta origin, instead of the whole-vault pass a replacement used to
pay for a lineage it could advance. The `cutover` block reports it as
`adoption: {"residue": N, "reason": "adopted"}`. A non-zero `residue` is an
adoption that succeeded *and* owes the drain that many pages — the deferred
writes the outgoing worker left behind. The standby only *records* that debt:
queueing the repair and withdrawing the availability marker are writes to state
the serving worker still owns, so they happen at promotion, and the handoff
record reports `residue_applied`. From then on, reads that require a current
projection refuse until that repair lands. A refused adoption names its
reason instead (for example `residue_exceeds_drain_limit`), leaves
`graph_snapshot` waiting, and the candidate is discarded on budget expiry.
`embeddings` joins `components` only when the process's mode and overrides allow
a model preload (`EXOMEM_PRELOAD_MODELS=1`). A standby answers its own probe as
serving-ready while it warms; `cutover_ready` is what the supervisor polls, and
`components` is what an operator reads to see which component a candidate is
waiting on.
`semantic_corpus` is in the set because the gate that admits governed writes
waits on it. Building it is a read — it walks and parses Markdown and caches in
memory, takes no lock and publishes nothing — so a standby may do it beside the
worker that still owns the vault. On the 0.85.0 upgrade it was not in the set:
the cutover itself took 1.6 s and the promoted worker then refused every
governed write for about thirty seconds with
`MUTATION_WARMING warming_component=semantic_corpus` while it built a corpus its
own standby could have built. A cutover that hands back a worker which refuses
writes has moved the outage, not removed it.
### What a promoted worker does not repeat
A promoted worker runs its own warm-up, and subtracts from it whatever its
standby already finished in the same process and promotion either re-verified or
cannot invalidate. `carried_from_standby` on the `cutover` block and on the
promotion record names what was subtracted, and the worker's `warm complete` log
line repeats it. The graph handoff is carried only when the promotion verdict is
`current`: the snapshot the standby proved is still on disk, or promotion
re-proved the one the old worker published after that proof. An `advanced`,
`unproven` or `rebuild-after-promotion` verdict leaves it out, and the adoption
runs again in full while the writers' gate refuses writes. What remains after a
carried promotion is the retrieval catalog check and any model the standby's
preload policy did not load. When promotion adopted a catalog the standby built
detached, `carried_from_standby` names `catalogue_handoff` and not `lexical`: the
promoted warm re-parses the pages that changed since that build in place of the
whole-catalog repair, then warms the lexical caches the standby deliberately
did not build over the old catalog.
A worker that starts cold — no standby, no promotion — carries nothing and runs
every warm-up step, unchanged. If a cutover is fast but the writes after it are
slow or refused, `carried_from_standby` is the first field to read: an empty
list after a promotion means the worker is paying the whole warm again.
### Budgets
| Budget | Default | Override |
| --- | --- | --- |
| Standby warm | 300 s | `EXOMEM_STANDBY_WARM_SECONDS` in the unit's environment file |
| Cutover (pause to the migrator) | 40 s | supervisor `transition_timeout` |
| Drain of active finite requests | 30 s | within the cutover budget |
| Replacement readiness after the stop | 300 s, never under 120 s | `EXOMEM_COLD_START_SECONDS` |
| Detached stream reattachment | 40 s | ingress `reattach_budget` |
The standby warm budget (300 s unless the unit sets it) is the ceiling on how
large a vault this handoff covers: the detached build measured 54-60 s at
4.6k pages, so a vault several times larger needs a longer
`EXOMEM_STANDBY_WARM_SECONDS` before a schema-bumping upgrade.
The cold-start budget sizes every wait that begins after the previous worker
has stopped: the supervisor's own start, the promotion of a standby, and the
cold one-worker start — in a fresh upgrade and in `--resume` alike. It is one
window, so an operator who lengthens it for a large vault lengthens all of
them. A promoted worker that is still building a component it could not warm as
a standby — a retrieval catalog delegated to background repair, for instance —
answers `/health/ready` with `not_ready` for as long as that repair takes, and
that is what this budget has to cover.
Admission is unchanged through the longer wait: ingress stays paused, and a
request that outlasts its own 45-second queue budget is answered explicitly as
undispatched rather than held. It is never left queued without a bound.
A candidate that does not reach cutover readiness inside the warm budget is
stopped, the handoff record names the component it was waiting on, and the
existing worker keeps serving. The upgrade then falls back to the one-worker
sequence — reported in the handoff record, never silent. A release that predates
standby mode reports `"standby": "unsupported"` and takes the same path.
### Polling a transition
`upgrade` is acknowledged, not awaited: the supervisor answers immediately with
`{"ok": true, "accepted": true, "transition": "<id>"}` and runs the sequence in
the background, because a standby warm is minutes long and holding the control
connection open for it would turn any client read timeout into a false failure
while promotion proceeded regardless. Poll `--status` for the outcome: it
carries `transition` (in flight) and `last_transition` (the finished record,
including the handoff). `scripts/upgrade.sh` does this for you, with a deadline
covering the warm, the cutover and the cold-start budgets — a client that gives
up sooner would report a failure for a handoff that was still going to succeed.
### The migration record
The offline state migrator runs only when the staged target declares a state
migration: the supervisor compares the descriptor set the candidate package
requires with the set the vault's state manifest was published with. When they
match and the manifest is complete, the step is recorded as skipped:
```json
{"handoff": {"standby": "ready",
"migration": {"state": "skipped", "reason": "declared_none"},
"promotion": {"snapshot": "current",
"revalidated": true, "reproved": false}}}
```
`handoff.unavailable_ms` is the window nobody was served in — pause to resume —
and is the number to compare across releases. `handoff.ready_after_ms` is the
part of it spent waiting for the replacement. It is measured from the old
worker's stop, so it includes the offline migrator's own run when a migration
ran; `migration.state` says whether it did. When it is close to
`unavailable_ms`, the cutover itself was cheap and the replacement's own warm
is what the outage was.
Both fields appear on a failed handoff too, and there `handoff.replacement_waiting`
carries what the replacement last reported waiting on — a readiness reason such
as `retrieval_unavailable`, with the lexical repair phase when there is one, or
the cutover component it was still warming. Together they separate a
replacement that burned the whole window on a background repair from one that
died two seconds after the stop, which need different responses. Both are
recorded from the readiness contract's own fixed vocabulary and are
observations only: no budget, wait or outcome depends on them. A failure before
the old worker stopped carries neither, because there was no replacement to
measure.
`replacement_waiting` is a separate field from `handoff.waiting`, which keeps
its own meaning: the component a *standby* was warming when it was discarded.
An upgrade can record both — a candidate discarded on its warm budget, then a
one-worker replacement that never reported ready — and that combination is
precisely the one worth reading closely, so neither value overwrites the other.
`migration.state` is `ran` with the reason (`descriptors_changed`, or the
manifest state that was not complete) when it runs. `promotion.snapshot` is
`current` when the checkpoint this process adopted is the one on disk,
`advanced` when it moved and promotion could not re-prove it, and
`rebuild-after-promotion` when the migrated re-proof failed — in that last case
promotion still proceeds and the coalesced rebuild path owns the repair. It is
`unproven` when the standby's proof declined the snapshot outright, for example
because the graph sidecar has an older schema than the build; promotion then
records the decline reason in `promotion.proof_declined`, and the promoted
worker runs the whole graph handoff.
`promotion.revalidated` says promotion re-checked the snapshot at all;
`promotion.reproved` says it re-ran the source proof. With a migration it re-runs
the whole proof, because the migrator may have rewritten what the standby proved.
Without one, promotion compares the checkpoint pair — a real re-validation
against the one writer the sequence has not already excluded. A whole source
proof would add seconds to the one window this sequence exists to shorten.
When the pair moved, the old worker published after the standby's proof, which
is the usual case under live writes. Promotion then re-proves the snapshot on
disk from the page reads of the standby's proof. Each of its one or two rounds
walks the vault once and checks the stat of every page. It reads again only the
pages that moved since, where the stat generation binds a page's bytes. On
Windows, and for a page written less than 2 s before its read, the check reads
the page's bytes again. An adoption records `current` with `reproved: true`.
The supervisor sends the budget of its promotion request as `request_seconds`
in the request body, at most 30 s. The re-proof may spend 40% of that budget,
so at most 12 s, and it stops at the first page it reaches after that. A decline
records `advanced` with the reason in `promotion.reproof_declined`, for example
`proof_deadline_passed`, and the promoted worker runs the handoff in full. On a
synthetic 13,000-page vault, on a host at load 8, the re-proof of one moved
page took 2.2 to 2.6 s.
Requests wait at the supervisor during the re-proof, and none is refused. The
re-proof adds its own time to `ready_after_ms` and `unavailable_ms`, and up to
12 s when it declines. Without it, the promoted worker refused every write as
warming while it ran the handoff in full, about 100 s on a large vault.
The supervisor runs from the launcher venv, so a supervisor that predates
`request_seconds` stays in place after a seamless upgrade. It caps the request
at 10 s and sends no budget, and the re-proof then has 4 s. The 12 s bound takes
effect after the offline maintenance that changes the stable supervisor.
### A collection store schema change
A release that changes only the collection store's schema records `migration`
as `skipped` with the reason `declared_none`, because the store schema is not a
state migration. The new worker migrates the store itself once it acquires the
writer lease. Until then, the old worker keeps the lease and serves.
A release from before the schema change cannot open the migrated store. When
the vault's marker routes collections to the store, or the store records a
pending intent, the supervisor compares the
newest store schema that an upgrade target reads with the schema of the live
store. It refuses a target that cannot read the live store before the serving
worker stops, so that worker keeps serving. Releases 0.114.0 through 0.118.x read
only up to schema 10, so the supervisor refuses them on a store at schema 11.
0.119.0 and later read schema 11 and stay valid targets.
The supervisor runs this check, so the supervisor's release decides it. A
supervisor started on 0.120.0 or earlier does not have the check. After a
seamless upgrade replaces its worker, it still accepts 0.114.0 through 0.118.x
until the service unit restarts. After an export to files, no collection routes
to the store, so the check refuses nothing and an earlier release is the way
back.
Started by hand on a migrated store, a release from 0.114.0 through 0.118.x
cannot open it. On a vault with a collection store fence, every write on that
host then refuses with `WRITER_LEASE_REQUIRED`. Roll forward to recover.
0.119.0 changed the store schema, and 0.120.0 is the first release that migrates
a store from an earlier release. Back up the store with the earlier release
before the upgrade, as "A release that changes the collection store's schema" in
`docs/deployment.md` describes.
### A supervisor older than the vault's state
A worker records each optional state format that it uses in the vault's state
manifest, for example the schema of a migrated collection store. The supervisor
runs from the launcher venv, which a seamless upgrade does not replace. A
supervisor whose release predates such a format cannot read the state
manifest, so it cannot verify any upgrade target.
A supervisor that has this check reports the case under `upgrade_blocker` in
`--status`. The staging operator then refuses before it stages a release, and
the supervisor refuses an upgrade request with the same message. The message
names the supervisor's release and the formats that it does not recognize. The
current worker keeps serving.
A supervisor without the check refuses the same upgrade with "target
interpreter verification failed; current admission unchanged".
To recover in both cases:
1. Install the serving release into the launcher venv through the offline maintenance described above.
2. Run `bash scripts/upgrade.sh` again with the release you want.
The upgrade then runs, and `--status` shows `upgrade_blocker` as `null`.
### Moving file collections into the store at a handoff
From the release that enables `collections-migrate-v1`, a seamless upgrade can
move the vault's file Records and Planning collections into the collection
store. A target that keeps that release fence off declares no such migration,
and its upgrade runs as the sections above describe.
The supervisor decides whether to ask for this step, so the launcher's release
decides it. A seamless upgrade replaces only the worker. Upgrade the launcher
offline to a release with this step before you rely on it.
The step has four parts:
- The standby pre-imports every file collection into private staging under the
vault's state directory, as the last part of its warm. It takes no lease and
changes no vault file. It does this only when the supervisor tells it that
this handoff can run the step: the target declares the migration and the
operator credential file is usable. Otherwise it removes the staging that an
earlier warm left. Its cutover component, `collections_migration`, is ready
once the pre-import ran, however it ended. A pre-import that fails or finds
nothing to move leaves no staging.
- After the old worker stops, the supervisor runs the migrator only if the
target declares the migration, the operator credential file is usable, and
the vault holds the staging or a published store that the migration has not
finished.
- The migrator compares every collection with what the standby imported. When
nothing changed, it keeps the staging. When a collection changed, it imports
all of them again inside its budget.
- The migrator then publishes the store, cuts the coordinator's store fence,
records the compatibility descriptor, publishes the replica and installs the
marker.
The supervisor waits 15 s for the migrator. The migrator abandons a migration
that it has not sealed 9 s after the supervisor started it. The other 6 s are
for the steps after the seal, which always finish once they start. They took
about 3 s on a test vault with about 3,900 items, and they grow with the store.
#### The operator credential file
The migrator needs the coordinator's operator credential to cut the store
fence. The supervisor reads it from one file, and only when it starts the
migrator. It gives the credential to the migrator's environment alone, never
to the serving worker or the standby, and never on a command line.
1. Find the directory that holds the unit's `service.env`, usually `~/.config/exomem`.
2. Create the file `operator.env` in that directory.
3. Write one line in it: `EXOMEM_LEASE_COORDINATOR_OPERATOR_TOKEN=<operator token>`.
4. Run `chmod 600 operator.env`.
5. Make sure that the service user owns the file and that the file is not a symlink.
Do not add `operator.env` to the unit as an `EnvironmentFile=`, and do not put
the operator credential in `service.env`. Either way, every worker would get it.
The supervisor reads the file at each handoff, so a new or changed file needs no
restart. A missing file, an unreadable file, a file that is not a regular file,
or a file that another user can read or write means no migration. The release
still completes, in file mode.
#### What the handoff record shows
`handoff.migration.collections` reports the step. It is absent when the target
declares no collections migration.
- `{"state": "skipped", "reason": "nothing staged"}`: the standby prepared
nothing, for example because every collection is already in the store.
- `{"state": "skipped", "reason": "operator credential file missing"}`: the
file is absent. The other reasons are `unreadable`, `is not a regular file`,
`is not private to its owner` and `holds no operator credential`.
- `{"state": "skipped", "reason": "target migration probe failed"}`: the
target release could not report whether it has the migration. Check that
release before you upgrade to it again.
- `{"state": "ran", "outcome": "migrated"}`: the vault now routes every
migrated collection to the store.
- `{"state": "ran", "outcome": "abandoned"}`: the migrator published nothing.
- `{"state": "ran", "outcome": "blocked"}`: a collection failed its proof, and
the migrator published nothing.
- `{"state": "ran", "outcome": "pending"}`: the migrator published the store
but did not finish, and the supervisor's rerun did not finish it either. The
service runs, and no write is lost. See below.
- `{"state": "ran", "outcome": "unsafe"}`: the migrator stopped while the live
store was set aside, or left a store or marker that cannot be read. The
upgrade fails. See below.
- `{"state": "ran", "outcome": "timed_out"}`: the supervisor stopped the
migrator at the end of its 15 s in a handoff that also migrated state, which
the stop may have cut. The upgrade fails. See below.
An outcome can carry a `code`, which names the refusal or error that the
migrator ran into. `"code": "timed_out"` means that the supervisor stopped the
migrator at the end of its 15 s, and the outcome is what the stopped migrator
left: `abandoned`, `pending`, `migrated` or `unsafe`. After a rerun, `first_attempt` holds the outcome and code of
the attempt before it. The migrator's full report goes to the service journal. For
an abandoned migration, `abandoned_because` is `deadline`, `live_store_changed`
or `refused`. For a blocked one, the report names the collection and the failed
check.
#### An abandoned migration
An abandoned migration publishes nothing. The new release serves the vault in
file mode, and the next handoff tries the migration again. A vault whose store
changes during every handoff, or whose collections take longer than the
budget, migrates offline instead:
1. Stop the Exomem service.
2. Run `(set -a; . ~/.config/exomem/operator.env; exomem collections migrate --vault <vault> --apply)`.
3. Start the Exomem service.
Step 2 reads the operator credential from the protected file in a subshell, so
the credential never enters your shell history or your login shell. Use the
directory that holds your `operator.env` if it is not `~/.config/exomem`. The apply reports
`marker_admitted` when the migration is finished. For a blocked collection,
fix its files as the report says, then repeat the steps.
#### A pending migration
When the migrator publishes the store and then fails before it installs the
marker, the supervisor runs it once more before it starts the new release. The
migrator that failed keeps its lease for the lease time-to-live,
`EXOMEM_WRITER_LEASE_TTL`, 30 s by default, so the supervisor waits that long
first. The rerun adds the time-to-live and up to 15 s to the handoff. When it
finishes the migration, the outcome is `migrated` and the new release serves
the store. The supervisor tries only once, because each attempt costs a
time-to-live of downtime, and a fault that persists does not clear on a second
try.
A `pending` outcome means that the rerun did not finish the migration either.
The supervisor starts the new release anyway, and the service answers reads.
On a first migration, what happens to writes depends on how far the migrator
got:
- When the migrator had cut the coordinator's store fence, the fence refuses
every write until the migration finishes.
- When it failed before the cut, the fence is not enrolled. The vault stays in
file mode, and writes go to the files, which are still canonical.
In both cases no write is lost. On a vault whose store was already live, the
next migration restores or supersedes the published store as the offline apply
does.
The next handoff finishes the migration. To finish it sooner, fix the cause
that `code` names, then do the offline steps under "An abandoned migration"
above. Either way, the migrator first compares every collection with the files
that it imported. When a file changed meanwhile, for example through a sync
client, it imports the collections again before it installs the marker. A
migration that runs within the time-to-live after a failed one is refused with
`COLLECTION_STORE_SERVICE_ACTIVE` and stays `pending`.
#### A migrator that timed out or left the store unsafe
For the outcomes `timed_out` and `unsafe`, the supervisor reports
`recovery-required`. A stopped migrator that left a published store gets the
same rerun and the same `pending` as one that failed.
1. For `timed_out`, wait for the lease time-to-live, 30 s by default.
2. For `unsafe` with no `collections.sqlite` in the vault's state directory,
rename the most recently modified `collections.sqlite.superseded-<suffix>`
there back to `collections.sqlite`.
3. Run the private `--resume` command above.
The resume runs the migrator again. The result is `migrated`, or `pending` as
described above.
### The service environment file
A worker child is spawned with every `EnvironmentFile=` the unit declares,
re-read at spawn time and overlaid in order on the inherited environment. systemd reads that file once,
when the supervisor starts, so without this an edited service environment would
reach a worker only after a full supervisor restart — which is exactly what a
seamless upgrade avoids. The supervisor's own environment is never changed.
This is how `EXOMEM_PRELOAD_MODELS=1` and `EXOMEM_STANDBY_WARM_SECONDS` reach
the next worker: edit `service.env`, then run the ordinary
`scripts/upgrade.sh`. A file that cannot be read is reported in the handoff
record as `environment: ["unreadable: <name>"]`, not just logged.
### Streams through promotion
A long-lived client `GET` stream detached for cutover is reattached with
backoff inside the reattachment budget while the standby is being promoted,
instead of being closed on the first non-success. A replacement that refuses the
original credential (401 or 403) is a definitive answer and still closes the
stream.
## When the service starts
The supervisor opens the public listener, the local listener and the control
socket before it starts the first worker. Until that worker reports ready, the
phase is `starting`, and every call gets an immediate HTTP 503 with the body
`Request not dispatched`. No call executes and none waits in the queue, because
a cold warm, such as a full catalogue rebuild, can outlast any queue budget.
When the first worker reports ready, ingress serves calls and the phase becomes
`ready`. A first worker that fails to start still stops the supervisor with a
failed exit status.
## When the worker crashes
A worker that exits outside a transition no longer stops the supervisor. The
public listener, the local listener and the control socket stay open, and the
supervisor replaces the worker in place:
1. The supervisor sets the phase to `restarting` and pauses ingress. New calls
queue under the admission budgets below.
2. The supervisor proves that the exited worker and every descendant it left
have exited. A child that called `setsid()` is included.
3. The supervisor starts the active release under the cold-start window.
4. When the replacement reports ready, ingress resumes and the phase returns to
`ready`.
A call that the dead worker was answering gets the retryable error described
in the managed-service spec. A call that arrives during the restart is held for
up to the 45-second queue budget. If the replacement reports ready inside that
budget, the replacement serves the call. Otherwise the call is answered as not
dispatched and can be retried. A production worker takes about 40 to 80 seconds
to warm, so a call that arrives early in the restart usually gets the
not-dispatched answer.
The supervisor makes one attempt for each exit, and restarts a serving worker
in place at most 3 times in any 10 minutes. The supervisor stops with a failed
exit status in these cases:
- A fourth exit falls inside the window.
- A descendant of the exited worker is still alive after 10 seconds.
- The replacement exits before it reports ready, or does not report ready
inside the cold-start window.
- The restart fails with an unexpected error.
systemd then cleans the unit's cgroup and applies `Restart=`, as it did for
every crash before in-place restarts. `--status` reports the restarts inside
the window, the limit, the window and the last exit status under
`worker_restarts`. The journal records each restart with the worker's exit
status. A negative status is the signal that killed the worker.
The supervisor runs from the launcher venv, so a worker handoff does not
change this behaviour. It takes effect after the offline maintenance that
changes the stable supervisor, described above, which restarts the unit once.
## Admission budgets
Admission during handoff allows at most 64 queued requests, 64 MiB total and
32 MiB per request. A queued request has 45 seconds from reservation, including
body intake. The cutover has a 40-second deadline, with up to 30 seconds for
active finite requests to drain. If draining expires, the old worker keeps
serving. Requests refused before dispatch report that they were not executed;
requests already dispatched are never replayed.
A queued request's 45 seconds is deliberately shorter than the replacement
readiness budget, so no client waits on a handoff without an answer: a request
that outlasts it is told explicitly that it was not dispatched and can be
retried. Clients and intermediaries need timeouts longer than the queue budget
to wait through an ordinary handoff; none of them should wait out a slow
replacement's whole warm.
## Independent read-only recall monitoring
The installed `exomem.recall_canary` module runs public MCP cases without a source checkout.
Its `--check availability` mode checks one response per case by default.
Each configured tool needs a successful positive case with its declared references and material.
An expected abstention checks only its declared negative case.
The report retains latency results, even when the availability check passes with insufficient latency samples.
An `expected` object uses either the existing indexed fields or `outcome` plus `response_schema`, never both.
The schema format uses JSON Schema Draft 2020-12 against the complete parsed response.
Use one `contains` witness to require source identity and nonblank material from the same result or parent.
Use `prefixItems` with `minItems` when the source contract requires rank.
Local schema references work; the observer never retrieves external schema resources.
An invalid schema prevents connection. A response mismatch fails the case with `unexpected-response`.
An evaluation error reports `invalid-expectation`, fails the check and leaves availability incomplete; it does not establish a service outage.
An independent author derives each key from source truth and checks wrong identity, missing identity, blank material and material under another parent.
Schema syntax validation cannot prove those obligations.
Keep frozen keys and earlier failures unchanged; replacement keys start a new declared series without exposing unused cases.
The default `--check acceptance` mode retains the complete warm programme contract.
It requires its sample count, warm proof, known contention and verified deployment identity.
The observer package identity describes the caller, while the deployment identity comes from the operator.
Neither identity proves the other.
The `exomem.canary_systemd` adapter installs only `exomem-recall-canary.service` and `exomem-recall-canary.timer`.
The oneshot has a finite timeout, and systemd prevents concurrent executions of that service.
It has no runtime dependency, restart policy or repair action.
An observer update requires an explicit install; worker upgrades never change its interpreter.
1. Choose an independent interpreter with a noneditable observer wheel, so its version stays pinned through worker upgrades.
2. Prepare owner-only environment and case files, so credentials and source truth stay outside repositories.
3. Run the adapter's `install` command with its configuration options; the timer stays disabled.
4. Run the adapter's `enable` command; systemd loads the named credential before it enables the timer.
5. Read the adapter's `status` command or doctor's `recall.canary` check; the latest evidence appears with its state.
The install options are `--observer-python`, `--environment-file`, `--mcp-url`, `--cases`, `--token-env` and `--report-file`.
Paths must be absolute; the report directory must be owner-only.
The adapter uses systemd's native loader for the existing environment file and never copies the credential.
A missing credential leaves the timer unconfigured.
An unavailable target or authentication failure becomes a failed observation while the timer stays scheduled.
`status` reads the unit, execution properties and latest atomic report without probing the target.
It distinguishes absent, disabled, unconfigured, failed, stale and healthy availability evidence.
An old successful report cannot hide a later failed execution.
Changing the endpoint or case truth invalidates the old report.
The adapter's `uninstall` command removes only these units and retains the private configuration and report.
Failures reach the failed unit, local journal and doctor.
They do not send a push notification; delivery beyond those local surfaces remains open.