docs/deployment.md
Rama main · réplica de lectura del repositorio original.
<!-- authority:non-specification -->
# exomem — remote deployment
This guide covers the **remote tier**: running exomem as an always-on HTTP service
behind a public HTTPS endpoint so you can reach your vault from **claude.ai** on
the web or mobile as a custom connector. The local-only path (Claude Code over
stdio, no cloud) is in [../QUICKSTART.md](../QUICKSTART.md); start there if you
don't need mobile access.
This personal remote profile is still one user and one vault. The managed hosted
service uses the same single-vault runtime as a private tenant cell behind a
separate shared control plane; it is not made multi-tenant by pointing several
accounts at the server configured below. Exomem Cloud operators should also
read the Cloud runbooks in [runbooks/](runbooks/) and the
[Substrate control-plane contract](substrate-control-plane-contract.md).
Throughout, replace `<your-host>` / `example.com` with your own hostname.
For the guided ≤15-minute bring-up, start with
[remote-quickstart.md](remote-quickstart.md); this document is the reference.
For structured logs, metrics, and diagnosing a failure after the fact, see
[observability.md](observability.md).
For host-configured content ceilings between owner clients, see
[connector boundaries](connector-boundaries.md).
## Architecture
```
┌──────────────────┐ HTTPS ┌──────────────────────────────┐
│ claude.ai │ ─────────▶ │ public edge (CDN / tunnel) │
│ (mobile/web │ bearer │ kb.example.com │
│ backend) │ │ TLS terminated at edge │
│ 160.79.104.0/21 │ └──────────────────────────────┘
└──────────────────┘ │
│ tunnel
▼
┌────────────────────────────────────┐
│ host: Linux / Windows │
│ │
│ FastMCP @ 127.0.0.1:8765 │
│ GitHub OAuth (single user) │
│ ↓ │
│ MCP tools (find, get, note, …) │
│ ↓ │
│ <vault>/Knowledge Base │
└────────────────────────────────────┘
```
**Why a public endpoint, not a tailnet-internal one?** claude.ai's MCP client
fetches the connector URL *from Anthropic's cloud infrastructure* (egress range
`160.79.104.0/21`), not from your phone. A purely internal hostname is therefore
unreachable. The auth boundary is not network membership but **GitHub OAuth**,
locked down to a single GitHub login via a custom `SingleUserGitHubVerifier`
wrapping FastMCP's `OAuthProxy`. claude.ai discovers the OAuth endpoints at
`/.well-known/oauth-authorization-server`, registers itself via Dynamic Client
Registration at `/register`, and walks the standard authorize → token → use flow.
**Downtime.** A single-host deployment without an always-on box accepts downtime:
when the host is asleep, mobile writes fail with a connection error — fall back to
editing the vault directly (the local capture path). Run on a box that stays up
(or a cheap VPS) if you want reliable mobile access.
### Managed hosted topology
The managed service has a different ingress and ownership boundary:
```text
client
-> Substrate public auth / gateway / Home
-> authoritative account mapping + internal entitlement
-> one private Exomem cell
-> one tenant vault + isolated state/log/secret roots
```
The public gateway derives the destination from the authenticated account. No
public body, URL, query, cookie, or header may select a tenant, cell, private
endpoint, credential, or vault path. Each cell accepts only its unique private
gateway/operator identity and exposes content-free readiness. Optional embedding
and media workers start only from provider-neutral cell policy and soft-fail to
durable capture and lexical recall.
Deployment supplies private networking, process/container isolation, encrypted
tenant volumes, secret injection, backup object storage, and KMS integration.
That is encryption at rest and owner-scoped access—not zero-knowledge or
end-to-end encryption. Search and indexing require plaintext in cell memory, so
operators must minimize and audit volume access and keep content, queries, paths,
and credentials out of operational logs.
The gateway and cell negotiate a versioned protocol and are tested as a pair.
Optional response fields may be added compatibly; command, parameter, or stable
error removal requires a coordinated rollout. Pin the cell release per tenant,
take a verified canonical snapshot before a file-format migration, and canary new
gateway/cell pairs. Rollback stops routing, drains the affected cell, and starts
the previous protocol-compatible image against the same vault. It never rolls
canonical Markdown/media backward; disposable sidecars can be rebuilt.
Paddle is not installed in, called by, or trusted by an Exomem cell. Substrate
owns Paddle checkout, webhooks, portal, reconciliation, and the provider-neutral
entitlement projection consumed before routing and at cell startup. Paddle MCP
is an operator/development aid for that control-plane lane, not production
runtime infrastructure.
## 1. Install dependencies
```powershell
cd /path/to/exomem
# Install Python deps (creates .venv automatically).
# --extra embeddings pulls torch + sentence-transformers for HYBRID search.
# --extra media pulls faster-whisper + Pillow + pymupdf + markitdown for
# SERVER-SIDE media extraction (auto transcribe/OCR/parse uploaded binaries →
# searchable). On Windows the [media] extra also pins the CUDA-12 runtime
# (cublas/cudnn/cudart) that ctranslate2 needs alongside torch's cu132 build.
uv sync --extra embeddings --extra media
```
Media extraction needs two **system** tools (not pip-installable):
- **Tesseract OCR** (images): `winget install --id UB-Mannheim.TesseractOCR -e`.
The installer doesn't add it to PATH; the server auto-discovers it at
`%ProgramFiles%\Tesseract-OCR\`, or set `EXOMEM_TESSERACT_CMD`.
- **ffmpeg** is bundled by PyAV (pulled via faster-whisper), so audio/video decode
works without a separate install.
Verify the GPU media path: `uv run python scripts/verify-media-gpu.py`.
The `media` extra reads PDF, Word, Excel, PowerPoint, HTML, EPUB, OpenDocument
(text, spreadsheet, presentation) and RTF files. Plain text, email and calendar
files need no extra.
OCR detects each image's script with Tesseract's OSD model first. It then reads the
image with the installed script models for that script and the installed language
packs written in it. A page with too little text for detection reads with
`EXOMEM_OCR_DEFAULT_LANGS` (for example `eng+jpn`), or with Tesseract's default
English when the variable is unset. Tesseract 4.x does not name its model
directory, so on it OCR skips script detection and reads each image once in
Tesseract's default language. To read Japanese, install the `jpn` and
`jpn_vert` packs and the Japanese script models (Debian: `tesseract-ocr-jpn`,
`tesseract-ocr-jpn-vert`, `tesseract-ocr-script-jpan`, `tesseract-ocr-script-jpan-vert`).
OCR leaves each word with a confidence under `EXOMEM_OCR_WORD_CONFIDENCE_FLOOR`
(default `70`, scale 0 to 100) out of search. The sidecar keeps those words in a
`## Low-confidence OCR` section. Set the variable to `0` to keep every word searchable in each image that OCR reads from then on.
`EXOMEM_MEDIA_ENGINES` limits which engines run, for example `documents,ocr`.
Unset, a personal install runs every engine and a Cloud cell runs none. cellctl
renders the variable into a Cloud cell, for one cell or as the fleet default.
HEIC photos are not decoded; see `docs/runbooks/cloud-media.md`.
Lean / CPU-only boxes can skip all of this — set
`EXOMEM_DISABLE_MEDIA_EXTRACTION`; uploads still work, just without server-side
searchable-text extraction. CPU also works (no GPU needed), just slower — pick a
smaller Whisper model with `EXOMEM_WHISPER_MODEL=base`. The CUDA-12 wheels above
are Windows + GPU only, unused on CPU.
To make **existing** Evidence/media files searchable (one-shot back-fill; new
uploads are handled automatically), run:
```powershell
uv run python -m exomem backfill-media --dry-run # preview (writes nothing)
uv run python -m exomem backfill-media # do it (CPU or GPU)
```
It writes a sidecar if missing, extracts text (OCR/ASR/PDF), and CLIP-embeds
images. Idempotent. Flags: `--no-ocr` (sidecar + CLIP only), `--no-clip`,
`--vault <root>`.
Media discovery, binary hashing, extraction compute, and the usual
sidecar/transcript/failure index fanout run outside the global vault mutation
boundary; only the revalidated per-artifact canonical commit is fenced. Lock
telemetry can still show short named `background_media_clip_commit` and
`background_media_scene_frame_commit` holders while CLIP vectors or persisted
scene frames commit. Those are the measured residual paths, not a stuck scan.
## 2. Set up a public HTTPS URL
You need a hostname for step 3. Pick **one**:
**Option A — Cloudflare Tunnel.** Needs a domain you own in Cloudflare; more
burst-tolerant under load. Prereq:
`winget install --id Cloudflare.cloudflared`.
```powershell
cloudflared tunnel login
pwsh -File scripts/setup-cloudflared.ps1 -Hostname kb.example.com -TunnelName exomem-host
# -> https://kb.example.com (script makes the tunnel + DNS + auto-start service).
```
In the Cloudflare dashboard for this hostname: Bot Fight Mode **OFF** + no WAF
managed ruleset (Security Level low); the edge caps requests at ~100s.
**Option B — ngrok.** No domain needed; every ngrok account gets one free static
dev domain. Claim it in the ngrok dashboard, then:
```powershell
ngrok config add-authtoken <your-authtoken>
ngrok http 8765 --url https://<you>.ngrok-free.dev
```
For an always-on Windows service instead of a foreground process, use
`pwsh -File scripts/setup-ngrok.ps1 -Domain <you>.ngrok-free.dev -Port 8765`.
Free-tier limits: 120 requests/min, ~20k requests/month, and a one-time browser
interstitial on first visit (click through once; it does not recur). Fine for
one person; upgrade or switch to Cloudflare if you outgrow the cap.
**Why not Tailscale Funnel?** Funnel is beta, has non-configurable bandwidth
caps, and its shared relay throttles claude.ai's connector request bursts —
producing a "looks disconnected" failure that has nothing to do with exomem
itself. It is no longer the recommended no-domain default (ngrok's static
domain is more burst-tolerant), but existing Funnel deployments keep working.
If you're already running it:
```powershell
tailscale funnel --bg --https=443 http://127.0.0.1:8765
tailscale funnel status # note the URL, e.g. https://<device>.<tailnet>.ts.net
```
## 3. Create a GitHub OAuth App (one-time, ~3 min)
At <https://github.com/settings/developers> → **OAuth Apps** → **New OAuth App**:
| Field | Value |
|---|---|
| Application name | `exomem` |
| Homepage URL | `https://kb.example.com` |
| Authorization callback URL | `https://kb.example.com/auth/callback` |
Save the generated **Client ID** and **Client Secret**.
## 4. Populate `.env`
Create `.env` in the repo root:
```
EXOMEM_BASE_URL=https://kb.example.com
EXOMEM_GITHUB_USERNAME=<your-github-login>
EXOMEM_GITHUB_USER_ID=<positive-numeric-id-from-GitHub>
GITHUB_CLIENT_ID=<from step 3>
GITHUB_CLIENT_SECRET=<from step 3>
# Required root for durable opaque sessions. Rotation revokes every session.
# Generate: python -c "import secrets;print(secrets.token_urlsafe(48))"
EXOMEM_JWT_SIGNING_KEY=<long-random-string>
# Required: vault root — the folder that contains Knowledge Base/
EXOMEM_VAULT_PATH=<your-Obsidian-vault-root>
```
`EXOMEM_BASE_URL` must match your public hostname (from step 2) exactly — no
trailing slash, no `/mcp` suffix. `EXOMEM_GITHUB_USERNAME` is case-insensitive but
must be the *login*, not the display name. `EXOMEM_GITHUB_USER_ID` is the positive
numeric `id` returned by `GET https://api.github.com/users/<login>`; authorization
must match both values. `EXOMEM_VAULT_PATH` is **required**:
claude.ai connects over HTTP and passes no environment, so the service resolves the
vault solely from this line in `.env` at startup.
## 5. Sanity-test locally
```powershell
# Configuration profile check
uv run python -m exomem doctor --profile remote
# stdio (no auth needed)
uv run python -m exomem --transport stdio
# Ctrl-C to stop
# HTTP (OAuth required)
uv run python -m exomem --transport streamable-http --host 127.0.0.1 --port 8765
# In another terminal:
# curl.exe -i http://127.0.0.1:8765/mcp → expect 401
# curl.exe -i http://127.0.0.1:8765/.well-known/oauth-authorization-server
# → expect JSON metadata
```
Once the tunnel is up (step 2) and `.env` is populated (step 4), `exomem doctor
--profile remote --probe` runs the equivalent checks above — plus the public
OAuth-discovery and bare well-known endpoints through the live tunnel — in one
command.
## 6. Install as a service (auto-start on boot)
**Option 0 (container):** `docker compose --profile cloudflared up -d` (or
`--profile ngrok`) runs the lean server plus the tunnel as one supervised unit.
Use `docker compose -f compose.yaml -f compose.ml.yaml ...` for CPU hybrid search,
or `docker compose -f compose.yaml -f compose.cuda.yaml ...` for NVIDIA/Linux
CUDA capability. The CUDA image remains CPU-default at idle unless you explicitly
set performance mode. See [docker.md](docker.md). The native services below suit
Windows live vaults (Docker Desktop/WSL2 bind mounts miss live file-watch events)
and hosts where Docker isn't wanted. macOS is not a host exomem can serve: the
held-filesystem substrate every governed write acquires a reserved-path root
through has no darwin backend, so the server refuses there.
Pick your platform — both run the same `streamable-http` server and differ
only in the OS service manager. The release commands create or update a separate
PyPI-backed service venv, load the repository `.env` into the service manager,
doctor-gate the selected profile and remote configuration, start the service, and
verify that local `/mcp` returns the expected OAuth `401`.
The default `standard` profile is multimodal but not permanently model-resident.
Evidence work is durable and serialized through one disposable child; after
`EXOMEM_MEDIA_IDLE_SECONDS` (default 300) without work, that child exits and returns
its RAM plus MPS/MLX/CUDA state. After the deadline, maintainers can verify the
persistent-core envelope with `python scripts/verify-resource-envelope.py` (targets:
one service, zero media children, <=512 MiB pre-cache RSS, and <1% idle CPU). GPU
acceptance is checked separately with `nvidia-smi` or Activity Monitor: the idle
core must not be a CUDA compute process and targets <200 MiB GPU delta.
On macOS the JSON also reports Activity Monitor's physical-footprint metric via
`proc_pid_rusage`; use `--max-memory-mb` for an optional physical-footprint
limit. That limit is evaluated only when every server PID has a native sample,
while `--max-rss-mb` keeps its existing RSS meaning.
**macOS (launchd) — does not work today:**
Kept for the darwin held-filesystem backend that does not exist yet. The
service starts and every governed write refuses at the reserved-path root, so
this is a record of the intended shape rather than a route to follow.
```bash
bash scripts/install-service.sh --release
# Re-run after a package or .env change to update and restart the service.
# Developer checkout mode: bash scripts/install-service.sh --repo-dev --profile standard
# Uninstall: launchctl bootout gui/$(id -u)/com.exomem && rm ~/Library/LaunchAgents/com.exomem.plist
```
**Linux (systemd --user):**
Add `--seamless` to a release install to preserve existing client connections
during later worker upgrades. First enablement requires one reconnect; see
[managed service upgrades](managed-service-upgrades.md) for setup and recovery.
```bash
bash scripts/install-service.sh --release
# The installer attempts to enable user linger so the service survives logout.
# Re-run after a package or .env change to update and restart the service.
# Developer checkout mode: bash scripts/install-service.sh --repo-dev --profile standard
# Uninstall: systemctl --user disable --now exomem && rm ~/.config/systemd/user/exomem.service
```
**Windows (NSSM):**
```powershell
# Prereq: NSSM must be installed and on PATH. Easiest:
# winget install NSSM.NSSM
# or download from https://nssm.cc/download and add nssm.exe to PATH
# (or pass -NssmPath "C:\path\to\nssm.exe" to the script below).
# The script self-elevates; approve the UAC prompt.
pwsh -File scripts/install-service.ps1 -Release
# The installer creates/updates the PyPI service venv and verifies /mcp -> 401.
# It is idempotent: re-running against a registered service reconfigures it in
# place and reuses the venv the service is already installed against.
# Developer checkout mode is still available when .venv already exists:
# pwsh -File scripts/install-service.ps1
# Uninstall:
# nssm stop exomem && nssm remove exomem confirm
# Restart (after .env edits): elevated shell required
# Start-Process -Verb RunAs -Wait sc.exe -ArgumentList 'stop','exomem'
# Start-Process -Verb RunAs -Wait sc.exe -ArgumentList 'start','exomem'
```
**Upgrading after a release — run this, not a bare `uv pip install`:**
```powershell
pwsh -File scripts/upgrade.ps1
# Explicit policy: -CliSync auto|always|never (default: auto)
```
Requires PowerShell 7+; Windows PowerShell 5.1 is refused up front rather than
allowed to fail obscurely partway.
The service runs a PyPI-backed venv that is **not** the repo checkout, so `git pull`
never touches it. `upgrade.ps1` locates that venv from the NSSM registry, prints the
`target:` version it resolved before touching anything, upgrades it, **fails if the
venv did not actually end up on that target** (a `uv` that exits 0 having applied
nothing is the failure this catches — #578), repairs the CUDA torch build, gates on
`doctor`, restarts, and then asserts the **live** `/health` version matches what it
just installed. It then records a
non-secret managed-install manifest and, in the default `auto` mode, aligns an
existing uv-managed `exomem`/`kb` command to that exact verified release. `auto`
never installs a command that was absent; `always` may install it; `never` leaves
CLI management alone. `-SkipRestart` stages only the service venv and deliberately
defers CLI alignment until a later run verifies the live release. No elevation is
needed.
The CLI and service need release parity, not dependency parity. The uv-tool CLI
stays lean while a `standard` or `media` service keeps its embedding/media extras;
duplicating Torch and model dependencies into the command environment is wasteful.
Check the resolved command and its paired service identity without loading those
optional stacks:
```powershell
exomem --version --json
kb --version
```
On macOS/Linux, `bash scripts/upgrade.sh` provides the same behavior with
`--cli-sync auto|always|never`.
A second service on the same machine (its own unit, port and vault, such as a
client cell beside a personal one) upgrades with `--unit-file <its unit>`. The
script reads that service's vault from the unit's own environment, never from the
checkout's `.env`, and refuses to continue when the unit names none: pass `--vault`
then. For a unit that is not the default one, `--cli-sync auto` leaves the uv-tool
CLI alone, and the managed-install manifest keeps naming the service it already
names. A CLI sync keeps an operator-owned launcher (a regular file where uv would
put its symlink) in uv's bin directory.
### Windows upgrade: legacy idempotency runtime DACL
Exomem 0.47.0 began requiring a protected, principal-private DACL on Windows
idempotency runtime state. An install upgraded from 0.46.0 or earlier can already
have a runtime directory whose inherited ACL does not meet that contract. Exomem
does not silently rewrite it: the service and `exomem doctor` fail closed and print
the exact offending path plus a one-line `icacls.exe` repair for the current runtime
identity.
Stop the process that owns the reported runtime, open an elevated PowerShell as
that intended owner, copy the complete command from the error, run it, then restart
the process. The command resets and protects only the named entry; it is not
recursive. Do not delete the idempotency database to bypass validation. A command
generated by a normal user is not a repair for a LocalSystem-owned path: run the
diagnostic under the identity that owns the runtime.
The trustee set is exact. LocalSystem state permits only LocalSystem and
Administrators; normal-user state permits that user's SID plus LocalSystem and
Administrators. One `EXOMEM_WRITER_LEASE_STATE_DIR` therefore cannot serve both an
NSSM LocalSystem service and a direct user process. More importantly, that directory
also anchors the process-safe mutation lock. Do not point the two identities at
separate state roots and then mutate the same vault concurrently. Route ordinary
mutations through the service. Before direct CLI maintenance such as `exomem index`,
stop the service, use a user-private state root, finish the command, and restart the
service. `exomem mode` only updates machine configuration and does not enter the
vault mutation path.
Releases carrying semantic parser changes bump the parser generation used by the
lexical, vector, graph, pack, and count sidecars. Canonical Markdown is not
rewritten. After upgrading, quiesce vault mutations and explicitly run
`exomem maintain --reconcile` on the host to
refresh stale derived rows; service restart alone does not promise this rebuild. Anonymous
unit references whose normalized content changed become explicitly stale rather
than resolving to a nearby unit; authored `^anchors` remain the stable reference
mechanism. A full `maintain_memory(mode="fix", rebuild_embeddings=true)` rebuild
is optional when exact vector regeneration is required.
Upgrading by hand is the trap this replaces: `uv pip install --upgrade exomem[...]`
ignores `[tool.uv.sources]` (only `uv sync` reads it), so it silently pulls the PyPI
**CPU** torch wheel over the CUDA build and moves embeddings/media onto the CPU with
no other symptom. `doctor` now fails on that state when an NVIDIA driver is present
(set `EXOMEM_ALLOW_CPU_TORCH=1` to accept CPU deliberately).
### Renaming an existing `kb-mcp` service
Boxes provisioned **before the `kb-mcp` → `exomem` rename** still run the Windows
service under the old name `kb-mcp`. The code (and `install-service.ps1` /
`restart.ps1`, which now default to `exomem`) are renamed, but the installed NSSM
service isn't — it must be re-registered once per box. `restart.ps1` falls back to
the legacy `kb-mcp` name automatically so it keeps working until you migrate; to
finish the rename, run these in an **elevated** PowerShell (the old service must be
removed first or it collides with the new one on port 8765):
```powershell
sc.exe stop kb-mcp
nssm remove kb-mcp confirm
pwsh -File scripts/install-service.ps1 # installs + starts 'exomem', re-grants no-UAC rights
```
The cloudflared tunnel/funnel targets the **port** (127.0.0.1:8765), not the
service name, so it keeps working across the rename. Verify with
`sc.exe query exomem` (RUNNING) and `sc.exe query kb-mcp` (should not exist).
## 7. Add to claude.ai
1. claude.ai → Settings → Connectors → **Add custom connector**
2. **Name**: `Knowledge Base` (or whatever)
3. **Server URL**: `https://kb.example.com/mcp` (this host's public hostname)
4. Leave **OAuth Client ID** and **OAuth Client Secret** blank — claude.ai uses
Dynamic Client Registration against your `/register` endpoint.
5. Save. claude.ai opens a GitHub login window → log in (only the user in
`EXOMEM_GITHUB_USERNAME` is allowed) → approve consent → redirects back to
claude.ai. The tools appear in the palette.
## Machine-local state placement (and file-sync agents)
Machine-local derived state — the semantic/lexical/graph index stores, the graph
epoch and receipt records, the deferred-index queue, media-job and governance
sidecars, due/review projections, and their rebuild scratch files — lives
**outside the vault**, in a per-user, per-vault state root:
1. `EXOMEM_STATE_ROOT` environment override (absolute path, used verbatim), else
2. `%LOCALAPPDATA%\exomem\state` on Windows, else
3. `$XDG_STATE_HOME/exomem/state` (default `~/.local/state/exomem/state`) on POSIX,
with one subdirectory per vault, keyed by the vault's resolved path. The vault
itself carries only user content, so a file-sync agent watching it (Syncthing,
Dropbox, OneDrive, iCloud) has no index store, epoch file, lock, or rebuild
scratch to hash, hold, or replace — the failure class that has caused
whole-vault rebuild chains and warm-up stalls when live state synced between
machines.
The structured collection store checks this before it writes its live
database. Store custody stays unverified while the vault or the state directory
is inside a file-sync root. Custody recognizes a root from the evidence in the
`sync-providers` vocabulary registry: client metadata files, cloud document
paths, platform variables, Windows sync folder names and vault sync plugins.
The built-in sync plugin's enabled bit does not identify a connection or sync
root, so the shipped pack does not treat that bit as root evidence. The
single-host adapter still checks configured roots, root metadata, platform
paths, and separation between the vault and live state. An overlay that
explicitly declares core-plugin evidence retains its conservative refusal.
No plugin toggle or declaration grants custody.
If your sync client is not in the shipped pack, add it with
`schema_memory(subject="sync-providers", operation="save")` or in
`_Schema/sync-providers.yaml`. You can also list sync roots as absolute paths
in `EXOMEM_COLLECTION_STORE_SYNC_ROOTS`. A vault entry can only add evidence;
it cannot change or remove a shipped entry.
Ordinary service and stateful CLI startup is read-only at this boundary. It
never creates the state root, copies or removes a legacy file, resumes an
interrupted transition, or upgrades a manifest. Until an explicit offline
migration has completed, it refuses with
`STATE_MIGRATION_OFFLINE_REQUIRED`. A complete manifest plus even one
reintroduced legacy duplicate refuses the same way. `exomem doctor` names the
local paths and conflict without exposing them in the public refusal.
Every desktop upgrade is one fail-closed transition:
**stop -> prove stopped/PID gone -> install target**, then
**offline migrate -> doctor -> start -> prove new PID/version**. The stop proof
checks both the SCM `Stopped` state and that the exact worker PID captured before
the stop no longer exists; checking the NSSM parent after stopping cannot detect
an orphan. Run the migration only from the target interpreter:
```powershell
pwsh -File scripts/upgrade.ps1 -Vault "C:\path\to\vault"
# Or, inside a manually proven stop window:
& $ServicePython -m exomem maintain --vault "C:\path\to\vault" `
--migrate-state --offline
```
The migration lock serializes two new migrators; it cannot exclude an older
Exomem release that does not know the lock exists. That is why the proven stop
window is mandatory. `-SkipRestart` is unavailable for this transition. Any
failure leaves the service stopped, including migration, doctor, start,
new-worker, and live-version failures. Fix the failure and rerun the whole
transition; never start an old installed release against state already migrated
by the target. If doctor reports two authorities, repeat the explicit offline
command with `--adopt-state external` or `--adopt-state vault` only after
deciding which copy to keep.
Belt and braces for vaults that carried historical state or may run an older
exomem again: keep sync-agent ignore patterns for the state families, e.g. in
Syncthing's `.stignore`:
```text
.graph-sync*.json
.graph-commit-receipts
.graph-reset-*
.graph-rebuild-*
.graph-coordination
.authorization-projections
*.sqlite
*.sqlite-wal
*.sqlite-shm
*.sqlite-journal
.lexical.sqlite.rebuild-*
.deferred-index.json
.media-jobs.json
.idempotency.json
.idempotency.jsonl
.voice_profiles.json
.due-state.json
.review-state.json
.media-worker.lock
```
After migration these names no longer exist under the vault, so the patterns
are inert on current versions — they only protect against re-introduction.
## Deploying on a second machine (multi-host)
Each machine has its own Exomem process and local vault replica. Replication moves
files; it does not decide which process may write. Choose the operating model that
fits the downtime, automation, and operational complexity you actually want:
| Model | Normal operation | What it gives you | Trade-off |
|---|---|---|---|
| Single host | One service and one public URL | Lowest operational overhead | The service is unavailable while that host is off or asleep. |
| [Manual cold standby](runbooks/cold-standby.md) | Desktop normally active; laptop is demand-start and promoted by an explicit guarded command | One stable connector URL and a recoverable copy without an always-on lease/edge control plane | Planned handoff has bounded downtime; ambiguous/partitioned evidence refuses; forced recovery can lose or fork unreplicated state. |
| Automatic writer-lease HA | Two replicas, coordinator/lease, and a stable edge route | Automatic active-writer selection and continued reads/failover under the documented HA contract | More moving parts: coordinator, lease, shared OAuth state, edge routing, and release-parity operations. |
None is universally preferred. Choose single host when downtime is acceptable,
manual cold standby when you prefer a deliberate recovery procedure, and automatic
HA when its availability benefit is worth operating the coordination stack.
### Independent second host with its own URL
If the second machine is an independent deployment rather than manual cold standby
or automatic HA, give it its own public hostname, OAuth app, `.env`, and connector.
A GitHub OAuth App accepts one callback URL, so create a separate app whose callback
is `https://<second-host>/auth/callback`; put that app's client ID and secret in the
second host's `.env`. Set its own `EXOMEM_BASE_URL` and `EXOMEM_VAULT_PATH`, and add
a second connector pointing at that host's `/mcp` URL.
Hybrid `find` also needs the embeddings stack on each host independently. Run
`uv sync --extra embeddings` there and verify the GPU path with
`uv run python -c "import torch; print(torch.cuda.is_available(), torch.cuda.get_arch_list())"`.
For an RTX 50-series GPU, expect `True`, `sm_120`, and the startup log line saying
the embedding model is ready on CUDA; otherwise `find` falls back to keyword/BM25.
### Automatic single-writer lease and laptop takeover
One managed service can also use the included coordinator locally for collection
store admission. Set the explicit single-host topology in that service's
environment:
```dotenv
EXOMEM_WRITER_LEASE_TOPOLOGY=single-host
EXOMEM_WRITER_LEASE_URL=http://127.0.0.1:8770
EXOMEM_WRITER_LEASE_VAULT_ID=main
EXOMEM_WRITER_LEASE_REPLICA_ID=local
EXOMEM_WRITER_LEASE_TOKEN=<coordinator-writer-secret>
```
Under this topology the service keeps the writer lease while it is idle: no other
replica can reach a loopback coordinator, so an idle release would hand authority
to nobody. Set `EXOMEM_WRITER_LEASE_IDLE_SECONDS` to a value of at least the TTL
to release an idle lease anyway. Owner offline steps still need the service
stopped first.
Run the coordinator under the service manager on the same host, bound to
`127.0.0.1`, with its database outside the vault. Give the coordinator the same
writer credential through `EXOMEM_LEASE_COORDINATOR_TOKEN`. Keep its distinct
operator credential in `EXOMEM_LEASE_COORDINATOR_OPERATOR_TOKEN` in the protected
coordinator environment and offline operator storage. Ordinary service requests
receive only the writer credential. A managed service that moves file collections
into the store at an upgrade reads the operator credential from `operator.env`
beside `service.env`, for its migrator alone, as `docs/managed-service-upgrades.md`
describes. Keep the existing vault identifier,
replica identifier, signing key, and FastMCP home stable.
The source template is `scripts/exomem-lease-coordinator.service`. Render
`__VENV_PYTHON__` with the installed interpreter, `__COORDINATOR_ENV_FILE__` with
the protected coordinator environment file, and `__PORT__` with the loopback
port. Render `__COORDINATOR_STATE_DIRECTORY__` and `__COORDINATOR_DATABASE__` with
the persistent external directory and database path. Keep these private inputs
outside the repository. The environment file must allow only its owner to read
it; the unit's `UMask=0077` protects files the coordinator creates. Run systemd's
native unit verification on the rendered file before installing it as
`exomem-lease-coordinator.service` in the user manager. The template restarts a
failed coordinator and binds only to loopback. It does not install or enable
itself.
Single-host accepts only a credential-free literal loopback HTTP origin,
including IPv6 loopback. Paths, queries, fragments, and hostname aliases refuse.
Its coordinator requests bypass proxies and refuse redirects to another origin.
This restriction prevents a shared remote coordinator from silently using local
session identities. A false refusal delays setup for the deployment owner.
The service keeps its existing durable sessions and FastMCP OAuth storage when
`EXOMEM_OAUTH_STORAGE_URL` is absent. Supplying that URL still selects shared
storage and requires its existing credentials and namespace. Omitting the
topology retains multi-host behavior and its shared OAuth requirement.
`exomem doctor --profile ha --probe` checks the local coordinator and lease
configuration without requiring an HA edge or two replica origins. It checks
authenticated lease status and requires HTTP 401 for a credential-free status
request. Other results fail the diagnostic, including unavailable transport.
This detects missing bearer enforcement; a false refusal delays setup for the
deployment owner.
The store still requires its capability fence, head checks, mutation lock, verified custody,
and supported reader admission. This setup does not enable store creation or
migrate an existing collection.
The collection store requires this coordination. On an install without a
configured coordinator, including a stdio install, the migration preflight
reports the vault `blocked`. Every store step refuses with
`COLLECTION_STORE_COORDINATION_REQUIRED` and points to this section: migration,
enrolment, a summary collection create, serving a store and adopt-local.
File-mode Records and Planning collections are unaffected.
The lease is replication-agnostic. It decides which replica may mutate; it does
not copy vault files. This desktop/laptop example uses Syncthing, but self-hosters
can use shared storage, Unison, rsync automation, Git-based replication, or another
external mechanism. Followers can only serve content that mechanism has delivered,
and its convergence/recovery behavior remains an operator concern.
Network reachability is a third, separate layer. In a LAN-only Syncthing setup,
an isolated laptop can take the lease and write its last local vault copy, but its
changes do not reach the desktop until the machines share a network again. If
takeover must work while the laptop is away from home, the coordinator must also
be reachable from both networks (for example through public TLS, a small VPS, a
managed linearizable service, or a private overlay network). Never put the only
coordinator on the desktop whose shutdown is meant to trigger failover.
Run one strongly consistent coordinator. The included reference service uses a
transactional SQLite database on one always-on node:
```powershell
$env:EXOMEM_LEASE_COORDINATOR_TOKEN = '<long-random-secret>'
uv run python -m exomem.lease_coordinator --host 0.0.0.0 --port 8770 `
--database C:\path\to\writer-leases.sqlite
```
Expose that service over private TLS or a protected tunnel. SQLite is consistent
on a single coordinator node but not highly available; for HA, implement the same
HTTP contract on a linearizable store such as a Durable Object, transactional SQL,
Consul, or etcd.
A reference Cloudflare edge worker for this shape used to ship in
`deploy/cloudflare-ha/`. **It has been removed.** Leaving retired routing
infrastructure deployed cost a total MCP outage: the worker had no way to
represent a legitimate single-node deployment, so once the second replica was
decommissioned it refused the only healthy origin on every request, and the
origin never learned it was being refused (#581, #550). Recover it from git
history if you genuinely run two replicas; do not resurrect it by default.
Nothing about the contract above depends on that worker. The coordinator and
state contracts are plain HTTP, so any reverse proxy, load balancer or edge that
implements them will do.
Configure both Exomem hosts with the same vault ID and coordinator, but unique
replica IDs. Desktop example:
```dotenv
EXOMEM_WRITER_LEASE_URL=https://lease.example.com
EXOMEM_WRITER_LEASE_VAULT_ID=personal-main
EXOMEM_WRITER_LEASE_REPLICA_ID=desktop
EXOMEM_WRITER_LEASE_TOKEN=<long-random-secret>
EXOMEM_WRITER_LEASE_TTL=30
EXOMEM_WRITER_LEASE_PREFERRED=1
# Required when both replicas serve one stable OAuth/MCP hostname.
EXOMEM_OAUTH_STORAGE_URL=https://exomem.example.com
EXOMEM_OAUTH_STORAGE_NAMESPACE=personal-main
EXOMEM_OAUTH_STORAGE_TOKEN=<same-edge-secret>
```
The coordinator's `EXOMEM_LEASE_COORDINATOR_TOKEN`, each replica's
`EXOMEM_WRITER_LEASE_TOKEN`, and `EXOMEM_OAUTH_STORAGE_TOKEN` must be the same
non-empty value. `exomem doctor --profile ha --probe` proves that anonymous
state reads receive `401` and an authenticated absent-key read succeeds; a
coordinator that accepts anonymous state access is not safe for session authority.
Laptop uses the same values except
`EXOMEM_WRITER_LEASE_REPLICA_ID=laptop` and omits the preferred flag. Both remain
readable. The first write acquires the lease; the server renews it while alive.
After a crash or sleep, another host can take over after the TTL. Writes fail closed
with `WRITER_COORDINATOR_UNAVAILABLE` if authority cannot be confirmed, but reads
continue locally. Check any replica with:
```powershell
uv run python -m exomem coordination_status --json
```
That command probes the configured vault's local process-safe OS boundary, not
just the calling process's memory. A held boundary includes only bounded holder
diagnostics and `verified: true` when the runtime-state sidecar is bound to the
current lock generation. `verified: false` means an external holder is real but
its identity could not be safely attributed. Vault paths, content, credentials,
and tenant identifiers are never included. This local boundary coordinates
service and worker processes on one host; it does not replace the remote writer
lease between replicas.
REST callers should attach one stable `Idempotency-Key` to every mutation attempt;
CLI callers can set `EXOMEM_IDEMPOTENCY_KEY`. Exact same-key/same-payload retries
return the stored canonical terminal, including its original request and receipt
identity; reusing a key with different input is rejected. Explicit completed and
recognized committed-failure receipts are retained locally for 24 hours. The
default successful product response is compact and decisive (`ok`,
`status="committed"`, `mutated`, path(s), request/receipt identity, and warning
count); `response_detail="full"` or `"legacy"` changes only presentation and is
excluded from mutation identity.
MCP does not expose the REST idempotency header through its tool schema. Exomem
therefore derives a privacy-safe local replay identity from the verified
principal, hashed bearer, or MCP session plus the exact command and arguments.
Its completed-result window is 600 seconds (10 minutes). Each MCP call also gets
a UUIDv4 `request_id` generated locally unless a valid `x-exomem-request-id`
arrives; that ID is correlation, not a substitute for the replay key. Never
change the payload or create a fresh explicit identity to recover a pending or
committed-uncertain acknowledgement: retry only with the same identity as
directed, or reconcile.
A REST write without an `Idempotency-Key` carries no replay identity: two
identical requests are two writes, and a busy boundary refuses it with a retryable
`MUTATION_BUSY` after one bounded acquire wait, without waiting out contention. Send one `Idempotency-Key` per intended write to have
the server wait out ordinary contention and to let a resend attach to the first
attempt. The bundled Studio does this for every action.
Expected MCP operation refusals such as `MUTATION_WARMING` and `MUTATION_BUSY` are normal tool content:
inspect top-level `success: false` and the structured `error` object, then follow
its retry fields. They are not transport failures. A `receipt_id` is diagnostic,
not a caller-supplied cross-session replay key; effective retry identity remains
the explicit idempotency identity where supported or the same bounded
principal/bearer/session scope plus unchanged command payload.
Idempotency records and credentials stay in per-machine runtime state outside the
synced vault. Both the 24-hour explicit window and the 600-second inferred window
bound completed and recognized committed-failure receipts per replica. Pending or
committed-uncertain markers fail closed until local reconciliation rather than
silently expiring into another write. The lease prevents concurrent writers, but
neither cache supplies cross-replica exactly-once durability if a retry reaches
another replica after the original disappears.
On Windows this runtime state is also principal-private. A LocalSystem service and
a normal-user CLI cannot share one exact DACL or DPAPI receipt envelope. Keep direct
user maintenance quiescent with the service stopped; separate state directories are
an ownership boundary, not permission to create two concurrent mutation locks for
one vault.
The reference Cloudflare edge therefore treats `tools/call` as an ambiguous
side-effect boundary. `ORIGIN_TIMEOUT_MS` remains the short connectivity window
for discovery/initialization traffic, while `MCP_TOOL_TIMEOUT_MS` defaults to 60
seconds for actual tools. While a lease holder exists, a tool call is sent only
to that replica and is never replayed to the passive origin after a timeout or
5xx response. Once the lease expires, the edge probes both origins, chooses one
runtime-compatible replica, and forwards the next tool call exactly once. Runtime
admission uses the content-free `/health/ready` contract: supported behavioral
contract, stateless HTTP transport, expected replica identity, healthy writer
coordination, and takeover eligibility. Admission is cached in the Durable Object
against the active lease holder and fencing token, so steady-state tool calls do
not pay another origin probe. The active origin
renews its lease independently while a long tool runs; correctness does not rely
on ordering `MCP_TOOL_TIMEOUT_MS` against `EXOMEM_WRITER_LEASE_TTL`.
Both replicas must also use the same stable `EXOMEM_BASE_URL`, GitHub OAuth client
ID/secret, `EXOMEM_GITHUB_USERNAME`, `EXOMEM_GITHUB_USER_ID`, and
`EXOMEM_JWT_SIGNING_KEY`. Durable session records are Fernet-encrypted on the
replica before leaving it. Session and auth-generation reads are always direct to
the coordinator: there is no read-through mirror or stale local fallback that
could resurrect a revoked session.
### When files change before a migration finishes
`exomem collections migrate --apply` publishes its store before it installs the
marker. Until the marker is installed, the files stay canonical. If an apply
stops in that window and a writer outside the writer lease, such as a sync
client or an editor, then changes an imported file, the next apply does not
admit the published store, because admitting it would lose that change:
- On a vault that no store served before, the apply imports the change into a
new store and moves the coordinator's fence to it at the next generation. Its
report names the replaced store under `superseded`, and the new store keeps
that store's replica as `.foreign-*` evidence.
- On a vault whose store already served collections, the apply first restores
that store, with its rows and its head, and then migrates again with the
change. Its report describes the restore under `restored`. When the migrated
store's replica or head already left this host, the restored store continues
as a new tenure, as adopt-local continues one.
To finish the migration:
1. Keep the Exomem service stopped on every host that serves the vault.
2. Run `EXOMEM_LEASE_COORDINATOR_OPERATOR_TOKEN=<operator token> exomem collections migrate --vault <vault> --apply`.
3. If the apply reports `status: restored`, wait until the writer lease expires, then run step 2 again.
4. If the apply reports `COLLECTION_STORE_SYNC_PENDING`, wait for the vault replica, start and stop the service once, then repeat step 2.
5. If the apply reports `COLLECTION_STORE_DIVERGED`, run `exomem collections adopt-local --vault <vault> --why "<reason>" --dry-run`, apply that preview with `--preview-id`, then run step 2 again.
6. If the apply reports `COLLECTION_STORE_MIGRATION_STALE`, put back the files of each collection that it names, then run step 2 again.
7. Start the Exomem services.
The apply reports `marker_admitted` when the migration is finished.
The apply never adopts a replica itself. The service adopts the vault replica when it starts, so step 4 starts it once on this host.
`status: restored` means that the restore finished but could not hand back its
lease. `COLLECTION_STORE_SYNC_PENDING` means that the coordinator's head or the
vault replica is neither the restored store nor the migrated store, usually
because a newer vault replica has not reached this host yet.
`COLLECTION_STORE_DIVERGED` means that the restored store records a divergence
that adopt-local must resolve first.
`COLLECTION_STORE_MIGRATION_STALE` means that the apply cannot prove that the
migrated store is the restored store plus only the migration's own imports, so
it restored nothing. The stores, the marker, the fence and the files stay as they
were. When the apply found the changed files, the refusal names the manifest of
each collection that holds them. Put back those files with the bytes that the
migration imported, for example from a backup or the vault's sync history.
An apply that stops during a restore, after the migrated store reached the vault
replica or the coordinator, leaves the restored store refused with
`COLLECTION_STORE_SYNC_PENDING` until the next apply finishes the restore. An
apply that finished its restore and then stopped, for example on a failed
round-trip proof, leaves the restored store serving. The services may run
meanwhile, and the next apply migrates that store with the writes it served.
### Summary collections
A summary collection is a NEW Records collection whose rows live in the
SQLite collection store outside the vault. It holds large preserved exports,
such as a health export, and answers daily count, sum and average queries
from rollups. Existing Records and Planning collections stay in their files;
a summary collection never migrates them. The store requires writer-lease
coordination (see "Automatic single-writer lease and laptop takeover").
A summary collection whose derived rows are missing or were built under an
earlier layout rebuilds them from its import log at the service's next start.
The rebuild gives way to requests between batches.
The owner creates a vault's first summary collection offline, which enrols the
vault's store with the coordinator: stop the service, then run
`exomem collections create --manifest-path <path> --manifest-file <file> --why "<reason>"`
with the service's lease environment and `EXOMEM_LEASE_COORDINATOR_OPERATOR_TOKEN`
set. Later summary collections are created through `manage_memory` with `records.create` on the
running service.
To turn summary collections off for one vault, stop the service and run
`exomem collections rollback --vault <vault>` with the same environment. The
rollback is one-way: it
keeps every row, history entry and preserved source, pauses import jobs and
refuses summary reads with `COLLECTION_STORE_DISABLED`. A later release does
not turn the vault's summary collections back on.
### A release that changes the collection store's schema
A release can change the schema of the live collection store. 0.119.0 changed
it from 10 to 11, but 0.119.0 cannot migrate a store that an earlier release
wrote. From 0.120.0, the first service that holds the writer lease migrates the
store before it serves the store's collections. The migration keeps every row
and the store head, so the coordinator's recorded head still matches it.
Until a writer has migrated the store, the new release does not serve its
collections:
- In a handoff, the earlier release's worker keeps the lease and serves until it
stops. The new worker refuses store reads and writes with
`WRITER_LEASE_REQUIRED`, and it migrates the store once it acquires the lease.
- An offline owner command that only reads the store, such as
`exomem collections backup`, refuses with `COLLECTION_STORE_SCHEMA_PENDING`.
Start the service once to migrate the store. `exomem collections create` holds
the writer lease itself, so it migrates the store before it creates.
File collections and knowledge writes are not affected.
If the migration fails, it rolls back and leaves the store at its earlier
schema. The service status then reports the store as `schema_pending`, with the
error in its reason. The service refuses the store's collections with
`COLLECTION_STORE_SCHEMA_PENDING`. A shutdown or an idle release hands the writer
lease back at once, not at its expiry. The service tries the migration again on
the next use, at least 10 seconds later. A rollback to the earlier release is
safe in that state.
Releases 0.114.0 through 0.118.x read the store only up to schema 10, so they
cannot open a migrated store. 0.119.0 and later read schema 11, so they stay
valid rollback targets.
The managed supervisor refuses an upgrade target that cannot read the live
store when the vault's marker routes collections to the store, or the store
records a pending intent. It compares the
newest store schema that the target reads with the schema of the live store, and
it refuses before the serving worker stops. The serving worker keeps serving.
The supervisor runs this check, so the supervisor's release decides it, not the
worker's. A supervisor started on 0.120.0 or earlier does not have the check.
After a seamless upgrade replaces its worker, it still accepts 0.114.0 through
0.118.x as a target until the service unit restarts.
The check refuses nothing in these cases:
- Without a live store, as in a copy of the vault on a fresh state root, the
supervisor has no schema to compare.
- After an export to files, no collection routes to the store. The exporting
host sets its store aside, and every other host keeps its store until
`exomem collections export --retire-local`. An earlier release is then the way
back.
- If the supervisor cannot read the store's schema, the target meets the store
as any release would. A refusal would also block the release that repairs it.
Started by hand on a migrated store, a release from 0.114.0 through 0.118.x
cannot open the store. On a vault with a collection store fence, its writer gets
no lease, so every write on that host refuses with `WRITER_LEASE_REQUIRED`. Roll
forward to recover.
From schema 12 on, the service also records a state compatibility ID for each
store schema, such as `collections-store-schema-12`, in the vault's state
manifest before it migrates the store. A failed migration removes the ID again.
A release that does not support the ID then refuses at its own startup. Schema
11 has no ID, because every release since 0.114.0 treats an unknown ID the same
way, so an ID would also refuse 0.119.0 and later.
If an earlier release refuses at startup with `STATE_MIGRATION_MANIFEST_INVALID`
or `STATE_COMPATIBILITY_UNSUPPORTED` after a rollback, the vault holds state
that the release cannot use. Roll forward to a release that supports it.
If you run 0.119.0 on a store that an earlier release wrote, 0.119.0 cannot back
it up: its backup refuses with `COLLECTION_STORE_SCHEMA_PENDING`. In step 1, back
up the store with the release that you ran before 0.119.0, then upgrade straight
to 0.120.0 or later.
To upgrade a vault with a collection store to a release that changes its schema:
1. With the release that you run now, back up the store: `exomem collections backup --vault <vault> --to <file outside the vault>`.
2. Upgrade the service.
3. Read a store collection, for example with `query_records`.
The read answers with the collection.
The backup in step 1 is a copy at the earlier schema. It does not make a rollback
safe, because restoring it loses every store write after the upgrade.
### Export the vault to files before a downgrade
A release from before the collection store cannot open a vault whose collections
live in the store. Before a downgrade, the owner exports the vault: the export
turns every store collection back into its file layout and then retires the
store. Only the owner can export, with the service stopped on every host and the
operator credential, `EXOMEM_LEASE_COORDINATOR_OPERATOR_TOKEN`, set.
1. Stop the Exomem service on every host that serves the vault.
2. Preview the export: `exomem collections export --vault <vault> --dry-run`
3. If the verdict is `blocked`, resolve each collection that the preview names, then preview again.
4. Note the `plan_id` value in the preview.
5. Apply it: `EXOMEM_LEASE_COORDINATOR_OPERATOR_TOKEN=<operator token> exomem collections export --vault <vault> --plan-id <plan_id> --why "<reason>"`
6. If the apply stops before it reports `exported`, run step 5 again with the same `plan_id`.
7. If the apply reports `COLLECTION_STORE_EXPORT_STALE`, start again at step 2.
8. If the apply reports `COLLECTION_STORE_COORDINATOR_UPGRADE_REQUIRED`, upgrade the coordinator, then start again at step 2.
9. If a rerun reports `COLLECTION_STORE_EXPORT_VIEW_EDITED`, `COLLECTION_STORE_EXPORT_UNVERIFIED` or `COLLECTION_STORE_SYNC_PENDING`, follow "When an export cannot finish" below.
10. On every other host, after the exported vault reaches it, run `exomem collections export --vault <vault> --retire-local`.
11. Install the older release on every host, then start the Exomem services.
The services then read every collection from its files. The `--retire-local`
step needs only the writer-lease credential, because it changes the host that
runs it and reads the coordinator's fence without moving it.
The preview blocks the whole vault while any one collection cannot leave the
store, and names the collection and the reason:
- The store created the collection, so file mode never held it. This rule
stands until the owner decides whether rollback must cover such collections.
- The rendered files exceed a file-mode bound: 2,000 item files, 512 KB in one
item, 8 MB in one collection or 2 MB in one log.
- A value or a held correction has no file form.
- A person changed a store view since the store published it. Restore the
view's bytes, or wait for the release that classifies view edits.
A collection that the store never wrote keeps its files byte for byte. The
export renders a collection that the store wrote from its rows, and records one
content-free checkpoint in the activity log, `log.md`, that names the store
transitions it summarizes. Inspection then reports `acknowledged_gap` for that
collection.
After it verifies that file mode reads every collection as the store served it,
the export records that result in its intent. Then it retires the coordinator's
store fence through the operator-only
`POST /v1/vaults/<id>/collection-store-fence/retire`. Then it renames
`<knowledge base folder>/_Collections/mode.json` (`Knowledge Base` by default)
to `mode.json.exported-<utc>`, renames the replica and the live store to
`collections.sqlite.exported-<utc>`, and removes the store's state compatibility
descriptors. No host adopts the renamed files; they stay as evidence of the
export.
An apply that stops after the fence retire finishes these renames without
reading the files again, so a file-mode edit made after the retire does not
stop it. First it reads the coordinator's newest retired fence through
`GET /v1/vaults/<id>/collection-store-fence/retired` and checks that the
recorded head is this store's own. A retired fence alone never lets a rerun
retire the marker: after a fence retire made by hand, the rerun first writes and
verifies the files.
The retire and the lease revocation are one coordinator transaction. The lease
holder at that moment loses its fencing token, so a store writer that opened
against the fence cannot renew or write. A repeated retire returns the same
answer and does not revoke the lease again.
After the retire, the hosts behave as follows:
- A host whose vault copy routes no collection to the store acquires the lease
as on a vault that was never enrolled.
- A host whose vault copy still has the store marker does not serve the store.
This release refuses its store requests with `COLLECTION_STORE_RETIRED` until
the exported vault reaches that host.
- A store-serving client from a release before the retire, such as 0.117, cannot
read a retired fence. It fails closed with `WRITER_COORDINATOR_UNAVAILABLE` and
does not serve the store. The cost is that the store collections stay
unavailable on that host until its vault copy no longer routes to the store.
Do not roll the coordinator back to a release from before the retire once a
fence is retired. An older coordinator ignores the retired generations, so it
enrols the vault at generation 1 again, and a stale store marker could then bind
that enrolment.
The coordinator keeps every retired generation. The fence read reports
`enrolled: false` with the newest retired generation, or 0 for a vault that was
never enrolled. A later enrolment, such as the owner's first
`exomem collections create` or `exomem collections migrate --apply` on an
exported vault, takes the next generation with a new store identity. Run that
enrolment with a release that has the retire, because an older release cannot
read a retired fence. An enrolment request based on an older generation gets
HTTP 409.
A vault that was migrated again after an export has fence generation 2 or later
in its marker. Only the coordinator that holds that fence can serve it. A
replaced coordinator, or one restored from a backup older than that migration,
does not hold the fence. The hosts then refuse the store with
`COLLECTION_STORE_MARKER_CONFLICT`, because only a marker at generation 1 may
make the first fence cut on a new coordinator. Keep the coordinator database in
the backups, and restore a copy taken after the latest migration. A running
coordinator keeps recent grants and fences in its `-wal` file, so the database
file alone is not current. Back it up with SQLite's online backup API, for
example `sqlite3 writer-leases.sqlite ".backup copy.sqlite"`. A plain file copy is
complete only after a clean stop; after a crash, copy its `-wal` file with it.
#### When an export cannot finish
An apply that stops leaves its intent in the live store. Once the apply has
published the replica, the replica carries the intent too. The store then keeps
every row and refuses writes. The services refuse its collections with
`COLLECTION_STORE_EXPORT_PENDING` until the export finishes or the owner
abandons it, on this host and on every host that adopts the replica.
- `COLLECTION_STORE_EXPORT_VIEW_EDITED` names each view that changed after the
export began. To export without the change, put back the bytes of each named
file, for example from the vault's sync history, then run the apply again
with the same `plan_id`. A rerun accepts a view that holds the bytes the
store published or the bytes this export wrote. To keep the change for a
later decision, abandon the export.
- `COLLECTION_STORE_EXPORT_UNVERIFIED` means that file mode does not read a
collection as the store served it. A rerun repeats the same check, so it
cannot finish. Abandon the export, then file a bug report with the refusal
text.
- `COLLECTION_STORE_SYNC_PENDING` means that another host adopted the vault
replica and wrote rows after the export began, and this host's store does not
have them. Keep the services stopped. Keep both stores, `mode.json` and the
coordinator's fence as they are, because the other host's store holds the
newer rows. File a bug report with the refusal text.
To abandon an export:
1. Keep the Exomem service stopped on every host that serves the vault.
2. Run `EXOMEM_LEASE_COORDINATOR_OPERATOR_TOKEN=<operator token> exomem collections export --vault <vault> --abandon --plan-id <plan_id> --why "<reason>"`.
3. Start the Exomem services. The services serve every store collection again.
The command reports `abandoned`. The vault stays routed to the store, and the
store keeps every row that it served before the export. The abandon rewrites no
vault file. Each file that the export already wrote keeps the export's bytes,
and the store records those bytes as that file's view. A view that a person
edited keeps the edit, and the store adopts no part of it. Before a later
export, restore the bytes of each edited view, or wait for the release that
classifies view edits. Then start again at step 2 of the export steps.
The abandon refuses with `COLLECTION_STORE_RETIRED` when the coordinator's
fence is already retired, because the store cannot serve again without a new
enrolment. After a fence retire made by hand, run the apply again instead: it
writes the files from the store, verifies them and finishes the renames.
Do not retire the fence by hand. If the apply after a retire made by hand
refuses, no step continues the export or abandons it. Keep the services
stopped, keep `mode.json`, the live store and the fence as they are, and file a
bug report with the refusal text.
#### When the exporting host loses its live store after the fence retire
The export verified every file before it retired the fence, so only renames
remain. Do these steps only when the export itself retired the fence:
1. Keep the services stopped on every host.
2. Pick one time for both renames, in the form `YYYYMMDDTHHMMSSZ` (UTC).
3. Rename `<knowledge base folder>/_Collections/mode.json` to `mode.json.exported-<time>`.
4. Rename `<knowledge base folder>/_Collections/collections.sqlite` to `collections.sqlite.exported-<time>`.
5. Run `exomem collections export --vault <vault> --retire-local`.
The command reports `retired` and removes the store's state compatibility
descriptors. Start the services, which then read every collection from its
files.
### Durable-session cutover and rollback
Treat the upgrade from FastMCP reference tokens as a coordinated HA cutover, not
a rolling mixed-provider deployment:
1. Back up `.env`, the coordinator database/shared OAuth state, and both service
installations. Preserve the existing `EXOMEM_JWT_SIGNING_KEY`; add the
immutable GitHub ID and matching coordinator/storage credentials.
2. Quiesce connector traffic. Upgrade active and passive replicas together so no
request can alternate between the legacy and durable token formats.
3. Run offline and probed doctor on both replicas, then bring up only the new
provider. Keep the public connector URL and connector registrations unchanged.
4. Each existing client receives one expected `401` for its legacy reference JWT
and completes one final GitHub authorization. It then holds an Exomem-owned
opaque session with no advertised expiry. Do not delete/recreate connectors.
5. Prove a session issued through one replica validates through the other. Run
single-session and `--all` revocation checks. Run the Codex harness and the
supported hosted-connector smoke test; block rollout if either client asks for
another login or discards the response because `expires_in` is omitted.
The Codex gate uses an isolated persistent home, performs exactly one interactive
login, then starts several fresh ephemeral processes:
```bash
python scripts/codex_auth_session_harness.py \
--url https://kb.example.com/mcp \
--codex-home .rollout/codex-auth-gate \
--codex-auth-source ~/.codex/auth.json \
--runs 3 \
--acknowledge-disposable-target
```
Run this only against a disposable staged KB. The harness refuses a non-empty
target `CODEX_HOME`, copies only the invoking Codex OpenAI `auth.json` into it
with owner-only permissions, and verifies `codex login status` before registering
Exomem. It never copies `config.toml`, `.credentials.json`, or prior MCP state.
When `--codex-auth-source` is omitted, it resolves to `$CODEX_HOME/auth.json` for
the invoking process, or `~/.codex/auth.json` when `CODEX_HOME` is unset. The
acknowledgement is deliberately mandatory.
Keep legacy JTI/upstream-token collections untouched for a bounded rollback
window, but never dual-read them while the new provider is active. Rollback means
deploying the prior provider as another coordinated cutover; clients already on
durable sessions authorize once into the old format. Only after the window closes
should obsolete JTI/upstream GitHub-token records be removed. New durable-session
collections can remain inert during rollback.
For troubleshooting, a malformed, unknown, revoked, or legacy bearer returns the
normal `401 invalid_token` challenge. A coordinator/network failure or current
session-namespace decryption problem returns `503` with no OAuth challenge. Repair
the authority and retry a 503; logging in again cannot fix an availability outage.
The coordinator contract is:
- `POST /v1/vaults/{vault}/lease/acquire`
- `POST /v1/vaults/{vault}/lease/renew`
- `POST /v1/vaults/{vault}/lease/release`
- `GET /v1/vaults/{vault}/lease`
- `POST /v1/state/{namespace}/{get|ttl|put|put-if-absent|list-keys|delete|get-many|ttl-many|put-many|delete-many}`
Lease POST bodies carry `replica_id`, `ttl_seconds`, and, for renew/release,
`fencing_token`. Lease responses carry only `holder`, `expires_at`,
`fencing_token`, and `granted`; no vault content is sent.
State POST bodies carry `collection` plus the operation's `key`, `keys`, `value`,
`values`, and optional positive `ttl`. Every successful state response uses the
`{"result": ...}` envelope. `put-if-absent` is transactional: it returns `true`
only when it creates the key, never replaces a live value, and treats a
TTL-expired value as absent. `list-keys` returns sorted live key strings for the
requested collection, removes expired entries, and never returns encrypted
values. Networked coordinator deployments require `Authorization: Bearer
<token>` on every lease and state operation. Missing or rejected credentials
return `401 {"error":"unauthorized"}`, malformed bodies return `400
{"error":"invalid request"}`, and unknown operations return `404
{"error":"unknown operation"}`.
With the edge deployment, clients register only the stable connector URL; the two
origin hostnames are operational endpoints, not separate connectors. After editing
`.env`, restart each service so it reloads the coordination settings. Automatic
two-way failover requires the replication layer to accept changes from either
host. For Syncthing that means **Send & Receive on both hosts**; users of other
replication systems should apply their equivalent. The writer lease guards Exomem
mutations, not direct Obsidian/filesystem edits, so do not edit a follower manually.
Before deliberately stopping the active writer, let replication reach **Up to
Date**; on an unclean crash, the lease prevents concurrent Exomem writers but cannot
recover bytes the old writer had not replicated yet.
Replica release drift remains deployment housekeeping, but runtime compatibility
is now enforced independently. Before enabling the stable route—or after every
release—check the repo, installed service environment, and readiness contract on
**both** machines:
On each machine, upgrade and self-verify in one step — it locates the service venv
itself, so there is no path to get wrong and no version to eyeball:
```powershell
pwsh -File scripts/upgrade.ps1
```
Then confirm the pair from either box:
```powershell
curl.exe -fsS https://exomem-desktop.example.com/health/ready
curl.exe -fsS https://exomem-laptop.example.com/health/ready
```
Then run the combined read-only diagnostic:
```powershell
uv run python -m exomem doctor --profile ha --probe `
--replica-url https://exomem-desktop.example.com `
--replica-url https://exomem-laptop.example.com
```
Different package releases are allowed when they advertise a supported common
runtime contract. A missing readiness route, unsupported contract, stateful
transport, duplicate replica identity, unhealthy coordinator, or takeover
ineligibility fails the HA gate. An old stateful laptop is therefore excluded
even if its port answers.
The Worker example sets `SUPPORTED_RUNTIME_CONTRACTS="1"`,
`REQUIRED_RUNTIME_TRANSPORT="streamable-http-stateless"`, and
`REQUIRE_COORDINATION="true"`. For an incompatible contract bump, first expand
the accepted set (for example `"1,2"`), roll every replica, confirm doctor passes,
then contract the set to the new version. The deployment platform owns immutable
release pinning, rollout, and rollback. Exomem deliberately has no cross-machine
auto-updater, and the readiness contract is independent of Syncthing or any other
replication product.
## Recall model and re-embedding
A personal server and an Exomem Cloud cell encode recall with `BAAI/bge-m3`,
served from a pinned int8 ONNX artefact on CPU, and activation shares that one
instance (about 0.65 GB resident). The first load fetches the published
artefact, or builds it from the pinned export: a 2.2 GB download and a
quantisation that peaks near 9 GB in a child process. The cloud image carries
the artefact, since a cell can fetch nothing. A hosted cell (the helm cell
chart) keeps `BAAI/bge-base-en-v1.5`. `EXOMEM_RECALL_MODEL` names the model
explicitly.
Every recall sidecar records the model, fingerprint and width of the vectors it
holds, and nothing reads it with another encoder. When the recall model changes,
the installed sidecar keeps serving with the model that wrote it while the
service builds a sidecar for the new model beside it
(`.embeddings.<16 hex>.sqlite` in the vault's state directory). The build runs
in the background, one passage at a time, and resumes after a restart. Pages
written meanwhile are picked up. When it finishes, the service switches
`.embeddings.active` to the new sidecar in one atomic write and releases the old
model; the old sidecar is deleted by the next start. Until then both models are
resident.
- Progress: `exomem doctor` (`embeddings.reembed`, read from disk) and
`exomem status` (`recall_reembed`: pages done, seconds per 1,000 chunks, ETA).
- `EXOMEM_RECALL_REEMBED=off` builds nothing and keeps the old sidecar serving
on a personal server.
- On a personal server both models are resident while the build runs, in every
mode, quiet included: warm-up loads the old model before writes are admitted
so a write never finds it cold. `EXOMEM_RECALL_REEMBED=off` is the way to
keep one model there.
- Rolling back to `BAAI/bge-base-en-v1.5` after a later start has retired the
old sidecar re-embeds the whole vault into the English space, the same way.
- A cloud cell holds one model. It never loads the old one: the old sidecar is
refused, the vector lane reports `vector_space_mismatch` and the lexical
lanes answer until the cutover, and doctor warns that dense recall is off.
Writes meanwhile are built by the job's catch-up. With
`EXOMEM_RECALL_REEMBED=off` a cell's dense recall stays off.
- `exomem index` maintains whichever sidecar is serving, with its own model.
- The sidecar for bge-m3 is about a third larger: 1,024 dimensions against 768.
## The activation packet mode
`recall_context` serves one of two packet
modes. `maps` is the default: the packet carries a map, an index of what the
vault holds, and the client pulls the entries that bear on the turn by ref
before it answers. `anchors` is the earlier packet, which resolves the turn to
anchors. The default changed to `maps` by an operator decision, while the
evaluation of the two modes continues; it is not the result of a measured gain.
To go back to `anchors`, add one line to the service environment and restart
the service:
```bash
EXOMEM_ACTIVATION_PACKET=anchors
```
1. If the install is managed (`--seamless`), add the line to `service.env`, usually `~/.config/exomem/service.env`, and run `bash scripts/upgrade.sh --package-version <the serving version>`. The next worker re-reads `service.env`. Without `--package-version`, the command also stages a newer release when one exists.
2. If the install is a release install that is not managed, add the line to the dotenv file that you gave to the installer, and run the same `bash scripts/install-service.sh --release` command again.
3. If the service runs from a source checkout, add the line to the checkout's `.env` and run `bash scripts/restart.sh`.
4. Call `recall_context` once. The packet has no `map` block.
To return to `maps`, change the line to `EXOMEM_ACTIVATION_PACKET=maps` and
restart in the same way, then call `recall_context` once: the packet has a
`map` block. Do not only remove the line. A managed supervisor that started
while the line was present keeps the value in its own environment, and each
new worker inherits it.
A cloud cell reads no `.env` file. Set the same variable in the cell model
environment that `cellctl` renders (`CELLCTL_CELL_MODEL_ENV`), then roll the
cell.
A value other
than `anchors` or `maps` logs one warning, `invalid EXOMEM_ACTIVATION_PACKET;
using maps`, and serves `maps`. The working-set kill switch disables both modes.
### The turn slice
The `maps` packet carries a `turn_slice` block beside the map. The map is the same
index on every turn. The slice lists the leads that the words of this turn and
its `conversation` reached: resolved and partial anchors, the senses of an
ambiguous turn, pages that the turn names by title, related pages and the carried
page. Each item names the match and the kinds of evidence behind it. A lead that
the conversation reached carries the evidence kind `conversation`. The slice is
not checked for relevance, so the client reads a ref before it relies on it.
The slice reads the `conversation` and the `trigger` of the request. It takes no
`continuity` refs, no `anchor` and no session, and it writes nothing.
The slice is on by default. The map compiles in a fixed share of the packet
budget on every turn, so the map is the same on a turn that resolves nothing.
Task context and the slice share the existing 25% reserve. The complete guarded
slice compiles in this reserve first. Task context uses the actual remaining
allowance and omits generic pointers only for routes that the slice retains.
A turn that reaches nothing gets `turn_slice.state` of
`empty`. A slice that cannot be built sets its `state` and leaves the map as it
is. The default changed to on by an operator decision, so that the `maps` packet
reaches pages that the map does not list; it is not the result of a measured gain.
To disable the turn slice, add one line to the service environment and restart
the service:
```bash
EXOMEM_ACTIVATION_TURN_SLICE=off
```
1. If the install is managed (`--seamless`), add the line to `service.env`, usually `~/.config/exomem/service.env`, and run `bash scripts/upgrade.sh --package-version <the serving version>`. The next worker re-reads `service.env`. Without `--package-version`, the command also stages a newer release when one exists.
2. If the install is a release install that is not managed, add the line to the dotenv file that you gave to the installer, and run the same `bash scripts/install-service.sh --release` command again.
3. If the service runs from a source checkout, add the line to the checkout's `.env` and run `bash scripts/restart.sh`.
4. Call `recall_context` once. The packet has no `turn_slice` block; task context keeps its independent behavior.
To return to the slice, change the line to `EXOMEM_ACTIVATION_TURN_SLICE=on` and
restart in the same way, then call `recall_context` once with a turn that names a
known page: the packet has a `turn_slice` block with `state` of `ready`. Do not
only remove the line. A managed supervisor that started while the line was
present keeps the value in its own environment, and each new worker inherits it.
A cloud cell reads no `.env` file. Set the same variable in the cell model
environment that `cellctl` renders (`CELLCTL_CELL_MODEL_ENV`), then roll the
cell.
Any value other than `off` serves the slice. A value other than `on` or `off`
also logs one warning, `invalid EXOMEM_ACTIVATION_TURN_SLICE; using on`. The
setting has no effect in the `anchors` mode.
### Activation time limit
An MCP `recall_context` call (the `activate_context` command) runs until the MCP
request budget is one second from its end. Set `EXOMEM_ACTIVATION_BUDGET_SECONDS`
to make it stop looking for more context sooner. When the limit arrives while
the call looks for task leads, the packet serves what the call already found:
the map, and any leads that the call reached. The `task_context` block is
`partial` with `budget` as the reason, and the packet has a `request_budget`
block that names what the call left out. If the limit arrives earlier, during
the turn slice (the heat ranking, resolution or the conversation carry), the
packet serves the map alone, without the turn slice or task leads. If it arrives
before the map is built, the call returns its budget abstention. On a host that
is short of CPU, a short limit can therefore cost many calls their slice and
leads, which is why the variable is unset by default. A call that finishes within the
limit returns the packet it always returned.
To change the limit, set the variable to a positive number of seconds in the
service environment and restart the service in the same way as for
`EXOMEM_ACTIVATION_TURN_SLICE`. A value that is not a positive number logs one
warning and the call runs without its own limit. The setting has no effect in the
`anchors` mode, which still abstains when the request budget ends. It does not
change the REST and CLI doors, which keep their own shorter limit.
Each row of `logs/activations.jsonl` records `activation_result` (whether the
call abstained and whether a budget limited it) and `budget` (seconds, remaining
milliseconds, and the stages skipped or cut). `budget` is null when no budget
was bound to the call, and `activation_result` is null when the packet cannot be
observed.
### Task context and live methods
The `task_context` block supplies bounded leads from admitted catalogue carriers,
including notes, episodes, collections and governed methods outside anchor
neighborhoods. These leads require source reads before the client uses them as
facts or ongoing intentions. A lead for an unrestricted caller also lists up to
five outbound links (relation, target ref and title), typed relations first, with
`links_omitted` counting the rest; the client chooses which to read. The retrieve
hook renders each lead's links on one line under the lead, with titles quoted, and
drops those lines before any lead or method line when its cap is tight. An incomplete
discovery returns `partial` or `unavailable`; it does not establish that the
knowledge is absent.
A method projection binds the whole source version, including scope and exclusions
outside a selected unit. Its `read_required` applicability requires the full source
read before use. Current user instructions can adapt a method for one task.
An authorized reusable revision uses the ordinary guarded writer.
Clients carry the activation continuity token across requests. Method invalidation
runs before task selection and does not depend on map freshness. A true
`task_context.methods.reset` discards the held method set before the client uses
new projections. Missing and withheld sources produce the same generic reset.
A client that omits method continuity must read a source again before reuse.
Task discovery has a fast path only when the caller has unrestricted content
access and the vault has no file release policy or bound writer. A vault whose
structured collections live in the collection store keeps the fast path for its
owner, and each page the store owns is still decided on its own. Otherwise
discovery must prepare the caller's complete released corpus on every request,
and that preparation does not yet meet the activation latency ceiling at scale.
This applies to a restricted caller, such as a shared or delegated audience. It
also applies to every caller on a vault with a file release policy. By default
such a request gets
`task_context.state` of `unavailable`, and `task_context.methods.reset` is true
when the client held methods.
To serve that discovery anyway, add this line to the service environment and
restart the service as described for the turn slice:
```bash
EXOMEM_ACTIVATION_RELEASED_CORPUS_TASKS=on
```
Then call `recall_context` once with a turn that names a known page:
`task_context.state` is `ready` or `partial`, not `unavailable`. Expect that call
to take longer on a large vault. To turn it off again, change the line to
`EXOMEM_ACTIVATION_RELEASED_CORPUS_TASKS=off` and restart in the same way. Do not
only remove the line, for the reason given for the turn slice. A value other than
`on` or `off` logs one warning,
`invalid EXOMEM_ACTIVATION_RELEASED_CORPUS_TASKS; using off`.
### Optional context-fit interpretation
Interpretation is disabled unless the process environment names an operator-owned
JSON file through `EXOMEM_CONTEXT_FIT_CONFIG`. Keep that file outside the vault.
It selects exactly one placement and supplies these deployment values:
- `placement`: `local` or `api`.
- `runtime`: `ollama` for local placement, or `openai` for a compatible API.
- `endpoint`, `model`, `model_revision` and `runtime_version`: the measured deployment identity.
- `timeout_seconds`, `max_input_chars`, `max_output_tokens` and `resource_allowance_gib`: the deployment allowance.
- For an API, `provider_audience` and `disclosure_authorized: true`: explicit provider disclosure authority.
- Optionally, `credential_env`: the name of the environment variable that holds the API credential.
Local placement requires a loopback IP endpoint. The adapter verifies the installed
model digest and runtime version before it sends request data. Ollama receives the
output schema through its documented [structured-output interface](https://docs.ollama.com/capabilities/structured-outputs).
The API placement separately checks provider disclosure for every candidate source.
Full source input requires full disclosure for both the caller and the provider.
Excerpt-only admission retains baseline discovery and ordinary client reads.
Neither placement changes the caller's admission or silently selects another provider.
The adapter validates every returned identity, source version and fit label.
Valid labels only order the existing pool; omitted judgments remain uncertain.
Invalid output, missing capability, input overflow and timeout preserve baseline
discovery. Turn text, prompts and fit judgments remain request-local.
Reported resource allowance comes from configuration; it is not measured resource use.
Keep interpretation disabled until a paired real-model check establishes benefit
for that deployment. Remove `EXOMEM_CONTEXT_FIT_CONFIG` from the process environment
to restore the agent-only path. This rollback leaves canonical methods intact.
### The map while the vault changes
A managed service that holds the owner's map does not rebuild it on the thread that serves the call.
When the vault changed since that build, the packet serves the earlier map with `map.stale` set to
`true`, `generation.index_stale` set to `true`, and `generation.index_generation` set to the index
generation the map was built from. One background pass rebuilds the map, and the next call serves
the new one. A restricted caller never gets the owner's map: its packet is built in the call.
The `budget` block has two sizes. `used_chars` is the prose the packet charges against `max_chars`.
`serialized_chars` is the length of the packet as sent, in compact JSON, without the
`serialized_chars` field itself. A map item omits `topics` when it has none, and omits
`lifecycle_class` and `state` when they are unset.
A client that holds the current map can send its `continuity` token back. When the map that the
call would serve has the generation in that token, the `map` block has `unchanged: true` and no
items. A token from the `anchors` mode, an older token and a malformed token get the whole map.
`generation.continuity` in a `maps` packet reports the token that the call received. It is `absent`
without a token. It is `applied` when the token carries a map generation and the service minted it
under the role registry and conventions that serve the call. A change of the map does not change
this report. It is `stale` for any other token, including one that the service minted before an edit
of the role registry or the conventions. `map.unchanged` alone decides whether the client keeps its
held map, and it can be `true` while `generation.continuity` is `stale`.
## Reranking and languages
The cross-encoder reranker reorders a query's top candidates. It runs on explicit
`rerank=True`, or automatically in performance mode on an accelerated device; CPU
services leave it off by default. `EXOMEM_DISABLE_RANKING` turns it off entirely.
The default CrossEncoder backend is PyTorch. To use ONNX, install
`uv sync --extra embeddings --extra reranker-onnx` and set
`EXOMEM_RERANKER_BACKEND=onnx` before starting the service. This uses the
model's published ONNX export; Exomem does not export the model during a request.
The first load may download that export if it is not already in the model cache.
The ONNX provider follows Exomem's device policy: CPU is the default, CUDA is
used only when that policy selects it, and MPS uses CPU. This setting changes
the scorer runtime, not candidate retrieval or the automatic reranking policy.
It does not establish a recall, latency, or memory improvement.
Each reranker declares what it can judge, and recall keeps the fused order outside
it, explicit `rerank=True` included (`retrieval_profile.rerank.reason` says why).
Governed projected recall applies the same gate.
| Reranker | Scripts | Across languages | Notes |
|---|---|---|---|
| `BAAI/bge-reranker-base` (default) | Latin, Han | no | Trained on English and Chinese. Skipped for a query written mostly in another script (`query_script_not_covered`). Also skipped when the best dense match shares no content word with the query and either the query's words matched pages in another script than that match (`cross_language_not_covered`) or matched nothing while the query itself is in another script than that match (`cross_script_lead_not_covered`). |
| `BAAI/bge-reranker-v2-m3` (opt-in) | all (assumed) | yes (assumed) | Select it with `EXOMEM_RANKING_MODEL=BAAI/bge-reranker-v2-m3`. Its model card says only that it is multilingual. The all-scripts, cross-language declaration is an assumption that has not been measured here. Its cost is a design estimate, not a measurement: about 1.3-1.8 GB more memory and roughly 45 s per 30-pair rerank on 2 CPU cores. So it is meant for accelerated hosts. |
A reranker without a declaration is not gated: the owner configured it and is
trusted with it. The declaration is looked up by the exact model name, so a local
path or a mirror of `BAAI/bge-reranker-base` is undeclared and not gated either.
The gate works by script, not by language, because no language is detected.
Two consequences follow:
- A German or Estonian query whose answer is an English page is Latin on both
sides, so nothing marks it as crossing and the default reranker runs. On the
multilingual fixture this reranking can put a same-language look-alike above
the English answer. This is an open gap.
- A Japanese query written mostly in kanji reads as Han and is reranked.
## GPU notes (CUDA / Blackwell / Apple Silicon MPS)
Blackwell GPUs (RTX 50-series, compute capability 12.0 / `sm_120`) need CUDA
wheels. Default PyPI torch on Windows is **still CPU-only**, so `pyproject.toml`
pins an explicit CUDA index (`pytorch-cu132`, CUDA 13.2) whose wheel ships
`sm_120` — `torch.cuda.get_arch_list()` includes it, so any RTX 50-series GPU
works. The index is scoped to win32/linux via a platform marker; macOS falls back
to default PyPI — whose arm64 torch wheel **includes the MPS (Metal) backend**, so
Apple Silicon gets GPU acceleration for the torch models with **no extra wheels**
(see "Apple Silicon" below).
**Apple Silicon (MPS / Metal).** Device selection is centralized in
`exomem.accel.select_device` (priority CUDA → MPS → CPU). The torch models — bge
text embedder, bge reranker, and CLIP — auto-select the Metal GPU when no CUDA is
present, so interactive `find`, note-write embedding, and CLIP image search run on
the GPU on an M-series Mac. `PYTORCH_ENABLE_MPS_FALLBACK=1` is set automatically so
any op MPS lacks degrades to CPU instead of raising.
**ASR on the Metal GPU (`media-mlx` extra).** faster-whisper/ctranslate2 has no Metal
backend, but ASR runs behind a `TranscriptionBackend` seam (`extract.get_transcriber`)
with two implementations: `FasterWhisperBackend` (CUDA/CPU) and `MlxWhisperBackend`
(**mlx-whisper**, Metal GPU). Install the extra — `uv sync --extra media --extra
media-mlx` — and `get_transcriber()` **auto-selects MLX on Apple Silicon**, putting
transcription on the GPU too. `EXOMEM_ASR_BACKEND=mlx|faster-whisper` forces the choice;
`EXOMEM_MLX_WHISPER_MODEL` picks the HF repo (default `mlx-community/whisper-large-v3-mlx`;
use `mlx-community/whisper-large-v3-turbo` for a lighter, faster run on a fanless Air).
Without the extra, transcription falls back to CPU faster-whisper — pick a smaller
`EXOMEM_WHISPER_MODEL` (e.g. `base`) there to cut cost. Audio is decoded via PyAV (the
shared 16 kHz whisper timebase) when the `media` extra is present, else mlx-whisper
decodes the file itself (needs `ffmpeg` on PATH).
On MPS, bge/CLIP also run in **fp16** by default (`EXOMEM_MPS_FP16`, set `0` to keep
fp32) — roughly half the memory and faster encodes, which these retrieval models
tolerate well. Storage is unchanged (vectors are upcast to float32 before the sqlite
blob); existing fp32 vectors differ from new fp16 ones by ~1e-3, harmless for ranking,
and `audit_fix(rebuild_embeddings=True)` re-embeds for exact consistency if wanted.
Two more notes: **Voiceprints** (ECAPA) and **diarization** stay on CPU by default for
cross-machine numeric parity — opt in per model with `EXOMEM_VOICE_DEVICE=mps` /
`EXOMEM_DIARIZE_DEVICE=mps`. And you can force every torch model to a device with
`EXOMEM_TORCH_DEVICE=cpu|mps|cuda` (handy to dodge thermal throttling on a fanless
MacBook Air during a large `backfill-media`).
The `media` extra adds a wrinkle: `faster-whisper` runs on **ctranslate2**, which
wants **CUDA-12** cuBLAS/cuDNN/cudart, while torch's `cu132` build ships cuBLAS
**13** (a different major). So on Windows the `media` extra additionally installs
`nvidia-cublas-cu12`, `nvidia-cudnn-cu12`, and `nvidia-cuda-runtime-cu12`, and the
server prepends their `bin` directories to PATH before load (see
`extract._ensure_cuda_dll_path()` — `add_dll_directory` alone is not enough).
Verify with `uv run python scripts/verify-media-gpu.py`. On Linux, ctranslate2
resolves CUDA via the wheels' RPATH.
CLIP visual search runs CLIP on **CPU when ASR is active on CUDA** (whisper's cu12
cuDNN PATH-prepend otherwise shadows torch's cuDNN and breaks CLIP's Conv2d). This
clash is CUDA-only, so on Apple Silicon CLIP keeps the **MPS** GPU even with ASR
running. Override with `EXOMEM_CLIP_DEVICE=cuda`/`mps`/`cpu` if needed.
Zero-shot **image tags** (`EXOMEM_IMAGE_TAGS`, default off) reuse that same loaded
CLIP model — no extra dependency. When set, each extracted image is cosine-scored
against a fixed generic tag vocabulary and the top matches are appended to its
indexed text as a `Tags: invoice, table, screenshot` line, so a photo is findable
by what it depicts (not just its OCR text). It is a frozen cosine measurement (no
LLM), inherits the CLIP device logic above, and soft-fails to no tags when CLIP is
absent. Tune with `EXOMEM_IMAGE_TAGS_TOPK` (default 5) and
`EXOMEM_IMAGE_TAGS_THRESHOLD` (raw cosine, default 0.22); only newly-extracted
images are tagged.
**Video scene frames** (`EXOMEM_VIDEO_SCENE_FRAMES`, default off) upgrade video
indexing from uniform-interval keyframes to visual-change scene detection, and
persist one representative JPEG per scene in a `<video>.frames/` directory next to
the video (each with a sidecar pointing back at the parent + timestamp). Frames
ride the normal image OCR path, so on-screen text (slides, stack traces) becomes
keyword-findable at its timestamp; `find` groups frame matches under the parent
video's hit and surfaces `scene_frame` + `scene_match_at`. Detection is a cheap
I-frame-only metrics pass (PyAV `skip_frame NONKEY`, 64×64 grayscale hash +
histogram) — no new dependency, and it soft-fails back to the uniform sampler.
The worker budget stays modest: decoding I-frames of an hour-long 1080p video
takes seconds, plus at most `EXOMEM_MAX_VIDEO_KEYFRAMES` (40) full-res seeks.
Tune boundaries with `EXOMEM_VIDEO_SCENE_THRESHOLD` (hash bits of 64, default 10)
and `EXOMEM_VIDEO_SCENE_MIN_SECS` (default 4). To upgrade already-indexed videos,
run `exomem backfill-media` with the flag set — idempotent, and it replaces the
old uniform CLIP rows with scene-aware ones.
**Semantic video segments** (`EXOMEM_SEMANTIC_SEGMENTS`, default off) make the
moment the retrieval unit for audio/video. Transcripts render as one timed line
per ASR segment (`[51:20] …`, diarized `[51:20] [Alice]: …`), and the embedding
chunker segments them at fused topic boundaries — bge similarity valleys plus
scene-change, speaker-turn, and OCR-change events from the persisted scene
frames. A transcript match then surfaces `transcript_match_at` on the hit with
the nearest scene frame attached. Pure measurement, no LLM, no new dependency;
gate off is byte-identical. The worker re-embeds a video's sidecar once after
its frame OCRs so segments see all signals (bounded, off the request path).
Upgrade existing recordings with `exomem backfill-media --retime` (opt-in —
re-runs ASR; combine with `--rediarize` to gain both markers in one pass).
**Install:** `uv sync --extra media --extra diarization`, then build the isolated
sidecar with `pwsh -File scripts/setup-diarizer.ps1 -Prewarm` on Windows or
`sh scripts/setup-diarizer.sh --prewarm` on Linux/macOS. On a managed service, run
`<active release python> -m exomem.diarizer_sidecar --prewarm` once, with the
service's `EXOMEM_STATE_ROOT` when its unit sets one. The sidecar
venv lives under the machine-local state root, keyed by the sidecar's lock, so a
release switch keeps it. `scripts/upgrade.sh` builds a new one after a switch to a
release that changes the lock, while `EXOMEM_DIARIZE` is on.
Named-speaker diarization's ECAPA voice embedder (`diarization` extra) runs on
torch and follows the same precedent: it defaults to **CPU when ASR is active** (and
to CPU on Apple Silicon for cross-machine voiceprint parity), with a
`EXOMEM_VOICE_DEVICE=cuda`/`mps`/`cpu` override. Enroll voices with
`exomem enroll-speaker --name <name> [--self] <sample.wav>` (profiles live in a local
`.voice_profiles.json` beside the embedding sidecar; `list-speakers` / `remove-speaker`
manage them). With ≥1 profile enrolled and `EXOMEM_DIARIZE` set, matched clusters render
as `[<name>]: …`; unknown voices stay anonymous. The ECAPA checkpoint
(`speechbrain/spkrec-ecapa-voxceleb`, override `EXOMEM_VOICE_EMBED_MODEL`) is gated like
pyannote's — set `HUGGINGFACE_TOKEN`. Attribution thresholds are tunable via
`EXOMEM_VOICE_MARGIN`, `EXOMEM_VOICE_MERGE_THRESHOLD`, `EXOMEM_VOICE_CONFIDENT_DELTA`, and
`EXOMEM_VOICE_REL_GAP` (defaults match the shipped evidence-based values). The whole path is
default-off + soft-fail: with no profiles or the dep absent it degrades to today's anonymous
`[Speaker A]: …` output.
## Revoke access
Pick the strongest option that fits the situation:
| Situation | Action |
|---|---|
| Suspect the GitHub OAuth grant is compromised | Revoke at <https://github.com/settings/applications> → find `exomem` → Revoke. claude.ai's token dies on the next call (the verifier hits `api.github.com/user` per request). |
| Suspect the GitHub OAuth App secret leaked | Rotate the secret at <https://github.com/settings/developers> → `exomem` → "Generate a new client secret". Update `GITHUB_CLIENT_SECRET` in `.env`, restart the service. |
| Want to disconnect just claude.ai | Delete the connector in claude.ai → Settings → Connectors. |
| Want to take the endpoint offline entirely | `tailscale funnel --https=443 off` (or stop the Cloudflare tunnel). The endpoint becomes unreachable from the public internet. |
| Want to stop the service but leave the public URL configured | Stop the service (e.g. elevated `Start-Process -Verb RunAs -Wait sc.exe -ArgumentList 'stop','exomem'`). The tunnel stays up but proxies to nothing. |
| Want a clean uninstall | Stop + remove service, turn off the tunnel/Funnel, delete the connector in claude.ai, delete the GitHub OAuth App. |
### The remote owner binding
`EXOMEM_OWNER_OAUTH_SUBJECT=github:<numeric id>` in the service environment
makes remote sign-ins by that GitHub account act as the owner, labelled remote
(`principal_kind: owner-oauth` in the call ledger, with the remote caller hash).
Unset, the remote sign-in is a separate non-owner principal, as before. The
value must equal `EXOMEM_GITHUB_USER_ID`: `github:` then ASCII digits, no
leading zero. Anything else is treated as unset, and
`exomem doctor --profile remote` reports it as `malformed` or `mismatch`. It is
read only from the process environment (`service.env`, or the working
directory's `.env`), never from the vault, and a hosted cell clears it. The
service refuses to load a `.env` that is in, or resolves into, a vault and logs
`event=dotenv_refused` naming the file and the remedy: move the `.env` out of the
vault, or put the settings in `service.env`. If that file held required settings,
startup then stops on the missing one.
Enable it with one line and a restart. Before enabling, read doctor's
`governance.remote_owner_former_audience`: policy rules and grants that name the
remote connector's former `principal:` audience stop applying to it. Nothing is
migrated; removing the line and restarting restores the former audience and its
state.
On an HA pair, set the same line on every replica; otherwise the audience flips
with the replica that answers.
| Situation | Action |
|---|---|
| A remote token may be stolen | `exomem auth revoke -- SESSION_ID` for one session (`exomem auth sessions` lists them, marking the owner-equivalent ones), or `exomem auth revoke --all`. Put any options before `--`: an opaque session ID can begin with a hyphen. To drop owner power only, without signing anyone out, remove `EXOMEM_OWNER_OAUTH_SUBJECT` and restart; this works even while an HA coordinator is down. |
| The GitHub account is taken over, or you move to another account | Set `EXOMEM_GITHUB_USER_ID` and `EXOMEM_GITHUB_USERNAME` to the new account, and set `EXOMEM_OWNER_OAUTH_SUBJECT` to its id (or remove it), restart, and **always** run `exomem auth revoke --all`. Changing the allowed account only suspends the former account's sessions: they would come back, with owner power if the binding names that account again, the moment it is re-allowed. `revoke --all` ends them for good; every client then signs in once more. |
### Same-machine clients: the local listener
On a managed Linux/WSL install, both Cloudflare tunnels deliver public traffic
to the public listener on `127.0.0.1:8765`, so a loopback address does not mean
a local caller. Same-machine clients get their own door instead: a second
listener, on `127.0.0.1` only, that accepts nothing but per-client local tokens.
It is off by default. Turn it on with one line in `service.env` and a restart of
the service unit (the supervisor itself, not just an upgrade handoff):
```bash
EXOMEM_LOCAL_PORT=8764
```
`python -m exomem.service_upgrade --runtime-dir <runtime dir> --status` then
reports `local_port`. A malformed value, the public
port, or a port already in use leaves the local listener closed with a logged
reason; the public listener is never affected. Both listeners share one ingress,
so an upgrade pauses and drains them together.
The local listener refuses, before anything reaches the worker, any request
that came through Cloudflare (`cf-ray` or `cf-connecting-ip`), any request with
an `Origin` header, any `Host` other than literal `127.0.0.1` or `[::1]` (not
`localhost`), and every path except `/mcp`, `/api/*`, `/upload`, `/upload/sessions*`
and `/health*`.
It accepts only local tokens: an OAuth session, the REST key and the upload
token are all refused there, and a local token is refused on the public path.
Issue one token per client, into a new file that only you can read:
```bash
exomem auth issue-local --client home --output ~/.config/exomem/home.token
```
The command needs the same environment as `exomem auth sessions`
(`EXOMEM_JWT_SIGNING_KEY`, `EXOMEM_GITHUB_USER_ID` and `EXOMEM_GITHUB_USERNAME`)
and prints only the session id. Local tokens do not expire. `exomem auth
sessions` lists them with `ingress: local`; `exomem auth revoke -- SESSION_ID`
ends one, and `exomem auth revoke --all` ends them with every other session.
A request with a local token acts as the owner and is recorded as
`principal_kind: owner-local`; access log lines carry `ingress=local` and the
client label, never the token. Tokens give attribution and revocation, not
isolation: any process running as your user that can read the file can use it.
A client then points at `http://127.0.0.1:8764/mcp` with
`Authorization: Bearer <token file contents>`. Two helpers use the token file:
- The retrieve hook's REST rung tries the local listener first when
`EXOMEM_LOCAL_TOKEN_FILE` names the file and `EXOMEM_LOCAL_PORT` is set (in the
hook's environment or `service.env`). For one release it still falls back to
the key it lifts from `service.env` when the local request fails.
- `exomem attach <file> --scope <scope> --category <category>` sends a file's
bytes to the local `/upload` and prints the handle the service returns. It
reads `EXOMEM_LOCAL_TOKEN_FILE` and `EXOMEM_LOCAL_PORT`, or `--token-file` and
`--port`. Without `--scope` and `--category` the service holds the bytes
outside the vault and prints a `file` handle for `save_evidence` (or,
with `--lane source`, `save_source`). A held handle is redeemable once,
within an hour, and only by a request using the same local token, so point
the client's MCP connection at the local listener with that token.
`--raw-protection` keeps a directly preserved original owner-only until you
release the whole artifact; a held file takes that choice when its command
redeems it. A direct preserve on the local listener never crosses the proxy
edge, so its cap is `EXOMEM_LOCAL_UPLOAD_MAX_BYTES` (default 1 GiB), not the
public `EXOMEM_UPLOAD_MAX_BYTES` (default 100 MB, Cloudflare's edge cap). A held
file stays within what `save_evidence` can fetch, which on this server is
the session maximum below. The held files of one local session together stay
within that same cap, so one file of the largest size always fits. The
listener refuses a hold that does not fit in free disk with
`HELD_UPLOAD_NO_SPACE`.
- A remote client with a shell or a networked sandbox imports a folder by
sending it as one plain or gzip tar or zip: `upload_files(lane="import")`
returns a one-use upload, and `manage_memory` `imports.file` with its
`upload` token unpacks it into `_Imports/<id>/` and files it. Exomem Cloud
refuses that lane. The unpack streams, refuses links, devices,
absolute and `..` paths, encrypted entries and more than 65,536 entries, and
leaves nothing behind when it refuses. `EXOMEM_IMPORT_MAX_EXPANDED_BYTES`
(default 512 MiB) caps the bytes that one archive may unpack to; the unpack
runs inside one MCP request, so lower it on a slow volume, and
`EXOMEM_IMPORT_RESERVE_BYTES` (default 256 MiB) is the free space that an
unpack keeps back. The archive is held for at most an hour and released once
it is unpacked or refused.
- A file over 64 MiB, a file that the listener refuses as too large, or any
file with `--resumable` goes through a resumable upload session. The
listener's cap is in the service's environment, so `exomem attach` does not
read it; the listener's refusal tells it to use a session. `exomem attach`
keeps the session's address and secret in a private file under your state
directory. If the upload stops, run the same command again: it continues from
the last byte that the service holds. After a timed-out part, the command
reads the accepted offset before sending more bytes and reduces its part size.
If the service holds every byte but
cannot preserve them yet, for example because the disk is full, the command
says so and keeps the session. Run it again to retry the commit without
sending the bytes again.
- `--archive members` keeps a zip as its members. The service stores each
distinct member once per Evidence family, owner-only, and writes a manifest
last, so a later export writes only the members that changed. An archive
that the family already records answers `already_stored`.
Other clients can use the same sessions at `/upload/sessions` on either
listener through the tus 1.0 protocol, with one addition that a stock tus
client does not make. The client must capture the `Exomem-Upload-Secret`
header that creation returns and send it on every later request; the secret
never goes in a URL. Create a session with the credential that `/upload` takes
on that listener. The credential fixes the lane; metadata cannot change it.
Evidence requires `filename`, `sha256` (of the whole file), `scope` and
`category`. Source requires `filename` and `sha256`; its optional fields are
`title`, `source_kind`, `domain`, `url`, `content_type` and `raw_protection`.
Source uses the canonical Source writer and stores its original beside the
Source page. It does not accept `archive=members`. Evidence also accepts
`raw_protection` and `archive`. One request carries at most 64 MiB, and one session holds at most
`EXOMEM_UPLOAD_SESSION_MAX_BYTES` (default 2 GiB). A session expires 24 hours
after its last part.
A Source capability does not grant read access to its destination. If governance
withholds the custody receipt, the session retains the bytes and reports
`SOURCE_OUTCOME_UNKNOWN`. The canonical commit may already exist.
Use an authorized reader to inspect
custody, and reuse this session for a retry. A different caller's credential
does not change this session's transfer principal.
The service preserves the file only when its SHA-256 matches. `GET` on the
session returns its state and, after the commit, the receipt. The state
`retryable` means that the service holds every byte but the commit failed for
a reason other than the bytes; the next request on the session retries it.
When the reason is a busy or still-warming vault (`MUTATION_BUSY`,
`COLLECTION_STORE_BUSY` or `MUTATION_WARMING`), the service also retries the
commit itself, after 2 seconds and then after twice the previous wait, up to
60 seconds, until the session expires.
A deployment behind Cloudflare Access must let `HEAD`, `PATCH`, `GET` and
`DELETE` requests to `/upload/sessions/*` through without an Access login.
These requests carry only the session secret. Creation, a `POST` to
`/upload/sessions`, still needs the upload credential.
The owner REST key and the static upload token keep working on the public path.
Their use on a Cloudflare-transited request is logged as
`event=owner_credential_transit`, so a later release can refuse them there once
no remote client uses them. It is kept out of `/metrics.json`, which is served
without authentication.
### Resumable uploads from a shell
`transfer_artifact(operation="upload", lane="evidence")` returns a `resumable`
object beside `upload_url`. A Source handoff advertises the same protocol: the
Source writer keeps every original opaque, in every storage authority mode.
Never change the storage authority mode or use Evidence to bypass a Source
refusal.
The handoff supplies `create_url`, `max_part_bytes`, `max_bytes` and the required
metadata keys. The minted token authorizes creation. Later requests carry the
session secret, so an existing upload can outlast that token. These personal
server routes do not lift the separate Cloud gateway exclusion.
For local Evidence uploads, use the maintained client:
```sh
exomem attach export.zip --scope Device --category Exports --resumable
```
The client reads its local token and port from the configuration above. It
keeps its session secret in a private file. Run the same command after an
interruption to resume accepted bytes or retry a pending commit. Held file
handles and `--handoff-file` keep their separate one-request contract.
For a remote or Source upload, use a tus client that retains the additional
session-secret header. Follow this sequence:
1. Mint a handoff for the intended lane; expect its supported `resumable` object.
2. Create one session with its token and required metadata; retain the returned address and secret privately.
3. Send bounded parts with the session secret; expect an advancing `Upload-Offset`.
4. After an uncertain part response, read `HEAD` before retrying; expect the service's accepted offset.
5. Resume from that offset with smaller parts; expect previously accepted bytes to remain stored.
6. Read the session until it commits; expect an exact-hash receipt or an explicit pending or failed state.
A lost creation response does not supply a session address. Reconcile that
creation before creating another session. An expired creation token needs a
new token; an existing session continues with its secret. An expired session
answers `UPLOAD_SESSION_EXPIRED` and needs a new session.
`retryable` means that the service still holds the bytes. Retry the same
session; do not create a new identity for an uncertain Source commit.
`SOURCE_OUTCOME_UNKNOWN` also retains the bytes when recovery proves a commit
without recovering its complete receipt. That state needs reconciliation and
does not authorize another Source publication.
A Source receipt preserves the original and its Source page. Its
`semantic_compilation` field remains `unassessed` until an agent compiles the
content into governed knowledge. Media status reports recorded processing and
activity separately. `health_scope="reconciliation"` does not prove active
inference; an unobserved worker remains `activity.state="unknown"`.
## Deploying a new version
For an opt-in managed Linux/WSL service, `bash scripts/upgrade.sh` stages a new
worker environment and replaces the worker while the public endpoint stays up.
The supervisor's interpreter stays fixed; the private managed status command
identifies the active worker release. Follow the
[managed upgrade procedure](managed-service-upgrades.md) for this mode.
**The service interpreter is the source of truth — not the checkout you are standing in.**
A service can run from a standalone venv while its `AppDirectory` points at a checkout, so
the checkout looks authoritative and is not. When that happens, `uv sync` in the checkout
changes nothing the service runs and `/health` keeps reporting the old version through any
number of clean restarts.
Ask the running server where it came from:
```bash
curl -s http://127.0.0.1:8765/health
```
```json
{"status": "ok", "service": "exomem", "version": "0.25.5",
"install_source": "wheel", "revision": null,
"torch": "2.13.0", "accelerated": false, "extras": ["embeddings", "media"]}
```
`install_source` is the field that matters. `wheel` with a null `revision` means the server
is **not** running a local checkout — upgrade the venv directly. `editable` reports the git
`revision` it is serving. For local CLI/service release, profile, route, and
interpreter identity, run `exomem --version --json` (or `exomem install-info`);
`/health` withholds local paths because it is unauthenticated and publicly
reachable. A false `version_match` means the shell command and managed service
are split and should be reconciled before treating the upgrade as complete.
The scripted path resolves the interpreter from NSSM, gates on `doctor` and accelerator
capability, restarts, and verifies the running version:
```powershell
pwsh -File scripts/deploy.ps1 -Version 0.25.5
```
It fails rather than reporting success if the running server does not serve the requested
version after restart.
### The CUDA pin does not survive a PyPI upgrade
`[tool.uv.sources]` pins torch to the `pytorch-cu132` index, but that is **repo
configuration, not wheel metadata** — it does not travel with the published `exomem` wheel.
Upgrading a PyPI-backed venv with `uv pip install` therefore resolves torch from default
PyPI, which on Windows is CPU-only, silently replacing `2.12.0+cu132` with a bare `2.13.0`.
`deploy.ps1` treats that as a hard failure. On a genuinely CPU-only host, pass
`-AllowCpuTorch`. To restore the pinned wheel:
```powershell
uv pip install --python <service venv python> --index-url https://download.pytorch.org/whl/cu132 --upgrade torch
```
Note the difference in meaning between the two signals: `/health` reports **which wheel is
installed** (a deploy property, read from metadata without importing torch), while `doctor`
reports **whether a GPU is usable right now**. Conflating them is what let this regression
hide as one advisory line among passing checks.
## Restarting the service
Remote Streamable HTTP runs statelessly: authenticated tool calls do not rely
on a process-local `Mcp-Session-Id`. A service restart or active/passive route
change therefore cannot strand a transport session on the old process. OAuth
is still enforced on every request; a genuinely expired, revoked, or invalid
credential still requires authorization, but a normal restart does not.
**macOS / Linux:** `bash scripts/restart.sh` — launchd `kickstart -k` on macOS,
`systemctl --user restart` on Linux. It truncates `logs/exomem.log` and tails it
(a repo-mode/dev install; a packaged install's `exomem.log` lives under its
resolved log directory instead — see § Logs below).
**Windows:** `install-service.ps1` grants your user account start/stop rights on
the service, so day-to-day restarts don't need UAC:
```powershell
sc.exe stop exomem
sc.exe start exomem
$logDir = ((exomem doctor --json | ConvertFrom-Json).checks | Where-Object { $_.id -eq 'observability' }).details.log_dir
Get-Content "$logDir\exomem.log" -Tail 6
```
(`install-service.ps1` pins `EXOMEM_LOG_DIR` — `%ProgramData%\exomem\logs` by
default — into the service's environment for a `-Release` install, so its
`exomem.log` is NOT under the repo checkout's `logs\` folder. `exomem doctor`
reports the resolved path directly, which the command above reads rather than
hardcoding it, so it stays correct whether the box uses the default or an
operator-pinned override.)
If you skipped the grant (or installed from an older version of the script),
re-run the install script — it's idempotent and will only add the ACE if it's
missing.
On a box still running the pre-rename `kb-mcp` service, `scripts/restart.ps1`
auto-falls back to that name; use `sc.exe stop/start kb-mcp` for the manual form,
and see [Renaming an existing `kb-mcp` service](#renaming-an-existing-kb-mcp-service)
to migrate it to `exomem`.
For a stuck restart (orphan python processes holding port 8765), force-clean:
```powershell
sc.exe stop exomem
Get-Process python -ErrorAction SilentlyContinue | Stop-Process -Force
sc.exe start exomem
```
## Logs
- `exomem.log` — application log (rotated in-process, 5 MB × 5 files; same on
every platform), under the **resolved log directory**
(`logging_config.resolve_log_dir()`, see `docs/observability.md`): the repo
`logs/` folder for a repo-mode/dev checkout, or the pinned/per-platform
location (`%ProgramData%\exomem\logs` by default on a Windows
`-Release`/service install) otherwise. `exomem doctor --json` always reports
the actual resolved path under the `observability` check's `log_dir` detail
— read it from there rather than assuming `logs/exomem.log`.
- `logs/service.out.log`, `logs/service.err.log` — service stdout/stderr,
always under the checkout's own `logs/` folder regardless of profile or
`EXOMEM_LOG_DIR` (NSSM's `AppStdout`/`AppStderr` redirect targets, set by
`install-service.ps1` relative to the checkout it was run from — a separate
concern from the app's own structured logging). On Windows NSSM writes and
rotates these; launchd/systemd write them but do **not** rotate them (the
app's own `exomem.log` is the durable, self-rotating record). On Linux,
`journalctl --user -u exomem` is the primary stdout/stderr view.
## Troubleshooting
| Symptom | Likely cause | Fix |
|---|---|---|
| claude.ai "Couldn't reach the MCP server" during connector add | OAuth discovery failed | `curl.exe -i https://<your-host>/.well-known/oauth-authorization-server` should return JSON (or run `exomem doctor --profile remote --probe`, which checks this plus the bare well-known path). If 404, the OAuthProxy isn't mounted — most likely `EXOMEM_BASE_URL` has a trailing slash or includes `/mcp`. |
| GitHub redirects to "The redirect_uri MUST match…" error | OAuth App callback URL mismatch | At github.com/settings/developers → exomem, set the callback to exactly `https://<your-host>/auth/callback` (no trailing slash). |
| GitHub: "The redirect_uri is not associated with this application" on a *second* machine | Reused another host's OAuth App client ID/secret (the app's one callback points at the other host) | Create a per-host OAuth App with callback `https://<this-host>.example.com/auth/callback`, put its client ID/secret in this `.env`, restart the service. See § Deploying on a second machine. |
| claude.ai connector connects but every tool call returns 401 | Wrong GitHub user | `EXOMEM_GITHUB_USERNAME` must equal the login of the GitHub account you authorized with. Check the exomem log for `rejecting token for github login=...`. |
| Existing client receives one `401 invalid_token` after the durable-session cutover | Legacy FastMCP reference JWT is intentionally not dual-read | Complete one final login without deleting or changing the connector URL. If a durable session later gets 401, inspect it with `exomem auth sessions`. |
| Authenticated tools return `503` without `WWW-Authenticate` | Authoritative session storage is unavailable or corrupt | Repair the coordinator/network/storage path. Do not reauthorize: a fresh login uses the same unavailable authority. |
| claude.ai shows "connector failed" | service down (host asleep, service stopped, crash loop) | Check the service status; tail `logs/service.err.log` and `exomem.log` (see § Logs for its resolved path — not always under `logs/`). Multiple startup banners within seconds = orphan python processes — kill them and force-restart. |
| Edits to `.env` not picked up | service didn't restart | Restart the service (elevated on Windows). Confirm the python process restarted. |
| 404 / Funnel "no service" | Tunnel disabled or pointing at the wrong port | `tailscale funnel status` (or check `cloudflared`); re-run the tunnel command from step 2. |
| `KB vault not found` on startup | vault path moved or `EXOMEM_VAULT_PATH` wrong | set `EXOMEM_VAULT_PATH` to the absolute vault root in `.env`. |
| Schema parse error on startup | `_Schema/references/frontmatter.md` shape changed | diff against the version that was working; the parser is conservative on purpose. |
| Connector setup stalls mid-burst behind ngrok | Free-tier rate limit (120 req/min) | Check the ngrok agent console for `429`s. Wait and retry, or upgrade / switch to Cloudflare Tunnel. |
| OAuth redirect hangs the first time behind ngrok free | The one-time browser interstitial | Open the public URL once in a browser and click through the interstitial; it does not recur. |
| `add` fails with `INVALID_SOURCE` | missing required field (url for article/paper/video; non-empty content/title) | the error payload names the missing field; fix and retry. |
## Out of scope
The remote tier is intentionally minimal. Not included:
- Auth layers beyond single-user GitHub OAuth (no mTLS, IP allowlist, multi-user
RBAC).
- Web UI.
- Highly available coordinator hosting (the bundled SQLite coordinator is a
strongly consistent single-node reference).
- Compiled-note creation from mobile (`add` only captures raw sources;
compilation stays a desk-side / Claude Code flow).