ARCHITECTURE.md

Rama main · réplica de lectura del repositorio original.

# Aura System Architecture

**Last updated:** 2026-09-30  
**Host:** Single Linux VPS (Europe/Berlin timezone on server; Kirill in JST)  
**Status:** Production live (`development_mode: false`)

This document describes the **runtime** of **Aura** — OpenClaw + RMP on this VPS. Binding constitution (why / must / conflict law): [`docs/CONCEPT_TREE.md`](docs/CONCEPT_TREE.md). Cursor agents must not violate that file.

Related logs:

- [`docs/CONCEPT_TREE.md`](docs/CONCEPT_TREE.md) — source of truth
- [`docs/CONCEPT_TREE_PROGRESS.md`](docs/CONCEPT_TREE_PROGRESS.md) — constitution work log
- [`docs/history/DEVELOPMENT_PLAN.md`](docs/history/DEVELOPMENT_PLAN.md) — Phases 0–11 (historical)
- [`docs/history/PRODUCTION_PLAN.md`](docs/history/PRODUCTION_PLAN.md) — Phases 12–20 (historical)
- [`docs/CONTROL_PLANE_PROGRESS.md`](docs/CONTROL_PLANE_PROGRESS.md) — Analyst control-plane log (append-only)
- [`docs/SYSTEM_AUDIT_PROGRESS.md`](docs/SYSTEM_AUDIT_PROGRESS.md) — Sep 29 2026 system audit and hardening log (append-only)
- [`docs/DEEP_MEMORY_PROGRESS.md`](docs/DEEP_MEMORY_PROGRESS.md) — deep memory and two-phase answering log (append-only)
- [`docs/runbooks/slack-sockets.md`](docs/runbooks/slack-sockets.md) — this-host Slack sockets only; never `apps.connections.open`

---

## 1. Executive summary

Aura is a **three-layer system** on one machine:

| Layer | Path | Role |
|-------|------|------|
| **Safe Harbor** | `/root/aura_safe_harbor` | Legacy scripts, Moltbook scanners, watchdog (`restore_powers.js`), Kairos modules |
| **OpenClaw** | `/root/.openclaw` | Agent runtime: Slack gateway, LLM, workspace, cron, plugins |
| **RMP** (Reliability & Memory Plane) | `/root/.openclaw/rmp` | Sidecar API + Temporal workflows + Postgres ledger + vector memory + LLM orchestration |

**How the layers combine:** OpenClaw is the **runtime** (Slack socket, gpt-6-luna + NVIDIA gpt-oss-20b fallback, tools, JSONL sessions, cron). RMP wraps it as a **sidecar control plane**: the `rmp_adapter` plugin intercepts inbound messages, creates durable tasks, and blocks the main/Slack session so work runs in isolated `rmp_task_*` sessions under Temporal. Safe Harbor supplies legacy scanners and watchdog scripts; RMP can invoke or sync them when not in development mode.

**Design intent:** User-facing work (Slack DMs, cron) is **routed through RMP** so every turn becomes a durable Temporal workflow with steps, observations, evidence checks, and idempotent Slack delivery. OpenClaw remains the **execution engine** (tools, LLM, JSONL sessions); RMP is the **control plane** — it owns plans, step predicates, process memory, completion gates, reconciliation, and **LLM key orchestration** (balanced rotation, concurrency caps, usage accounting).

**Primary LLM (chat):** OpenAI GPT-6 Luna (`openai/gpt-6-luna`) over the OpenAI Responses API on OpenClaw runtime (`agentRuntime.id: "openclaw"`), with NVIDIA-hosted GPT-OSS 20B (`nvidia/openai/gpt-oss-20b`) as the fallback after auth-key rotation. Chat Completions is not used: it rejects function tools with any reasoning effort on gpt-6-luna and has no `max`. **Intake** uses the same chain; subagent sessions, including the Process Evaluator, run on gpt-6-luna. Thinking: `max` for Aura's user-task runs (`send_to_openclaw` passes it per run), `medium` for everything else (`agents.defaults.thinkingDefault`: intake, evaluator, canaries). GLM-5.2, DeepSeek V4 Flash and MiniMax M3 are dropped (NVIDIA HTTP 410; MiniMax ended 2026-09-09). HTTP 410 is treated as skip, not a 5s idle retry.

**Vector embeddings:** Advisory only. Live embedder is OpenAI `text-embedding-3-small` (1536-d) after NVIDIA NIM embeddings returned HTTP 410 EOL. `/health` `ready=true` only after a successful embed probe. Conversational continuity still uses Postgres `task_messages` + process memory and must not depend on embeddings. Postgres holds every memory; Qdrant is its index, kept in step by a transactional outbox and a daily reconcile, with Postgres full-text search as the fallback.

**OpenAI storage:** off (`store: false`, patch `RMP_OPENAI_NO_STORE`). Reasoning carries between turns as encrypted content, so OpenAI keeps no prompts or replies.

**Rate-limit policy:** On NVIDIA **rate limits**, RMP **waits, rotates keys, and tracks usage** (does not hop providers for 429s). Separate from that, OpenClaw keeps an ordered **model fallback chain** for unavailable/broken models. OpenAI key: `OPENAI_API_KEY` in `/etc/openclaw/openclaw.env` (SQLite profile `openai:default`). Three NVIDIA accounts (`nvidia:default`, `nvidia:key2`, `nvidia:key3`) with **balanced load** and a **max concurrent agent-run cap** of 4 (3 user slots + 1 canary), plus a 1-slot intake lane outside the cap so intake never waits behind Aura's long runs. Never pin `nvidia:keyN` on an `openai/*` session.

---

## 2. High-level architecture

```
┌─────────────────────────────────────────────────────────────────────────────┐
│                           Kirill (Slack DM)                                 │
└───────────────────────────────────┬─────────────────────────────────────────┘
                                    │ Socket Mode
                                    ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│  OpenClaw Gateway (:18789)          /root/.openclaw/openclaw.json           │
│  • Slack provider                                                           │
│  • Heartbeat off (every "0m", enforced by RMP model_policy)                 │
│  • Cron (MoltMarket notifications, etc.)                                    │
│  • Plugin: rmp_adapter (/root/.openclaw/plugins/rmp_adapter)                │
│  • auth.order: nvidia:default → nvidia:key2 → nvidia:key3                   │
└───────────────┬───────────────────────────────┬─────────────────────────────┘
                │ before_message_write          │ hooks/agent (internal)
                │ (sync block → RMP)            │
                ▼                               ▼
┌───────────────────────────┐     ┌───────────────────────────────────────────┐
│  RMP API (:8000)          │     │  OpenClaw Agent (gpt-6-luna / NVIDIA fb)  │
│  FastAPI                  │     │  • Tools, skills, workspace files           │
│  POST /tasks              │────▶│  • Per-task session: agent:main:rmp_task_*│
│  signals, memory, export  │     │  • JSONL poll for completion              │
└───────────────┬───────────┘     └───────────────────────────────────────────┘
                │
                ▼
┌─────────────────────────────────────────────────────────────────────────────┐
│  Temporal (:7233)  +  rmp-worker                                            │
│  • GenericTaskWorkflow — plan-driven steps + child workflows                │
│  • CatalogTaskWorkflow — 6 step templates                                   │
│  • Reconciler + janitor — stale/stuck/orphan repair                         │
└───────────────┬─────────────────────────────────────────────────────────────┘
                │
    ┌───────────┼───────────┬──────────────┬────────────────┐
    ▼           ▼           ▼              ▼                ▼
 Postgres    Qdrant     Artifacts      Side-effect       Safe Harbor
 (rmp_db)   (vectors)   (SHA-256 FS)   receipts          scanners/watchdog
              ▲
              │ embeddings (advisory; fail-soft if embedder down)
              └── Mem0 + local Qdrant

        ┌─────────────────────────────────────┐
        │  LLM orchestration (RMP)            │
        │  quota_broker + usage_monitor       │
        │  balanced rotation · max 2 slots    │
        │  /data/llm_quota.json               │
        │  /data/llm_usage.json               │
        └─────────────────────────────────────┘
```

### systemd services

| Unit | Purpose |
|------|---------|
| `openclaw-gateway` | Slack + agent gateway; env from `/etc/openclaw/openclaw.env`; **ExecStartPre:** `ops/sync_nvidia_keys.py`, non-blocking (`-`) |
| `rmp-api` | FastAPI; `/etc/rmp/rmp.env` + `/etc/openclaw/openclaw.env`; non-blocking key sync pre-start |
| `rmp-worker` | Temporal worker (`openclaw-tasks` queue); same env + non-blocking key sync |
| `temporal` | Official Temporal server 1.30.1 on this host's Postgres (`temporal`, `temporal_visibility`), frontend `127.0.0.1:7233`. One node, not Temporal Cloud. |
| `rmp-canary.timer` | Hourly health check (`*:07`) |
| `rmp-canary-sentinel.timer` | Every 30 min: health/memory canaries, runtime code sync, LLM usage (24 h budget, live context > 60k, abort rate > 10%) → ops Slack alert with a 4 h cooldown |
| `rmp-temporal-watchdog.timer` | Every 5 min Temporal probe. Full recovery (`ops/temporal_recover.sh`) only when the probe itself exits 1 |
| `rmp-code-watch` | Restarts `rmp-api` + `rmp-worker` on `app/*.py` or `worker.py` changes, only once no user task is active |
| `rmp-memory-canary.timer` | Memory canary at 04:37, 10:37, 16:37, 22:37 CEST |
| `rmp-janitor.timer` | Daily workflow janitor (`ops/workflow_janitor.py`) |
| `rmp-backup.timer` | Full nightly backup at 03:15 UTC (`ops/backup.sh`) with a verified manifest; local only (accepted exception K-7) |

Every RMP unit file is tracked in `ops/systemd/`, with drop-ins in `ops/systemd/<unit>.d/` and the units that must be enabled in `ops/systemd/enabled-units.txt`. The deploy never installs them: it names the changed files in its result and in Kirill's note, and readiness's `host_file_drift` warns until they are installed. After the deploy, an operator reviews each new or changed file's diff and the commit that brought it, and gets Kirill's explicit OK first for a unit change that does not come from a reviewed programme PR. The operator then runs the command below, a dry run that prints the commit it planned from on a `head:` line and lists the new and changed files, host-only units and units to enable, and runs it again with `--apply --head <sha>` appended in an idle window, `<sha>` being that `head:` line's commit. `--apply` refuses unless `HEAD` is still that commit, and while the checkout has a git replace ref, a user task is active or the code-reload lock is held. It installs that commit's bytes, backs up every file it replaces, runs one `daemon-reload`, enables the listed units that are not enabled and prints a `rollback:` line, after a failed step too. It never starts, restarts, disables or deletes a unit: the operator starts each unit on a `start:` line in an idle window.

```bash
systemd-run --wait --pipe --collect --unit=rmp-install-units -p WorkingDirectory=/root/.openclaw/rmp \
  -p Environment=PYTHONPATH=/root/.openclaw/rmp -p EnvironmentFile=-/etc/rmp/rmp.env \
  -p EnvironmentFile=-/etc/openclaw/openclaw.env \
  /root/.openclaw/rmp/venv/bin/python /root/.openclaw/rmp/ops/install_units.py
```

---

## 3. Message flow (Slack → reply)

### 3.1 Normal Slack DM

1. Kirill sends a DM → OpenClaw Slack provider receives it (session key `agent:main:slack:channel:…`).
2. **`rmp_adapter`** claims the turn (`inbound_claim` / `message_received`) and `POST /tasks` with the **full Slack text** (not a truncated intent). **No native OpenClaw Slack fallback** — fail closed (claim + suppress) if intake/API is down.
3. **`before_message_write`**: blocks Slack DM persistence / native assistant turns while RMP owns delivery (`{ block: true }`).
4. **Intake Analyst** (not Aura) classifies the message against hybrid-retrieved evidence into one of four relation classes, then applies a decision (`clarify` / attach / wait / rebuild / guided / fresh). See §5.0.0. The request's own reservation row (`intake_reserved`) is never evidence; when intake handles the message elsewhere, it closes as `cancelled` with `closed_reason: intake_placeholder` and stays out of the task registry.
5. If work proceeds, Temporal starts **GenericTaskWorkflow** or **CatalogTaskWorkflow** (catalog type is **intake-LLM only**). Unless intake answered `recall_depth: none`, the run starts the IA's deep recall beside it (§6.3), which never delays it.
6. Aura executes in `agent:main:rmp_task_*` via **`send_to_openclaw`** (`deliver: false`), with one memory block: the IA's fast context (§6.3), plus the recall report once it is ready.
7. **Process Evaluator** (not Aura; session `rmp_verify_*`) must **accept** the result before Slack. Conversational replies are gated too. Canary/system stay on the short deterministic path. It sees Aura's action trace (tool calls and results from her transcript) and the run's artifacts. If the evaluator itself fails, the run waits for it on durable timers (1, 2, 4 … 60 min), sends a hold notice after two failures, and after the last one closes the task without sending the unchecked draft. Messages that arrive while Aura works, while she is judged or during rework are folded into the draft, which is judged again.
8. **`notify_slack_user`** → idempotent RMP `chat.postMessage`, in ordered parts of at most 3,500 characters. Transient Slack errors retry; a permanent one is recorded (`slack.delivery_failed`) and alerted, and the task ends `failed` with `closed_reason: slack_delivery_failed` (Kirill, Sep 30 2026), with no further notice Slack would refuse too. Insufficient work is reworked (attempts 1–19, with one strategy change at 10) or escalated with a diagnosis (attempt 20). Messages that arrive after the reply go back to intake (`resubmit_user_messages`).
9. **Two-phase answering.** When the recall report arrives after the reply, the task stays running for at most `followup_wait_sec`. The IA's novelty judge decides whether the report adds to or corrects the reply. If it does, RMP sends a notice, Aura refines in `rmp_task_*__recall`, the evaluator judges the refinement as a follow-up, and only an accepted one is sent (`task_messages.kind = followup`). At most one per task; a stop or a new message during the wait lets the recall go.

### 3.2 Cron (e.g. MoltMarket)

Same path via plugin (`tags: cron`). OpenClaw cron job uses `delivery.mode: none`; RMP owns Slack delivery when actionable.

### 3.3 Heartbeat

**Off.** `apply_openclaw_policy` enforces `agents.defaults.heartbeat.every: "0m"`, so OpenClaw keeps its `heartbeat-main` monitor job disabled across upgrades. RMP canaries own liveness. The old `agent:main:heartbeat` session is archived with its transcript kept. It never used a tool, yet it re-billed its growing context every 30 minutes (89% of transcript tokens). If a heartbeat ever runs again, the plugin still creates no RMP task, **`HEARTBEAT_OK`** is not delivered, and reserve/release skip `trigger === 'heartbeat'`.

### 3.4 Hourly canary

`rmp-canary.timer` → `ops/canary.sh` → `POST /tasks` with intent `RMP CANARY: Reply with exactly CANARY_OK`.  
Success/failure is logged internally; **no Slack notification** to Kirill (canary/system suppression).

---

## 4. OpenClaw layer

| Item | Location / value |
|------|------------------|
| Config | `/root/.openclaw/openclaw.json` |
| Workspace | `/root/.openclaw/workspace` (`USER.md`, `MEMORY.md`, `memory/*.md`, `HEARTBEAT.md`) |
| Agent sessions | SQLite `/root/.openclaw/agents/main/agent/openclaw-agent.sqlite` (`session_nodes`, `session_windows`, `transcript_events`); legacy JSONL under `agents/main/sessions/` |
| Primary model | `openai/gpt-6-luna` (`api: openai-responses`, `agentRuntime.id: "openclaw"`); fallback `nvidia/openai/gpt-oss-20b`; intake same chain; subagents gpt-6-luna; thinking `max` for user-task runs, `medium` default |
| Model fallbacks | gpt-6-luna → gpt-oss-20b on NVIDIA (no GLM, no DeepSeek, no MiniMax). HTTP 410 = skip to next model |
| Concurrency | OpenClaw `maxConcurrent: 2`; RMP `max_concurrent: 3`, `min_interval_sec: 5` |
| LLM keys | `/etc/openclaw/openclaw.env`: `OPENAI_API_KEY` (gpt-6-luna), `NVIDIA_API_KEY`, `NVIDIA_API_KEY_2`, optional `_3` |
| Auth profiles | Shared SQLite `authProfiles.store` in `state/openclaw.sqlite` (2026.9+); synced from env via `ops/sync_nvidia_keys.py`. Do not recreate leftover `auth-profiles.json` (triggers AUTH_PROFILE_MIGRATION_REQUIRED). |
| Auth rotation order | `auth.order.openai`: `openai:default`. `auth.order.nvidia`: `nvidia:default` → `nvidia:key2` → `nvidia:key3`. Never pin `nvidia:keyN` on `openai/*` sessions.
| Plugin | `/root/.openclaw/plugins/rmp_adapter/index.js` |
| Slack sockets | This host: exactly one `openclaw-gateway`. See [`docs/runbooks/slack-sockets.md`](docs/runbooks/slack-sockets.md). Never `apps.connections.open`. Other hosts undetectable. |
| Slack streaming | `{"mode":"off","nativeTransport":false}` (RMP owns Slack delivery; object form required by OpenClaw 2026.7+) |
| OpenClaw version | `2026.9.7` on Node 24 from NodeSource `node_24.x` (2026.9.7 requires Node ≥ 24.16.0 on 24.x) |
| Utility model | `agents.defaults.utilityModel: ""` (RMP model policy; the verifier checks it). Unset, 2026.9.7 derives `gpt-5.6-luna` for its own activity recaps and progress narration, which send transcript excerpts |
| Hook sessions | `/hooks/agent` uses `sessionMode: persistent` plus `hooks.allowedSessionKeyPrefixes: ["hook:", "agent:main:rmp_"]` (2026.9 defaults isolated and will not persist `rmp_*` keys otherwise) |
| Post-update patches | `ops/upgrade_openclaw.sh` (or `patch_openclaw.sh` then `ops/verify_openclaw_patch.sh`; both finish with `ops/openclaw_patch_audit.py` over every `.js` and `.mjs` file; see §4.1) |

OpenClaw also has built-in profile cooldown/rotation on 429; RMP’s quota broker **caps cooldowns**, **balances load across keys**, **limits concurrent agent runs**, and syncs `authProfiles.state` `usageStats.cooldownUntil`.

### Plugin hooks (`rmp_adapter`)

| Hook | Behavior |
|------|----------|
| `message_received` / `inbound_claim` / `before_dispatch` | Slack DM → mark claimed and return `{ handled: true }` at once (native OpenClaw never replies); `POST /tasks` runs in the background, one route per session in arrival order, with timeout recovery and RMP notices (fail closed) |
| `before_message_write` | Synchronous (OpenClaw ignores a returned Promise). Blocks Slack DM / assistant writes while RMP owns delivery; cron task creation runs after the hook returns; skips internal heartbeat routing |
| `message_sending` | Suppress native Slack during active RMP user task; strip interim tool-planning text; cancel pure-ack messages |
| `before_agent_run` | **`POST /api/llm/reserve`** — balanced key + concurrency slot for gateway sessions (not `rmp_task_*`, `rmp_verify_*`, `rmp_intake_*`, Slack/main, heartbeat) |
| `agent_end` | **`POST /api/llm/release`** — free slot for same session exclusions |
| `llm_output` | **`POST /api/llm/record-gateway`** — token/request accounting for gateway LLM turns |
| `before_tool_call` | In `rmp_task_*` / `rmp_verify_*` / `rmp_intake_*` sessions, blocks `ask_user` and `secrets` `request`: they ask Kirill through a channel RMP suppresses, so the run would sleep until their 15-minute timeout. Aura is told to ask in her reply |
| Assistant ack block | Pure system acks not written to `agent:main:main` transcript |

The plugin never blocks the gateway's event loop: every RMP call is async `fetch` with `AbortSignal.timeout`, and the session-key lookup reads the OpenClaw store in-process (`node:sqlite`). The same gateway serves intake's LLM leg while `POST /tasks` waits on it; a synchronous call froze it for the whole intake (~70 s).

On task create the plugin also **prefetches** process memory (`GET /memory/process/{id}/context`) and passes `initial_memory_block` into the workflow payload.

Tools exposed to the agent: `rmp_task_create`, `rmp_task_status`, `rmp_memory_recall` (optional; normal path is automatic routing). OpenClaw 2026.9 calls plugin tools as `execute(toolCallId, params, signal)`; `rmp_task_create` is a tool factory so it gets the session from the factory context. `tests/node/*.test.js` call every plugin tool that way.

### 4.0.1 Galaxy web capability stack

OpenClaw plugin **`aura_web`** (`/root/.openclaw/plugins/aura_web`) plus localhost FastAPI **web-stack** (`http://127.0.0.1:8791`, systemd `aura-web-backends`) give Aura multi-backend web tools:

| Class | Tools |
|-------|--------|
| Search | Brave `web_search` (default), `langsearch_search` (key in `plugins.entries.langsearch`) |
| Fetch | `jina_reader` (r.jina.ai), built-in `web_fetch`, `crawl4ai` |
| Crawl / extract | `crawl4ai`, `crawlee_crawl`, `scrapling`, `scrapegraph_extract` |
| Interact | OpenClaw `browser`, `browser_use`, `obscura_browse` |
| Status | `web_capability_status` |

RMP **`WebCapabilityAnalyzer`** (`app/orchestrator/web_capability.py`) classifies intake intents (`search|fetch|crawl|adaptive_extract|schema_extract|interact|none`), injects a **WEB CAPABILITY BRIEF** into execute prompts / `initial_memory_block`, and may *suggest* `browser_automation` as a soft note. **Catalog workflow assignment is intake-LLM only** (`catalog_hint` → policy). Keyword/regex catalog patterns are advisory (`soft_catalog_candidates` in the intake prompt) — they must not hard-route Slack DMs after intake. Agent-visible docs: workspace `TOOLS.md`.

Paid APIs (Perplexity, Firecrawl, Tavily) are **not** configured unless keys are added later.

### 4.1 OpenClaw dist patches (re-apply after every `npm install -g openclaw`)

Run: `bash /root/.openclaw/rmp/ops/upgrade_openclaw.sh` (`make upgrade-openclaw`). Never `openclaw onboard`, never `doctor --force`, never hand-edit dist.

| Patch | Why |
|-------|-----|
| Hook persistence | Always run `before_message_write` when `hookRunner` exists (typed `hasHooks` can miss the plugin) |
| Announce suppress | Skip native subagent announce for `rmp_(task\|verify\|intake)_` sessions |
| Minimal bootstrap | TOOLS.md only for those RMP sessions |
| Slack suppress | `deliverReplies` calls `__RMP_SUPPRESS_NATIVE_SLACK` so RMP owns delivery |
| allowUnsafe passthrough | OpenClaw 2026.7 dropped `allowUnsafeExternalContent` from HTTP `/hooks/agent` normalize; RMP needs it (or auto-enable for `rmp_*` keys) to avoid EXTERNAL wrap → `NO_REPLY` on JSON intake |
| LLM idle 5s | `DEFAULT_LLM_IDLE_TIMEOUT_MS` 120s → 5s so idle silence fails fast and NVIDIA keys rotate |
| OpenAI first byte 20s | `RMP_OPENAI_FIRST_BYTE_20S`: for `provider === "openai"` stream creation, the first chunk and the provider first-event guard wait up to 20s (gpt-5-nano's first byte was ~4s median, often over 5s on intake prompts). Gaps between chunks and every other provider keep 5s |
| OpenAI max effort 120s / 30s | `RMP_OPENAI_MAX_EFFORT_120S`: OpenAI calls with `options.reasoning === "max"` wait up to 120s for the first byte (wrapper and provider first-event guard) and 30s between chunks. gpt-6-luna at max sent nothing for 69s on one probe and paused 5s mid-stream on another. Every other call keeps the rows above |
| GPT-6 thinking levels | `RMP_GPT6_THINKING_BACKPORT`: 2026.9.1's OpenAI thinking policy offers `max` only to `gpt-5.6*` and ignores declared efforts outside the codex runtime, so gpt-6-luna sessions clamped `max` to `high`. Backport of 2026.9.6's GPT-6 branch: levels come from `compat.supportedReasoningEfforts`. Skipped on releases that ship `OPENAI_GPT_6_MODEL_IDS` (2026.9.7 does) |
| Session canonical scan | Skip in-flight `{}` placeholders with `entry_valid != 1`; do not fail-closed the whole store on parseable pending rows (2026.9 `entry_valid` triggers otherwise poison every `/hooks/agent`). 2026.9.7 moved the rule into `validateCanonicalSessionRow`/`…RowEntry`, which still refuse `entry_valid` 0 on admission |
| Local placement cleanup | `RMP_SKIP_LOCAL_PLACEMENT_CLEANUP`, 2026.9.1 only: its first reconcile re-read the session store once per local placement and froze startup. 2026.9.7 reworked startup placement work (ready in 54 s on real data without it), so the audit requires the patch only where the old loop exists |
| Session timestamp drift | Ignore `session_nodes.updated_at` vs JSON `updatedAt` mismatch (often tens of ms); stock parser returns null and `/hooks/agent` throws `SESSION_CANONICAL_KEY_MIGRATION_REQUIRED` |
| OpenAI no storage | `RMP_OPENAI_NO_STORE`: `storeMode: "disable"` and no server-side compaction for provider `openai`, so requests send `store: false` and replay `reasoning.encrypted_content` (stock sends `store: true` with no config switch) |
| HTTP 410 skip | Classify 410 as `model_not_found` (next fallback), not timeout/idle retry |
| Model fallbacks | **Left enabled** — gpt-6-luna → gpt-oss-20b (do not re-apply legacy no-fallback disable; do not restore GLM, DeepSeek or MiniMax) |

Upgrade checklist: `ops/upgrade_openclaw.sh` (pre-flight `ops/openclaw_preflight.py`: the target is downloaded to a scratch directory and refused, before anything is stopped, when this host's Node is outside its range, RMP's patches do not apply to it, or it stores transcript events in a layout `app/openclaw_transcripts.py` does not decode → backup: online copies of both SQLite stores (`ops/backup_openclaw_state.py`), config, local plugins, workspace files and `VERSIONS.txt` → Node ≥ 24.16.0 → `npm install -g openclaw@latest` (or `OPENCLAW_PACKAGE`) → `openclaw plugins update --all` (newest plugin versions compatible with the core) → `OPENCLAW_SERVICE_REPAIR_POLICY=external openclaw doctor --fix --non-interactive` → RMP model policy, config keys and `TOOLS.md` → `ops/settle_openclaw_sessions.py` (never drop `session_nodes` entry_valid triggers) → key sync → `patch_openclaw.sh` → skills → verify → restart if no user tasks → `make production-check`). Do not run `openclaw update` (it re-runs doctor/restart on its own). Rehearse a new release first on a copy of production's data (`ops/openclaw_staging.py build` then `start`, and `ops/openclaw_probe.py --target staging`); after the cutover run `ops/openclaw_probe.py --target production`. Roll back with `ops/rollback_openclaw.sh [backup dir]`, rehearsed on staging with `ROLLBACK_TARGET=staging`. Plugins must not reach into a sibling plugin's folder by relative path: 2026.9.7 loads a captured copy of each plugin.

---

## 5. RMP (Reliability and Memory Plane) layer — orchestration, workflows & process control

RMP is the **control plane** for Aura. OpenClaw provides LLM + tools + Slack transport; RMP provides **durability**, **process semantics**, **memory injection**, **completion gates**, **reconciliation**, and **LLM resource orchestration**. Every user-visible turn becomes a Postgres-backed **Task** driven by a Temporal **workflow**; OpenClaw runs inside isolated **child workflows** / activities as the execution engine.

### 5.0.0 Analyst control plane (binding)

**Constitution:** [`docs/CONCEPT_TREE.md`](docs/CONCEPT_TREE.md). This section is the runtime summary; if it drifts, the constitution wins.

Two **non-Aura** RMP agents own judgment. Retrieval is **evidence only** — never assignment. Regex catalog patterns and vector similarity are advisory.

| Role | Session | Job |
|------|---------|-----|
| **Intake Analyst** | `agent:main:rmp_intake_*` | Classify each user message against running work, finished work, global memory, or “this is new.” If unsure, **clarify** via RMP Slack. |
| **Process Evaluator** | `agent:main:rmp_verify_*` | Aura never reaches Slack first. Dual gate: deterministic evidence, then semantic judge. Walks gpt-6-luna (thinking `medium`) → `nvidia/openai/gpt-oss-20b` with an explicit model, own session per model, within its activity deadline. Fail closed on parse/tool errors. |
| **Aura** | `agent:main:rmp_task_*` | Execution engine only (tools, code, search, replies). |

**Four relation classes (intake):**

1. Related to a **currently running** process → `attach_active`, `wait_active`, or `rebuild_stale`
2. Related to **finished** work → `create_guided` (citations of past task + memory)
3. Related only to **global/user memory** → guided/conversational with memory brief
4. **New** → `create_fresh` (new Task row for this message — **not** conversational amnesia)

**Decisions:** `clarify`, `attach_active`, `wait_active`, `rebuild_stale`, `create_guided`, `create_fresh`, `skip_valid`, `skip_noop`, `supersede`, `spawn_process`. `skip_*` still persist an intake decision and a short user-visible ack — messages are never silently dropped. Low confidence **must not** default to `create_fresh` (clarify, or wait if a single clear **user** active target exists). Health canary/heartbeat/system tasks are not user work: never `wait_active` / `attach_active` onto them. If intake LLM/workflow fails, user DMs still **`create_fresh` a Task and start `GenericTaskWorkflow`**. `execution_mode=conversational` only selects the one-step RMP plan (still `rmp_task_*` + RMP `chat.postMessage`); it is **not** a native Slack reply and must never skip the plugin → `/tasks` → intake → Temporal path.

**Always-gated Slack:** including greetings. Canary/system tasks stay on the short deterministic path (no 20-turn loops, no user clarify).

**Session continuity:** `create_fresh` still starts a new Task/workflow, but `POST /tasks` injects **RECENT DIALOGUE** from prior same-`session_key` `task_messages` (user + assistant Slack turns). User-local clock is injected as a fact (`USER LOCAL TIME`), not a greeting-phrase policy. Vector recall is advisory (OpenAI `text-embedding-3-small`); dialogue injection must not depend on embeddings.

**Attempt law:** attempts 1–9 rework with evaluator critique; attempt 10 `strategy_change`; 11–19 continue; attempt 20 stop and Slack a diagnosis of what was tried and why it was insufficient.

**Hybrid retrieval:** Postgres FTS + Qdrant dense + user memory + active-task metadata, fused with Reciprocal Rank Fusion (k=60). Intake Analyst adjudicates.

### 5.0.1 Universal Task Intake (Phase 4–5 + analyst)

Before `POST /tasks` starts Aura execution, intake runs when `task_registry.enabled`:

1. **Layer 1 — Fast path** — idempotency; duplicate same-session intent → `attach_active`; active recurrence → `wait_active`; health-canary LLM bypass; deterministic `skip_valid` / `skip_noop` / `supersede` for recurrent jobs (still logged + ack)
2. **Layer 2 — Hybrid evidence pack** — FTS + dense + memory + active metadata (RRF). Vector similarity is **advisory**; it must not auto-attach or auto-create.
3. **Layer 3 — Intake Analyst** — first one typed Jev request (`app/decisions/intake.py`, `jev.intake_mode`), used only above its thresholds; otherwise the LLM turn `classify_task_intake` on `rmp_intake_*`. The API fallback also asks Jev before `create_fresh` at confidence 0.
4. **Policy engine** (`intake_decision_engine.py`) — validates catalog ids, low-confidence clarify, modes: `off` | `shadow` | `enforce`

**Plugin:** all Slack DMs route through `POST /tasks`; stop commands still signal directly.

**Durable tasks:** `task_kind=durable` + `spawn_leg`; `spawn_process` API; legs linked via `ProcessRun.parent_process_run_id`.

Supplementary user/cron text is stored in `task_messages`. Terminal tasks are indexed into `task_registry_entries` + Qdrant (including evaluator summary) on every terminal path: workflow status updates, reconciler repairs and orphaned-reply recovery (after its commit), and `POST /tasks/{id}/cancel`. `ops/backfill_task_registry.py --task-id …` indexes specific tasks.

**Completion rework:** global attempt counter on the process run; strategy change at 10; user diagnosis at 20. Canaries exempt.

### 5.0 Control-plane overview

```
Slack / cron
     │
     ▼
rmp_adapter (sync block → POST /tasks)
     │
     ▼
RMP API ── Intake Analyst (hybrid evidence; decisions including clarify)
     │
     ▼
Task + ProcessRun + Event rows
     │
     ▼
Temporal workflow (GenericTaskWorkflow | CatalogTaskWorkflow)
     │
     ├── generate_process_plan
     ├── for each plan step:
     │      GenericExecuteChildWorkflow | CatalogStepChildWorkflow
     │           ├── PROCESS BRIEF + build_process_memory_context
     │           ├── reserve_profile → send_to_openclaw → release_profile
     │           ├── extract_agent_facts + evaluate_step_predicate
     │           └── decide_step_outcome (orchestrator)
     ├── Process Evaluator (always-gated: evidence then semantic judge)
     ├── notify_slack_user only after accept (or attempt-20 diagnosis)
     └── execute_compensation on terminal failure
```

**Enhancement model:** OpenClaw alone = stateless agent turns in JSONL. With RMP modules, each turn gains:

| Capability | Without RMP | With RMP |
|------------|-------------|----------|
| Durability | Session JSONL only | Postgres ledger + Temporal replay |
| Multi-step work | Single agent loop | Plan-driven steps with child workflows |
| Completion | LLM self-reports | Predicate gates + evidence + quality review |
| Memory | Workspace MEMORY.md | Process-scoped inject + vector recall + promotion |
| Slack delivery | Gateway streaming | Idempotent, suppressed during tasks, policy-filtered |
| LLM keys | OpenClaw rotation only | Balanced 3-key pool, concurrency cap, usage ledger |
| Failure recovery | Manual | Reconciler, janitor, compensation, retry signals |

### 5.1 Task lifecycle & routing

**Creation (`POST /tasks`):**

1. Plugin or API submits `intent`, `session_key`, optional `tags`, `task_type`, `idempotency_key`.
2. API deduplicates by idempotency key; returns existing task if terminal and caller retries.
3. Creates `Task` (status `created` → `running`), `ProcessRun`, initial `Event`.
4. Starts Temporal workflow on queue `openclaw-tasks` with payload including `initial_memory_block` when prefetched.

**Signals (`POST /tasks/{id}/signal`):**

| Signal | Effect |
|--------|--------|
| `user_input` | Resume blocked step / inject user reply |
| `cancel` | User stop; workflow sets `stopped_by_user` |
| `approve` | Human gate for sensitive catalog steps |
| `retry` | Re-run failed/compensated task |

**Leases:** Worker activities acquire short-lived leases on ProcessRun to prevent duplicate side effects during Temporal retries.

**Idempotency:** Slack receipts stored in `SideEffectReceipt`; duplicate `notify_slack_user` calls are no-ops.

### 5.2 Workflow types

#### GenericTaskWorkflow (`app/workflows/generic_task.py`)

Primary path for Slack DMs, cron, and most automation.

1. **`initialize_process_run`** — binds task to ProcessRun, sets status `running`.
2. **`_plan_driven_loop`** — program-owned step machine (legacy monolithic loop removed):
   - **`generate_process_plan`** — single LLM call produces JSON plan (`steps[]` with `name`, `prompt`, `predicate_id`).
   - **`save_process_plan`** — persists `plan_json` on ProcessRun.
   - **`build_process_memory_context`** — once at start; refreshed after each completed step (skipped vector for canaries via `skip_vector`).
   - For each plan step, up to **3 attempts** via **`GenericExecuteChildWorkflow`** child.
   - On step `blocked` → task `pending_user_input`; on exhausted failures → **`execute_compensation`**.
3. **`decide_completion_gate`** — evidence check, then Process Evaluator (quality LLM) for user work. Quality LLM is skipped only for internal/canary/heartbeat (`is_internal_task`). `_judge_and_deliver` leaves only through an accept (then delivery) or an escalation; evaluator failures retry the evaluator (`EvaluatorRetry`, `app/workflows/judgment.py`), never Aura.
4. **`notify_slack_user`** — final delivery unless internal/canary/heartbeat suppression applies.

Messages signalled while the run works are taken at each turn and folded into the draft (`AttachedMessages`, `app/workflows/user_messages.py`); leftovers after delivery, stop or escalation are resubmitted to intake. A run the reconciler restarts after a crash carries `recovered_draft`, which is judged before anything is sent.

Deep recall (`DeepRecallPhase`, `app/workflows/recall_phase.py`): the run starts `{task_id}-recall` when the payload's `deep_recall.enabled` says so. A finished report is merged, without waiting, before each plan step, judgment and rework brief. A report that arrives after the reply leads to the follow-up phase of §3.1 step 9. The report row records its consumer (`steps`, `evaluator`, `followup`, `none`), and every exit settles it.

Exception path: any uncaught workflow error triggers compensation + user-facing error message (formatted via `notification_policy`).

#### CatalogTaskWorkflow (`app/workflows/catalog_task.py`)

Structured multi-step flows for repeatable business processes. Seven templates in `app/workflows/catalog.py`:

| # | Template | Typical predicates |
|---|----------|-------------------|
| 1 | Account registration | form fill, confirmation |
| 2 | Login / sign-in | auth success |
| 3 | Email verification | inbox / code |
| 4 | Procurement | cart, checkout |
| 5 | Outreach / email | draft, send |
| 6 | Browser automation | navigation, extract |
| 7 | Tool self-upgrade | draft → tests → approval → controlled restart → verify |

Each catalog step maps to a **`predicate_id`** and runs in **`CatalogStepChildWorkflow`** (same OpenClaw dispatch pattern as generic children). A catalog run reads the deep recall report only at step boundaries and never follows up.

#### Child workflows

| Workflow | Role |
|----------|------|
| `GenericExecuteChildWorkflow` | One OpenClaw dispatch per generic plan step; isolated session `agent:main:rmp_task_{id}` |
| `CatalogStepChildWorkflow` | One dispatch per catalog step with template-specific prompt |
| `DeepRecallWorkflow` | The IA's deep recall: plan, retrieve, read (`app/workflows/deep_recall.py`), bounded by `recall_deadline_sec` |

Child IDs use pattern `{task_id}-plan-{step_name}-{attempt}` and `{task_id}-recall` for orphan detection by reconciler/janitor.

### 5.3 Process management (Postgres entities)

| Entity | Purpose |
|--------|---------|
| **Task** | User-visible unit: intent, status, session_key, tags, next_check |
| **ProcessRun** | Execution instance: `plan_json`, lease holder, terminal reason |
| **Step** | Named plan step: status (`pending`/`running`/`completed`/`failed`/`blocked`/`compensated`), attempt count |
| **Observation** | Agent output snapshot per step attempt |
| **Event** | Audit trail (created, status changes, signals) |
| **MemoryItem** | Process-scoped and promoted memories |
| **SideEffectReceipt** | Dedup keys for Slack posts |
| **Artifact** | SHA-256 content-addressed outputs |

**Status machine:** a task has one of ten statuses, in four groups:

```
IN_PROGRESS  created, running                 a workflow runs the task, or is about to
WAITING      pending_user_input, blocked      waiting for Kirill, a coding slot or Claude's usage limit
HANDED_OFF   deploying                        the coding workflow handed the task to the self-deploy
TERMINAL     completed, failed, compensated, stopped_by_user, cancelled
```

Terminal is final: no write moves a finished task again (a retry is a new task). The full table of allowed moves is `TASK_TRANSITIONS` in `app/status.py`, beside `LIFECYCLES`, what each task workflow writes. `app/transitions.py` is the one writer of task statuses and run states: a compare-and-set that refuses and logs any move the table forbids.

**Compensation (`execute_compensation`):** Releases lease, marks open steps `compensated`, writes process memory annotation, sets terminal task/process state — without overwriting with a generic `failed` when already compensated. A finished task is not compensated afterwards: it keeps the status it ended in, and no `task.compensated` event is written.

### 5.4 Orchestrator modules (`app/orchestrator/`)

The orchestrator keeps **program logic** in charge; the LLM proposes actions and facts, but **does not** unilaterally declare step completion.

| Module | File | Role |
|--------|------|------|
| **Step predicates** | `step_predicates.py` | `extract_agent_facts()` parses fenced JSON `{ "facts": {...} }`; `evaluate_step_predicate(predicate_id, ...)` returns pass/fail/issues |
| **Decision engine** | `decision_engine.py` | `decide_step_outcome()` — maps predicate result + attempt budget to `completed`/`pending`/`failed`/`blocked`; `decide_completion_gate()` — evidence + quality LLM (user work always evaluated) |
| **Prompt policy** | `prompt_policy.py` | Forbids workspace `memory_search` during RMP steps. Regex `GENERIC_PROFILES` exist as leftover tables; `resolve_generic_profile` returns `None` so they do not assign tool budgets on user DMs |

**Predicate examples:** `generic_deliver`, read/summarize/recall-specific gates, catalog-specific gates. Legacy `task_status` JSON in agent output is still parsed as fallback but **predicates are authoritative**. A deliver step passes any real answer however short ("Pong!"); whether it is enough is the Process Evaluator's call. Only an empty or `NO_REPLY` turn fails it.

### 5.5 OpenClaw execution path (`openclaw_activities.py`)

**`send_to_openclaw`** is the bridge from Temporal to OpenClaw:

1. **`wait_for_dispatch_sync()`** — quota broker pacing (per-key interval, cooldown-aware key pick).
2. **`reserve_profile(session_key)`** — atomic concurrency slot + balanced key assignment; pins `authProfileOverride` on session.
3. **`POST /hooks/agent`** — dispatches prompt to isolated RMP session; `deliver: false` (RMP owns Slack); `allowUnsafeExternalContent: true` so OpenClaw does not EXTERNAL-wrap trusted RMP prompts (otherwise structured JSON intake returns `NO_REPLY`).
4. **JSONL poll** — waits for terminal assistant message; scans up to 3 recent session files if session ID rotated. Each dispatch starts with `[RMP_DISPATCH <nonce>]`, and only an assistant turn after that user turn counts, so a previous turn's reply or verdict is never taken for this one. Any non-empty finished turn is the reply, however short.
5. **`release_profile()`** — always in `finally`.
6. On 429: `record_rate_limit`, rotate, retry (up to 12× for chat).

**Gateway path (non-RMP sessions):** Plugin hooks call **`POST /api/llm/reserve`** at `before_agent_run` and **`POST /api/llm/release`** at `agent_end` — same broker pool, so gateway cron/misc sessions share the concurrency cap with worker dispatches.

**Session bootstrap:** OpenClaw patch loads minimal TOOLS.md for `rmp_task_*` / `rmp_verify_*` / `rmp_intake_*` sessions — no MEMORY.md pollution.

**Session profile assign:** `assign_openclaw_session_profile` only updates keys that already have a `sessionId` (call after `/hooks/agent` creates the session). Pre-creating keys races OpenClaw 2026.7+ `CronSessionLifecycleClaimError`.

### 5.6 Reconciliation, janitor & background control

| Component | File / unit | Behavior |
|-----------|-------------|----------|
| **Reconciler** | `app/reconciler.py` | Every 60 s in the API: stale tasks (>20m), stuck RUNNING workflows (>45m terminate + repair), orphan `*-plan-*` child cleanup; skips internal/canary Slack nudges. A reply left by a dead run: if Temporal shows the workflow closed or missing, restart GenericTaskWorkflow with the draft to judge (a catalog run closes with a notice instead of repeating side effects). Closes process runs whose task already ended |
| **Vector outbox** | `app/memory/vector_sync.py` (API loop) | Drains `vector_outbox` every 15 s; daily reconcile queues missing rows and unindexed finished user tasks, deletes unreferenced points |
| **Workflow janitor** | `ops/workflow_janitor.py`, `rmp-janitor.timer` | Daily sweep of orphaned Temporal executions >24h |
| **Canary timers** | `rmp-canary.timer`, `rmp-memory-canary.timer` | Hourly E2E + 6h memory/vector canary; results in logs + `data/last_memory_canary.json` |
| **Canary sentinel** | `app/production/canary_sentinel.py`, `rmp-canary-sentinel.timer` | Every 30 min: canary health with remediation, then the invariant checks; pages Kirill by Slack DM on a failure (4 h cooldown per incident) |
| **Readiness** | `GET /api/production/readiness` | Stuck workflow count, canary freshness, LLM orchestration snapshot, deep memory (`app/deep_memory/health.py`: ingest lag, enrichment, index drift, memory-lane budget, recall latency and follow-up rate), and the invariants (`app/production/invariants.py`): completions without an accept, attached messages neither answered nor resubmitted, Slack delivery failures, internal traces in shared memory, vector outbox and drift, orphan recoveries, finished user tasks without an enriched task document after 30 min, internal content in deep memory, follow-ups without an accept; and host-file drift (`host_file_drift`, warn only, names only): installed unit files that are missing or differ from `ops/systemd/` at `main`, RMP units installed with no tracked file, and mirror files whose live copy is missing or differs from `main` |
| **Deep ingest** | `app/deep_memory/ingest.py` (API loop) | Drains `dm_ingest_queue` every 10 s (claim lease, backoff up to 1 h; a spent model budget defers to the next UTC day) |

This layer makes the single-VPS deployment **set-and-forget**: transient worker crashes, hung OpenClaw sessions, and orphaned children are repaired without operator intervention.

### 5.7 API (`app/api/server.py`)

Key endpoints:

| Endpoint | Purpose |
|----------|---------|
| `POST /tasks` | Create task + start Temporal workflow |
| `GET /tasks/{id}` | Status |
| `POST /tasks/{id}/signal` | `user_input`, `cancel`, `approve`, `retry` |
| `POST /tasks/{id}/cancel` | Cancel workflow |
| `GET /api/coding/status` | Coding: Claude Code version, token expiry, the coding slot, live units, repositories |
| `GET /api/coding/jobs/{task_id}` | One coding task: its job, each Claude Code run, each verification, the deploy and its events |
| `POST /api/claude/sessions` | A direct Claude session for a running task (`repo` or `scratch` workspace) |
| `POST /api/claude/sessions/{id}/messages` | The session's next turn |
| `GET /api/claude/sessions/{id}/turns/{n}?wait=` | A turn's progress or Claude's answer (long poll up to 55 s) |
| `POST /api/claude/sessions/{id}/end` | End a session, stopping a running turn |
| `POST /api/replies/files` | `attach_file`: a file from one of the task's Claude sessions or OpenClaw's outbound media folder, sent to Kirill with her accepted reply |
| `GET /api/claude/sessions[/{id}]` | Sessions, by task |
| `GET /api/production/readiness` | Go-live readiness score |
| `GET /api/deep_memory/status` | Switches, ingest queue, documents, index points vs objects, memory lane, recall outcomes of the last 24 h |
| `GET /api/deep_memory/reports/{task_id}` | A task's deep recall reports: plan, candidates, report, latency, use, novelty |
| `GET /metrics` | Prometheus counters |
| `GET /tasks/{id}/export` | Postmortem bundle |
| `/memory/*` | Write, lookup, compact, graph, process context |
| `GET /memory/vector/status` | Qdrant / embedder health |
| `POST /api/notify-user` | RMP-owned Slack notice without Aura (intake unavailable / idle stop) |
| `POST /api/llm/reserve` | Acquire concurrency slot + balanced NVIDIA profile |
| `POST /api/llm/release` | Release slot for session |
| `GET /api/llm/orchestration` | Active slots, in-flight per key, rotation mode |
| `GET /api/llm/usage` | Daily per-key requests/tokens by source |
| `POST /api/llm/record-gateway` | Record gateway LLM token usage (plugin hook) |

Auth: `X-RMP-API-Key` (settings + `/etc/rmp/rmp.env`).

### 5.8 Activities (selected)

| Activity | File | Role |
|----------|------|------|
| `generate_process_plan` | `plan_activities.py` | One-shot LLM plan; conversational fast-path for trivial intents |
| `send_to_openclaw` | `openclaw_activities.py` | Reserve → quota gate → dispatch → JSONL poll → release |
| `build_process_memory_context` | memory activities | Ordered Postgres + vector merge for inject block |
| `notify_slack_user` | `openclaw_activities.py` | Idempotent Slack delivery |
| `verify_response_quality` | `openclaw_activities.py` | LLM quality review (extra NVIDIA call, same broker) |
| `parse_agent_evaluation` | `openclaw_activities.py` | Facts-first via `extract_agent_facts` |
| `execute_compensation` | compensation activities | Terminal failure cleanup |
| DB activities | `db_activities.py` | ProcessRun, Step, Observation, Event, leases |
| Side effects | `side_effects.py` | Slack receipt dedup |

### 5.9 Notification policy (`app/notification_policy.py`)

Central rules for **what reaches Slack**:

- Suppress canary, heartbeat, system-tagged tasks (success and failure)
- Strip leading `CANARY_OK` / `HEARTBEAT_OK` from user-facing text
- Suppress native OpenClaw Slack during active RMP user tasks (plugin `message_sending`)
- Format workflow errors (timeout, NVIDIA rate limit, key rotation) instead of generic “Activity task failed”

### 5.10 LLM orchestration — quota broker & usage monitor

All RMP-issued NVIDIA calls (worker dispatch, embeddings, quality review) and gateway agent runs share one orchestration layer.

#### Quota broker (`app/llm/quota_broker.py`)

| Mechanism | Behavior |
|-----------|----------|
| **Keys** | `NVIDIA_API_KEY` → `nvidia:default`, `NVIDIA_API_KEY_2` → `nvidia:key2`, `NVIDIA_API_KEY_3` → `nvidia:key3` |
| **Rotation mode** | `balanced` — pick key with lowest daily load score: `requests + tokens/5000 + in_flight×50` |
| **Per-key pacing** | `min_interval_sec: 5` (live settings); no global gap across all keys beyond broker pacing |
| **Concurrency** | `max_concurrent: 4` (3 user + 1 canary) — `reserve_profile()` / `release_profile()` track `active_slots`, `session_slots`, per-key `in_flight`. `rmp_intake_*` sessions reserve from a 1-slot intake lane outside `max_concurrent` (`INTAKE_LANE_SLOTS`) |
| **Waits** | The broker lock covers one attempt, never a sleep, so releases are never blocked by a waiting reserve. Callers pass a `deadline` (intake and the evaluator: their activity budget). Waits over 2 s are logged with their reason (slots full, key cooling down, key pacing) |
| **On 429** | Escalating cooldown per key (15s → 30s → 60s → 120s); rotate; retry up to 6× (embed) or 12× (chat) |
| **Session pinning** | `assign_openclaw_session_profile()` sets `authProfileOverride` on the SQLite auth store (do not recreate leftover `auth-profiles.json`) |
| **State** | `/root/.openclaw/rmp/data/llm_quota.json` |
| **OpenClaw sync** | Syncs SQLite `authProfiles` `usageStats.cooldownUntil` to match broker |
| **Config** | `settings.json` → `llm_quota.*` |

```json
"llm_quota": {
  "provider": "nvidia",
  "min_interval_sec": 5.0,
  "max_wait_sec": 1800.0,
  "cooldown_steps_sec": [15, 30, 60, 120],
  "rotation_mode": "balanced",
  "max_concurrent": 4
}
```

**Key sync:** `ops/sync_nvidia_keys.py` writes env keys into SQLite `authProfiles.store` on service start. It never blocks a start: `rmp-api`, `rmp-worker` and the gateway run it as `ExecStartPre=-`, so a missing key or an auth-store error is logged in the unit's journal and the unit starts anyway (readiness's `openai_key` reports a missing primary key). It exits 0 when any key synced, NVIDIA or OpenAI, and 1 when none did. Manual: `make sync-nvidia-keys`. Live probe: `ops/nvidia_key_probe.py`. Do not recreate leftover `auth-profiles.json`.

#### Usage monitor (`app/llm/usage_monitor.py`)

Tracks **requests + tokens** per key per day, by source:

| Source | Origin |
|--------|--------|
| `openclaw_llm` | Worker `send_to_openclaw` dispatches |
| `openclaw_hook` | Gateway sessions recorded via plugin `llm_output` |
| `embed` | `nvidia_embed.py` vector calls |
| `rate_limit_429` | Cooldown events |
| `probe` | `nvidia_key_probe.py` |

- **State:** `/root/.openclaw/rmp/data/llm_usage.json`
- **Scrape:** Reads OpenClaw's SQLite `transcript_events` (resumes by rowid; first run starts 24 h back) for gateway turns not captured by hooks
- **Transcript report:** `transcript_usage(hours)` per UTC day and category (heartbeat, intake, evaluator, task, canary, cron, slack_main, other). Aborted attempts record no usage, so their sent prompt is counted separately as `aborted_prompt_tokens` (an upper bound on re-billing). It also reports the largest live session context. `usage_alerts()` flags a 24 h prompt budget breach (`llm_usage.input_budget_24h`, default 5M), a live context above 60k and an abort rate above 10%. Budget and abort breaches must still hold in the last 2 h
- **Report:** `ops/llm_usage_report.py`, `GET /api/llm/usage` (`transcripts_24h`, `transcript_alerts`); healthcheck prints the per-key summary, an `llm_transcripts_24h:` line per category and `WARN: llm_usage …` per breach
- **Honesty:** `nvidia:unknown` is treated as unset when a model id is present (`openai/` → `openai:default`, `nvidia/` → `nvidia:default`). Only `rolling_24h` events with a model are rewritten. Day-bucket `nvidia:unknown` totals stay; summary annotates those UTC days. Never invent `nvidia:keyN`.

**Design choice:** Chat and embeddings share one key pool — avoids total quota overrun; bulk vector seeding may pace Slack turns slightly. Balanced rotation **aims for equal load** but instant parity is not guaranteed under burst traffic.

### 5.11 Claude Code

Claude Code is Aura's tool in any task (direct sessions, the default). When Kirill explicitly asks for a reviewed coding job, the coding workflow runs it as before.

**Direct sessions** (`app/coding/direct.py`; tools `claude_start`, `claude_send`, `claude_status`, `claude_end` in `plugins/rmp_adapter/claude_tools.js`; API `/api/claude/*`):

- **Turns.** Each turn is `claude -p --model opusplan` as root, in a transient unit `aura-direct-<task>-<session>-<n>`: `--session-id` the first time, `--resume` after.
  - **Models and effort:** a planning turn (`plan`) runs in plan mode on Opus and answers with a plan, changing nothing; every other turn runs in Claude's auto permission mode on Sonnet. Aura can set a turn's effort: `medium` (the default), `high` or `xhigh`. The records name the models each turn ran on.
  - **Policy and config:** the host's relaxed policy `/etc/claude-code/managed-settings.json`, and `CLAUDE_CONFIG_DIR=/root/.claude`.
  - **Workspace:** a clone of her repository from GitHub (`repo`, for work on her own code) or an empty folder (`scratch`, for everything else), never the live checkout. Only a change to her code goes through a pull request.
  - **Limits:** one turn at a time per session, two on the host, 60 minutes per turn.
- **Aura's turn time.** While she works with Claude in the task (a turn running, or one that ended less than 10 minutes ago), her reply deadline stays open (`_reply_deadline`, `task_working`), and her turns' activities allow 4 hours (`app/workflows/timeouts.py`).
- **Files to Kirill.** `attach_file` takes a file from one of the task's Claude sessions or from OpenClaw's outbound media folder, where Aura puts files she makes herself (at most 50 MB and 10 waiting; one that looks like it holds a secret is refused) and records its checksum (`app/coding/outbox.py`). Once the evaluator accepted her reply and RMP delivered it, `deliver_reply_files` checks each file again and uploads it into Kirill's DM, once. A file that changed or that Slack refuses is not sent, and Kirill gets a notice; a file problem never fails the task.
- **Stops.** A stop, cancel or supersede ends the task's sessions (`abort_task_runs`, `stop_task_units`). The reconciler ends the sessions of finished tasks and prunes old workspaces.
- **Memory and evidence.** What Claude did is in memory: a "Claude sessions" section in the task document, and a `claude_session` document per conversation (`app/coding/records.py`). The evaluator gets the same records as external evidence, along with GitHub's word on the pull requests they link.

**Shipping** (`app/coding/github.py`, `app/coding/deploy.py`, `ops/coding_deploy.py`):

- **Protection.** GitHub's `main` is protected for everyone: a pull request is required, the `test` check is required, and admins are not exempt.
- **Pull requests.** Claude pushes a branch and opens a pull request with `aura-github`, never `main`. Aura reviews what it reports and approves.
- **Merge.** Claude merges it with `aura-github gh pr merge --squash` once Aura approves; the protection makes GitHub take it only after the `test` check passed.
- **Watcher.** Every minute the reconciler loop's `watch_main` looks at GitHub's `main`. Once it moved past the live `main` and CI's `test` check passed on that commit, it queues the deploy of exactly that commit; a newer commit replaces it while it is still queued and no deploy unit of GitHub's `main` is running or still stopping. A commit whose deploy was blocked or rolled back is not queued again; a later one is.
- **Queue and launcher.** A deploy waits queued on disk (`/srv/aura-code/runs/<task>/deploy.json` and its state, `deploy-state.json`), not inside a unit. One launcher, `next_deploy`, starts the unit `aura-deploy-<task>` of the next queued run, runs to resume first, then reviewed runs, only through the gate: no deploy unit running, nothing holding the code-reload lock and no active user task by the strict count (a count that fails keeps it closed). The reconciler runs it every minute and a reviewed job's hand-off once; a lock file keeps the two from launching at once. A run still queued 2 hours after it was queued is launched to record `postponed` at the first pass that finds Aura busy; the postponed unit holds the code-reload lock while it records. Kirill gets a note for GitHub's `main`, whose commit the watcher then queues again. A run keeps one log, `deploy.log`, across its launches: the launcher empties it before the run's first launch, and every launch appends to it.
- **Deploy.** The unit does its slow work first, without the lock: it reports a commit of GitHub's `main` that the live `main` already contains as `unchanged`, with no note, and a reviewed run lands its pull request and waits for CI on the merge commit (below). When the change adds, changes or deletes `requirements.txt` or `requirements-dev.txt`, it builds RMP's venv of the head's lock files, `/root/.openclaw/venvs/rmp/<first 16 hex of their lock hash>`: one venv per lock, built in place by the builder the coding jobs share (`app/coding/venvs.py`: a fresh venv, the install of `requirements-dev.txt` with `--require-hashes` when there is one, else `requirements.txt`, then `pip check`, then a stamp written last), and kept as it is when a venv is stamped for those files already. When `web-stack/backends/requirements.txt` changed, it copies the web backends' venv to `.venv.before-<task>` next to it; a launch that starts before the change again drops an earlier copy and takes it afresh. The copy is deleted with `deployed`, `rolled_back` and every result reached before the change (`blocked`, `failed`, `postponed`, a run ended as dead, an abort); any other result reached from `applying` on (a resume that finds a live `main` it did not make) keeps it, since it may be the only way back, and names it in its summary. Then it takes the code-reload lock (waiting up to 5 minutes), refuses to deploy when a tracked file in `plugins/`, `web-stack/` or `.cursor/rules/` was edited by hand on the host, and takes the readiness baseline. It starts the change only while 90 of its 180 minutes are left and when the strict count, taken last, finds no active user task; otherwise it ends and the run waits for the launcher's next pass. Then it:
  - marks the run `applying`, with its venv plan when it built a venv: that venv, and the venv the link `/root/.openclaw/rmp/venv` serves now, which a revert returns to. The plan is recorded once, its paths absolute and normalised (so a link made by hand, relative or with a trailing slash, names the same venv), and the log names it (`venv plan: target <path>, previous <path>[, adopt <hash>]`); the window, the revert and every resume read it from the run's state and never work it out again. Then it stops the units the change affects, or whose mirror files the reconcile will write: `rmp-api` first, whose graceful stop lets requests in flight finish against the running worker, then the count again (a task that arrived meanwhile starts `rmp-api` again, and the run waits), then the others. With the API down, no task can start;
  - points the venv link at the new venv, by renaming a new link over it. The first switch finds production's venv still a real directory: it stamps it, makes a link to the new venv at `/root/.openclaw/venvs/rmp/<first 16 hex of the lock hash of the requirements it served>` and swaps the two in one `renameat2(RENAME_EXCHANGE)` call, so the live path is never missing and the directory lands, stamped, where a revert of that lock change finds it without a build. A kernel or filesystem that refuses the exchange fails the switch before anything moved, an error before the change; there is no two-step fallback. Every unit and script reaches the venv through the link's stable path;
  - fast-forwards the live `main` to GitHub's (`merged`): the venv moves before the tree, so a kill between the two leaves the old code on the new venv, which a change that adds packages keeps importable;
  - installs the web stack's changed requirements into its venv in place (the backends are stopped meanwhile);
  - reconciles `plugins/`, `web-stack/` and `.cursor/rules/` in full: it writes every tracked file whose live copy is missing, behind or at the wrong mode, deletes only what the range deleted, and never touches untracked files. A deleted live file comes back at the next deploy, so switch a plugin off in `openclaw.json` or through the repo, never by deleting its files;
  - starts the units in one call (`restarted`). No affected process runs while the venv, the tree or the mirrors change, and whatever fails in between, the units are started before the unit goes on. A failure before the tree moved first points the venv link back and counts as an error before the change; a kill there leaves the link on one venv or the other, never none, and a run whose link is on the new venv stays `applying` until a later launch completes its window;
  - never installs a unit file: it names the changed `ops/systemd/` files in its result and in Kirill's note, and an operator installs them (§"systemd services");
  - checks health (and, when it restarted them, that the web backends answer `/health` within 60 s), readiness and a canary. The canary runs without its sentinel, at most 10 minutes a try, and a check that cannot finish is a failure, never an error. Readiness is compared like with like with the baseline: a check that passed before and fails now fails the deploy; one that fails now and was absent or a warning before (a check the change adds or tightens) is named in the result (and in Kirill's note for GitHub's `main`) and does not count against it; one that failed before too is ignored;
  - on failure, records the revert commit (`reverting`), stops the same units, moves the live code back at once, then the venv link back to the venv the plan recorded and the web backends' venv copy back (the venv it replaces is deleted), and starts them, then lands the revert as a pull request, then checks health again, leaving out `claude_direct` while the revert waits on GitHub (its clause that GitHub's `main` contains the live one fails until the revert merges). Each of these three steps runs even when the one before failed, the result names a step that failed, and a resumed rollback does not publish the revert twice but tries again a publish that failed with an error. A revert window that fails before the live code, the venv link and the web backends' venv are all back is published all the same; the run then keeps `reverting`, and the next idle pass resumes the window without publishing again (three such failures leave it stuck, as below);
  - once `deployed`, deletes every RMP venv but the new one and the one before it.

  Then it sends Kirill a note with the links.

  Every result is recorded under the code-reload lock: one reached under it without letting go of it, one reached before it (`unchanged`, `blocked`, `failed`) only when the lock is free at once; otherwise nothing is recorded, and the run waits for the launcher's next pass. Kirill's note goes first, on its own, tried three times over about 2.5 minutes, and the log says whether it went; then the `coding.deploy` event, with whether the note was delivered; then a reviewed job's reply run, each whatever the one before did. The reconciler's report of runs whose units kept dying sends its notes the same way.
- **Resume, retries and aborts.** Each phase is written to the run's state (`deploy-state.json`) as it happens. A unit killed during or after its change (its 3-hour cap, a reboot, `systemctl stop`) is launched again at the next idle pass and goes on from its phase without redoing anything before the change: a window it was killed in is done again, then the checks. While a window holds units stopped it lists them in `stopped-units`, and the unit's `ExecStopPost=` starts them after any end of the unit, a `SIGKILL` included. An error before the change is retried quietly, at most three times; the third ends the run `failed` with a note. The unit runs on the venv the link resolves to when it is launched, so its own imports never follow the link's switch. Each launch checks in at its start and marks its end when it returns, so a unit killed without a word (`SIGKILL`, the OOM killer) or ended by an error it did not catch counts too, once. A unit that is still stopping counts as running: its run is left alone until it has ended. A run whose unit dies three times before it starts, or before it finishes and before the change, ends `failed` (Kirill's note for GitHub's `main`, the reply for a reviewed job). One whose unit dies or fails three times mid-change keeps its state, Kirill is told once, and nothing else deploys until it is resumed by hand (specs/WP-09.md §0). A coding task stays `deploying` while its run waits. To abort a run by hand: `/root/.openclaw/rmp/venv/bin/python -c 'from app.coding import deploy; print(deploy.abort_run("<task>"))'`. A run before the change then ends `failed` and its commit of GitHub's `main` is not queued again; a run past it keeps its state and resumes at the next pass. Stopping the unit alone counts an attempt, and the next pass launches the run again.

**Reviewed coding jobs** are catalog type `coding_task`, only when Kirill explicitly asks for one, run by **`CodingTaskWorkflow`** (`app/workflows/coding_task.py`, activities in `app/activities/coding_activities.py`).

- **Isolation:** Claude Code (`claude -p`, stream-json, pinned version) runs as the unprivileged user `aura-coder` in a transient systemd unit per run (`aura-claude-<task>-<n>`): read-only system, writes only to its job checkout and home, secrets and RMP's sockets hidden, the strict coding policy (`/srv/aura-code/policy`) bound over Claude Code's policy directory, and an nftables table that keeps it off this host's services (`app/coding/units.py`, `firewall.py`).
- **Flow:** one coding job at a time (the coding slot). Aura writes the brief, RMP prepares a job checkout (`app/coding/workspace.py`), then rounds: Claude Code (a rework resumes its session), RMP collects the work into a root-owned review repository through a bundle and runs the repository's tests itself as `aura-coder` (`app/coding/verify.py`; on the shared venv `/srv/aura-code/venvs/rmp`, which the deploy's venv builder builds from the live checkout's lock files, the dev lock with its hashes once there is one, and rebuilds when they change; a job whose lock files differ installs its own the same way), Aura reviews, and the Process Evaluator judges her reply against that external evidence. Then the approval card; only Kirill's own Slack *approve*, recorded after the card, ships (`confirm_approval_provenance`).
- **Shipping:**
  - Only Aura's own repository (`rmp`, Hyper-AI-Lab/openclaw-jev) is registered.
  - The approved commit must fast-forward `main` (else a rebase round) and pass the full suite on the exact commit. It is then queued for the launcher; its deploy unit refuses hand-edited mirrors and lands the commit through its own pull request before `main` moves. A retry after an error first looks on GitHub for that pull request merged by the try before, whose answer was lost: if it is there, the retry records it as landed instead of landing again; otherwise it lands as usual. It then waits for CI's `test` check on the merge commit and deploys exactly that commit as above, under the code-reload lock; when the check does not pass, nothing is deployed (`blocked`), and the change goes live with GitHub's `main` once CI passes on a later commit.
  - A fresh run of `workflow-<task>` then sends the judged reply on the new code.
- **Durability:** long runs heartbeat and reattach to their unit after a worker restart; a stop stops the unit within seconds and keeps the checkout; the reconciler closes a coding task whose workflow is gone and stops its units, and never re-judges it.

**Observability:**
- **Readiness checks:** `claude_code`, `coding_isolation`, `coding_jobs` and `claude_direct` (the host policy and `aura-github` as in the repository, `main` protected, the live `main` contained in GitHub's).
- **Invariants:**
  - `approved_deploys` (reviewed jobs only);
  - `deploy_verification` (RMP's suite, or CI's check at the merge, plus the canary);
  - `coding_units` and `direct_units`;
  - `main_deployed` (GitHub's `main` goes live within 3 hours);
  - `claude_records`.
- **API:** the coding and Claude endpoints above.

---

## 6. Data & memory

### 6.1 Postgres (`rmp_db`)

Core entities: `Task`, `ProcessRun`, `Step`, `Observation`, `Event`, `MemoryItem`, `Artifact`, `SideEffectReceipt`, `memory_links`, `task_messages`, `task_registry_entries`, `vector_outbox`. Postgres is the record of memory; Qdrant indexes it. `events` is indexed by `(event_type, occurred_at)` and `entity_id`.

### 6.2 Vector memory (Mem0 + Qdrant)

| Setting | Value |
|---------|-------|
| Qdrant mode | **server** (Docker on `127.0.0.1:6333`) |
| Qdrant data | `/root/.openclaw/rmp/data/qdrant-server` (bind mount) |
| API key | `QDRANT_API_KEY` in `/etc/rmp/rmp.env` (`qdrant_api_key()` in `app/memory/vector.py`: the environment first, then the file). Every client and script sends it: Mem0, the registry client on REST and gRPC, and the scripts through `ops/qdrant_curl.sh`, which puts it on curl's stdin. `ops/start_qdrant.sh` gives it to the container by name. Qdrant refuses requests without it |
| Collection | `rmp_memories_openai_3small` |
| Embedder | OpenAI `text-embedding-3-small` (1536-d). NVIDIA NIM embedders remain HTTP 410 EOL. See live `settings.json` `vector_memory.*` |
| Config | `settings.json` → `vector_memory.*` |
| Implementation | `app/memory/vector.py` (`embed_query_text`), `app/memory/vector_sync.py` (outbox, drain, reconcile) |

**Why server mode:** Embedded Qdrant uses a single-process `.lock` file. With both `rmp-api` and `rmp-worker` using vector memory, only one process could hold the lock — the other reported `ready: false`. A shared Qdrant server lets both connect over HTTP.

**Ops:**

```bash
# Start, stop, or recreate after a flag change (rmp-qdrant.service runs ops/start_qdrant.sh and ops/stop_qdrant.sh)
systemctl start rmp-qdrant
systemctl stop rmp-qdrant
systemctl restart rmp-qdrant

# One-time migration from embedded storage
make -C /root/.openclaw/rmp migrate-qdrant
```

`ops/start_qdrant.sh` stops the container `rmp-qdrant` gracefully, removes it and recreates it from its `docker run` flags at every start; the data stays in the bind mount. `ops/stop_qdrant.sh` stops it. `rmp-api` and `rmp-worker` systemd units `After=rmp-qdrant.service`.

**Key resolution:** Chat NVIDIA keys from `/etc/openclaw/openclaw.env` → SQLite `authProfiles.store`. Embeddings use `OPENAI_API_KEY` → `openai:default`.

**Rate limits:** NVIDIA chat/embed helpers still go through `wait_for_dispatch_sync()`. OpenAI embeddings use the OpenAI embeddings API directly.

**Chat vs embed “compatibility”:** Chat models never see raw vectors. Embeddings only rank text chunks for retrieval. Conversational continuity must not depend on embeddings. NVIDIA NIM embedders probed 2026-09-05 return HTTP 410 EOL; live replacement is OpenAI `text-embedding-3-small`.

**Semantic recall:** Memory lookup with `query=` merges Postgres rows and Qdrant vector hits (`MemoryRouter.read`). When the index or the embedder does not answer, recall falls back to Postgres full-text search (`to_tsvector('simple')`, GIN index). Procedural recall is narrowed to its process type (`procedural_scope_id`).

**Writes and sync:** `MemoryRouter.write` commits the memory row and its `vector_outbox` row in one transaction. The API's drain (every 15 s, per-row backoff up to 1 h) embeds and upserts with point id = row id in Mem0's payload shape, so a retry overwrites instead of duplicating. Registry entries go through the same outbox. A daily reconcile (by hand: `ops/reconcile_vectors.py`, dry run by default) diffs Postgres with both indexes: it queues missing rows and finished user tasks without a registry entry, deletes points no row references, and refuses to delete most of an index.

**Seeding:** `app/memory/seed.py` — workspace markdown chunks become memory rows (stored ones are skipped); the outbox indexes them:

```bash
make -C /root/.openclaw/rmp seed-vector-memory
```

Wipe Qdrant before changing embedder model or dimensions, then reconcile to re-queue every row.

**Index (2026-09-29):** memory 2,584 rows = 2,584 points; registry 201 entries = 201 points. The legacy 4096-d collections (`rmp_memories`, `rmp_task_registry`) are retired.

**Future option:** Local embeddings (Ollama / sentence-transformers) — no API quota, ops tradeoff; not required for chat-model compatibility.

### 6.3 Deep memory and the Internal Agent (`app/deep_memory/`)

The **Internal Agent (IA)** does RMP's memory work with direct OpenAI Responses API calls (`app/llm/openai_direct.py`): `gpt-6-luna`, `reasoning.effort=medium`, `store: false`, strict JSON schemas. It gets one retry, then falls back to `nvidia/openai/gpt-oss-20b` with the same schema. Its own lane limiter lives in the broker state: concurrency, per-minute rate, circuit breaker, a daily token budget, recall before enrichment, and enrichment down to 1 slot while user tasks run. Usage is recorded as `memory_llm`. Timeouts follow the idle law: 20 s to the first output, with the reasoning phase inside it, then 5 s gaps.

| Part | What it does |
|------|--------------|
| **Index** (`index.py`) | Qdrant `rmp_deep_memory_v1`: dense `text-embedding-3-large` at 1536-d plus server-side BM25 (IDF), fused with weighted RRF. Levels: chunk, section, document, fact. Postgres text search (OR semantics) answers when the index cannot |
| **Ingestion** (`ingest.py`, `chunking.py`) | Queue `dm_ingest_queue`, drained by the API. Stage 1 (no model): one task document per user task (Conversation, Path history, Deliverables, Actions), long deliverables, pages and files Aura read (web content marked untrusted), text attachments. Markdown-aware ~1,400-character chunks with 200 characters of overlap; each chunk carries its section path and conversation metadata. User work only |
| **Enrichment** (`enrich.py`) | Stage 2: section summaries, 50–100-token context headers per chunk, document summaries, and for a task its outcome and answer (the registry leads with them) |
| **Facts** (`facts.py`) | Stage 3: atomic facts from delivered conversations, compared with their nearest neighbours. Same, update (the old fact gets `valid_to` and a `supersedes` link), contradicts (a `contradicts` link), or new. No credentials; Jev reviews in its promotion mode |
| **Fast context** (`curator.py`) | One memory block per run within 3 s and 6,000 characters: dialogue, relevant facts, linked tasks, the run's memory, procedures |
| **Deep recall** (`recall.py`, `app/workflows/deep_recall.py`) | Plan (need, up to 4 sub-queries, levels, a time window that ranks rather than filters), hybrid retrieval never returning the current task, expansion (section and document summaries with the TOC, neighbours, linked tasks, fact versions), and a cited context report in `dm_context_reports`. Novelty judge for reports that arrive after the reply |
| **Health** (`health.py`) | Readiness checks, invariants, `GET /api/deep_memory/status` and `/reports/{task_id}`; healthcheck prints the lines |

Switches in `settings.json` `deep_memory`: `enabled` (ingestion, index, fast context), `recall_enabled` (deep recall during user tasks) and `followups_enabled` (the follow-up phase), with `recall_deadline_sec` (180) and `followup_wait_sec` (300). Intake's `recall_depth` (Jev typed question, or the analyst's answer; deep below thresholds) skips recall for requests that need no memory.

### 6.4 Memory promotion (`app/memory/promotion.py`)

Episodic → semantic/procedural pipeline (partial vs original 4-stage vision). Canary, heartbeat and system runs are never promoted and never indexed in the registry (`app/memory/hygiene.py` defines their traces; `ops/purge_internal_memory.py` removes any). Procedural memory stores a procedure — task, steps, tools used, failed calls, result — not the reply. User facts come from the deep memory facts stage (§6.3).

### 6.5 Artifacts

Content-addressed store under `/root/.openclaw/rmp/data/artifacts`; completion outputs registered per task.

### 6.6 OpenClaw workspace memory

Daily notes: `/root/.openclaw/workspace/memory/YYYY-MM-DD.md`  
Long-term notes: `MEMORY.md`, `USER.md`

Workspace files are **human notes** for main/heartbeat. They are **not** the production recall path for RMP-owned Slack DMs (process-scoped Postgres + RECENT DIALOGUE). Qdrant is advisory dense retrieval (OpenAI `text-embedding-3-small`). Conversational continuity must not depend on embeddings.

---

## 7. Safe Harbor (`/root/aura_safe_harbor`)

Operational scripts outside RMP:

- **Moltbook scanners** — `moltbook_*_scan.js`, swarm runners, rate-limit tracking
- **Kairos core** — `restore_powers.js` (cron every 5m), `task_watchdog.js`, `self_auditor.js`
- **Stubs / partial** — `deep_core.js` (calls RMP compact), `auditor.js`, `memory_chunker.js`

**Retired (2026-06):** Agent Cerebro (`agent-cerebro` pip package, `cerebro_data/`) — superseded by RMP memory (§6).

Scanner catalog synced by RMP (`app/scanners/`) when `development_mode: false`.

---

## 8. Observability & ops

| Mechanism | Status |
|-----------|--------|
| Readiness API | ✅ includes the invariant checks; 30 pass, 1 warn, 0 fail at the Sep 29 2026 audit close |
| Invariant monitors | ✅ `app/production/invariants.py`; the canary sentinel pages Kirill on a broken one |
| Deep memory | ✅ readiness checks and invariants in `app/deep_memory/health.py`; `GET /api/deep_memory/status` |
| Prometheus `/metrics` | ✅ |
| Hourly canary | ✅ `:07` past each hour |
| Daily backup | ✅ `ops/backup.sh`; readiness validates the newest backup's manifest (`backups`, warn only): its recorded failures, its components, each file's size, and the backup's age |
| Runbooks | ✅ `docs/runbooks/` |
| OTLP trace backend | ⚠️ `make observability` brings up Phoenix + an OTel collector, but `telemetry.otlp_endpoint` is empty on this host, so traces stay in-process (readiness warns) |
| Production Temporal | One official server on Postgres. Not a cluster and not Temporal Cloud. |
| Ops Slack DM | ✅ `production.ops_slack` (on by default): canary failures and broken invariants DM Kirill, deduped per incident |
| Alerting webhook | ⏳ Config present; disabled by default |

### Operator commands

```bash
cd /root/.openclaw/rmp
make production-check       # health, LLM usage, patch verify, intake canaries (LLM path ≤ 45 s)
make canary                 # manual E2E canary
make readiness              # full readiness JSON
make backup
make sync-nvidia-keys       # env → SQLite authProfiles.store
make seed-vector-memory     # workspace chunks → memory rows → outbox → Qdrant
make restart-rmp            # restart API, worker, gateway; waits up to 60 s for health
make rollback               # return to dev quiet mode (ops/rollback_dev.sh)

venv/bin/python -m ops.reconcile_vectors [--apply]    # Postgres vs Qdrant diff (dry run by default)
venv/bin/python -m app.production.canary_sentinel     # canaries + invariants now (pages on failure)
venv/bin/python -m pytest tests/ -q                   # hermetic: always a private SQLite database

python3 ops/llm_usage_report.py      # daily per-key usage by source
python3 ops/nvidia_key_probe.py      # live NVIDIA smoke test on all keys

systemctl status temporal rmp-api rmp-worker openclaw-gateway
systemctl status rmp-memory-canary.timer rmp-janitor.timer rmp-canary.timer
journalctl -u rmp-worker -f
journalctl -u openclaw-gateway -f
tail -f /root/.openclaw/logs/rmp_adapter.log
curl -s http://127.0.0.1:8000/memory/vector/status | jq
curl -s -H "X-RMP-API-Key: $RMP_API_KEY" http://127.0.0.1:8000/api/llm/orchestration | jq
curl -s -H "X-RMP-API-Key: $RMP_API_KEY" http://127.0.0.1:8000/api/llm/usage | jq
cat /root/.openclaw/rmp/data/llm_quota.json   # key cooldown + slot state
cat /root/.openclaw/rmp/data/llm_usage.json   # daily usage ledger
```

---

## 9. What is complete vs. partial (Phase 2 honest status)

**Phase 1 overstated behavioral dominance.** Phase 2–4 close the gaps below. Full orchestration detail is in **§5**; this section is a completion checklist. See `VISION_COMPLETION_PROGRESS.md` §Phase 2 for step-by-step evidence.

### Layer A — Control plane ✅ (Phase 2)

- **`execute_compensation`** — releases lease, marks open steps `compensated`, writes process memory annotation, sets terminal task/process state  
- **Generic/catalog exception paths** — compensation without overwriting with `failed`  
- **Reconciler** — syncs ProcessRun on repair; skips internal tasks before Slack notify  
- **Child workflow hygiene** — lease release + step finalization; dead `_dispatch_step` removed  
- **`GenericExecuteChildWorkflow`** — one isolated OpenClaw dispatch per plan step  

### Layer B — Non-LLM orchestration ✅ (Phase 2)

- **Persisted `plan_json`** on ProcessRun; **`generate_process_plan`** (one LLM call at start)  
- **`step_predicates.py`** — facts JSON + deterministic predicate gates (not LLM `task_status`)  
- **Plan-driven generic loop** — program owns step advance/retry/fail (legacy `user_evaluation_loop` removed Phase 3)  
- **Catalog steps** map to `predicate_id`; finish path uses **`decide_completion_gate`** (user work always hits Process Evaluator)  
- **Intent profiles** — leftover tables only; `resolve_generic_profile` is disabled (no regex assignment)  

### Layer C — Process memory daily use ✅ (Phase 2)

- **Minimal bootstrap** for `rmp_task_*` / `rmp_verify_*` sessions (TOOLS.md only — no MEMORY.md) via OpenClaw patch  
- **Unified inject** — all dispatch uses `build_process_memory_context`; child workflows receive prebuilt block  
- **Memory-first executor prompts** — forbid workspace `memory_search` during RMP steps  
- **Empty-process fallback** — user pinned/semantic from `read_ordered` when process pool empty  
- **Plugin** — no `[AUTO-ROUTED]` pollution; no `process_type_hint` (intake LLM assigns); prefetch `/memory/process/{id}/context`; **`rmp_memory_recall`** tool  
- **Vector `pinned` index** on write; promotion dedup by content prefix  

### Verification ✅ (Phase 2)

- **Full pytest suite** (zero exclusions)  
- **`make production-check`** — health + readiness including stuck-workflow and memory-canary checks  
- **`ops/canary_slack_memory.sh`** — live canary; writes `data/last_memory_canary.json`  

### Phase 3 — Production hardening ✅ (set-and-forget on single VPS)

- **Vector search timeout** (20s) — postgres-only fallback via `MemoryRouter`  
- **Memory built once per plan loop** — refresh only after completed steps; `skip_vector` for canaries  
- **Reconciler active repair** — terminate stuck RUNNING workflows after 45m; orphan `*-plan-*` child cleanup  
- **Daily workflow janitor** — `ops/workflow_janitor.py` + `rmp-janitor.timer`  
- **`STALE_TASK_MINUTES` = 20** — internal/canary tasks skip Slack nudge spam  
- **Scheduled memory canary** — `rmp-memory-canary.timer` (6h); readiness checks `last_memory_canary.json` age  
- **OpenClaw poll fallback** — scan last 3 session JSONL files on session-id rotation  
- **API prefetch** — `initial_memory_block` in workflow payload at task create  
- **`parse_agent_evaluation`** — facts-first via `extract_agent_facts`  

### Phase 4 — LLM orchestration ✅

- **Balanced key rotation** — `rotation_mode: balanced` across 3 NVIDIA accounts  
- **Concurrency cap** — `max_concurrent: 3` with `reserve_profile` / `release_profile`  
- **Per-key pacing** — `min_interval_sec: 5`  
- **Usage monitor** — per-key daily requests/tokens by source (`llm_usage.json`)  
- **Plugin LLM hooks** — `before_agent_run`, `agent_end`, `llm_output` on gateway sessions  
- **Session profile pinning** — `authProfileOverride` on RMP dispatches  
- **Heartbeat isolation** — no RMP task creation or LLM reserve on internal heartbeat triggers  

### Implemented well (foundation, pre–Phase 2) ✅

- External RMP sidecar with Temporal ledger  
- Plugin fail-closed routing (sync HTTP)  
- Evidence-based completion + idempotency + reconciler  
- Seven workflow catalog templates  
- Vector memory infrastructure + graph API  
- **RMP keys module** — `app/llm/model_policy.py` + SQLite auth sync; gpt-6-luna primary and subagent model, NVIDIA gpt-oss-20b fallback  
- **Multi-key NVIDIA rotation** + RMP quota broker (chat + embeddings) + balanced load + usage ledger  
- Backups, canary, readiness, go-live/rollback  
- Slack noise suppression (canary/heartbeat/ack stripping)  

### Still partial vs. original vision ⚠️

| Area | Gap |
|------|-----|
| **OTLP live backend** | Phoenix UI at `http://127.0.0.1:6006`; OTLP HTTP `4318`; RMP exports to `http://127.0.0.1:4318/v1/traces` |
| **Production Temporal** | Official server on this host's Postgres. Single frontend. Not Cloud and not a second node. |
| **Safe harbor integration** | `deep_core` compact wired; `auditor.js` / `memory_chunker.js` exist under `/root/aura_safe_harbor` (deeper consolidation optional) |
| **72h soak** | Recommended post-go-live; not formally signed off (Phase 7) |
| **NVIDIA free-tier quota** | Mitigated by 3-key balanced rotation + concurrency cap + usage monitor; org-level TPM may still cap throughput |
| **Live Slack memory canary** | Scheduled via `rmp-memory-canary.timer`; result file at `data/last_memory_canary.json` |
| **Stuck Temporal workflows** | Reconciler terminates at 45m; daily janitor for orphans >24h |

**Phase 6 completed:** spawn leg lineage (`parent_process_run_id`), skip_noop/supersede fast paths, MoltMarket skill symlink, intake canaries including attach/wait and supersede.

---

## 10. Recommended next steps

Prioritized for stability first, then capability.

### P0 — Stability (do first)

1. **Confirm timers** — `systemctl status rmp-memory-canary.timer rmp-janitor.timer rmp-canary.timer`.  
2. **Complete 72h soak** — monitor `make readiness`, canary timers, `llm_quota.json`, stuck-workflow count.  
3. **Verify Slack path** — DMs; single reply, Postgres task rows, process memory context API hits.

### P1 — Observability & ops

4. **OTLP stack** — done on this host (`make observability` + `telemetry.otlp_endpoint`).  
5. **Enable alerting webhook** — internal channel for canary failures.  
6. **Upgrade Temporal** — Temporal Cloud or self-hosted HA cluster.

### P2 — Product depth

7. **Catalog workflow usage** — test registration/login templates against real targets.  
8. **Embedding eval** — optional hit@3 benchmark on real Slack questions.

### P3 — Architecture completion

9. **Safe harbor consolidation** — optionally deepen wiring of `auditor.js` / `memory_chunker.js` into RMP (implementations already present).  
10. **MoltMarket skill** — fix missing `SKILL.md` path so cron stops erroring.

---

## 11. Key file index

```
/root/.openclaw/
├── openclaw.json                 # Gateway, model, auth.order, heartbeat, Slack
├── openclaw.env → /etc/openclaw/openclaw.env
│                                 # NVIDIA_API_KEY, NVIDIA_API_KEY_2, NVIDIA_API_KEY_3
├── agents/main/agent/
│   └── (auth: SQLite authProfiles.store — do not recreate auth-profiles.json)
├── workspace/                    # Agent memory & bootstrap files
├── plugins/rmp_adapter/          # Slack → RMP routing + LLM reserve/release hooks
├── agents/main/sessions/         # JSONL transcripts
└── rmp/
    ├── ARCHITECTURE.md           # ← this file
    ├── DEVELOPMENT_PLAN.md
    ├── PRODUCTION_PLAN.md
    ├── settings.json             # vector_memory, llm_quota, production, …
    ├── data/
    │   ├── llm_quota.json        # Per-key cooldowns, slots, in_flight
    │   ├── llm_usage.json        # Daily per-key usage by source
    │   ├── last_memory_canary.json
    │   └── qdrant-server/        # Vector index (Qdrant server bind mount)
    ├── app/
    │   ├── orchestrator/         # decision_engine, step_predicates, prompt_policy, process_evaluator
    │   ├── workflows/            # generic_task, catalog_task, child workflows, deep_recall,
    │   │                         # judgment (evaluator retry), user_messages (fold-in, resubmit),
    │   │                         # recall_phase (merge in time, else the judged follow-up)
    │   ├── production/           # readiness, invariants, canary_sentinel, ops_notify
    │   ├── reconciler.py         # Stale task + stuck workflow repair, orphan re-judging, run closing
    │   ├── deep_memory/          # index, chunking, ingest, enrich, facts, curator, recall, health
    │   ├── llm/
    │   │   ├── openai_direct.py  # The IA's direct Responses API client and memory lane
    │   │   ├── quota_broker.py   # NVIDIA key gate, balanced rotation, slots, memory lane state
    │   │   └── usage_monitor.py  # Per-key daily usage ledger
    │   └── memory/
    │       ├── vector.py         # Mem0/Qdrant service
    │       ├── vector_sync.py    # Outbox drain + Postgres/Qdrant reconcile
    │       ├── hygiene.py        # What counts as an internal trace in shared memory
    │       ├── nvidia_embed.py   # NVIDIA embeddings + broker
    │       ├── mistral_embed.py  # Legacy adapter (unused in prod config)
    │       └── seed.py           # Workspace chunks → memory rows
    ├── ops/
    │   ├── sync_nvidia_keys.py   # Env → SQLite auth (NVIDIA + optional OpenAI)
    │   ├── nvidia_key_probe.py   # Live NVIDIA smoke test
    │   ├── llm_usage_report.py   # Usage summary CLI
    │   ├── workflow_janitor.py   # Orphan Temporal cleanup
    │   ├── reconcile_vectors.py  # Postgres/Qdrant diff and repair
    │   ├── purge_internal_memory.py  # Remove internal traces from shared memory
    │   ├── canary.sh, backup.sh, …
    └── tests/                    # pytest (orchestration, usage, workflows, …)

/root/aura_safe_harbor/           # Scanners, watchdog, legacy Kairos
/etc/rmp/rmp.env                  # RMP_API_KEY, DATABASE_URL
/etc/openclaw/openclaw.env        # OPENAI_API_KEY, NVIDIA_API_KEY, NVIDIA_API_KEY_2, NVIDIA_API_KEY_3
/etc/systemd/system/              # installed copies of ops/systemd/ (ops/install_units.py)
```

---

## 12. Secrets & config (reference)

| Secret | Location | Used for |
|--------|----------|----------|
| RMP API key | `/etc/rmp/rmp.env`, `settings.json` | RMP API auth |
| **NVIDIA API keys** | `/etc/openclaw/openclaw.env` → SQLite `authProfiles.store` | NVIDIA fallback (gpt-oss-20b) + embeddings (advisory) |
| Slack bot/app tokens | `openclaw.json` | Slack gateway |
| Postgres | `DATABASE_URL` in `/etc/rmp/rmp.env` | RMP ledger |
| Mistral API key | `/etc/openclaw/openclaw.env` (optional) | **Legacy** — not used by current RMP config |
| **OpenAI API key** | `/etc/openclaw/openclaw.env` `OPENAI_API_KEY` → SQLite `openai:default` | gpt-6-luna primary. Never git, Slack, or `openclaw.json`. Do not set `models.providers.openai.apiKey` to `${OPENAI_API_KEY}` — that marks the provider cold if the gateway process env is stale. Missing key: readiness **warn** `openai_key_missing`; OpenClaw falls back to gpt-oss-20b. |

Do not commit secrets to git.

---

*For day-to-day status, run `make production-check` and inspect the dashboard at `http://127.0.0.1:8000/` (localhost only).*

## Jev decisions

TypeSafe Jev (`jev-1.13.0`) answers typed questions in two places, each with its
own `off` / `shadow` / `enforce` mode under `settings.json` `jev`:

- **Intake** — one request per DM (relation, target task, action, execution mode,
  catalog, web intent). An answer above its thresholds becomes the same result the
  intake LLM returns, so `apply_intake_policy` still decides; anything else goes to
  the LLM turn. Shadow stores the proposal in `task_intake_decisions.llm_raw.jev`.
- **Memory promotion** — support, durability and user-scope checks before a fact
  becomes semantic or pinned user memory.

Jev calls use no OpenClaw session and no LLM quota slot. See
[the Jev runbook](docs/runbooks/jev.md) for evaluation, rollout and rollback.