docs/OPERATIONS.en.md
Rama main · réplica de lectura del repositorio original.
# Production Operations Guide
This guide covers stable self-hosted operation of BailingHub: topology, readiness, capacity, upgrades, backups, and recovery. Use [DEMO.en.md](DEMO.en.md) for evaluation and [QUICKSTART.en.md](QUICKSTART.en.md) for the first installation.
## 1. Deployment Topology
For development and small pilots, run one BailingHub instance behind an HTTPS reverse proxy with MySQL. Jobs, leases, approvals, distributed rate limits, and audit records are persisted in MySQL. The in-process `Queue` only limits local concurrency; it is not the durable job queue.
For rolling upgrades and instance failover, run multiple identical BailingHub replicas behind a load balancer. Every replica must share the same MySQL database, security configuration, build version, and migration level. Use shared object storage for chat media in multi-replica deployments; local media storage is intended for a single instance.
Incremental chat events use a short process-local replay window by default. In a multi-replica deployment, route job creation and the SSE connection for the same `job_id` to one replica, or inject a shared `JobStreamBroker`. Without either, the canonical `done` result remains available from MySQL, but the client may not see incremental text.
```text
Business systems / channels
|
Load balancer / Ingress
|
+----+----+
| |
Hub A Hub B
| |
+----+----+
|
Shared MySQL
|
Shared object storage
```
## 2. Health Endpoints
| Endpoint | Purpose | Failure action |
|---|---|---|
| `GET /health` | Liveness: the process responds | Restart the process |
| `GET /health/ready` | Readiness: MySQL is reachable and migrations are complete | Remove the instance from new traffic |
A false readiness result / HTTP 503 signals the Host or load balancer to remove traffic; it does not automatically reject every `/run` request. Deployment admission must honor that signal. The probe compares the artifact inventory with the migration ledger, not every physical table definition: manually dropping a business table while leaving its ledger intact can escape this check.
Readiness checks the actual state and configuration pools. Both databases must be complete when separate; one shared pool is checked once. JSONL state does not skip an attached MySQL configuration store. The installed artifact owns the active/retired migration inventory and SHA-256 checksums: missing active rows are pending; unknown rows, conflicting/missing/empty checksums, or an unreadable ledger are not-ready. Retired records are optional history but must match when present, and are never pending SQL. Probes only read: they do not create tables, backfill checksums, or run migrations. Legacy ledgers require the official migrator's validation/backfill in a controlled upgrade step first.
Custom MySQL state stores can expose their existing pool/query resource through the optional `RuntimeStateStore.schemaReadinessDb` contract. If absent, readiness fails rather than opening another connection or substituting the configuration database. This tightens the readiness result for older extensions while preserving type compatibility. Pure JSONL instances without SQL configuration still skip SQL checks. Existing status fields are retained; connection details and raw SQL exceptions are not returned. `migrations.pending` counts distinct missing migration filenames across resources, not databases. Distinct pools are checked separately even if they point to one schema; connection settings are never used to guess that resources can be merged.
Readiness intentionally does not probe models, tool providers, or channels. A third-party outage must not remove the entire hub from the load balancer. Use system diagnostics and alert rules for those dependencies.
`/health` exposes the process-local `observability.audit_write_failures` counter and `last_audit_failure_at`. Any non-zero value should trigger an investigation of structured `audit_write_failed` runtime log events and database health. The counter resets on process restart and does not replace external monitoring. Security-critical audit remains fail-closed; best-effort callers may degrade, but their write failures still pass through this observer first.
```yaml
livenessProbe:
httpGet: { path: /health, port: 18900 }
readinessProbe:
httpGet: { path: /health/ready, port: 18900 }
```
## 3. OpenMetrics Operational Metrics
`GET /metrics` exposes low-cardinality OpenMetrics text for Prometheus-compatible monitoring systems. It is disabled by default and returns 404 while disabled so an unconfigured deployment does not accidentally expose runtime state.
Enable it with:
```bash
BAILING_METRICS_ENABLED=true
BAILING_METRICS_TOKEN="$(openssl rand -hex 32)"
BAILING_METRICS_SCRAPE_TIMEOUT_MS=5000
```
`BAILING_METRICS_TOKEN` must be a dedicated random token of at least 24 characters and must not equal `BAILING_TOKEN`. Send it only through the `Authorization: Bearer ...` header; query-string authentication is not supported. Restrict `/metrics` to the monitoring network and retain HTTPS or a controlled cluster network.
Verify a scrape:
```bash
curl -fsS \
-H "Authorization: Bearer ${BAILING_METRICS_TOKEN}" \
https://hub.example.com/metrics
```
Prometheus example:
```yaml
scrape_configs:
- job_name: bailinghub
metrics_path: /metrics
authorization:
type: Bearer
credentials_file: /etc/prometheus/secrets/bailinghub_metrics_token
static_configs:
- targets: ["bailinghub:18900"]
```
Primary metrics:
| Metric | Meaning |
|---|---|
| `bailinghub_info` | Current version and build commit |
| `bailinghub_up` | The metrics endpoint served this scrape |
| `bailinghub_runtime_paused` | Global runtime pause state |
| `bailinghub_runtime_queue` | Process-local running and waiting counts |
| `bailinghub_job_records` | Current jobs by fixed lifecycle status |
| `bailinghub_jobs_terminal_15m` | Jobs entering a terminal state in the last 15 minutes |
| `bailinghub_queue_oldest_queued_age_seconds` | Age of the oldest non-monitor queued job |
| `bailinghub_queue_delayed_jobs` | Queued jobs whose `run_after` is still in the future |
| `bailinghub_jobs_expired_leases` | Running or dispatched jobs with explicitly expired leases |
| `bailinghub_threads_blocked` | Conversation threads blocked by in-flight work |
| `bailinghub_approvals_pending` | Tool approvals awaiting a decision |
| `bailinghub_executors` | Registered executors by online or offline heartbeat state |
| `bailinghub_audit_write_failures_total` | Audit-write failures observed by this process |
| `bailinghub_background_work_in_flight` / `bailinghub_background_work_oldest_pending_age_seconds` | Accepted background work and oldest in-flight age by fixed source / kind |
| `bailinghub_background_work_finished_total` | Continuation completion/failure outcomes observed by this owner, by fixed outcome |
| `bailinghub_background_work_last_run_timestamp_seconds` / `bailinghub_background_work_last_run_age_seconds` | Latest started / completed / failed time and age; timestamp=0 and no age sample when never observed |
| `bailinghub_background_draining` / `bailinghub_background_drain_timeouts_total` | Usage/notification tracker drain state and drain calls that exhausted their budget |
| `bailinghub_maintenance_last_page_records` / `bailinghub_maintenance_last_page_has_more` | Current/latest usage and approval continuation page progress; usage has_more=1 when the bounded page size is reached |
| `bailinghub_usage_maintenance_consecutive_failures` / `bailinghub_usage_maintenance_retry_after_seconds` | Bounded usage reconciliation failure streak and remaining retry backoff |
| `bailinghub_metrics_collector_available` | Whether each optional collector is implemented |
| `bailinghub_metrics_collector_success` | Whether each collector succeeded during this scrape |
| `bailinghub_metrics_scrape_duration_seconds` | Collection and rendering duration |
Labels are fixed and low-cardinality. They never include job IDs, tenants, principals, route arguments, or business payloads. If an optional collector times out or fails, the endpoint still returns the remaining metrics and sets that collector's `bailinghub_metrics_collector_success` to 0. Logs contain only the collector class and failure class, never exception text or business data. The JSONL state backend can expose job metrics; without the MySQL control-plane aggregation, `control_plane` is reported as unavailable.
The `maintenance` collector reads owner-local work snapshots and read-only notification outbox aggregates. It does not expose notification bodies, trigger reconciliation or call a model provider. Fixed sources are `usage`, `notification` and `approval`; kinds cover only existing model completion, settlement, response cleanup, usage reconciliation, reference price refresh, callbacks, delivery, approval notifications, alerts, delivery DLQ and approval continuation work. Completion counters describe Promise/continuation outcomes, not recipient delivery acknowledgements. An approval pass can complete with failed individual records; check the last-page `failed` sample as well. Counters reset when the process/owner is recreated. Last-page progress is not the persisted pending/unknown backlog or its oldest age, and has_more is not an exact backlog size. Unavailable optional modules do not emit invented zero samples.
At minimum, alert on collector failures, expired leases, sustained oldest-queue age above the service objective, increases in audit-write failures, expected executors becoming offline, and abnormal recent error or rejection ratios.
Also watch increases in failed background continuations or drain timeouts, sustained in-flight age beyond the work/shutdown objective, and recent-run age plus last-page failures to confirm maintenance is progressing.
Metrics feed external monitoring. They do not replace `/health/ready`, final business authorization, the audit ledger, or console diagnostics.
## 4. Concurrency And Database Connections
- `concurrency` limits tasks running concurrently in one replica.
- `BAILING_MYSQL_CONNECTION_LIMIT` controls the pool size per replica; the default is 15.
- Approximate total connection budget as `replicas × pool size + migration and operator connections`.
- Do not size the system by HTTP QPS alone. Model latency, tool calls, job duration, and SSE connections materially change capacity.
- Disable reverse-proxy buffering for SSE and set the upstream read timeout above the maximum chat wait. BailingHub sends `X-Accel-Buffering: no`, but CDN and Ingress behavior must still be verified independently.
Measure queue depth, oldest queued age, success rate, P95/P99 duration, tool latency, MySQL connections and lock waits, CPU, memory, and event-loop delay. BailingHub does not publish an unverified universal QPS number; load-test with your actual task mix before making capacity commitments.
### Tool semantic retrieval safeguards
Tool semantic retrieval is an optional context enhancement. If the index store or query-embedding service exceeds its deadline, that turn falls back to progressive tool discovery instead of leaving the whole conversation waiting indefinitely for retrieval. Two deployment environment variables set independent deadlines:
After startup, the process prewarms persisted indexes sequentially in the background; HTTP readiness does not wait for prewarm. An expired in-memory cache continues serving its previous snapshot while a single background refresh runs. Only a true cold cache waits, bounded by the index deadline below.
| Variable | Default | Allowed range | Purpose |
|---|---:|---:|---|
| `BAILING_TOOL_INDEX_LOAD_TIMEOUT_MS` | 5000 ms | 100–60000 ms | Bounds each database wait for tool vectors and their embedding credential metadata |
| `BAILING_TOOL_QUERY_EMBEDDING_TIMEOUT_MS` | 15000 ms | 250–120000 ms | Bounds the wait while embedding the current query |
These are infrastructure safeguards for one BailingHub deployment, not route-level or provider-level business policy, so they apply to every tool semantic retrieval performed by the process. Validate the deployment's normal latency distribution before lowering them; increasing them prolongs conversation waits during dependency failures.
The new stage-diagnostic event records cache state, index/credential/query-embedding latency, and stable reason codes, but carries no user query, service URL, or credential. Diagnostic writes are best-effort and never hold up retrieval fallback.
## 5. Upgrades
1. Identify source/target artifacts and state/configuration stores using the [compatibility policy](COMPATIBILITY.en.md) and target-version guide. A [read-only upgrade plan](UPGRADE_PLAN.en.md) can help review the supplied evidence.
2. Run `npm run release:check` outside production and rehearse the upgrade and consistent backup restoration.
3. Stop admission to every instance and other writer sharing those stores, drain accepted work, and retain a consistent backup. Then run the target migrator once per distinct database. Do not assume online migration or mixed-version operation is safe without separate validation.
4. Start a new replica and wait for `/health/ready` to return 200.
5. Replace old replicas gradually while observing queue, errors, and lease recovery.
Migration state is recorded in `bz_schema_migrations`.
## 6. Backup And Recovery
Back up the complete MySQL database, object-storage media and knowledge-source files, and a secure copy of deployment configuration and external secrets. Use a consistent MySQL snapshot or `mysqldump --single-transaction`, and regularly perform restore drills.
Recovery order:
1. Stop admission to all hub replicas and other writers sharing the stores, and confirm that background SQL, approvals and notification work have drained. Removing incoming traffic alone does not stop writes.
2. Restore MySQL and shared object storage.
3. Start one build compatible with the restored migration level.
4. Verify readiness, diagnostics, and a representative trace.
5. Restore remaining replicas and traffic gradually.
## 7. Failure Semantics
- A hub crash during work is recovered through the database lease and reaper.
- Executor report failures are retried; unreported work is eventually recovered by lease expiry.
- Approval intents remain durable in MySQL.
- Tool and model failures are recorded in job trace; retry behavior depends on route policy and error type.
- Security-critical tool audit is fail-closed. Best-effort operational audit must still feed metrics and alerts.
- Graceful close drains accepted callback, delivery, approval-notification, alert and DLQ continuations within the shared Kernel budget before closing database/channel resources. A timeout preserves resources for another close attempt. Background failure logs contain fixed work kinds only.
- Native delivery DLQs retain channel credentials. Custom `X-notify` DLQs requeue only the original failed notification child; queued acknowledges executor admission, not receipt. A permanent DLQ marker is saved atomically with admission, so repeated calls or lost resolve replies do not admit it again or rerun the business parent. Legacy entries are checked against the original job; binding conflicts return 409 and adapters without replay ports return 503.
- Historical v0.9.3 drain does not provide a complete SIGKILL-safe notification outbox. Core v0.9.4 adds [durable notifications](NOTIFICATION_DURABILITY.en.md) and migration 068: new terminal transitions and approval creation commit alongside intents, and recovery sends notifications without rerunning business tasks. Ordinary alerts and legacy adapters retain best-effort boundaries. Receivers deduplicate stable event IDs; partial or unknown receipt must not be blindly resent.
### Standalone boot-recovery close barrier (unreleased candidate)
This candidate adds a close barrier that is not included in the published Core v0.9.4 artifact. An unfired recovery timer is cancelled; once recovery has started, `kernel.close(drainMs)` waits for target refresh, queries, audits and requeue work to settle. On `boot recovery drain timed out`, Kernel preserves resources and stops admitting new work. Investigate blockers such as database lock waits, then retry `close()` on **the same Kernel instance**; do not force-close its connection pool to bypass the barrier.
Retry applies to the Kernel API. The default CLI exits with status 1 after a failed signal shutdown and does not retry automatically; another `server.shutdown()` call is not a Kernel close retry. This barrier does not protect against forced process exit or replace durable recovery and business-result reconciliation.
## 8. Security Baseline
- Expose public traffic only through HTTPS.
- Use separate credentials for admins, callers, executors, and tool providers.
- Use a dedicated `BAILING_METRICS_TOKEN` for metrics; never reuse the administrative root token.
- Do not expose MySQL directly to the Internet.
- Unattended deployments should set `BAILING_BOOTSTRAP_ADMIN_USERNAME` and `BAILING_BOOTSTRAP_ADMIN_PASSWORD` together. They only create the first account when the admin table is empty; they are not continuously reconciled configuration. Rotate passwords through the console or explicit `admin:create`, not by restarting the service.
- Rotate credentials and remove demo defaults in production.
- Keep API keys, database passwords, and sensitive payloads out of public errors, logs, and metric labels.
## 9. Go-Live Checklist
- `/health` and `/health/ready` return 200.
- Migration pending count is zero.
- If operational metrics are enabled, `/metrics` succeeds with the monitoring-only token and collector availability matches the selected backends.
- System diagnostics report no configuration errors or expired leases.
- A real `/run`, tool call, trace, and result delivery have completed.
- Backup restoration has been verified.
- Alerts cover queue backlog, error rate, executor offline state, and delivery dead letters.
### Core v0.9.4 durable-notification metrics
`bailinghub_notification_outbox_available{source}` reports durable port availability; `bailinghub_notification_outbox_records{source,status}` reports status counts. `bailinghub_notification_outbox_oldest_pending_age_seconds{source}` and `bailinghub_notification_outbox_expired_leases{source}` report oldest pending age and expired claims. Sources are fixed to `jobs` / `approvals`. These are store aggregates rather than owner-local counters. Unsupported stores emit available=0 without invented zero backlogs. Query errors follow collector failure/timeout handling and never trigger delivery. See [durable notifications](NOTIFICATION_DURABILITY.en.md) for migration 068 and recovery/rollback boundaries; historical v0.9.3 graceful drain alone does not provide this recovery.