docs/adr/0094-launch-posture-single-replica-external-probe.md
Rama master · réplica de lectura del repositorio original.
# ADR-0094: One replica, one external probe: the launch posture is measured, not promised
## Metadata
| Field | Value |
| -------------- | --------------------------------------------------------- |
| **Status** | Accepted |
| **Date** | 2026-10-20 |
| **Authors** | Alessio Brillo |
| **Deciders** | Alessio Brillo |
| **Supersedes** | — |
| **Relates to** | ADR-0026, ADR-0033, ADR-0054, ADR-0061, ADR-0090 |
| **Project** | DOMINUS |
## Context
The Cloud offering is a two-node deployment: an app node (API, worker,
scheduler, Redis, Unbound, Prometheus, Alertmanager, Grafana, Caddy) and a
database node (PostgreSQL 17 with PITR). The SLA commits to 99.5% monthly
availability, measured "by an external probe, configured per deployment",
and states plainly that there is no standby replica: recovery after a node
loss is a restore, not a failover.
Three things about that posture were decided but never implemented:
1. **The monitor could not report the outage.** Prometheus, Alertmanager and
Grafana run on the app node. An app-node outage takes the alerting with
it, so the event that matters most is the one nothing reports. The
availability figure in the SLA had no instrument.
2. **The infrastructure had never been applied.** The Terraform exists, the
CI validates it, and no `terraform apply` had ever been observed. The
restore drill was a static inspection of a base-backup directory: it
proved the file layout and not the data, so the RTO in
`docs/operations/rto-rpo.md` was a target and not a measurement.
3. **The ingress was open by default.** `ssh_allowed_cidrs` defaulted to
`0.0.0.0/0` on both nodes — key-only SSH with fail2ban, from any
address — because the default existed to "preserve existing
deployments". A default that is safe for the incumbent and unsafe for a
public launch is a default that has to be decided, not inherited.
## Decision Drivers
1. **A commitment nobody measures is a marketing sentence.** The SLA is
honest about the limits; it must also be honest about the instrument.
2. **The first launch has to be rehearsable.** A deployment that has never
been applied cannot be rolled back, and a restore that has never been
performed cannot be promised.
3. **One replica, stated plainly, beats two replicas that lie.** The
in-process rate limiter counts per replica; a second API replica without
a shared store makes the effective limit `RATE_LIMIT_MAX × replicas` and
resets a per-token counter when the balancer moves a caller.
4. **Safe defaults must fail closed.** The operator has to write the
dangerous decision down, not inherit it from a previous deployment.
## Considered Options
### Option A: Two API replicas behind Caddy, for zero-downtime rollouts
**Advantages:** no maintenance window; a single container crash is not an
outage.
**Disadvantages:** the protected rate limiter becomes wrong (ADR-0033, ADR
0041/0052 note the shared store comes with the multi-replica topology); the
SLA would have to be rewritten; the operational surface doubles on the one
node that everything else already sits on.
### Option B: One replica, plus a second node for the proxy
**Advantages:** the proxy survives an app-node failure.
**Disadvantages:** a proxy that cannot reach the app node is not
availability; it is a second thing to patch, for a topology that the
Terraform deliberately keeps on one node.
### Option C: One replica, an external probe, and a rehearsed restore
**Advantages:** every promise in the SLA becomes measurable; the failure
modes are the ones the architecture has; the operator's remaining work is
operational (apply the Terraform, run the drill) rather than architectural.
**Disadvantages:** a rollout has a short window in which the API is
restarted; a node loss is still a restore.
## Decision
**Option C.**
- **One application replica, stated in the SLA and in the launch
checklist.** The rollout's restart is a maintenance window outside
business hours, as the SLA already allows.
- **An external availability probe** (`deploy/monitoring/uptime-probe.yml`,
`docs/operations/external-monitoring.md`): a monitor outside the app
node, pointed at `/api/health`, with a dead-man's switch for the opposite
condition. The probe must not depend on an upstream service: a degraded
provider reduces the number of verdicts, not the availability figure.
`EXTERNAL_PROBE_URL` is read by the launch gate, which warns without it
and fails with `--paid`.
- **A real restore rehearsal.** `scripts/restore-drill.mjs --pg-restore`
restores a base backup, replays the WAL, waits for the recovery to
finish, proves it is talking to the restored instance and compares row
counts. The static `--pg-base` inspection stays, and is documented as
what it is: a layout check.
- **Fail-closed ingress.** The Terraform's `ssh_allowed_cidrs` defaults to
empty and the apply is refused until the operator either lists their own
CIDRs or sets the explicit `allow_public_ssh = true`. The state moves to
remote storage (Backblaze B2, S3-compatible) because it holds the
database passwords, the B2 key and the whole `app_env` map: it is a
secret store, and a secret store does not live on a laptop.
## Consequences
### Positive
- The availability figure in the SLA is measured by something that does not
live on the node that fails.
- The RTO is a rehearsed number: the drill restores, replays and compares
rows, and it can fail.
- The dangerous ingress decision is written down at apply time.
- The remote state survives the loss of the machine that created it.
### Negative
- A rollout has a maintenance window. The SLA already announces one, and
the SLA does not promise zero downtime.
- The operator has to configure a monitor and keep the heartbeat alive.
This is work that no amount of automation removes: somebody has to answer
the alert.
- The `ssh_allowed_cidrs` change refuses an apply for an existing
deployment that never set it. That is the point: the decision becomes
explicit, and the firewall is updated in place, without recreating a
node.
## Links
- `deploy/monitoring/uptime-probe.yml` — the probe specification.
- `docs/operations/external-monitoring.md` — how to configure it.
- `docs/operations/rto-rpo.md` — the recovery objectives and the drill.
- `scripts/restore-drill.mjs` — the rehearsal.
- `deploy/terraform/main.tf` — the fail-closed ingress and the state.
- `docs/operations/sla.md` — the commitment being measured.