docs/deployment/first-apply.md
Rama master · réplica de lectura del repositorio original.
# First apply — from zero to a staging Cloud
This runbook turns an empty Hetzner project into a staging DOMINUS Cloud
that the launch gate can judge. It is the execution half of
[launch-checklist.md](../releases/launch-checklist.md); the code half is
already in the repository.
**Status: written on 2026-11-02, never executed.** No credential, domain or
account existed when it was written. That is stated here on purpose: a
runbook that has never been run is a hypothesis with good formatting, and
the first person to follow it is expected to correct it. The local
rehearsal (`npm run launch:rehearse`) covers the application topology; it
does not cover Hetzner, DNS, Let's Encrypt, Backblaze B2 or the real
providers.
The staging environment is a **second Hetzner project**, never the
production one: its own state, domain, B2 bucket and secrets. A rehearsal
that shares a bucket with production can delete production's backups.
---
## 0. What you need before starting
| # | Account / asset | Why | Where it is used |
|---|---|---|---|
| 1 | Hetzner Cloud project `dominus-staging` | the two nodes | `hcloud_token` |
| 2 | An SSH key pair | operator access, no passwords | `ssh_public_key` |
| 3 | A domain you control, e.g. `staging.example.com` | TLS and canonical URLs | `domain` |
| 4 | Backblaze B2: bucket `dominus-tfstate` | remote Terraform state (it holds secrets) | `backend.hcl` |
| 5 | Backblaze B2: bucket `dominus-staging-pitr` | base backups + WAL archive | `b2_backup` |
| 6 | An alert webhook (Slack, Discord, ntfy, …) | Alertmanager and dead-letter jobs | `alertmanager_webhook_url`, `NOTIFIER_WEBHOOK_URL` |
| 7 | An external uptime monitor (UptimeRobot, Better Stack, Checkly, …) from a vendor **different** from the alert webhook | availability measured outside the app node | `EXTERNAL_PROBE_URL` |
| 8 | Stripe **test** account | the billing surface in staging | `STRIPE_*` |
| 9 | An OIDC provider test tenant (Auth0 free tier is enough) | the interactive login the acceptance suite cannot exercise | manual smoke |
| 10 | Optional: EUIPO API credentials | second trademark source | `EUIPO_CLIENT_ID` / `EUIPO_CLIENT_SECRET` |
Nothing in this list requires a paid plan except the two Hetzner nodes
(≈ €15/month) and, later, the domain. B2 and the uptime monitor have free
tiers that are sufficient for staging.
### 0.1 Hetzner
1. Create a project named `dominus-staging`.
2. Security → API tokens → **Generate token**, read/write, scoped to that
project. Copy it once; it is not shown again.
3. Security → SSH keys → add the **public** half of your key.
4. Note the location you want (`fsn1`, `nbg1` or `hel1`).
### 0.2 Backblaze B2
Two buckets, two application keys, never the master key:
| Bucket | Contents | Key scope |
|---|---|---|
| `dominus-tfstate` | `dominus/terraform.tfstate` | read/write, this bucket only |
| `dominus-staging-pitr` | `base/` and `wal/` | read/write, this bucket only |
Create each key under **Application Keys → Add a New Application Key**,
restricted to its bucket. Note the `keyID` and the `applicationKey`; the
second is shown once.
### 0.3 The domain
The Terraform module can create the A record in Hetzner DNS when
`hetzner_dns_token` and `dns_zone_name` are set. If the domain lives at
another registrar, leave both empty and create the A record yourself after
`terraform apply` prints `app_public_ip`.
Caddy obtains the Let's Encrypt certificate on the first HTTPS request.
Ports 80 and 443 must reach the app node from the internet; the Terraform
firewall already admits them.
---
## 1. Rehearse locally first
The application topology can be proven before any account exists:
```bash
git clone https://github.com/alessiobrillo/dominus.git
cd dominus
git switch <the released tag> # the release being deployed
npm ci
npm run launch:rehearse # Docker required; builds the Cloud stack,
# runs the gate against it, writes evidence
```
A green rehearsal means the images, the migrations, the RLS policies, the
public surface and the gate agree. A red one is cheaper to fix here than
on a node you are paying for. The evidence lands in
`.acceptance/launch/` (git-ignored) and is described in
[evidence-index.md](../security/evidence-index.md).
The local gate, without Docker:
```bash
npm run gate
npm run launch:check -- --community --evidence .acceptance/launch
```
---
## 2. Prepare the state backend
The state is not a cache. It carries the rendered cloud-init, and
therefore the three database passwords, the B2 key and the whole
`app_env` map, in plaintext. It must not live on a laptop.
```bash
cd deploy/terraform
cp backend.hcl.example backend.hcl
```
Edit `backend.hcl`:
```hcl
bucket = "dominus-tfstate"
key = "dominus-staging/terraform.tfstate"
region = "us-east-005" # the B2 region of the bucket
endpoint = "https://s3.us-east-005.backblazeb2.com"
skip_credentials_validation = true
skip_region_validation = true
use_path_style = true
```
Export the state bucket's key in the shell, not in the file:
```bash
export AWS_ACCESS_KEY_ID="<tfstate keyID>"
export AWS_SECRET_ACCESS_KEY="<tfstate applicationKey>"
```
`backend.hcl` must be git-ignored. The directory's `.gitignore` covers
`terraform.tfvars` and the state files; add `backend.hcl` before creating
it:
```bash
grep -qxF 'backend.hcl' .gitignore || echo 'backend.hcl' >> .gitignore
```
Then:
```bash
terraform init -backend-config=backend.hcl
```
Terraform asks no questions if the backend is correct. If it offers to
copy a local state, stop: you are initialising in the wrong directory or
with the wrong backend.
---
## 3. Write `terraform.tfvars`
```bash
cp terraform.tfvars.example terraform.tfvars
```
Generate the three database passwords; never reuse one:
```bash
openssl rand -base64 32 # db_password
openssl rand -base64 32 # db_backup_password
openssl rand -base64 32 # db_app_password
openssl rand -hex 32 # METRICS_TOKEN
```
A complete staging file:
```hcl
hcloud_token = "<the dominus-staging token>"
ssh_public_key = "ssh-ed25519 AAAA... operator@laptop"
domain = "staging.example.com"
location = "fsn1"
image_tag = "<the released tag>" # must name a RELEASED tag that exists on GHCR
# The db node is operator-only and can be restricted freely.
# The app node is reached by the rollout script from the operator host
# egress; until a static-egress runner exists, the explicit opt-in is the
# honest choice. See deploy/terraform/README.md § SSH ingress.
ssh_allowed_cidrs = ["<your public IP>/32"]
allow_public_ssh = true
db_password = "<generated>"
db_backup_password = "<generated>"
db_app_password = "<generated>"
alertmanager_webhook_url = "https://hooks.slack.com/services/..."
app_env = {
# Identity: mark this as staging, never as the production operator.
LEGAL_ENTITY_NAME = "DOMINUS Staging (not a service)"
LEGAL_ENTITY_KIND = "individual"
LEGAL_REGISTERED_ADDRESS = "Staging environment — no offer to the public"
LEGAL_VAT_ID = "IT00000000000"
LEGAL_CONTACT_EMAIL = "staging@example.com"
LEGAL_PRIVACY_EMAIL = "privacy@example.com"
# The public origin: without it the SPA publishes canonicals that point
# at dominus.app, and the launch gate fails a public deployment.
PUBLIC_APP_URL = "https://staging.example.com"
TRUST_PROXY_DEPTH = "1" # Caddy in front of the API
ALLOWED_HOSTS = "staging.example.com"
# Cloud boot requirements.
METRICS_TOKEN = "<generated>"
# Monitoring: the same destination as Alertmanager, so one inbox.
NOTIFIER_WEBHOOK_URL = "https://hooks.slack.com/services/..."
EXTERNAL_PROBE_URL = "https://staging.example.com/api/health"
BACKUP_RETENTION_DAYS = "14"
# Stripe TEST keys only. A live key in staging is a live key with a
# staging blast radius.
STRIPE_SECRET_KEY = "sk_test_..."
STRIPE_PUBLISHABLE_KEY = "pk_test_..."
STRIPE_WEBHOOK_SECRET = "whsec_..."
STRIPE_PRICE_ID_PRO_MONTHLY = "price_..."
STRIPE_PRICE_ID_PRO_YEARLY = "price_..."
STRIPE_PRICE_ID_TEAM_MONTHLY = "price_..."
STRIPE_PRICE_ID_TEAM_YEARLY = "price_..."
# AUTH_PROVIDER=db is the managed identity the Cloud uses; the OIDC
# interactive login is a separate smoke (§8).
AUTH_PROVIDER = "db"
AUTO_PROVISION_TENANTS = "true"
USAGE_ENFORCEMENT_ENABLED = "true"
# Optional second trademark source.
# EUIPO_CLIENT_ID = "..."
# EUIPO_CLIENT_SECRET = "..."
}
b2_backup = {
enabled = true
bucket = "dominus-staging-pitr"
account_id = "<PITR keyID>"
application_key = "<PITR applicationKey>"
endpoint = "https://s3.eu-central-003.backblazeb2.com"
}
```
Two things to check before planning:
- `image_tag` names a tag that exists on GHCR. A tag that was never
released produces a first boot that pulls nothing.
- `PUBLIC_APP_URL` is the exact origin the browser will use, including
`https://`. The gate compares it with the `--url` it is given.
---
## 4. Plan, then apply
```bash
terraform fmt -check
terraform validate
terraform plan -out=staging.tfplan
```
Read the plan. It creates two servers, a private network, two firewalls,
an SSH key, optionally the DNS record, and the B2-backed state. It does
not create the B2 buckets: those are the two manual steps in §0.2.
```bash
terraform apply staging.tfplan
```
Expected outputs:
```text
app_public_ip = "203.0.113.10"
db_private_ip = "10.0.0.2"
app_url = "https://staging.example.com"
```
The apply is **refused** until the SSH posture is written down: either
`ssh_allowed_cidrs` is non-empty or `allow_public_ssh = true`. That is a
precondition in `deploy/terraform/main.tf`, not a warning.
If the apply fails halfway, the state is still the source of truth:
`terraform plan` again before retrying, and read the error rather than
deleting the state file. A state file deletion orphans paid servers.
---
## 5. DNS, TLS and the first health check
If `hetzner_dns_token` was set, the A record exists. Otherwise create it
at the registrar: `staging.example.com → app_public_ip`, TTL 60.
Wait for propagation, then:
```bash
curl -fsS https://staging.example.com/api/health
# {"status":"ok", ...}
```
The first request triggers the certificate. If it fails:
```bash
ssh root@<app_public_ip>
docker compose -f /opt/dominus/docker-compose.yml logs --tail=100 caddy
```
The usual causes are a DNS record that does not point at the node yet, or
port 80 blocked upstream. Do not disable HTTPS to "make it work": the
gate treats a plain-HTTP public deployment as a failure.
---
## 6. Bootstrap the first API key
The Cloud has no static `API_KEYS`; the first key is minted against the
database after the stack is up:
```bash
ssh root@<app_public_ip>
cd /opt/dominus
docker compose exec api node dist/cli.js keys create \
--tenant staging-ops --name ops --role admin
```
The command prints the key once. Store it in the operator's secret store;
do not paste it into the shell history of a shared machine. Then:
```bash
curl -fsS -H "Authorization: Bearer <key>" \
https://staging.example.com/api/v1/usage
```
A `401` means the key was copied wrong or the app is not connected to the
same database.
---
## 7. Run the launch gate against staging
The gate reads the **environment**, not the repository. Run it where the
deployment's environment is available, from a checkout of the same tag:
```bash
# On the operator workstation, outside the repository.
ssh root@<app_public_ip> 'cat /opt/dominus/.env' > ~/.dominus/staging.env
chmod 600 ~/.dominus/staging.env
cd dominus
git switch <the released tag>
node --env-file="$HOME/.dominus/staging.env" \
scripts/launch-check.mjs --cloud \
--url https://staging.example.com \
--evidence .acceptance/staging
```
`--cloud` judges the managed edition. Do **not** pass `--paid` on staging:
`--paid` fails on a missing third-party audit, and the audit is a
production precondition. The staging run should be green with warnings;
the warnings are the staging work list.
Expected warnings at this point:
| Warning | Closed by |
|---|---|
| third-party audit not on file | the audit, before production |
| Alertmanager placeholder | a real webhook in §3, then re-run |
| external probe | §8 |
| canonical origin | `PUBLIC_APP_URL` in §3 |
| job notifications | `NOTIFIER_WEBHOOK_URL` in §3 |
If any of those is still a warning after the corresponding step, the step
did not reach the process: check `/opt/dominus/.env` on the node.
---
## 8. Close the staging-only gaps
Each of these produces an evidence artifact; the index is
[evidence-index.md](../security/evidence-index.md).
### 8.1 The external probe
Configure the probe with the contract in
[deploy/monitoring/uptime-probe.yml](../../deploy/monitoring/uptime-probe.yml):
`GET https://staging.example.com/api/health`, expect 200 and
`"status":"ok"`, every 60 s, from a region outside the app node. Configure
the dead-man's switch on a host that is not the app node. Point
`EXTERNAL_PROBE_URL` at the same URL and re-run the gate.
### 8.2 Alerting
Send a test alert through the real webhook:
```bash
curl -fsS -X POST https://hooks.slack.com/services/... \
-H 'content-type: application/json' \
-d '{"text":"DOMINUS staging alert test"}'
```
Then stop the API for two minutes and confirm the `ApiDown` alert
arrives. A webhook that accepts a `curl` is not the same as an
Alertmanager route that fires; this is the check that closes the
"monitoring does not alert anyone" blocker.
### 8.3 The restore drill (the real one)
The Cloud acceptance suite names PITR as out of scope: it needs a real
base backup and a WAL archive. The first staging drill is where the RTO
stops being a target.
The db node has `psql`, `pg_ctl`, `rclone` and the archive. Install Node
once and clone the same tag:
```bash
ssh root@<db_private_ip> # via the app node or the private network
apt-get update && apt-get install -y nodejs npm git
git clone --depth 1 --branch <the released tag> \
https://github.com/alessiobrillo/dominus.git /opt/dominus/repo
cd /opt/dominus/repo && npm ci --omit=dev
```
Pull the newest base backup from B2 if it is not local:
```bash
rclone copy b2:dominus-staging-pitr/base/<base-dir> /var/backups/dominus/base/
```
Inspect the layout first, then restore:
```bash
node scripts/restore-drill.mjs --pg-base /var/backups/dominus/base/<base-dir>
node scripts/restore-drill.mjs --pg-restore \
--base /var/backups/dominus/base/<base-dir> \
--wal /var/lib/postgresql/17/main/archive \
--target-time "<UTC timestamp before the drill>" \
--expect-rows schema_migrations:<n> \
--expect-rows <a table you know>:<n>
```
Record: the command, the timestamp, the recovery duration, the row
counts, and the newest recoverable point. Those four numbers are the
evidence row in [rto-rpo.md](../operations/rto-rpo.md).
### 8.4 Stripe test mode
The three scenarios (IT B2C, IT B2B, DE B2B) are in
[billing-tax-verification.md](../releases/billing-tax-verification.md).
Record the outcome in its table; the table is the evidence, not a
screenshot of the dashboard.
### 8.5 The OIDC login
The acceptance suite runs `AUTH_PROVIDER=db`. Before production, one
manual smoke with the real provider: sign in, land on the dashboard,
confirm the tenant is the invited one, and confirm that a user without a
seat is refused. Record the timestamp and the outcome.
### 8.6 The real pipeline
Run one real scoring pipeline against the real providers and require a
**green** run, not a degraded one:
```bash
ssh root@<app_public_ip>
cd /opt/dominus
docker compose exec api node dist/cli.js run \
--sync --closeout-csv examples/closeout-sample.csv
```
A degraded DNS stage means the resolver is not validating, and a
degraded run is a partial answer (ADR-0002). Record the run id.
---
## 9. Promote to production
Production is the same Terraform, a different project, domain and B2
bucket, plus the paid preconditions:
1. the fiscal position and the operator identity are real, not staging
placeholders;
2. the third-party audit report is in `docs/security/` with "external" or
"third-party" in the filename;
3. `EXTERNAL_PROBE_URL` and the Alertmanager webhook are real;
4. the Stripe **live** keys and price ids are configured, and the
test-mode matrix has been repeated live;
5. `node --env-file=<production.env> scripts/launch-check.mjs --cloud \
--url https://dominus.example --paid --evidence .acceptance/production`
is green.
Then the release path is the documented one, not a manual compose edit.
GitHub Actions is a billing-blocked mirror, so the release is cut with the
local-first engine and rolled out with the shared rollout script:
```bash
node scripts/release-cut.mjs --publish
# Then, on the target host (staging or production):
DOMINUS_TAG=<the released tag> ./scripts/deploy/rollout.sh
```
The rollout script is the low-level primitive. The preferred operator entry point
is `deploy-cut.mjs`, which runs preflight (tag + image verified), migrate,
rollout, health poll, `launch:check --cloud --url`, and writes evidence — in
one command:
```bash
# From the operator workstation, against a provisioned app node:
npm run deploy:staging -- --tag <the released tag> --url https://staging.example.com
```
The tag alone does not change what customers are served; the rollout does.
The rollback is the previous `DOMINUS_IMAGE_TAG` through the same pipeline,
with the migration gate refusing a downgrade across a schema change
([migration-policy.md](../releases/migration-policy.md)).
---
## 10. Evidence to keep
| Step | Artifact | Where |
|---|---|---|
| local rehearsal | `rehearsal-*.json` / `.md` | `.acceptance/launch/` |
| staging gate | `launch-evidence-*.json` / `.md` | `.acceptance/staging/` |
| acceptance | `report.json` | `.acceptance/` |
| restore drill | the command output and the four numbers | operations log |
| Stripe | the scenario table | [billing-tax-verification.md](../releases/billing-tax-verification.md) |
| OIDC smoke | timestamp and outcome | operations log |
| production gate | `launch-evidence-*.json` / `.md` | attached to the release |
| audit | the report | `docs/security/` |
The rule from [evidence-index.md](../security/evidence-index.md) applies
to all of them: an artifact without the command that produced it is a
screenshot, and a screenshot is not evidence.
---
## 11. What this runbook does not do
- It does not create the Hetzner, B2, domain, Stripe or OIDC accounts.
Those are the operator's, by necessity: they are contracts, not
configuration.
- It does not perform the fiscal registration, the legal review or the
third-party audit.
- It does not promise a cost. The two nodes are the only fixed cost;
egress, B2 storage and the domain depend on usage.