docs/deployment/README.md
Rama master · réplica de lectura del repositorio original.
# Deployment Guide
DOMINUS is designed to run anywhere — from a laptop to a Terraform-provisioned cloud node. Choose the option that fits your scale.
## Quick Start (Docker)
```bash
# Create the SQLite bind-mount dir FIRST as your own user. The services run
# as the non-root `dominus` user: if the daemon auto-creates ./data it is
# root-owned and the API crashes with SQLITE_CANTOPEN. If the container
# still cannot write (your host UID differs from the image user), grant
# write access explicitly (e.g. `chmod 775 data` with a shared group) —
# never 777 on a machine holding a real portfolio.
mkdir -p data
# Generate an API key: the compose stack binds 0.0.0.0 and refuses to boot
# without authentication (fail-closed — bypassing it is not supported).
export API_KEYS="$(openssl rand -hex 32)"
# Build and run (api + worker + scheduler — the full application)
docker compose up -d
# Production profile: GHCR images, resource limits, healthchecks, the
# full Cloud stack (PostgreSQL + Redis) and the monitoring stack
# (Prometheus + Alertmanager + Grafana)
docker compose -f docker-compose.yml -f docker-compose.prod.yml up -d
```
> The base compose file keeps PostgreSQL and Redis behind the `cloud`
> profile so community deployments stay single-container-file simple. The
> production profile clears that gate — the command above starts the
> complete stack. Worker and scheduler start by default in both variants.
The server listens on `http://localhost:3000` by default. Set `HOST=0.0.0.0` in `.env` to expose on all interfaces (required behind a reverse proxy).
## Immutable Deployments
The Deploy workflow publishes `master`, `sha-<commit>` and `vX.Y.Z` tags
to `ghcr.io/alessiobrillo/dominus` (with `-worker` / `-scheduler`
variants). The production compose profile references images through
`DOMINUS_IMAGE_TAG`:
```bash
# Rolling default (mutable — fine for personal use)
docker compose --env-file .env -f docker-compose.yml -f docker-compose.prod.yml up -d
# Immutable production rollout — pin the exact commit you verified
DOMINUS_IMAGE_TAG=sha-6f4b3c2a1d0e9f8a7b6c5d4e3f2a1b0c9d8e7f6a \
docker compose --env-file .env -f docker-compose.yml -f docker-compose.prod.yml up -d
```
Resolve the tag for a released version with:
```bash
# Tags published for the latest commit on master
git fetch origin
git ls-remote --tags origin # semantic version tags
DOMINUS_IMAGE_TAG=v0.10.0
```
Pinning to a `sha-` tag means the deployed artifact is immutable and
reproducible — you always know exactly which commit is running, and
rollback is a one-line `.env` change to a previously verified tag.
### Release flow: gate → migrate → roll
Tagged releases (`vX.Y.Z`) deploy through the workflow in three explicit
phases (see `scripts/deploy/` and [ADR-0061](../adr/0061-release-migrate-before-roll.md)):
1. **Migration gate** — `migration-gate.sh` refuses the deploy when the
database is ahead of the target image's manifest (downgrade/divergent
lineage) and blocks rollbacks against a post-release schema.
2. **Migrate-before-roll** — `migrate.sh` pulls the target image and runs
its migrations against the live database while the previous image keeps
serving, under a dedicated timeout (`MIGRATE_TIMEOUT_SECONDS`, default
600 s). A failed or timed-out migration aborts the deploy — nothing is
rolled. See the [migration-policy runbook](../releases/migration-policy.md)
for the failure procedures.
3. **Roll** — `rollout.sh` bumps `DOMINUS_IMAGE_TAG` in `.env`, rolls the
stack, verifies the running image and health, and auto-rolls on failure
when the migration gate permits.
### Two targets: production and staging
The Deploy workflow has two targets, and they are not interchangeable:
| Target | Trigger | Environment | Secrets |
| ------------ | -------------------- | ------------ | ----------------------------------- |
| `production` | a `vX.Y.Z` tag | `production` | the `production` environment secrets |
| `staging` | an explicit dispatch | `staging` | the `staging` environment secrets |
```bash
# Rehearse a release on staging. Never happens by accident: a tag does
# not reach staging, and a dispatch does not reach production.
gh workflow run deploy.yml -f environment=staging -f tag=v1.3.1
# Production: the tag is the event. The environment requires an
# approval, so the tag alone is not enough to change what the customers
# are served.
git tag v1.3.1 && git push origin v1.3.1
```
The deploy secrets (`DEPLOY_SSH_HOST`, `DEPLOY_SSH_KEY`) live at the
**environment** level, not at the repository level: a repository secret
would reach both targets, and the whole point of the split is that it
does not. The `production` environment accepts only `v*` tags and
requires a reviewer; `staging` accepts `master` and `v*` tags.
The staging stack is a **second Hetzner project** with its own state,
domain, B2 bucket and secrets — see the
[staging runbook](../releases/cloud-acceptance.md#2-the-staging-runbook).
## Backups
The SQLite database lives at `DATABASE_PATH` (default: `./data/dominus.db`). **This file is your entire portfolio and configuration — back it up.**
```bash
# Built-in backup (VACUUM INTO) via the scheduler
# SCHEDULER_ENABLED=true + SCHEDULER_BACKUP_CRON="0 4 * * *" (default)
# Manual backup
dominus maintenance backup ./backups/dominus-$(date +%Y%m%d).db
# Or with SQLite directly
sqlite3 data/dominus.db ".backup ./backups/dominus-$(date +%Y%m%d).db"
```
### Keeping backups off the DB disk
In the base and production compose profiles, the worker and scheduler
mount a dedicated `backups` volume at `/backups` (`BACKUP_DIR=/backups`).
The image pre-creates that mount point owned by the non-root `dominus`
user, so the volume works without extra chown steps.
The `backups` volume isolates backups from the `./data` volume that
holds the database. It still lives on the same host disk, so it protects
against accidental wipes and write amplification, not against disk
failure.
### Off-disk durability with a Hetzner Volume
For real off-disk durability on a self-managed VPS (e.g. Hetzner):
1. Create a Volume in the Hetzner Cloud console (same location as your
VPS) and attach it to the server.
2. Mount it on the host and add it to `/etc/fstab`:
```bash
mkfs.ext4 /dev/disk/by-id/scsi-0HC_Volume_<id>
mkdir -p /mnt/dominus-backups
mount /dev/disk/by-id/scsi-0HC_Volume_<id> /mnt/dominus-backups
echo '/dev/disk/by-id/scsi-0HC_Volume_<id> /mnt/dominus-backups ext4 defaults,nofail 0 2' >> /etc/fstab
```
3. Point the backup volume at the mount:
```yaml
# docker-compose.override.yml
services:
worker:
volumes:
- /mnt/dominus-backups:/backups
scheduler:
volumes:
- /mnt/dominus-backups:/backups
volumes:
backups:
external: false
```
(or replace the `backups` named volume with the bind mount in your own
compose override).
4. Add a periodic snapshot of the Volume from the Hetzner Cloud API
(snapshot = off-site copy recoverable even if the VPS itself dies).
5. Test a restore at least once:
```bash
# SQLite restore
sqlite3 data/dominus.db ".restore /mnt/dominus-backups/dominus-<date>.db"
# PostgreSQL restore (cloud mode)
pg_restore --dbname=dominus /mnt/dominus-backups/dominus-<date>.db
```
Backup retention defaults to 30 days (`BACKUP_RETENTION_DAYS`) and the
scheduler prunes expired backups automatically.
### Restore drill
Backups are worthless until a restore has been proven. Run the drill at
least once per release — it takes the live database through the online
backup API, re-opens the copy, verifies `integrity_check` and compares
every table row count:
```bash
node scripts/restore-drill.mjs ./data/dominus.db
# restore-drill: OK - 52 tables, 123456 rows, integrity verified
```
Exit code 0 = green, 1 = red (do not ship). On a scheduled basis this can
be wired into the scheduler, but a manual run before each release is the
requirement.
### Point-in-time recovery (DOMINUS Cloud / PostgreSQL)
The daily `pg_dump` backup has an RPO of up to 24h. For the Cloud stack
(Dominus Cloud), PostgreSQL additionally ships with WAL archiving
(ADR-0054) so a restore can replay the database to _any_ point in time:
- **WAL archiving** is enabled in `docker-compose.prod.yml`:
`archive_mode=on` copies each switched segment into
`PGDATA/archive` (inside the `pgdata` volume; `archive_timeout=300`
forces a segment at least every 5 minutes of activity).
- **Base backups** anchor the archive. Run `deploy/postgres/base-backup.sh`
daily on the host (cron/systemd timer, `pg_basebackup` from the
postgresql-client package):
```bash
crontab -e
30 4 * * * PG_PASSWORD=<pass> /opt/dominus/deploy/postgres/base-backup.sh
```
Base backups land in `$BACKUP_DIR/base-<timestamp>` next to the
`pg_dump` files; the script prunes bases older than
`PG_BASE_RETENTION_DAYS` (default 14).
- **Health**: the scheduler's `pitr-health` job (PostgreSQL only) polls
WAL archiving lag every 15 minutes and the age of the newest base
backup; Prometheus alerts `PitrWalLagHigh`, `PitrBaseBackupStale` and
`BackupStale` (the latter fires whenever the daily dump has not
succeeded for 26h — including a failed or silently skipped backup).
- **Restore**: `deploy/postgres/restore-base.sh <base-dir> <archive-dir>
[recovery_target_time]` replays the archive into a fresh cluster on a
spare port. Verify, then point `DATABASE_URL` at it and run
`dominus maintenance vacuum`-style checks before promoting.
> **Trade-off (documented in ADR-0054):** the WAL archive shares the DB
> disk, so PITR protects against _logical_ corruption (bad migration,
> accidental delete, buggy write) — the realistic failure mode — not
> against host death. If the VPS dies, the archive dies with it: ship
> the `backups` volume off-host (Hetzner Volume + snapshot, or object
> storage) and keep the daily `pg_dump` as the transportable artifact.
> **Never place `dominus.db` itself on a network filesystem** (NFS/SMB,
> Hetzner Volume, etc.). SQLite WAL mode is unsafe over network storage —
> the WAL/SHM coordination assumes a local POSIX filesystem and risks
> corruption under concurrent writes. Network storage is for _backups_
> only.
## SQLite Concurrency (community edition)
The community edition runs on SQLite in WAL mode. The API, worker and
scheduler all open the same `dominus.db` file (shared `./data` volume),
which SQLite supports on a local filesystem — with these limits:
- **One writer at a time.** WAL allows concurrent readers plus a single
writer. Writes are serialized at the file-lock level; contention
surfaces as `SQLITE_BUSY` and is absorbed by the 30s busy timeout
(`DATABASE_BUSY_TIMEOUT`). High write throughput (many concurrent
pipeline runs) degrades throughput rather than failing.
- **Long-lived readers stall WAL checkpoints.** A slow read transaction
keeps the WAL growing. If you run long queries, keep them short and
commit promptly.
- **Single-node only.** No network access to the DB and no horizontal
write scaling — that is what PostgreSQL in cloud mode (`DATABASE_URL`)
is for. Migrate with a single database dump when you outgrow SQLite.
If you see frequent `SQLITE_BUSY` errors, scale back concurrency (fewer
parallel runs, `WORKER_CONCURRENCY`) before considering PostgreSQL.
## DNS Resolution in Containers
Availability verdicts run through the co-hosted **Unbound** recursor
(ADR-0072): a private, DNSSEC-validating resolver on the compose network,
reached as `unbound:5300`. The `docker-compose.unbound.yml` overlay adds it
and wires api/worker/scheduler to it with `DNS_UNBOUND_STRICT=true`.
Inside a container the **Node.js DNS fallback is the Docker embedded
resolver (127.0.0.11)** — a stub that forwards to the host's
`/etc/resolv.conf`, which on many hosts is systemd-resolved (127.0.0.53)
with its own negative cache and search domains. Consequences to know:
- **Stale negative cache.** A cached NXDOMAIN can serve "available" for
a domain that just got registered. The app-level cache honours
`forceRecheck` (used on closeout imports), but resolver-level caches
are outside its control. This is exactly why the fallback is not
allowed to serve _Available_ verdicts (ADR-0002 conservatism).
- **Search-domain mangling.** Single-label candidates can be rewritten
by search domains into a "resolved" verdict — a false _registered_,
which is the conservative direction (missed opportunity, never a
wasted buy).
The overlay is not optional for the DNS stage to be useful: without it the
first start waits ~30 s for a resolver that is not there and the run is
marked degraded (`consensus-unverified`), which never produces a buy
verdict. Community edition uses the fallback for non-Available verdicts
only; DOMINUS Cloud stays strict (ADR-0076).
```bash
docker compose -f docker-compose.yml -f docker-compose.unbound.yml up -d
```
The overlay adds the `unbound` service and points api/worker/scheduler at it
with `DNS_UNBOUND_HOSTS=unbound:5300`. If the resolver subnet collides with
a network already on your Docker host, move it with
`DOMINUS_UNBOUND_SUBNET=<free-cidr>` and retry. Run
`docker compose config --quiet` to validate the merged topology.
## Monitoring (production profile)
The prod compose profile ships a €0 self-hosted monitoring stack:
| Service | Role |
| -------------- | ---------------------------------------------------------------------------------------------------------------------------------------------- |
| `prometheus` | Scrapes `http://api:3000/api/v1/metrics/prometheus` every 30s, 30-day retention |
| `alertmanager` | Routes alerts to the webhook in `deploy/prometheus/alertmanager.yml` |
| `grafana` | Provisioned dashboard "DOMINUS Overview" (`GRAFANA_ADMIN_PASSWORD` is **required** by the prod overlay — no default `admin/admin` credentials) |
Alert rules live in `deploy/prometheus/rules.yml` and cover: API down,
provider error rate > 25%, stage errors, stuck job queue (jobs queued with
no runner), queue backlog and new dead-letter jobs. Alertmanager does not
expand environment variables, so set the webhook URL inside
`deploy/prometheus/alertmanager.yml` before deploying.
The metrics endpoint is intentionally unauthenticated (same policy as
`/api/health`) but is only reachable on the Docker network — **do not
proxy `/api/v1/metrics/*` publicly** in your nginx/Caddy config.
## Redis Degradation (DOMINUS Cloud)
In cloud mode (`DATABASE_URL` set), Redis powers distributed rate
limiting, caching, circuit breakers and locks. The application treats
Redis as an availability optimization, not a hard dependency:
- `REDIS_REQUIRED` (defaults to `true` in cloud mode) only enforces that
`REDIS_URL` is configured; it never probes connectivity.
- If Redis is down, every Redis-backed component (`RedisRateLimitStore`,
`RedisRateLimiter`, `RedisCacheProvider`, `RedisLock`,
`DistributedCircuitBreaker`) falls back to its in-memory equivalent
and logs a warning — rate limiting stays enforced per-instance, the
API keeps serving.
- The API refuses to boot only when `REDIS_URL` is missing while
`REDIS_REQUIRED` is true (multi-instance split-brain protection).
Expected behaviour: with Redis down, per-IP limits still hold per
container but are no longer shared across replicas. Restore Redis and
the next operation resumes the shared counters automatically.
## Options
| Method | When to use | Commands |
| -------------------------- | ---------------------------------- | ------------------------------------------- |
| **CLI only** | Personal use, one-off scoring | `dominus run --closeout-csv candidates.csv` |
| **Docker** | Growing portfolio, REST API needed | `docker compose up -d` |
| **Docker + reverse proxy** | Public-facing API | Add nginx/Caddy in front |
| **systemd** | Bare-metal Linux server | `systemctl enable dominus` |
| **PM2** | Node.js process management | `pm2 start ecosystem.config.cjs` |
| **Terraform (Hetzner)** | DOMINUS Cloud / dedicated node | see `deploy/terraform/README.md` |
## Architecture
```
Internet ──► Reverse Proxy (nginx/Caddy) ──► DOMINUS (port 3000) ──► SQLite (data/dominus.db)
│
├── CLI (direct access)
└── Scheduler (cron jobs)
```
## Reverse Proxy
### Nginx
Copy `docs/deployment/nginx.conf` to your nginx configuration directory, adjust the `server_name` and SSL certificate paths, then reload nginx:
```bash
sudo cp docs/deployment/nginx.conf /etc/nginx/sites-available/dominus
sudo ln -s /etc/nginx/sites-available/dominus /etc/nginx/sites-enabled/
sudo nginx -t && sudo systemctl reload nginx
```
### Caddy
```caddyfile
dominus.example.com {
reverse_proxy 127.0.0.1:3000
}
```
## Bare Metal
### systemd
```bash
sudo useradd -r -s /bin/false dominus
sudo mkdir -p /opt/dominus/data
sudo cp -r dist node_modules package.json /opt/dominus/
sudo cp docs/deployment/dominus.service /etc/systemd/system/
sudo systemctl daemon-reload
sudo systemctl enable --now dominus
```
### PM2
```bash
npm install -g pm2
cp docs/deployment/ecosystem.config.cjs .
pm2 start ecosystem.config.cjs
pm2 save
pm2 startup # follow the instructions to enable boot-start
```
## Environment Variables
All configuration is via environment variables. See `.env.example` for the full reference.
Key variables for deployment:
| Variable | Default | Notes |
| ------------------- | ------------------- | -------------------------------------------------------------------------------------------------------------------------- |
| `HOST` | `127.0.0.1` | Set to `0.0.0.0` behind reverse proxy |
| `PORT` | `3000` | Container port mapping |
| `DATABASE_PATH` | `./data/dominus.db` | Must be writable; use a volume mount in Docker |
| `API_KEYS` | (empty) | **Set this in production** to enable authentication |
| `SCHEDULER_ENABLED` | `false` | Enable for automated renewal checks, rescoring, pruning |
| `LOG_LEVEL` | `info` | Set to `warn` in production to reduce noise |
| `DOMINUS_IMAGE_TAG` | (required) | Compose-only (ADR-0046): no default — pin to `vX.Y.Z` or a `sha-…` tag for immutable rollouts; compose fails fast if unset |
| `BACKUP_DIR` | `./data/backup` | In the prod profile: `/backups` (dedicated volume) |
## Security Checklist
- [ ] Set `API_KEYS` to enable REST authentication
- [ ] Run behind a reverse proxy with HTTPS (TLS 1.2+)
- [ ] Restrict `HOST` to `127.0.0.1` unless proxied
- [ ] Set `RATE_LIMIT_MAX` to protect against abuse
- [ ] Use a non-root user (Docker: `USER dominus`, systemd: `User=dominus`)
- [ ] Set `DOMINUS_IMAGE_TAG` to a `vX.Y.Z` or `sha-…` tag in production (required, immutable rollouts — ADR-0046)
- [x] Keep backups off the DB disk (dedicated `backups` volume, ideally a Hetzner Volume or S3 target)
- [x] Test a backup restore at least once per release (`node scripts/restore-drill.mjs`)
- [ ] Keep `dominus.db` on local disk only — never on a network filesystem
- [ ] Do not proxy `/api/v1/metrics/*` publicly (internal monitoring only)
- [ ] Set `GRAFANA_ADMIN_PASSWORD` (production profile)
- [ ] Set the Alertmanager webhook URL in `deploy/prometheus/alertmanager.yml`
- [ ] Keep the `data/` directory in `.gitignore`
- [ ] Review logs periodically (`journalctl -u dominus`)