docs/DEPLOYMENT.md
Rama main · réplica de lectura del repositorio original.
# Deployment
Maestro deploys as a single Docker Compose stack on one host: Postgres, MongoDB,
Qdrant, an embedding server, the API, the web app, and Caddy for TLS. Caddy is
the only service that opens a port. Everything reachable from the internet is
served from **one origin**, which is why the frontend image carries no domain
and there is no CORS to configure.
Any VM with 4 GB of RAM will do. Oracle Cloud's Always Free Ampere instance
(4 ARM cores, 24 GB) runs it comfortably at no cost; the published images are
built for `arm64` as well as `amd64` on every release tag.
## Why not Vercel, Railway, or a serverless platform
The API cannot run as a serverless function:
- **WebSockets.** `/api/v1/tasks/{id}/stream` and `/api/v1/architect/live` are
long-lived upgrades. Vercel Functions do not serve them.
- **Task duration.** `TASK_TIMEOUT_SECONDS` defaults to 1800. Vercel's ceiling
is 300 seconds.
- **Background work.** A task keeps running after the HTTP response is returned.
A serverless runtime freezes or kills the process at that point.
The frontend alone would sit happily on Vercel. See
[Running the frontend elsewhere](#running-the-frontend-elsewhere) if you want
that split.
---
## First deploy
### 1. Prepare the host
Install Docker Engine with the Compose plugin (**v2.17 or newer** — the stack
relies on `service_completed_successfully`):
```bash
curl -fsSL https://get.docker.com | sh
docker compose version
```
Point a DNS `A` record at the host and open ports 80 and 443.
#### Oracle Cloud (Always Free Ampere) specifics
The reference free host is a `VM.Standard.A1.Flex` instance (4 OCPU / 24 GB,
arm64). Three things bite on Oracle that don't elsewhere:
- **Ingress is blocked at two layers.** Add an ingress rule for TCP 80 and 443
in the VCN **Security List** (or NSG) *and* open the host firewall — Oracle's
Ubuntu images ship iptables rules that REJECT everything but SSH, and `ufw`
does not remove them:
```bash
sudo iptables -I INPUT 6 -m state --state NEW -p tcp --dport 80 -j ACCEPT
sudo iptables -I INPUT 6 -m state --state NEW -p tcp --dport 443 -j ACCEPT
sudo netfilter-persistent save
```
- **A1 capacity is scarce.** "Out of capacity" on launch is normal for Always
Free accounts: retry, try another availability domain, or upgrade the account
to Pay As You Go — Always Free resources stay free, but provisioning gets
priority.
- **Use a release tag, not `latest`.** arm64 app images are only built on `v*`
release tags; pushes to `main` publish amd64 only. On an Ampere host,
`IMAGE_TAG` in `.env.prod` must always be a release version or the pull will
fail (or run nothing) for lack of an arm64 manifest.
### 2. Place the deployment files
Only four files live on the server, and one of them holds every secret you
have. None of them come from a `git clone`:
```bash
sudo mkdir -p /opt/maestro && cd /opt/maestro
# copy docker-compose.prod.yml, Caddyfile, scripts/backup.sh and
# .env.prod.example from the repo
cp .env.prod.example .env.prod
chmod 600 .env.prod
chmod +x backup.sh
```
After the first placement, `deploy.yml` keeps `backup.sh` in sync with the
repo on every tagged rollout; the other three files are host-managed only.
### 3. Fill in `.env.prod`
Generate real secrets — the defaults in `.env.example` are placeholders, and with
`ENVIRONMENT=production` the backend refuses to start until each one is replaced:
```bash
openssl rand -hex 32 # JWT_SECRET
openssl rand -base64 32 # API_KEY_MASTER_KEY (AES-256-GCM, encrypts BYOK keys)
openssl rand -base64 24 # POSTGRES_PASSWORD, MONGO_PASSWORD, QDRANT_API_KEY
```
Set `DOMAIN` to your hostname, then substitute the passwords you generated into
`POSTGRES_URL` and `MONGODB_URL`. Keep `?authSource=admin` on the Mongo URL: the
root user cannot authenticate without it and index creation fails at startup.
The boot guard checks the substitution for you. With `ENVIRONMENT=production` the
backend refuses to start while `CHANGE_ME` is still in `POSTGRES_URL`,
`MONGODB_URL` or `REDIS_URL`, while either of the first two is still the
development default, or while a password is guessable — and the error in
`docker compose logs backend` lists every variable at fault in one line, without
echoing any value. It also refuses `RATE_LIMIT_ENABLED=false` and an unset
`TRUST_PROXY_HEADERS`. It does not demand a password where there is none: a
datastore reachable only on the compose network may run without auth. Note that
the compose-level `${POSTGRES_PASSWORD:?}` guards cover the *container's*
variables, not the URLs the backend reads — the two can be wrong independently,
which is what this guard closes.
### 4. Bring it up
```bash
docker compose -f docker-compose.prod.yml --env-file .env.prod up -d
docker compose -f docker-compose.prod.yml --env-file .env.prod ps
```
`migrate` and `ollama-pull` run once and exit `0`; the rest should report
`healthy` or `running`. `backend` does not start until `migrate` succeeds, so a
broken migration leaves the previous release serving rather than starting a new
one against an unmigrated schema.
Caddy requests a certificate on first request. Then:
```bash
curl https://your-domain/health # {"status":"ok"}
```
### 5. Seed the marketplace (optional, once)
```bash
docker compose -f docker-compose.prod.yml --env-file .env.prod \
run --rm --no-deps backend python -m app.scripts.seed_marketplace
```
### 6. Schedule the account purge
Deletion requests are irreversible after a 30-day grace period, and something
has to do the purging. Compose has no scheduler, so use the host's:
```cron
0 3 * * * cd /opt/maestro && docker compose -f docker-compose.prod.yml --env-file .env.prod run --rm --no-deps backend python -m app.scripts.purge_deleted_accounts
```
The script takes a Postgres advisory lock and is idempotent, so an overlapping
run is harmless.
### 7. Schedule the email-token sweep
`email_tokens` rows are otherwise only removed by the account-deletion cascade,
so verification and password-reset rows accumulate forever. This sweep deletes
rows whose links died more than 30 days ago:
```cron
30 3 * * * cd /opt/maestro && docker compose -f docker-compose.prod.yml --env-file .env.prod run --rm --no-deps backend python -m app.scripts.purge_email_tokens
```
Offset half an hour from the account purge so the two never contend. It takes
its own advisory lock, deletes in batches, and accepts `--dry-run` to report
what a run would remove without touching anything.
### 8. Schedule the backups
See [Backups](#backups) below for what gets backed up and how restore works;
the cron line lives there next to the rest of the backup documentation.
---
## Continuous deployment
Pushing a `v*` tag triggers `.github/workflows/deploy.yml`, which SSHes to the
host and runs `docker compose pull && up -d`. Migrations are part of the stack,
not a separate step.
After `up -d` the workflow **gates on health**: it polls the backend's
`/health/ready` and the frontend's `/` from inside the containers for up to
~3 minutes. If the gate fails, it dumps the last container logs into the
Actions output, **rolls back automatically** to the image tag that was serving
before the rollout, and fails the job. A release that starts but cannot serve
never stays up just because `docker compose up -d` exited 0. (On a first
deploy there is nothing to roll back to; the job just fails loudly.)
Repository secrets:
| Secret | Purpose |
| --- | --- |
| `DEPLOY_HOST` | Server hostname or IP |
| `DEPLOY_USER` | SSH user with docker access |
| `DEPLOY_SSH_KEY` | Private key for that user |
Add required reviewers to the `production` environment in repository settings to
gate rollouts behind an approval.
Images are published to `ghcr.io/yigtwxx/maestro-backend` and
`ghcr.io/yigtwxx/maestro-frontend` on every push to `main` (amd64) and every
release tag (amd64 + arm64). If you keep the packages private, run
`docker login ghcr.io` once on the host with a read-only token.
### Rolling back
A rollout whose health gate fails is rolled back automatically (see above).
For a manual rollback — images are immutable per tag, so it is a re-deploy of
the previous one:
```bash
cd /opt/maestro
IMAGE_TAG=1.2.2 docker compose -f docker-compose.prod.yml --env-file .env.prod \
up -d --no-deps backend frontend
```
`--no-deps` matters: a plain `up -d` re-runs the `migrate` one-shot with the
*old* image, whose alembic cannot resolve a head revision created by the newer
release — the migration fails and `backend` (which waits on it) never starts.
Skipping `migrate` leaves the schema where it is.
Note that this does **not** roll back the database. Migrations must be
backward-compatible with the release before them, or a rollback needs
`alembic downgrade` run by hand first.
---
## Models
The `ollama` service exists for one reason: **embeddings**. RAG retrieval and
document upload call it no matter which chat provider a user picked, so without
it document upload fails and memory retrieval silently degrades. It pulls only
`nomic-embed-text` (~275 MB), which is fast on CPU.
**No chat model is served.** Users bring their own key (Gemini, OpenAI,
Anthropic). `.env.prod` sets `OLLAMA_CHAT_ENABLED=false`, so picking the free
local model rejects the task start with an explicit 400 telling the user to
connect a key or self-host — instead of spawning a task whose every subtask
fails with `ollama chat failed` and still reports `completed`. The UI shows the
same guidance next to the provider selector.
> A user's own Ollama, running on their laptop, **cannot** be used by a hosted
> instance. Every LLM call is made by the backend, so `localhost:11434` is the
> server, not the visitor's machine. The free tier only works when the whole
> stack runs on the user's own machine — or if you, the operator, opt in below.
To actually offer the free chat tier from the server, all three steps are
required:
1. Raise the `ollama` service memory limit in `docker-compose.prod.yml` well
above its default `2g` (a 9B model needs roughly 8 GB to load; the limit
exists for the embedding-only default and the model will be OOM-killed
inside it).
2. Pull a chat model:
```bash
docker compose -f docker-compose.prod.yml --env-file .env.prod exec ollama ollama pull qwen3.5:9b
```
3. Set `OLLAMA_CHAT_ENABLED=true` in `.env.prod` and `up -d backend`.
If you want to point the backend at an Ollama on the host rather than in a
container, set `FREE_MODEL_ENDPOINT=http://host.docker.internal:11434/v1` and
start that Ollama with `OLLAMA_HOST=0.0.0.0` — bound to loopback it will
refuse the connection.
---
## Scaling to multiple workers
The backend defaults to a single uvicorn worker. To run more, set
`WEB_CONCURRENCY` (the Dockerfile passes it to `uvicorn --workers`):
```env
WEB_CONCURRENCY=4
```
**Multi-worker (>1) requires `REDIS_URL` to be set — enforced at boot.** The
backend refuses to start (clear config error in the logs) when
`WEB_CONCURRENCY>1` and `REDIS_URL` is empty, instead of silently degrading to
process-local state. The production config guard runs first, so clear anything it
reports before expecting to see this one. With more than one worker,
task execution, the live event stream, human-in-the-loop answers, and task
cancellation must coordinate across processes; they do so over Redis:
- **Event bus** — an event emitted by the worker running a task is published on
`maestro:events:{task_id}` so a WebSocket subscribed on any worker receives it.
With no `REDIS_URL`, the bus is in-process and a client connected to a
different worker sees no live updates.
- **Control channel** — cancel and HITL answers are routed on
`maestro:ctrl:{task_id}` to whichever worker owns the task.
- **Reconciliation** — every worker sweeps for tasks orphaned by a crashed peer
and atomically re-claims them (a single conditional `UPDATE` guarantees exactly
one winner), so a mid-run crash never leaves a task stuck at `running`.
Durable state (Postgres `task_runs`/checkpoints, Mongo `agent_logs`) is always
authoritative, so a Redis outage degrades liveness (missed live ticks, slower
cancel) but never correctness — clients recover on reconnect via the
`?after_seq=` snapshot cursor. Leave `WEB_CONCURRENCY` unset (single worker) if
you are not running Redis.
---
## Security notes
- **`CODE_EXECUTION_ENABLED`** ships `false` and must stay that way in
production. The tool shells out to the `docker` CLI, so enabling it means
mounting `/var/run/docker.sock` into the backend — which hands agent-generated
code the ability to start privileged containers on the host. Two independent
things have to go wrong before that happens now (the socket mounted *and* the
variable set), and the missing-daemon probe is a third, but it is an
availability check rather than a security boundary — do not treat it as the
gate. Leaving the tool off costs nothing but that one feature.
- **The refresh cookie.** The three `REFRESH_COOKIE_*` variables default to the
secure setting and need no entry in `.env.prod`; the boot guard refuses
production if `REFRESH_COOKIE_SECURE` is turned off. Two things are worth
knowing on the release that introduces it. Everyone is signed out once, because
sessions predating it carried their refresh token in `localStorage` and nothing
reads that any more — say so in the release note. And those tokens stay valid
server-side for up to `REFRESH_TOKEN_EXPIRE_DAYS`; if you want them retired at
the same moment, run once after the deploy:
```bash
docker compose -f docker-compose.prod.yml exec postgres \
psql -U maestro -d maestro \
-c "UPDATE refresh_tokens SET revoked_at = now() WHERE revoked_at IS NULL;"
```
This is a one-off operational choice, not a schema change, which is why it is
not an Alembic migration — the cookie change itself needs none.
- **`TRUST_PROXY_HEADERS` has no safe default**, which is why production will not
boot without it. `.env.prod.example` ships `true`, correct for this stack: Caddy
is the only service that opens a port, and it appends the peer it actually saw.
Set `false` only if you put the backend on a public port yourself — leaving
`true` there would let any client forge `X-Forwarded-For` and get a fresh
rate-limit bucket per request. Upgrading a deployment whose `.env.prod` predates
this guard means adding the one line before the next `up -d`.
- **`PAYMENT_PROVIDER=mock`.** No real money moves, and no real card should ever
be entered: `payment_methods` would fall under PCI scope. Ship a real
processor adapter before advertising billing.
- **Transactional email.** Five variables in `.env.prod`: `EMAIL_PROVIDER`,
`RESEND_API_KEY`, `EMAIL_FROM`, `SITE_URL`, `EMAIL_VERIFICATION_REQUIRED`.
A hosted instance sets `EMAIL_PROVIDER=resend`, a real `RESEND_API_KEY`, and
`SITE_URL=https://<your-domain>` (the base for verification/reset links in
emails). The default `EMAIL_PROVIDER=console` only logs messages, so
verification and password-reset links would never reach users. The backend
reads these via `env_file: .env.prod`, so no compose change is needed.
`EMAIL_VERIFICATION_REQUIRED` ships `false` for exactly that reason — never
turn it on before a real sender works end to end, or every account is locked
out of task start and API-key creation with no way to verify. When you do,
flip `EMAIL_VERIFICATION_LIVE` in `frontend/src/lib/legal/config.ts` in the
same change: it is a build-time constant the backend cannot set.
- FastAPI's Swagger UI, ReDoc and `/openapi.json` are disabled when
`ENVIRONMENT=production`.
- **Security headers come from Caddy** (`header` blocks in the `Caddyfile`),
on every response — app, API and Umami alike: HSTS (one year,
`includeSubDomains`, no `preload`), a Content-Security-Policy,
`X-Content-Type-Options: nosniff`, `X-Frame-Options: DENY`,
`Referrer-Policy` and `Permissions-Policy`; `Server` and `X-Powered-By` are
stripped. The CSP is pragmatic rather than strict: `script-src` and
`style-src` keep `'unsafe-inline'` because Next.js hydration scripts, the
JSON-LD block in the root layout and React style attributes all break
without nonces. The only external origin allowed is `https://*.sentry.io`
in `connect-src` — if you point `FRONTEND_SENTRY_DSN` at a self-hosted
Sentry, edit that line to match its host. Header changes take effect with
the zero-downtime reload documented under "Enabling on an existing
deployment" (`exec caddy caddy reload`).
- Datastores publish no ports. They are reachable only from the compose network.
- `.env.prod` holds `API_KEY_MASTER_KEY`, which decrypts every user's stored
provider keys. Losing it means losing them; leaking it means leaking them.
---
## Monitoring
The stack ships a lightweight observability setup: dependency health probes,
self-contained operator alerting, Prometheus-format metrics, structured logs, and
optional error tracking. No metrics *scraper* (Prometheus / Grafana) runs on the
host — deliberately, to keep RAM free on a single small box. The app exposes the
metrics; point your own scraper at them if you want a time series.
**Set `ALERT_WEBHOOK_URL` or `ALERT_EMAIL_TO`.** Without one of them nothing tells
you the site is down or erroring — the health probes are pull-based, and Sentry
alert rules are configured in Sentry's own UI, not here. The alerting below costs
nothing and needs no extra service.
### Health probes
Two endpoints, both reachable through Caddy without authentication:
- **`GET /health`** — liveness. Returns `{"status":"ok"}` without touching any
dependency. The Docker `HEALTHCHECK` uses this; keep an uptime monitor on it.
- **`GET /health/ready`** — readiness. Pings Postgres, Mongo, Qdrant and Redis
and returns `200 {"status":"ready"}` or, if any required service is down,
`503 {"status":"degraded"}`.
Point a free uptime monitor (UptimeRobot, Better Stack, …) at both:
```
https://<your-domain>/health # expect 200 — process is up
https://<your-domain>/health/ready # expect 200 — dependencies are up
```
Set a 1–3 minute interval and an email/Slack alert on a non-200 response. The
`/health/ready` monitor catches "the API is running but Mongo/Qdrant is
unreachable" — a state `/health` alone would miss.
**Which** dependency is down is not in the public response: naming it hands an
anonymous caller a live map of your infrastructure, including the moment Redis
drops and rate-limit buckets fall back to process-local counters. Set
`HEALTH_DETAIL_TOKEN` in `.env.prod` (any random string — it grants nothing else)
and pass it to get the per-dependency breakdown:
```bash
curl -H "X-Health-Token: $HEALTH_DETAIL_TOKEN" https://<your-domain>/health/ready
# {"status":"degraded","checks":{"postgres":"ok","mongo":"error", …}}
```
Leave it unset and the map is withheld from everyone; the status code an uptime
monitor alerts on is unaffected either way. `redis: "skipped"` means no
`REDIS_URL` is configured, which is a supported topology, not a failure.
### Operator alerts
The backend watches itself and pushes an alert when something breaks. Two
channels, either or both:
```bash
ALERT_WEBHOOK_URL=https://hooks.slack.com/services/… # or a Discord webhook
ALERT_EMAIL_TO=ops@example.com # via EMAIL_PROVIDER
```
One payload serves Slack and Discord (Slack reads `text`, Discord reads
`content`), so there is no per-platform setting — paste an incoming-webhook URL
from either. Anything that accepts a JSON POST works, including an internal
notifier on the compose network. **With both empty, alerting is a silent no-op
that makes no network call at all**, exactly like an empty `SENTRY_DSN`.
Configuring a channel *is* the enable; there is no separate switch.
Two things fire:
| Alert | Fires when | Recovers |
|---|---|---|
| **Readiness** | `check_readiness()` fails `ALERT_READINESS_FAILURES` ticks in a row (2 × 60s by default) | One good tick, with the downtime |
| **Error rate** | 5xx share of served requests exceeds `ALERT_ERROR_RATE_THRESHOLD` (5%) over `ALERT_ERROR_RATE_WINDOW_SECONDS` (300s), given at least `ALERT_ERROR_RATE_MIN_REQUESTS` (20) requests | — |
Alerts fire on a **state transition**, never on a tick: a dependency that stays
down pages once and then goes quiet until it recovers. `ALERT_COOLDOWN_SECONDS`
(900) is a second line of defence, not the first. Degraded and recovered carry
different dedupe keys, so a recovery is never swallowed by the outage's cooldown.
Two defaults that look arbitrary and are not:
- **Two failing ticks, one recovering tick.** A restarting Postgres finishes well
inside 120s, and a backend that boots before its dependencies pass their own
healthchecks must not wake anyone. Recovery is immediate because there is no
cost to being told early that things are fine.
- **The 5xx threshold is a ratio, not a count.** Each uvicorn worker serves
roughly 1/N of the traffic, so a raw count would silently become N times
stricter per worker; a ratio means the same thing at any `WEB_CONCURRENCY`.
With `REDIS_URL` set, the "send this alert" right is claimed with `SET NX EX`, so
N workers page once rather than N times. If Redis is itself the dead dependency
the claim falls back to process-local — each worker then reports the Redis
outage, which is the right failure mode for an outage that would otherwise
silence its own alert.
The alert body **names the failing dependency**, which `/health/ready`
deliberately withholds from an anonymous caller. That is not an inconsistency:
the recipient here is the operator who has to go fix it. Everything on its way
out passes a redactor that rewrites URL credentials (`postgres://user:pw@host`)
and masks the exact value of every configured secret, so a body cannot carry a
key even if a future code path interpolates one.
**Known limit:** this runs inside the backend it watches. It reports a dead
*dependency*, not a dead *backend*. See the uptime sidecar below.
### Metrics
`GET /metrics` serves the Prometheus text exposition format from in-process
counters — no scraper, no extra dependency, no `path` label (unbounded
cardinality is how a hand-rolled registry becomes an OOM).
```bash
METRICS_TOKEN=$(openssl rand -hex 16) # in .env.prod, then restart the backend
curl -H "X-Metrics-Token: $METRICS_TOKEN" http://localhost:8000/metrics
```
| Metric | Type | Labels |
|---|---|---|
| `maestro_build_info` | gauge | `version`, `environment` |
| `maestro_process_start_time_seconds` / `_uptime_seconds` | gauge | — |
| `maestro_http_requests_total` | counter | `status_class` (`1xx`…`5xx`) |
| `maestro_http_request_duration_seconds` | histogram | `le` (+ `_sum`, `_count`) |
| `maestro_dependency_up` | gauge | `dependency` |
| `maestro_readiness_up` | gauge | — |
| `maestro_alerts_sent_total` / `_suppressed_total` | counter | `kind` |
Every series carries a `worker` label. Each worker keeps its own counters and
publishes a snapshot to Redis; `/metrics` returns its own plus every peer's, so
`sum(rate(maestro_http_requests_total[5m]))` is correct and a worker restart
resets one series instead of making a summed counter run backwards.
Health-probe and `/metrics` traffic is excluded from the counters. That is
load-bearing: `/health/ready` answers 503 while degraded, so counting it would
make every dependency outage also trip the 5xx alert.
Three deliberate choices worth not "fixing":
- **Empty `METRICS_TOKEN` → 404, not 401.** An install that never configured it
is indistinguishable from one where the route does not exist.
- **A separate token from `HEALTH_DETAIL_TOKEN`.** This one is pasted into a
long-lived scraper config and exposes far more; their rotations should not be
coupled.
- **The `Caddyfile` does not route `/metrics`.** Its `@api path /api/* /health*`
matcher does not match it, so the endpoint is reachable only from the compose
network or an SSH tunnel — the same posture as the Umami dashboard. The token
is defence in depth, not the boundary. Scrape it over a tunnel:
```bash
ssh -L 9090:localhost:8000 user@host # then scrape localhost:9090/metrics
```
### Uptime sidecar (optional)
The in-process watchdog cannot report a backend that is itself wedged,
OOM-killed or deadlocked — the failure you most need to hear about. The `uptime`
profile adds a container that probes from outside the process and POSTs to the
webhook itself:
```bash
# .env.prod
COMPOSE_PROFILES=uptime # or analytics,uptime
ALERT_WEBHOOK_URL=https://… # required by this profile
docker compose -f docker-compose.prod.yml --env-file .env.prod up -d
```
It polls `backend:8000` directly rather than through Caddy, so a failure means
the app is down and not the proxy, and it alerts on a transition with the same
two-failure discipline. It is `curlimages/curl` running a shell loop: 3–6 MB RSS,
capped at 64 MB. Uptime Kuma would give you a dashboard for 150–250 MB plus a
volume and an interactive first run — install it separately if you want that and
have the memory; it is the wrong default for the single small box this compose
file targets.
**It shares the host it watches**, so it closes the "backend is wedged" blind
spot and not the "host is gone" one. Keep a free external monitor
(UptimeRobot, Better Stack) on `https://<your-domain>/health` — that is the only
thing that can tell you the machine itself disappeared.
### Error tracking (Sentry)
Optional and off by default, on both sides of the stack.
**Backend** — to enable:
1. Create a project at [sentry.io](https://sentry.io) (free tier is enough) and
copy its DSN.
2. Set `SENTRY_DSN=<dsn>` in `.env.prod` (see also `SENTRY_ENVIRONMENT`,
`SENTRY_TRACES_SAMPLE_RATE`) and restart the backend.
3. Configure an alert rule + email in the Sentry project.
Unhandled errors — including background task failures, which never reach the
request handler — are reported automatically via the logging integration. PII is
scrubbed before events leave the process: `send_default_pii=False` plus a
`before_send` hook that masks credential headers and drops request bodies, so
API keys, prompts and card data are never sent. With `SENTRY_DSN` empty, Sentry
is a no-op and the app makes no external calls.
**Frontend** — a *second* Sentry project (platform: Next.js), because backend
and frontend events need separate DSNs and dashboards:
1. Create the project and copy its DSN.
2. Set `FRONTEND_SENTRY_DSN=<dsn>` in `.env.prod` and
`docker compose -f docker-compose.prod.yml --env-file .env.prod up -d frontend`.
The DSN is read at runtime (server-side, like `SITE_URL`), so the image stays
domain-agnostic. Server-side render errors are captured via `onRequestError`;
browser errors are captured by the error boundaries once the lazily-loaded SDK
initializes. With the variable empty, no Sentry chunk is ever served to
browsers and the app makes zero requests to any ingest host.
Deliberate trade-offs (all cheap to revisit): no source-map upload (client
stack traces are minified; enabling it would need `withSentryConfig` plus a CI
auth token), no tunnel route (ad-blockers may drop *browser* events; server
capture is unaffected), no session replay (bundle size + PII surface).
### Logs
`LOG_FORMAT=json` (the production default in `.env.prod.example`) emits one JSON
object per line — pipe `docker compose logs` into any aggregator. `LOG_FORMAT=text`
keeps the readable format for local debugging.
Every HTTP response carries an `X-Request-ID` header, and each request emits one
`maestro.access` log line with `request_id`, `method`, `path`, `status` and
`duration_ms` (health probes excluded to keep the noise down). The same id is
attached to error logs and Sentry events, so a user-reported failure correlates
directly:
```bash
docker compose -f docker-compose.prod.yml logs backend | grep <request-id>
```
Caddy writes JSON access logs to stdout (`log` directive in the `Caddyfile`),
giving request-level visibility in front of both apps — including uptime-monitor
hits on `/health*`, which are not filtered at the proxy.
Container logs are rotated by the `json-file` caps in `docker-compose.prod.yml`
(`max-size: 10m`, `max-file: 5` per service — roughly 550 MB worst case for the
whole stack). Changing the caps requires recreating the containers; `up -d`
does that and drops the old log files.
---
## Analytics (optional, self-hosted Umami)
Off by default. When enabled it is first-party (data never leaves the host),
cookieless, consent-gated (the notice becomes a real Accept/Reject, the script
loads only after an explicit yes), and counts the public marketing pages only —
never the signed-in app. The Umami dashboard is deliberately not exposed to the
internet; only `/a/script.js` and `/a/api/send` pass through Caddy.
### Enabling
1. In `.env.prod`, set:
```env
COMPOSE_PROFILES=analytics
UMAMI_DB_PASSWORD=<openssl rand -base64 24>
UMAMI_APP_SECRET=<openssl rand -hex 32>
```
2. `docker compose -f docker-compose.prod.yml --env-file .env.prod up -d`.
`umami-db-init` runs once and creates the `umami` role and database inside
the existing Postgres (idempotent — safe on an already-initialized `pgdata`
volume, which is why an initdb script would not work); `umami` then runs its
own Prisma migrations and comes up healthy.
3. Reach the dashboard through an SSH tunnel — it listens on the host's
loopback only:
```bash
ssh -N -L 3001:127.0.0.1:3001 <user>@<host>
```
Open `http://localhost:3001`, log in as `admin` / `umami`, and **change that
password immediately**. Then Settings → Websites → Add website (name:
maestro, domain: your `DOMAIN`) and copy the Website ID.
4. Put the id in `.env.prod` as `UMAMI_WEBSITE_ID=<id>` and restart the
frontend so it picks up the new env:
```bash
docker compose -f docker-compose.prod.yml --env-file .env.prod up -d frontend
```
Visitors now get the Accept/Reject notice; views appear in the dashboard only
for those who accept, and only on marketing pages. Consent can be changed any
time at the bottom of `/cookies`.
### Enabling on an existing deployment
Copy the updated `docker-compose.prod.yml` and `Caddyfile` to the host first,
then follow the steps above. `up -d` recreates Caddy with the `/a/*` route (or
reload in place: `docker compose -f docker-compose.prod.yml --env-file .env.prod
exec caddy caddy reload --config /etc/caddy/Caddyfile`). Nothing else in the
stack restarts.
### Disabling
Set `COMPOSE_PROFILES=` and `UMAMI_WEBSITE_ID=` back to empty and `up -d
--remove-orphans`. The frontend reverts to the informational notice; the
`umami` database stays in `pgdata` (harmless) unless you drop it.
---
## Backups
`backup.sh` (from `scripts/backup.sh` in the repo, kept in sync by `deploy.yml`)
dumps every durable store into `/opt/maestro/backups`, applies retention and
optionally pushes the files offsite. It is scheduled from the host crontab,
exactly like the purge job.
### What is backed up
| Store | What | File |
|---|---|---|
| Postgres | `maestro` DB — accounts, subscriptions, quota ledger; the one that matters most | `maestro-pg-<ts>.sql.gz` (`pg_dump --clean --if-exists`) |
| Postgres | `umami` DB — only when the `analytics` profile is active in `.env.prod` | `maestro-umami-<ts>.sql.gz` |
| MongoDB | `maestro` DB — agent logs, task sessions, marketplace, agent configs | `maestro-mongo-<ts>.archive.gz` (`mongodump --archive --gzip`) |
| Qdrant | every collection, enumerated dynamically — per-collection snapshots over the HTTP API | `maestro-qdrant-<collection>-<ts>.snapshot` |
Qdrant is distroless and publishes no ports, so the script talks to it through
a one-shot `curlimages/curl` container joined to the compose network; each
snapshot is downloaded over HTTP and then deleted server-side so nothing
accumulates inside the `qdrantdata` volume.
> **Qdrant snapshots require named-volume storage.** On the production
> stack (`qdrantdata` named volume) a snapshot of a near-empty collection is
> ~140 KB and restores cleanly. On a Windows Docker Desktop **bind mount**
> (the dev stack's `./.data/qdrant`) the same snapshot balloons to ~400 MB
> (sparse WAL files get materialized) and its `wal/first-index` is zeroed,
> so the restore fails with a WAL deserialize error. This is a dev-only
> filesystem artifact — verified 2026-07-13 against `v1.18.2` both ways —
> not a production risk.
**Deliberately not backed up:** `redis` (ephemeral rate-limit buckets,
persistence is disabled on purpose), `ollamadata` (models are re-pulled by
`ollama-pull`), `caddydata` (TLS certificates re-issue automatically on the
first request after a rebuild).
**`.env.prod` is never touched by the script.** Back it up manually,
encrypted (e.g. in a password manager) — losing `API_KEY_MASTER_KEY` makes
every BYOK key in a Postgres dump permanently undecryptable, and leaking it
decrypts all of them. The script pushes only the backups directory offsite,
so secrets structurally cannot leave the host through it.
### Schedule and retention
Daily at 03:30 (offset from the 03:00 purge), guarded by `flock` so an
overlapping run exits instead of stacking:
```cron
30 3 * * * cd /opt/maestro && RCLONE_REMOTE=oci:maestro-backups flock -n /opt/maestro/backup.lock ./backup.sh >> /opt/maestro/backup.log 2>&1
```
Retention is applied by the script itself, locally and remotely: `daily/`
keeps 7 days, `weekly/` (a copy made every Sunday) keeps 28 days. Pruning is
scoped to the `maestro-*` naming pattern and never deletes anything else.
Run the script once by hand over SSH before trusting the cron line, and note
`backup.log` grows about a line per day — truncate it ad hoc.
### Offsite copy (Oracle Object Storage via rclone)
The offsite push is env-gated: leave `RCLONE_REMOTE` unset (drop it from the
cron line) and the script stays local-only. To enable it:
1. In the OCI console create a **private** bucket, e.g. `maestro-backups`
(Always Free includes 20 GB of Object Storage).
2. Create a **Customer Secret Key** for a least-privilege IAM user (Identity →
Users → Customer Secret Keys) — this is OCI's S3-compatible credential.
3. Install rclone on the host (`sudo apt install rclone` or the arm64 static
binary) and configure `~/.config/rclone/rclone.conf`:
```ini
[oci]
type = s3
provider = Other
access_key_id = <customer secret key id>
secret_access_key = <customer secret key>
endpoint = https://<namespace>.compat.objectstorage.<region>.oraclecloud.com
region = <region>
```
4. Set `RCLONE_REMOTE=oci:maestro-backups` in the cron line.
The script uses `rclone copy` (additive), deliberately not `sync`: syncing
from a freshly rebuilt host with an empty backups directory would delete
every offsite backup — exactly the disaster the offsite copy exists to
survive. Remote retention is pruned separately with `rclone delete --min-age`.
### Restore runbook
All commands run from `/opt/maestro`. Define the compose prefix once:
```bash
DC="docker compose -f docker-compose.prod.yml --env-file .env.prod"
```
**Postgres** — the dump carries `--clean --if-exists`, so piping it in
drops and recreates every object:
```bash
$DC stop backend
gunzip -c maestro-pg-<ts>.sql.gz | $DC exec -T postgres psql -U maestro -d maestro
$DC up -d backend
```
Umami restores the same way into `-d umami`. For a cheap drill without
touching live data, restore into a scratch DB first:
`$DC exec -T postgres createdb -U maestro scratch` then `psql ... -d scratch`.
**MongoDB** — `--drop` replaces each collection in place:
```bash
$DC exec -T mongo mongorestore --archive --gzip --drop --nsInclude 'maestro.*' \
-u "$MONGO_USER" -p "$MONGO_PASSWORD" --authenticationDatabase admin \
< maestro-mongo-<ts>.archive.gz
```
**Qdrant** — upload the snapshot per collection; the upload recreates the
collection even if it no longer exists (read `QDRANT_API_KEY` from `.env.prod`
rather than exporting it into shell history):
```bash
cat maestro-qdrant-<collection>-<ts>.snapshot | docker run --rm -i \
--network maestro_default curlimages/curl:8.11.1 -fsS \
-H "api-key: $QDRANT_API_KEY" \
-F "snapshot=@-;filename=restore.snapshot" \
"http://qdrant:6333/collections/<collection>/snapshots/upload?priority=snapshot"
```
**Full disaster recovery**, in order: provision a fresh host → place the four
files (compose, Caddyfile, `.env.prod` from your encrypted copy, `backup.sh`)
→ `up -d` (this runs `migrate` against the empty database) → restore Postgres,
Mongo, then Qdrant from the offsite bucket (`rclone copy oci:maestro-backups/daily .`)
→ `$DC restart backend`. If the dump predates the current migration head,
restoring it rewinds the schema to the dump's state; run
`$DC up -d migrate` afterwards to bring it forward again.
---
## Local smoke test
Verify the production stack on your own machine before touching a server. Set
`DOMAIN=http://localhost` in `.env.prod` — the `http://` prefix tells Caddy to
serve plain HTTP and skip certificate provisioning.
```bash
cp .env.prod.example .env.prod # DOMAIN=http://localhost, fill the secrets
docker compose -f docker-compose.prod.yml --env-file .env.prod up -d --build
```
> **Keep real secrets out of the repo tree.** A `.env.prod` filled with
> production values (notably `API_KEY_MASTER_KEY`, which decrypts every stored
> BYOK key) must not sit inside the working tree — it is `.gitignore`d, but one
> `git add -f` away from being committed. Keep the real file outside the repo
> (e.g. `~/.maestro/.env.prod`, locked to your user) and pass its absolute path
> to `--env-file`. On the production host the file already lives beside the
> compose file at `/opt/maestro/.env.prod`, so the relative form is correct
> there; only local runs off an out-of-tree copy need the absolute path:
>
> ```bash
> docker compose -f docker-compose.prod.yml \
> --env-file /absolute/path/to/.env.prod up -d --build
> ```
Then check, in order:
1. `docker compose -f docker-compose.prod.yml --env-file .env.prod ps` —
`migrate` and `ollama-pull` `exited (0)`, the rest `healthy`.
2. `curl http://localhost/health` → `{"status":"ok"}`.
3. `curl http://localhost/api/v1/billing/plans/public` → JSON.
4. Open `http://localhost/pricing`. Prices must render — the "unreachable"
banner means the server-side fetch failed.
5. Open `http://localhost/docs`. You should get the marketing page, not Swagger.
6. Register, log in, start a task. In DevTools → Network → WS, the connection to
`/api/v1/tasks/<id>/stream` must report **101 Switching Protocols** and
stream events.
7. Upload a `.txt` under Documents. A `200` proves the embedding service is
wired up.
8. If testing analytics (`COMPOSE_PROFILES=analytics` + the `UMAMI_*` vars): in
a fresh browser profile the notice must show Accept/Reject with **no**
`/a/script.js` request before you answer. After Accept, each marketing page
navigation sends one `POST /a/api/send`; navigating into `/login` or
`/dashboard` must send **zero**. `curl http://localhost/api/send` must still
reach the backend (a FastAPI 404/405, not Umami) — that proves the `/a`
matcher split.
---
## Running the frontend elsewhere
The frontend image is domain-agnostic, and the code supports a split deploy
without a rebuild. On Vercel (or any Node host), set:
| Variable | Value |
| --- | --- |
| `BACKEND_ORIGIN` | `https://api.your-domain` — proxies `/api/*`, keeping the browser same-origin |
| `INTERNAL_API_ORIGIN` | `https://api.your-domain` — used by the server-rendered `/pricing` and `/templates` |
| `NEXT_PUBLIC_WS_BASE_URL` | `wss://api.your-domain` — WebSockets cannot be proxied through Next rewrites |
| `SITE_URL` | `https://your-domain` — origin for canonical URLs, `sitemap.xml`, OG tags |
The backend then needs `CORS_ORIGINS` set to the app's origin, since the
WebSocket handshake no longer shares it. The API itself still has to run as a
long-lived container somewhere.
`BACKEND_ORIGIN` is not optional in this topology, and the refresh cookie is why:
it is `SameSite`, so it only travels on requests the browser considers same-site.
Proxying `/api/*` through Next keeps them that way. Calling the API host directly
from the browser would leave every sign-in unable to survive a reload. If app and
API share one registrable domain (`app.example.com` / `api.example.com`), the
alternative is `REFRESH_COOKIE_DOMAIN=.example.com` — at the cost of handing the
session cookie to every subdomain. There is deliberately no `SameSite=None`
option; see CLAUDE.md §8.
`SITE_URL` is what keeps the image domain-agnostic despite SEO needing an
absolute origin: it is read at request time, not baked in, so the same image
serves any domain and changing it needs only a container restart, not a
rebuild. It is deliberately not `NEXT_PUBLIC_` (that would inline it at build
time) and not derived from `DOMAIN` (which may carry a scheme). Unset, the
pages fall back to a placeholder domain — visibly wrong rather than silently
plausible.