docs/deployment.md
Rama main · réplica de lectura del repositorio original.
```
╔════════════════════════════════════════════════════════════════════════════╗
║ ║
║ DEPLOYMENT ║
║ Production operations for the single-process self-hosted setup. ║
║ ║
╚════════════════════════════════════════════════════════════════════════════╝
```
# Deployment
Cloud Codex is designed to run on **one box**: one Node container, one
MySQL container, one named volume for the database. This document covers
production deployment, the operational rhythm that comes with it
(backups, upgrades, log handling), and the few production-specific
gotchas.
For local development setup, see [getting-started.md](./getting-started.md).
For things that go wrong, see [troubleshooting.md](./troubleshooting.md).
---
## Production topology
```
Internet
│
▼
┌──────────────────────────────────────────────────────┐
│ Reverse proxy (Caddy / nginx / Cloudflare Tunnel) │
│ · TLS termination │
│ · forwards Upgrade headers for /collab and │
│ /notifications-ws │
│ · sets X-Forwarded-For for rate-limit accuracy │
└─────────────────────────┬────────────────────────────┘
│
▼
┌──────────────────────────────────────────────────────┐
│ Cloud Codex Node container (docker-compose-prod) │
│ · vite-express serves dist/ + Express API │
│ · port published on 127.0.0.1 only (APP_BIND) │
│ · network pinned to 172.29.0.0/16 (gateway .1) │
│ · 2 WS servers attached (collab + notifications) │
│ · reads .env, exits if admin credentials missing │
│ · one process per database (a per-schema lock) │
│ · SMTP is optional; mail degrades if unconfigured │
│ · named volume app_public (avatars, doc-images) │
│ · daily activity_log prune │
└─────────────────────────┬────────────────────────────┘
│ mysql2 pool (10)
▼
┌──────────────────────────────────────────────────────┐
│ MySQL 8 container │
│ · named volume db_data │
│ · 3306 not published by docker-compose-release.yml │
│ · 3306 on 127.0.0.1 in docker-compose-prod.yml │
└──────────────────────────────────────────────────────┘
```
---
## Compose files
| File | When to use |
|-------------------------------|----------------------------------------------------|
| `docker-compose.yaml` | **Dev** — MySQL only, app runs from `npm run dev` |
| `docker-compose-release.yml` | **Prod, published image** — no build toolchain |
| `docker-compose-prod.yml` | **Prod, from source** — builds `./cloudcodex` |
| `docker-compose.linux.yml` | Native-Linux override for the dev file: `:Z` SELinux labels on its bind mounts, merged by `start.sh` on Linux but not WSL; publishes no port |
Running a published release, which is the recommended path unless you are
deploying modified source:
```bash
cp .env.example .env # fill every required variable, APP_URL included (see below)
docker compose -f docker-compose-release.yml up -d # init.sql builds the schema
docker compose -f docker-compose-release.yml run --rm app \
npm run migrate -- --adopt-fresh-install
```
This pulls `ghcr.io/cloud-city-computing/cloud-codex`, pinned by
`CLOUDCODEX_VERSION` (default `0.14.0`), so nothing is compiled locally and the
version does not move under you on the next publish. The published image is
`linux/amd64`; Apple Silicon runs it under Docker Desktop's emulation.
> **If the pull answers `unauthorized`,** the GHCR package is still private.
> Container packages created by Actions start private regardless of repository
> visibility, and it has to be flipped once by an org owner in Package settings.
> See the release section of
> [`docs/maps/build-test-and-ops.md`](maps/build-test-and-ops.md).
Building from your own source instead:
```bash
cp .env.example .env # fill every required variable, APP_URL included (see below)
docker compose -f docker-compose-prod.yml up -d --build # init.sql builds the schema
docker compose -f docker-compose-prod.yml run --rm app \
npm run migrate -- --adopt-fresh-install
```
The second command, in either snippet, runs once, on a brand-new install only:
it records that the schema `init.sql` just built already has every migration,
and applies nothing, so the app can stay up for it. Until it runs, `/readyz`
answers `503 {"ready":false,"reason":"migrations"}`. An install you are
upgrading follows [Upgrades](#upgrades) instead, and
[First run: record a starting point, once](#first-run-record-a-starting-point-once)
says which command an older database needs.
The app container builds the Vite frontend during `docker build`. It
exits at startup, with a sentence naming the variable, if the admin
credentials are missing or `APP_URL` is unset. **Mail is optional**: with no
SMTP the server starts anyway, invitations show a link to copy instead of
being emailed, and password reset, email two-factor codes (so turning
two-factor off too) and notification emails are unavailable until it is
configured.
---
The compatibility promises for 1.x, and the breaking changes since 0.12.0, are in [stability.md](./stability.md).
## Required environment for production
**Every variable the server reads is listed in
[`cloudcodex/env-contract.js`](../cloudcodex/env-contract.js)**, with whether
it is required, required in production, defaulted (and to what) or optional,
and why. A test fails when the server reads a variable that file does not
list, so it is complete by construction; `.env.example` carries a comment for
each one, and [getting-started.md](./getting-started.md) walks through them.
Production-specific notes:
| Variable | Production note |
|----------------------------|----------------------------------------------------------|
| `APP_URL` | **Required in production**: without an `http://` or `https://` URL the server exits at boot, and a `localhost` one boots with a warning. The public address people use, `https://` behind a TLS proxy; invitation, reset and notification links carry it |
| `CORS_ORIGIN` | Leave empty. The app's own origin and `APP_URL`'s are always allowed; set it only for a separate front end |
| `TRUST_PROXY` | Which proxies to believe about the client address, which is what the rate limiters count, named by address. Unset is `127.0.0.1/32, ::1/128, 172.29.0.1/32`, right for nginx or Caddy on the same host in front of either compose file; without Docker, set `127.0.0.1/32, ::1/128`. A proxy container, a load balancer, or `docker run` must be listed. A hop count, `true`, or a range past the width thresholds stops the boot. See [Rate limiters](#rate-limiters) |
| `TRUST_PROXY_ALLOW_HOP_COUNT` | Leave unset. `true` accepts a hop count, `true` or an over-wide range in `TRUST_PROXY` anyway, knowing that any client able to reach the app's port can then choose its own address |
| `APP_BIND` | Compose only, not read by the server: the host address the app port is published on. Unset or blank is `127.0.0.1`. **An IPv4 address only**: `::` or an IPv6 address publishes to IPv6 clients, who arrive as the network's gateway and are trusted. See [TLS and reverse proxy](#tls-and-reverse-proxy) |
| `DB_BIND` | `docker-compose-prod.yml` (and the dev file), not read by the server: the host address MySQL's 3306 is published on. Unset or blank is `127.0.0.1`, which a mysql client or `npm run migrate` on the host reaches. Widen it only on purpose; the release file does not publish 3306 at all |
| `DB_POOL_SIZE` | MySQL connections the app holds open, 1 to 100. Unset is `10` |
| `SMTP_*` | Optional; the server starts without them. Without them invitations show a copyable link, and password reset, email two-factor codes and notification emails are unavailable |
| `SMTP_FROM` | Blank sends as `Cloud Codex <noreply@` the host in `APP_URL` `>`, so mail comes from this instance's own domain. Set it to a sender your mail host accepts |
| `ADMIN_*` | `ADMIN_EMAIL` is always required. `ADMIN_USERNAME` and `ADMIN_PASSWORD` are required while local sign-in is on, and a [OIDC-only instance](#oidc-only-mode) sets neither. **On a local instance they reset the admin's email and password at every boot**; see [The boot admin](#the-boot-admin) |
| `GITHUB_CLIENT_SECRET` | Doubles as the AES-256-GCM seed for stored OAuth tokens. **Never rotate without re-encrypting** existing rows or all linked GitHub accounts go invalid |
| `GOOGLE_OAUTH_DOMAIN` | Locks SSO to a specific domain — leave unset to allow any Google account to *link*, but only same-domain users can *sign up* |
| `AUTH_PROVIDERS` | Leave unset for a self-hosted instance. If set, it must include `local` and agree with the Google variables, or the server exits at boot with a sentence saying which. Without `local` it is a [OIDC-only instance](#oidc-only-mode): `oidc` alone, with the Google variables unset |
| `LEGACY_SESSION_COOKIE` | Leave unset. On https the session cookie is `__Host-sessionToken`; unset keeps a browser that still holds the older `sessionToken` signed in and moves it across on its next visit. `0` (https only) makes a lone `sessionToken` sign nobody in: set it when hosts you do not control share your domain. See [security.md](./security.md#the-session-cookie) |
| `WEBHOOK_ALLOW_PRIVATE_TARGETS` | Optional. `1` lets a webhook receiver sit on a loopback or private address. Subscriptions are created in Admin, Webhooks ([api/webhooks.md](./api/webhooks.md)) |
Add new env vars to `.env.example` (with a comment) when introducing them.
### OIDC-only mode
An OIDC-only instance signs people in through one OIDC issuer and nothing else. Set
`AUTH_PROVIDERS=oidc`, with `OIDC_ISSUER_URL`, `OIDC_CLIENT_ID` and `OIDC_CLIENT_SECRET`
(and `APP_URL`, which OIDC needs). Leave `GOOGLE_CLIENT_ID` and `GOOGLE_CLIENT_SECRET`
unset. The server then refuses, on the server and not only in the page:
- sign-up, password login, password reset, two-factor, the username check, and the
local email change (`/api/create-account`, `/api/login`, `/api/forgot-password`,
`/api/reset-password`, every `/api/2fa/*`, `/api/update-account/confirm-email`): 404;
- an email or password change through `POST /api/update-account`: 403. A display name still changes;
- Google sign-in (`/api/oauth/google` and its callback): 404.
`ADMIN_USERNAME` and `ADMIN_PASSWORD` may be unset. The boot admin is matched by
`ADMIN_EMAIL` alone and created with no password; its username is `ADMIN_USERNAME` when no
account holds it, and otherwise a name derived from `ADMIN_EMAIL`. Its starter workspace is
named from that account, and it has the `General` squad that synced members join. A boot
that sets `AUTH_PROVIDERS` without `oidc`, or sets Google's variables, exits with a sentence
naming the variable. There is no machine credential on any instance (removed in 0.15.0), so a
OIDC-only instance has nothing to configure for machine callers.
An administrator's two-factor reset (`POST /api/admin/users/:id/2fa/reset`) still works on an OIDC-only instance, so a local two-factor setting can be cleared before the instance switches back to local accounts. Provider setup is in [sign-in.md](./sign-in.md).
**Invitations bind on verified email.** An OIDC sign-in whose verified email matches an open
invitation creates the account with that invitation's squad and flags, once. With no
invitation the sign-in is `no_account`: the issuer's own account never admits anyone. The
issuer's self-registration is therefore not an admission.
**Deactivation.** `users.deactivated_at` (set by hand until an admin control exists) refuses a
user at their next request, and every sign-in path refuses them as `account_deactivated`. Their
open WebSocket connections are closed by a periodic re-check that runs every 60 seconds and closes
any socket whose session is no longer live for an active user, so a deactivation takes effect within
about a minute. Their authorship stays.
**Existing sessions at the switch.** A session minted by local or Google sign-in ends when an
instance switches to OIDC-only mode. Validation refuses it, and the next boot on the OIDC-only instance
deletes every session that is not an OIDC session, logging how many it ended, so none can be used
again. Those users sign in through OIDC. Sessions minted by OIDC keep their own expiry
(`OIDC_SESSION_TTL_HOURS`).
**Local credentials on an OIDC-only instance are kept.** A password hash and a local two-factor secret that
were set before the switch stay in their rows. Nothing on an OIDC-only instance reads them: local login,
password reset, two-factor and the local email change answer 404, and `validateAndAutoLogin` refuses any
session that is not OIDC. The rows are kept on purpose: clearing them is destructive, and switching the
instance back to local sign-in restores the same passwords without a reset. Clearing them is a product
decision, not a boot step, so it is not automated.
**Deep links.** A signed-out deep link is carried through sign-in as `returnTo` only when it passes
the same validator as the sign-in return path (`services/return-to.js`). A link that fails validation, or whose encoding does not
decode, falls back to `/`, which is the landing page, rather than being refused with an error.
**An account with two-factor on before the switch.** Its two-factor routes are unmounted, so it
cannot turn two-factor off itself, and its first OIDC sign-in is refused as `two_factor_enabled`,
because an email link would let it skip the second factor. An operator clears it on the user row,
as for an `identity_conflict` relink, and the next OIDC sign-in links the account:
```sql
UPDATE users SET two_factor_method = 'none', totp_secret = NULL WHERE id = <id>;
```
### The boot admin
Every boot reconciles one admin from `ADMIN_USERNAME`, `ADMIN_EMAIL` and
`ADMIN_PASSWORD` (`ensureAdminUser` in `cloudcodex/routes/admin.js`), before
the port opens, and logs one line saying which of three things it did. None of
them logs the password.
- **Created.** No account has that name or that email: boot creates the admin.
- **Synced.** The account with that name or that email is already an admin:
**boot resets its email to `ADMIN_EMAIL` and its password to
`ADMIN_PASSWORD`, at every boot.** `.env` is the source of truth for the
admin's credentials, so a password or address the admin changes in the app
lasts only until the next restart; change it in `.env` instead. Its name is
never rewritten.
- **Refused.** An account with that name or that email is **not** an admin.
Boot never promotes one: it changes nothing, not that account and not the
admin's, and logs
```
admin sync: <ADMIN_USERNAME> / <ADMIN_EMAIL> matches an existing non-admin account (user <id>), refusing to promote it. Promote it in the admin console if that is intended.
```
It happens when a member holds a name or address the admin has given up
(the admin renamed, or changed email, and a member took the old one), or
when `.env` is edited to name an existing member. If that account is meant
to be the admin, promote it in the admin console (Users, click its **User**
badge); from the next boot on it is synced as above, so its email and
password become `ADMIN_EMAIL` and `ADMIN_PASSWORD`. If it is not, set
`ADMIN_USERNAME` and `ADMIN_EMAIL` to the admin's current name and address,
or to a name and address no account uses to have boot create a fresh admin,
and restart. The refusal repeats at every boot until one of those is done.
A refused sync is not fatal: the instance starts, an existing admin keeps the
credentials it had, and the first-boot seed does not run on that boot.
---
## TLS and reverse proxy
Cloud Codex serves plain HTTP on port 3000 inside its container, or on `PORT`
if that is set. Both production compose files publish it as
`${APP_BIND:-127.0.0.1}:${PORT:-3000}:${PORT:-3000}`: `PORT` moves the port on
both sides of the mapping (update `APP_URL` to match, and remember the smoke
test below uses that port), and **the host side is `127.0.0.1`**, so only this
machine reaches the app. A browser on the same machine opens
`http://localhost:3000` as before, and so does a reverse proxy running on the
host.
**Production belongs behind a TLS-terminating reverse proxy**, not on the app
port. Docker's published ports are a DNAT rule that sits in front of the host
firewall, so a port published on every interface is reachable past `ufw deny`
by anything that can route to the machine, and a client reaching the app
directly skips TLS altogether.
To expose the port deliberately (a load balancer on another machine that must
reach it, or an evaluation from another computer on your network), set
`APP_BIND` in `.env` to `0.0.0.0`, or to one interface's IPv4 address to publish
on that interface only, and restrict who can reach it at the network edge
(a cloud security group, not a host firewall rule Docker bypasses). Four
conditions come with it:
- **With nothing in front of the app, also set `TRUST_PROXY=false`.** An
evaluation from another computer reaches the app directly, so there is no
proxy to believe, and `false` counts every client by the address that
connected. That holds on every Docker runtime, including the ones below that
present every client as the gateway, so it is the rule rather than a guess
about which runtime you have.
- **IPv4 only.** `0.0.0.0` publishes on IPv4 alone (measured: an IPv6 client is
refused). `APP_BIND=::`, or any IPv6 address, publishes to IPv6 clients, and
the pinned network is IPv4-only, so Docker's userland proxy carries each one
in from the network's gateway, `172.29.0.1`, which the default trusts:
measured, an IPv6 client then named a new address on every attempt and
never met a limit. The general rule: **an IPv4-only Docker network presents
every IPv6 client of an all-interfaces or IPv6 publish as its gateway.**
- **A load balancer that reaches a widened bind must be listed in
`TRUST_PROXY` itself**: Docker hands the container its real address, which
the default does not name, so until you list it every client behind it
shares its one rate-limit bucket. See [Rate limiters](#rate-limiters).
- **Every container on the host can then reach the port, and arrives as the
gateway.** Another container on the compose network going to `172.29.0.1:PORT`, and a
container on another Docker network going to the host's own address, are
presented as `172.29.0.1`, so each is trusted and can name its own address.
On a box that also runs another Docker Compose project, its containers are such peers. With
`APP_BIND` on `127.0.0.1`, no container reaches the port except one sharing
the host's network namespace.
Some Docker runtimes carry **all** published traffic through a userland
proxy, not only IPv6 and loopback: `dockerd` with `"iptables": false`, rootless
Docker with its default (`builtin`) port driver, and Docker Desktop on macOS,
Windows and Linux. There every client, local or not, arrives as the gateway.
Behind a proxy on the host, keep `APP_BIND` on `127.0.0.1`; with it widened
and nothing in front, `TRUST_PROXY=false` (above) is what keeps each client in
its own bucket. A widened bind behind a load balancer on such a runtime is not
safe while anything but the load balancer can reach the port.
A reverse proxy running in **another container** cannot reach the host's
loopback: join it to this compose project's network, proxy to `app:3000` (or
`app:$PORT`) instead of widening `APP_BIND`, and list its address, as shown in
[Rate limiters](#rate-limiters).
Requirements the proxy must satisfy:
1. **WebSocket upgrade passthrough.** Both `/collab/:logId` and
`/notifications-ws` rely on the HTTP upgrade dance. A proxy that strips
`Upgrade` / `Connection` headers will silently break collab and
notifications.
2. **Same-origin headers.** `services/user-channel.js` enforces an
`Origin` host check against `Host`. If your proxy rewrites either,
make sure both end up matching the public hostname.
3. **A client address the app can believe.** The proxy connects from an
address `TRUST_PROXY` names (a proxy on the host does, by default), and it
**sets** `X-Forwarded-For`. For nginx as the only proxy, overwrite it with
the address nginx saw:
```nginx
proxy_set_header X-Forwarded-For $remote_addr;
```
Caddy's `reverse_proxy` already does the equivalent. With a load balancer in
front of nginx, append instead, so the client's address the load balancer
added survives, and list the load balancer in `TRUST_PROXY`:
```nginx
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
```
Appending is safe under address trust **as long as the address the proxy
appends is the real client's, and `TRUST_PROXY` does not also name it**:
the app reads the chain from the right and stops at the first address it
does not trust, which is that client's, so whatever the client put on the
left is never reached. It stops being safe when the proxy sees a trusted
address instead of the client's, which is what happens to a proxy in a
container whose published port reaches IPv6 clients over an IPv4-only
network: it appends the gateway, the app trusts the gateway, and the walk
goes on to the client's own entry. **The fatal configuration is setting
nothing**: nginx then passes the client's own header through, the app trusts
the proxy that delivered it, and the client names its own address. See
[Rate limiters](#rate-limiters).
Both WebSocket servers refuse a cross-origin upgrade themselves: each requires
an `Origin` whose host equals `Host`. Helmet's CSP (`connect-src 'self' ws: wss:`)
is not what enforces that. In production it does govern the page that opens the
sockets, but it allows any `ws:` or `wss:` host, and a CSP binds only the
browser that honours it, never a script calling the socket directly.
---
## Backups
There are **two** stateful volumes, and a MySQL dump alone is not a complete
backup: `db_data` holds the database, and `app_public` holds uploaded avatars
and the images extracted out of documents. When an image is pasted into a
document, `routes/helpers/images.js` extracts it to disk and **replaces the
base64 data URI in `html_content` with a `/doc-images/` URL**, so after
extraction the file on disk is the only copy. Lose the volume and every
affected document shows a broken image while the database still points at it.
One command backs up both, and one restores both. From the repository root,
with the stack's `.env` in place:
```bash
make backup OUT=backups/c2-$(date +%F).tar.gz # scripts/backup.sh
make restore IN=backups/c2-2026-09-28.tar.gz # scripts/restore.sh
```
Both drive the stack through `docker compose`, on
`docker-compose-release.yml` unless `COMPOSE_FILE` names another (for example
`COMPOSE_FILE=docker-compose-prod.yml` when you build from source), and
`COMPOSE_PROJECT_NAME` works as it does for Compose. The two production files
share this directory's project and container names, so each script compares
`COMPOSE_FILE` with the file the stack's containers were created from and
refuses a mismatch, naming the right one: a restore through the wrong file
would migrate and restart your stack on the other file's image. Neither needs
the MySQL root password: they run inside the database service as the app's own
MySQL user (`DB_USER`), which the image grants everything on the app's database
and nothing else, and both refuse a MySQL user that holds anything more.
**What an archive holds.** One gzipped tar of three files: `database.sql`
(`mysqldump --single-transaction --routines --triggers --hex-blob` of the app's
database, including `schema_migrations`, so the migration ledger comes back
with the data), `app_public.tar.gz` (the uploads volume) and `manifest.json`
(the format, when it was taken, the database name, the app version, and the
SHA-256 of the other two). No password is in any of them, and the manifest
names no host or user; `database.sql`'s header comment does name the database
host it was dumped from, as every `mysqldump` does.
**Keep it the way you keep the database.** The archive is written readable by
its owner only (mode 0600) and never over an existing file. It holds
everything the database holds: password hashes, two-factor secrets, GitHub
tokens (encrypted), session digests and every document. It does **not** hold
`.env` or any key material, so back those up separately and just as carefully:
`GITHUB_CLIENT_SECRET` in particular, because every stored GitHub token is
encrypted under a key derived from it, and a restore under a different secret
leaves each linked account to link again. The sign-in providers' client
secrets, SMTP credentials and `ADMIN_PASSWORD` are in `.env` too.
**What a backup is consistent to.**
- The dump is one consistent snapshot of InnoDB tables, which is every table
`init.sql` creates. A table added by hand with another engine is not covered.
- A backup can run while the app serves. But the app keeps up to the last
three seconds of live collaborative edits in memory before saving them, so
only a backup taken **after a graceful stop** (`docker compose stop app`,
see [Stopping cleanly](#stopping-cleanly)) is sure to have every edit. Stop,
back up, start is the safe rhythm for a nightly job.
- The uploads are archived after the dump, so an image pasted between the two
is in the archive with no document pointing at it, which is harmless.
**Restoring.** `make restore` restores into a stopped stack, then starts it:
```bash
docker compose -f docker-compose-release.yml stop app # if it is running
make restore IN=backups/c2-2026-09-28.tar.gz
make restore IN=... ARGS="--replace" # over existing data
```
It checks everything before it writes anything, and each check is a refusal:
the archive holds exactly its three files and each matches the manifest's
checksum; the uploads hold only plain files and directories; no line of the
dump starts with a statement that switches, creates or drops a database or
with a `mysql` client command (the client runs with `--binary-mode`, so a
client command anywhere else on a line never runs either); the backup was not
taken on a newer release than this stack runs (pass `--allow-newer-backup` to
restore it anyway, knowing the older app does not know the newer schema); the
archive is a backup of a database with the same name as this stack's (pass
`--into <this stack's DB_NAME>` to restore it under a different name on
purpose); the MySQL user holds privileges on that database and nothing else;
the app is not running; and the database holds no rows.
Tables with no rows at all are what the database service builds from
`init.sql` when a new stack first starts, so a restore onto a new machine needs
no flag. Over real data it needs `--replace`, which drops every table in the
database and empties `avatars/` and `doc-images/` before loading. The load runs
as that confined MySQL user, so a statement naming another database fails on
the grant, wherever it sits on a line. Its first statement takes the database's
[instance lock](#one-process-per-database), or fails with nothing written if
any process holds it, and the load keeps it, so an app that starts meanwhile
refuses. Then it runs `npm run migrate` (a backup from an older release is
brought up to date) and `docker compose up -d`. `--no-start` stops after the data and prints the two
commands instead.
A load that fails part way leaves the database holding part of the backup, and
says so; fix the cause and run the restore again with `--replace`.
**Scheduling and keeping them.** Nothing here rotates or ships archives. A
nightly cron entry, for example:
```cron
0 3 * * * cd /srv/cloudcodex && docker compose -f docker-compose-release.yml stop app && make backup OUT=/var/backups/cloudcodex/c2-$(date +\%F).tar.gz; docker compose -f docker-compose-release.yml start app
```
then copy the archive off the machine, encrypted, and prune old ones. A
backup you have never restored is a guess: restore one onto a spare machine now
and then. `cloudcodex/tests/integration/backup-restore.test.js` runs that drill
in CI against a live MySQL, and the Compose path was drilled by hand when the
scripts landed: both volumes destroyed, then every document, comment, image and
avatar checked after the restore.
**Without Docker.** `scripts/backup.sh --local [--uploads DIR] <file>` and
`scripts/restore.sh --local [--uploads DIR] ... <file>` do the same with the
`mysql` and `mysqldump` clients on `PATH`, connecting over TCP as `DB_USER`
with `DB_PASS` to `DB_NAME` on `DB_HOST`, and with the uploads directory on
disk (default `cloudcodex/public`). `DB_USER` must be granted on `DB_NAME`
alone, as the app's own user should be (`GRANT ALL ON c2.* TO ...`): root, a
global privilege, a grant on another database or a role is refused, because
that grant is what keeps a restore inside its own database. They need MySQL's
clients: MariaDB's `mysqldump` writes the values of generated columns
(`logs.plain_content`), which MySQL refuses on restore, so `--local` refuses
it. A MySQL server shared by several instances is backed up the same way, one
instance at a time, each with its own user.
---
## Upgrades
**Applying migrations is a deliberate step on every upgrade path, including the
published image.** Bumping `CLOUDCODEX_VERSION` and running
`docker compose up -d` starts a newer app against an older schema, and it boots
cleanly before throwing 500s on whatever column it expects and cannot find.
`init.sql` will not save you: MySQL executes `docker-entrypoint-initdb.d` only
when it initialises an **empty** data directory, so on an existing `db_data`
volume it is skipped entirely.
There **is** a migration runner, `npm run migrate`. It applies every pending
file in `migrations/` in lexicographic order and records each one in a
`schema_migrations` table (filename, sha256 checksum, elapsed ms, timestamp), so
a second run is a no-op and an applied file that has since been edited is a hard
stop rather than a silent re-apply. Where you run it from depends on the
deployment: see "Running the migrations" below.
Take a dump before starting, and read the release's CHANGELOG entry for schema
changes.
```
+------------------+ +------------------+ +------------------------+ +----------------+
| 1. pull the new |-->| 2. stop every |-->| 3. run the migration |-->| 4. start the |
| image, or | | WRITER | | in a ONE-OFF | | new image |
| git pull main | | (the app, not | | container | | |
| | | the database)| | - applies pending in | | docker |
| docker compose | | | | lexicographic order | | compose up -d |
| ... pull app | | docker compose | | - records each in | | [--build] |
+------------------+ | ... stop app | | schema_migrations | +----------------+
+------------------+ | - init.sql is for new |
| installs only |
+------------------------+
```
### Upgrading to 0.15.0 (the suite connection is removed)
0.15.0 removes the connection to the other product, so it changes what an install may set and what it
deletes. Read this before the upgrade; the full list of removed variables and routes is in
[getting-started.md](getting-started.md#environment-variables) and the CHANGELOG.
- **Unset the removed variables.** `SERVICE_TOKEN`, `SERVICE_TOKEN_USER`, `MACHINE_OIDC_AUDIENCE`,
`MACHINE_OIDC_SUBJECTS`, `WEBHOOK_URL`, `WEBHOOK_SECRET`, `WEBHOOK_WORKSPACE_ID`, and any suite or
account variable from before 0.14.0. A set one prints a warning at boot and changes nothing else.
- **Recreate an env webhook as an admin subscription first**, if you had one: in Admin, Webhooks, with
the same URL and workspace filter. Give the receiver the new secret the console shows once. Unset the
three `WEBHOOK_*` variables after that.
- **Search and browse need a session.** A script that sent the service token now signs in as a user.
- **The migration deletes data.** `2026-10-17-remove-sibling-connection.sql` expires every revoked open
invitation and every open invitation the sync made for a workspace owner, deletes the env webhook
subscriptions and their queued deliveries, and drops the `grants_admin` and `revoked_at` columns on
`user_invitations`. Take a dump first.
- **Review the Admin users list after the upgrade.** The sync made some accounts instance admins (its
workspace owners, once they signed in). Nothing records which accounts those were, so the upgrade cannot
demote them. Open Admin, Users and demote any account that should no longer be an admin.
- **Other open invitations from the membership sync stay open** until they expire (a year from their last
refresh). They now admit a plain member, never an admin. To find them (`invited_by` is the former
`SERVICE_TOKEN_USER`'s id):
```sql
SELECT id, email FROM user_invitations WHERE invited_by = <former SERVICE_TOKEN_USER id> AND accepted = FALSE;
```
Revoke the ones you do not want in Admin, Invitations.
- **The order is free.** The 0.15.0 image reads neither column, so it may start before or after the
migration runs.
- **Rolling back to 0.14.0 needs the undo first.** 0.14.0 refuses to start without the two columns.
Run this before starting the 0.14.0 image (the values are not restored):
```sql
ALTER TABLE user_invitations
ADD COLUMN grants_admin BOOLEAN NOT NULL DEFAULT FALSE,
ADD COLUMN revoked_at TIMESTAMP NULL DEFAULT NULL;
```
Removed routes (`GET /api/suite`, `GET /api/suite/links`, `GET /api/workspaces/:workspaceId/reader-check`,
`GET /api/documents/state`, `PUT /api/machine/members`, `DELETE /api/machine/members`, and
`DELETE /api/machine/members/:email`) answer `404`.
### The pinned compose network (upgrading from 0.11.0 or earlier)
Both production compose files pin their default network to `172.29.0.0/16`,
gateway `172.29.0.1`, because `TRUST_PROXY`'s default names that gateway. An
install created by an earlier file has its `<project>_default` network (the
project is the directory name, so `c2_default`) on whatever subnet Docker
chose. **For this upgrade, take the project down before the migration**, so
the network is created fresh:
```bash
docker compose -f docker-compose-release.yml pull app
docker compose -f docker-compose-release.yml down # NOT down -v: that deletes db_data and app_public
docker compose -f docker-compose-release.yml run --rm app npm run migrate
docker compose -f docker-compose-release.yml run --rm app npm run backfill:doc-images # once, from 0.11.0 or earlier
docker compose -f docker-compose-release.yml up -d
```
`down` stops and removes both containers and the old network and keeps both
volumes. The usual order below (stop only the app, then migrate) is **not
safe here**, measured on a live 0.11.0 install with Compose 5.3.1: `run`
replaces the changed network, and when the database container's own
configuration has not changed Compose reconnects it to the new network
**without its `database` alias**, so the migration fails with
`getaddrinfo ENOTFOUND database`, and the following `up -d` starts an app that
cannot reach its database at all (measured before this release's instance lock,
while the health check still answered 200; with the lock, the app exits at boot
and restarts in a loop).
With `down` first, the same install migrated, came back on `172.29.0.0/16` and
kept both volumes, and so did one whose images changed as well.
`docker network inspect c2_default` should then show `172.29.0.0/16`. If another network on the host already uses that range,
Compose cannot create this one: change the subnet and gateway in the compose
file, and put the new gateway's `/32` in `TRUST_PROXY` in place of
`172.29.0.1/32`. The same applies to `docker-compose-prod.yml`.
**Rule:** `init.sql` and `migrations/*.sql` must stay in sync. Every
column or table added in a migration also lives in `init.sql` so a fresh
install converges to the same schema. New migrations are additive, no
file in `migrations/` is ever rewritten after it ships. The runner enforces
that second half: it checksums every file when it applies it, and refuses to
run again if one of them has changed since.
### Stop every writer first
**Order: pull, stop every writer, apply the migrations, start the new image.**
Not the other way round. The runner does not stop anything for you.
A migration that adds a `NOT NULL` column with no `DEFAULT` makes the schema
incompatible with the application in *both* directions, and no compose file
overrides `sql_mode`, so MySQL 8's default `STRICT_TRANS_TABLES` applies:
- **Old code against the new schema** fails every insert that omits the column
(error 1364).
- **New code against the old schema** fails every insert that names it
(error 1054).
Either way the affected endpoints 500 for real users for as long as the window
is open, and 500s are not always harmless: `2026-09-08-token-purpose.sql` would,
mid-window, make `POST /api/forgot-password` fail for an address that exists
while still answering 200 for one that does not, which is an account enumeration
oracle the code goes out of its way to close.
"Every writer", not "the app container": `docker-compose.yaml` (dev) defines a
single service, `database`. There is **no app container in dev**: the app runs
on the host under `npm run dev`, and that is the writer to stop. The
single-process architecture already makes a restart a brief total outage, so a
planned one costs nothing extra.
Stopping the writers also closes a partial-failure race for any migration that
deletes rows and then tightens the column: a row inserted in between makes the
tightening `ALTER` fail (each migration header names the exact error it would
raise), and MySQL implicitly commits DDL, so the table is left half-migrated.
The runner records nothing for a file that failed, so `schema_migrations` will
not paper over it, but nothing undoes the DDL either. For
`2026-09-08-token-purpose.sql` the error is 1265, `Data truncated for column
'purpose'`, because the `CHECK` in the same `ALTER` forces the table-copy path;
a bare `MODIFY` would report 1138 instead. Recovery is the same either way:
drop the column and re-apply with the writers down.
**Assume no rollback.** Reverting the application after applying a migration
lands you in old-code-against-new-schema. Getting back means undoing the DDL by
hand; each migration header says what that is.
`schema_migrations` is runner-owned bookkeeping and is deliberately **not** in
`init.sql`. If a fresh install arrived with the table already present and empty,
the runner would read "nothing applied" and try to replay every shipped delta
against the schema those deltas are already folded into.
### Running the migrations
**Where you run it depends on which compose file you deploy with**, because the
runner needs three things at once: the `migrations/` directory, the app's Node
dependencies, and a reachable MySQL.
| Deployment | Command | Why |
|---|---|---|
| `docker-compose-release.yml` (published image) | `docker compose -f docker-compose-release.yml run --rm app npm run migrate` | 3306 is **not** published to the host, so the runner has to be inside the compose network. `run` builds a one-off container from the **current** compose file and the **new** image, so it has both `scripts/migrate.js` and the `./migrations` mount, and it removes itself afterwards. |
| `docker-compose-prod.yml` (built from source) | `docker compose -f docker-compose-prod.yml run --rm app npm run migrate` | Same shape, after `docker compose -f docker-compose-prod.yml build app`. 3306 *is* published here, on 127.0.0.1 unless `DB_BIND` says otherwise, so `cd cloudcodex && npm run migrate` on the host also works if you have run `npm install` there. |
| `docker-compose.yaml` (dev) | `cd cloudcodex && npm run migrate` | Dev has **no app container**: the writer to stop is `npm run dev` on the host. MySQL publishes 3306 on 127.0.0.1 and `node_modules` is installed, so the runner just runs there. |
**`run --rm`, not `exec`.** This matters most on the one upgrade every existing
operator performs: the one that installs the runner. `exec` runs inside the
container that is **already running**, which at that moment is the old one, and
it fails twice over:
- the running image is the previous `CLOUDCODEX_VERSION`, which predates
`scripts/migrate.js` and the `migrate` script, so `exec` gets
`npm error Missing script: "migrate"`;
- and even with the new image pulled, the running container was **created**
before `./migrations:/migrations` existed in the compose file. `docker compose
pull` does not recreate a container, so `/migrations` is simply absent inside
it.
`run` has neither problem, because it creates a container from the compose file
and image you have right now, and it does not disturb the running app. From the
second upgrade onward `exec` would work, which is exactly why the broken
instruction reads fine.
The runner reads `DB_HOST`, `DB_USER`, `DB_PASS` and `DB_NAME` from the
environment, the same variables the app uses. Inside the container `DB_HOST` is
already `database`; on the host it comes from `.env`. There is no `DB_PORT`:
the runner uses 3306, matching `mysql_connect.js`.
On an **SELinux** host (Fedora, RHEL, CentOS, Rocky) the `migrations/` bind
mount carries `:ro,z` in both compose files. Without a relabel flag the
directory keeps its host label and is unreadable inside the container, and the
runner would fail on a directory that is plainly there. `z` (shared) rather than
`Z` (private), because more than one container reads it.
Earlier releases of this document told you to run
`source /var/lib/mysql/migrations/<file>.sql` inside `make db-shell`. That never
worked: no compose file mounts `migrations/` into the **MySQL** container
(`docker-compose.yaml`, `docker-compose-prod.yml`, `docker-compose-release.yml`
and `docker-compose.linux.yml` mount only the data directory and `init.sql`), so
that path does not exist there. The mount added for the runner is on the **app**
service, which is where the runner runs.
### First run: record a starting point, once
The runner refuses to guess. Against a database with no `schema_migrations`
table it stops with instructions rather than applying anything, because without
bookkeeping it cannot tell a fully-migrated database from a partly-migrated one,
and either guess is destructive. Which command you run once depends on where the
schema came from, and **they are not interchangeable**:
| Situation | Command | What it records |
|---|---|---|
| **Upgrading** a database that already existed before this release. This is the usual case, and it includes every install that predates the runner | `npm run migrate -- --baseline` | Only the **thirteen** files that shipped before the runner existed. Anything added since is genuinely missing from that database and stays **pending**, to be applied for real by the next ordinary run. |
| A database `init.sql` **built minutes ago** and that has never been upgraded | `npm run migrate -- --adopt-fresh-install` | **Every** file in `migrations/`, without running any of them. Correct only because `init.sql` is kept in sync with all of them, so the schema already has every change. |
Both record checksums and apply nothing. After either, `npm run migrate` is the
only command you need, from then on.
**Upgrading a 0.9.0 install is the one exception.** Two of the thirteen,
`widen_log_content.sql` and `drop_squad_permissions.sql`, shipped after 0.9.0 was
tagged, so a database that has only ever run 0.9.0 does not have them, and
`--baseline` records them anyway. Apply both by hand after `--baseline`; they
are idempotent. The exact commands are in the 0.10.0 entry of
[`CHANGELOG.md`](../CHANGELOG.md), under Migration.
**If you are not sure, it is not a fresh install.** "Brand-new install" means the
database, not the release: an install you are upgrading to a new version is an
existing database, however new the image is.
The thirteen are a closed list hardcoded in `scripts/migrate.js`
(`LEGACY_BASELINE`), deliberately not a scan of the directory. If `--baseline`
swept the directory, an operator upgrading across a release that adds a
fourteenth migration would mark it applied without running it, and the app would
then run against a schema that never got the change while the runner reported
success.
`--adopt-fresh-install` is the only mode that adopts everything, so it is the
one that can do that damage on purpose, and it is guarded twice:
- it **refuses** once `schema_migrations` holds a single row, which keeps it away
from a tracked install;
- and for every file that postdates the thirteen, it **checks the live schema
first**, through `information_schema`, and refuses unless the table or column
that file adds is already there. Empty bookkeeping proves nothing (an install
that predates the runner has none **by definition**), so the guard is that
positive check rather than the absence of rows. It prints the exact list of
files it is adopting before it adopts them.
The runner never bootstraps a schema, and refuses outright against a database
with no `users` table. Run `init.sql` first (a fresh Docker volume does this for
you).
So, end to end on a new published-image install:
```bash
docker compose -f docker-compose-release.yml up -d # init.sql builds the schema
docker compose -f docker-compose-release.yml run --rm app \
npm run migrate -- --adopt-fresh-install
```
Nothing is applied there, so the app can stay up for that one. Until it runs,
`/readyz` answers `503 {"ready":false,"reason":"migrations"}` and Docker
reports the container `unhealthy`, because a database with no bookkeeping
cannot be shown to be migrated; the app serves requests either way. Every
upgrade after that is the four-step order above (the upgrade from 0.11.0 or
earlier takes the project `down` once instead of `stop app`, and runs the
document-images backfill before the start; see
[The pinned compose network](#the-pinned-compose-network-upgrading-from-0110-or-earlier)):
```bash
docker compose -f docker-compose-release.yml pull app # 1. new image
docker compose -f docker-compose-release.yml stop app # 2. stop the writer
docker compose -f docker-compose-release.yml run --rm app \
npm run migrate # 3. migrate
docker compose -f docker-compose-release.yml up -d # 4. start it again
```
`run --rm` rather than `exec` on the upgrade path: `exec` runs inside the
container that is **already** running, which on the upgrade that first installs
the runner is the old image, and that image has neither the `migrate` script nor
the `migrations/` mount. `run` builds a one-off container from the updated
compose definition, so it has both.
### Concurrency
The runner takes a MySQL advisory lock for the duration of a run and waits up to
10 seconds for it. A second run started while the first is working refuses with
a clear message instead of racing it.
The lock name is `cloudcodex_migrate:<database>`, built server-side with
`GET_LOCK(CONCAT('cloudcodex_migrate:', DATABASE()), 10)`. `GET_LOCK` names are
scoped to the MySQL **server**, not to a database, so an unqualified name would
make two Cloud Codex schemas on one server serialise against each other while
the loser was told the contention was against its own database. MySQL caps a
lock name at 64 characters, so for a schema name longer than 45 the name is
`cloudcodex_migrate#` and the first 40 hex characters of the schema's SHA-256
instead; every shorter name is unchanged. The runner also
distinguishes MySQL's two negative answers: `0` is "someone else holds it" and
`NULL` is "the attempt itself errored", and they get different messages.
### If a migration fails
**MySQL implicitly commits DDL.** The runner opens a transaction per file, which
covers the DML inside it, but no `ROLLBACK` can undo a `CREATE`, `ALTER` or
`DROP` that has already run. A file that fails halfway therefore leaves the
database **partially migrated**, and the runner says exactly that rather than
promising a rollback it cannot deliver. It stops at the failing file and does
not continue to the next one. Recovery is to inspect the schema, or restore the
dump you took before starting, and retry. This is why that dump is not optional.
**One failure means the opposite of that.** If MySQL reports `ER_DUP_FIELDNAME`,
`ER_TABLE_EXISTS_ERROR` or `ER_DUP_KEYNAME`, the object the file adds is already
present and the run may have changed nothing at all. The usual cause is a
starting point recorded with the wrong command: a database `init.sql` built
contains every migration already, so `--baseline` leaves the newer files pending
and the next ordinary run dies on their first `ALTER`, on an install minutes old
with no dump to restore. The runner says so, and prints the `INSERT INTO
schema_migrations` that records the file as applied by hand once you have
confirmed the schema really has the change.
**A refusal is a third case.** Some migrations check the data before they
change the schema and refuse when it would break the change, for example
`2026-09-25-oauth-one-link-per-provider.sql` on a user linked twice to one
provider. The runner then reports `Migration <file> refused to run:` followed
by the file's own message, which names what to look for; it records nothing,
drops the throwaway guard procedure the file created, and the file stays
pending. Resolve what the message names (the file's header and the CHANGELOG
say how), then run `npm run migrate` again. Such a file needs the runner: the
`mysql` client cannot run it, because it splits the guard's body on its
semicolons.
### The document-images backfill, once
`2026-09-27-who-may-see-doc-images.sql` creates `doc_images`, the table that says which
documents hold which image, and the `/doc-images` handler now serves an image
only to its uploader and to readers of a document named there. The table starts
empty, so **on an install that already has documents with images, every image
is hidden from its readers until the backfill runs.** Run it once, after the
migration and before starting the new image, the same way as the runner:
| Deployment | Command |
|---|---|
| `docker-compose-release.yml` | `docker compose -f docker-compose-release.yml run --rm app npm run backfill:doc-images` |
| `docker-compose-prod.yml` | `docker compose -f docker-compose-prod.yml run --rm app npm run backfill:doc-images` |
| `docker-compose.yaml` (dev) | `cd cloudcodex && npm run backfill:doc-images` |
It records a row for every `/doc-images/` image in `logs.html_content` and
`versions.html_content` and prints `backfill-doc-images: recorded N image
reference(s) from D document(s) and V version(s)`. It trusts every reference
already stored, which is why it belongs before the app starts: run later, it
also trusts whatever was saved in between, including references pasted by
people who cannot see the image. So it refuses to run over a `doc_images` table
that already has rows. If an interrupted first run needs finishing before
go-live, add `--again` (`npm run backfill:doc-images -- --again`, or the same
after `run --rm app`); it is idempotent, so the rerun records only what was
missed. Do not rerun it after go-live. If the app has to start first, set
`DOC_IMAGES_PUBLIC=1` (images served to anyone with the address, as before),
run the backfill (it runs without `--again` while that is set), then unset it
and restart. A fresh install needs none of this.
---
## Several instances on one MySQL server
The compose files give each install a MySQL server of its own, with one
account (`MYSQL_USER`) that holds every privilege on `DB_NAME`. That is right
for one instance on its own server. To run several instances against **one**
server, give each its own schema and two accounts of its own: an app account
that can read and write rows in that schema and nothing more, and a migration
account that can change that schema's tables and nothing more. Neither account
can read, write, list (in `SHOW DATABASES`) or grant anything in another
instance's schema. Each can still learn the other schemas' **names**, and
their tables' names, as the last part of this section explains.
**Name the schema with lowercase letters and digits only, and make the name
opaque.** In a database-level `GRANT`, MySQL reads `_` and `%` in the schema
name as wildcards (unless the server runs with `partial_revokes`, which is off
by default), so a grant on `c2_acme` would also cover `c2xacme`, or any schema
whose name differs only where the underscore is. A name with neither character
means the same thing whatever the server's settings. Since every account on
the server can read the list of schema names, do not name a schema after the
customer: use something like `c2` followed by `openssl rand -hex 6`
(`c2a41f09c7d3e2`). The readable `c2acme` below stands in for that name.
Account names are not patterns, so they may carry underscores. MySQL limits an
account's user name to 32 characters, so with `_app` or `_mig` appended the
schema name can be at most 28 characters.
```sql
-- As root, once per instance. Put the instance's schema name where c2acme is,
-- and a generated password (openssl rand -hex 24) in place of each <...>.
CREATE DATABASE c2acme;
CREATE USER 'c2acme_app'@'%' IDENTIFIED BY '<app password>' WITH MAX_USER_CONNECTIONS 15;
GRANT SELECT, INSERT, UPDATE, DELETE ON `c2acme`.* TO 'c2acme_app'@'%';
CREATE USER 'c2acme_mig'@'%' IDENTIFIED BY '<migration password>' WITH MAX_USER_CONNECTIONS 3;
GRANT ALL PRIVILEGES ON `c2acme`.* TO 'c2acme_mig'@'%';
```
`'%'` lets each account sign in from any address that reaches the server, so a
leaked password works from anywhere with a route to MySQL. Where you know the
address the instance connects from, put it in place of `%` in all four
statements (`'c2acme_app'@'10.0.1.23'`, or a subnet such as `'10.0.1.%'`), and
the account works from there alone.
| Account | Holds | Used by |
|---|---|---|
| `c2acme_app` | `SELECT`, `INSERT`, `UPDATE`, `DELETE` on `c2acme`; at most 15 connections | the running app: its `.env` sets `DB_HOST` to the shared server, `DB_USER=c2acme_app`, `DB_PASS` and `DB_NAME=c2acme` |
| `c2acme_mig` | every privilege on `c2acme`, without `GRANT OPTION`; at most 3 connections | building the schema and `npm run migrate`, and nothing else. Its password never goes in the app's `.env` |
Neither account holds anything global: no `PROCESS`, `FILE`, `SUPER`,
`CREATE USER` and no `*.*` grant, so neither can list other sessions, read or
write the server's files, change its settings or make accounts. The app account
has no DDL even on its own schema, and the app needs none: every query it
issues, at boot and after, is a `SELECT`, `INSERT`, `UPDATE` or `DELETE`, and
the instance lock (`GET_LOCK`) needs no privilege.
**Point the compose file at the shared server.** Both production compose files
start their own `database` service and set the app's `DB_HOST` to it, which
wins over `.env`. An instance on a shared server needs neither, so save this
override next to the compose file as `shared-mysql.yml`:
```yaml
# The app's MySQL is a shared server, named by DB_HOST in .env.
services:
database:
profiles: [bundled-database] # never started unless asked for by name
app:
depends_on: !reset {} # Compose 2.24 or later
environment:
DB_HOST: ${DB_HOST:?set DB_HOST in .env to the shared MySQL server}
```
and pass both files to **every** `docker compose` command for that instance,
the ones under [Upgrades](#upgrades) included:
`docker compose -f docker-compose-release.yml -f shared-mysql.yml up -d` (the
override works the same over `docker-compose-prod.yml`).
Leave `MYSQL_ROOT_PASSWORD` out of this instance's `.env`. A command that
forgets the override then fails at the bundled database ("Database is
uninitialized and password option is not specified") instead of starting a
new, empty MySQL beside the shared one. It still stops the app: run it again
with both files, then remove the stray container with
`docker compose -f docker-compose-release.yml -f shared-mysql.yml rm -sf database`.
The compose files name their containers (`cloudcodex-app`), so they run one
instance per Docker host; the instances that share a MySQL server run on hosts
of their own.
**Build the schema as the migration account**, from a host with a `mysql`
client that reaches the server, then record it as a fresh install with the
runner, also as the migration account. The runner reads `DB_USER`, `DB_PASS`
and `DB_NAME` like the app, so give it the migration account's in place of the
app's:
```bash
mysql -h <mysql host> -u c2acme_mig -p c2acme < init.sql
read -rs DB_PASS && export DB_PASS # the migration password
docker compose -f docker-compose-release.yml -f shared-mysql.yml run --rm \
-e DB_USER=c2acme_mig -e DB_PASS app npm run migrate -- --adopt-fresh-install
```
Every later [upgrade](#upgrades) runs `npm run migrate` the same way, as the
migration account. A migration that created a trigger or a stored function
would also need `log_bin_trust_function_creators` on a server that keeps a
binary log (MySQL 8.4's default); none does, and the integration tests run every
migration file as this account, so one that did would fail there first.
**Connections.** The cap on the app account is what stops one instance taking
every connection the server has. 15 covers the pool (`DB_POOL_SIZE`, 10 by
default), the instance lock's own connection and room to reconnect; if you
raise `DB_POOL_SIZE`, raise the cap to at least `DB_POOL_SIZE` + 5
(`ALTER USER 'c2acme_app'@'%' WITH MAX_USER_CONNECTIONS 25;`). The server's
`max_connections` (151 by default) must cover every instance's two caps, 18 at
these values, plus what you keep for yourself: eight instances take 144 and
leave 7 (MySQL holds one more back for `root`), and a ninth needs
`max_connections` raised.
**What this does not protect against.** Anyone holding the server's root
password reaches every schema. Every account on the server, these included,
can read `information_schema.TABLESPACES_EXTENSIONS`, which MySQL lists without
any privilege check: it names every schema on the server and each of its tables
(`c2a41f09c7d3e2/users`). That is the one system view that names another
instance, and the reason for opaque schema names; the rows themselves stay out
of reach. Server-wide counters (`SHOW GLOBAL STATUS`) are readable too, so one
instance can see how busy the server as a whole is. Each instance also needs
its own `app_public` volume, because avatars and document images live on disk,
outside the grant. And the instances share the server's CPU, memory and disk,
so one heavy instance slows the others. Section 9 of
[`docs/maps/access-control.md`](maps/access-control.md) sets out where the
boundary between instances is and where it is not.
`cloudcodex/tests/integration/tenancy.test.js` runs the SQL block above, as it
is written here, for two instances on one server, checks that every
cross-schema statement it knows fails with MySQL's privilege error, and reads
every system view the app account can, to show that `TABLESPACES_EXTENSIONS`
is the only one naming the other instance; `grants-sufficient.test.js` runs the
app on the app account alone. The record of the run is in
[`docs/research/instance-isolation-2026-09-28/`](research/instance-isolation-2026-09-28/).
---
## Logs
Cloud Codex writes to **stdout/stderr only** — no logging library, no
file rotation. Capture with whatever your container runtime provides
(`docker logs`, journald, your cloud's log drain). The project format
for error lines is:
```
[2026-04-29T17:14:21.000Z] POST /api/save-document: <error message>
```
Anywhere `console.error` is used, this prefix is the convention. Don't
introduce a structured-logging library without discussing first.
---
## Background work
There are three scheduled tasks running inside the Node process, all `setInterval`s
in `server.js`, with no separate worker process or cron container:
- the outbound webhook delivery worker, which sends queued events every two
seconds (`services/webhook-worker.js`). An install with no webhook
subscription runs one short query per tick and sends nothing;
- the `activity_log` daily prune (rows older than 365 days);
- the webhook prune, which deletes finished deliveries and unreferenced events
older than 30 days, once a day.
If you ever need a true background worker, prefer adding a deliberate
single-process scheduler (BullMQ-like) over splitting into a second
container. The single-process story is load-bearing for self-hosting.
---
## Static asset caching
| Path | Cache headers |
|-----------------------|-----------------------------|
| `/avatars/*` | `max-age=604800, immutable` (7 days) |
| `/doc-images/*` | `private, max-age=86400` (1 day); a refusal is `no-store`. With `DOC_IMAGES_PUBLIC=1`, `public, max-age=2592000, immutable` (30 days) |
| Vite-built assets | hashed filenames + long max-age (Vite default) |
Avatars and doc images are content-addressed by SHA, so a new upload gets a new
URL and long cache windows are safe. Document images are `private` because they
are served per user: a reverse proxy or CDN in front of the app must not cache
`/doc-images/`, and `private` tells a standards-following one not to.
---
## Rate limiters
`express-rate-limit` runs in-process and resets when the container does.
For a multi-replica deployment you'd need to switch to a shared store —
but the single-process architecture is the recommended topology, so
this is not a typical concern.
| Scope | Limit |
|----------------------|---------------------------------|
| Auth endpoints | 20 / 15 minutes per IP |
| User search | 60 / 15 minutes per IP |
| WebSocket messages | 60 / second per connection |
The limiters count per client address. Express takes it from the connection,
and from `X-Forwarded-For` only while each hop it walks through, from the
right, starting with the peer that connected, is a proxy `TRUST_PROXY` names.
So `TRUST_PROXY` lists the proxies **by address**: never by count, and not by
range, since a range believes every client inside it too.
- **Unset** (the default) is `127.0.0.1/32, ::1/128, 172.29.0.1/32`: a proxy
on this host in front of either production compose file. Directly, it
connects from loopback; through the loopback publish, Docker presents it to
the container as the pinned network's gateway, `172.29.0.1`. Anyone else is
counted by the address that connected, whatever `X-Forwarded-For` it sends: a
public client, a machine on the same LAN, VPN or VPC (AWS's default VPC is
`172.31.0.0/16`), and another container on the compose network. The
default bridge's gateway, `172.17.0.1`, is deliberately not in it; see
`docker run` below. **An install run without Docker** (`npm run start` on the
host) sets `TRUST_PROXY=127.0.0.1/32, ::1/128`: `172.29.0.1` means nothing
there, and on a LAN that uses `172.29.0.0/x` it may be a real router.
- **A list replaces the default**, so keep the entries you still need and add
your proxy. The setups the default does not cover:
- *A proxy running as another container on the compose network.* Pin its
address and list that `/32` with loopback, and **not** the gateway. Docker
hands out addresses from the start of the range, so pick one near the end.
In the proxy's own compose file, with this project's network (named after
its directory: `c2_default` for a checkout in `c2/`):
```yaml
services:
caddy:
image: caddy:2
ports:
- "0.0.0.0:80:80"
- "0.0.0.0:443:443"
networks:
codex:
ipv4_address: 172.29.255.10
networks:
codex:
name: c2_default
external: true
```
```dotenv
TRUST_PROXY=127.0.0.1/32, ::1/128, 172.29.255.10/32
```
Publish the proxy's ports on explicit IPv4 addresses, as above, or give the
pinned network IPv6 (`enable_ipv6: true`, and a ULA subnet such as
`fd29::/64` beside the IPv4 one in its `ipam` config). A plain `"443:443"`
also publishes on `[::]`, and over an IPv4-only network every IPv6 client
reaches the proxy as `172.29.0.1`: all of them then share one bucket, and
had the gateway been listed, a proxy that appends would let each choose its
own address.
With IPv6 on the network, pin the proxy's IPv6 address too and list it:
`app` then resolves to an AAAA record as well as an A record, and nginx or
Caddy sends some requests from the proxy's IPv6 address, which an
IPv4-only list does not name. In this project's compose file:
```yaml
networks:
default:
enable_ipv6: true
ipam:
config:
- subnet: 172.29.0.0/16
gateway: 172.29.0.1
- subnet: fd29::/64
```
and in the proxy's, beside its `ipv4_address`:
```yaml
networks:
codex:
ipv4_address: 172.29.255.10
ipv6_address: fd29::ff10
```
```dotenv
TRUST_PROXY=127.0.0.1/32, ::1/128, 172.29.255.10/32, fd29::ff10/128
```
- *A load balancer with a private address*, reaching nginx on the host or,
with `APP_BIND` widened, the app itself. List the subnets the load balancer
runs in, beside the defaults:
```dotenv
TRUST_PROXY=127.0.0.1/32, ::1/128, 172.29.0.1/32, 10.0.1.0/24, 10.0.2.0/24
```
Every address in a listed subnet is believed, so give the load balancer
**dedicated subnets** that nothing else runs in, and a security group that
lets only the load balancer reach the port. With nginx between the load
balancer and the app, nginx appends (`$proxy_add_x_forwarded_for`);
overwriting would make every client the load balancer. **A layer-4 load
balancer that sets no `X-Forwarded-For` (an AWS Network Load Balancer, for
one) must not be listed at all**: it passes each client's own header
through, so trusting it lets every client choose its address.
- *The image run with `docker run` behind a proxy on the host.* Publish on
loopback and add the default bridge's gateway:
```bash
docker run -d -p 127.0.0.1:3000:3000 --env-file .env ghcr.io/cloud-city-computing/cloud-codex:<version>
```
```dotenv
TRUST_PROXY=127.0.0.1/32, ::1/128, 172.17.0.1/32
```
Only with that loopback publish. `-p 3000:3000` publishes on `[::]` too,
the default bridge is IPv4-only, and every IPv6 client then arrives as
`172.17.0.1`: with it listed, each one chooses its own address with no
proxy involved. **An IPv4-only Docker network presents every IPv6 client of
an all-interfaces or IPv6 publish as its gateway.** Without the entry, a
bare `docker run` behind a proxy on the host puts every client in one
bucket, and the warning below names `172.17.0.1`.
- Subnet names (`loopback`, `linklocal`, `uniquelocal`) are still accepted,
but each believes every client in its range, which is the gap the default
closed: behind a proxy that appends, a client in a trusted range names its
own address with the left entry.
- **`false`** believes no proxy; every request is counted by the address that
connected. Right with nothing in front of the app.
- **A hop count (`1`, `2`, ...) or `true` stops the server at boot.** Either
believes the `X-Forwarded-For` of whoever connects, so any client that can
reach the app's port directly sends a new address with every request and
gets a fresh bucket each time: unlimited password and two-factor guessing
(GHSA-9fmx-frrf-xxmq). If the port truly is reachable only through the proxy
and its address cannot be known, `TRUST_PROXY_ALLOW_HOP_COUNT=true` accepts a
hop count anyway, and with it that risk.
- **A range past the width thresholds stops the server at boot too**, naming
the entry: wider than an IPv4 /8 (`0.0.0.0/1,128.0.0.0/1` is every
address), wider than an IPv6 /16 outside `fc00::/7` and `fe80::/10`, or
holding more than an IPv4 /8 of the IPv4-mapped block `::ffff:0:0/96` (all of
it is every IPv4 client, which Express matches in that form). Those are the
thresholds, not "anything public": a public /8, or an IPv6 /16 to /31, is
accepted and believes everyone in it, so list only what your proxy uses.
`TRUST_PROXY_ALLOW_HOP_COUNT=true` accepts an over-wide range, with the same
risk as a hop count.
- **Every entry is a subnet name or an address in standard notation**, with an
optional prefix length or IPv4 netmask, or the server stops at boot even
with the opt-in. Express's parser accepts more, and reads it in ways nobody
expects: `0/1` is half of IPv4, and `010.0.0.0/8` is octal, `8.0.0.0/8`,
which is public space.
**One limit the default cannot remove.** Every process on the host that
reaches the published port, and every container sharing the host's network
namespace, arrives as the gateway, so it is trusted and can name its own
address. With `APP_BIND` widened, so does every container on the host (see
"TLS and reverse proxy"), and on a runtime that carries all published traffic
through the userland proxy, so does every client, which is why a widened bind
with nothing in front sets `TRUST_PROXY=false`. The loopback bind decides who
can reach the port, not whether they are believed once they do.
**A proxy the list leaves out is logged, not silent.** When a peer
`TRUST_PROXY` does not name sends `X-Forwarded-For`, the server logs one line,
once per address (an IPv6 peer once per /64) and for at most 32 of them per
process:
```
⚠ <address> sent X-Forwarded-For, but TRUST_PROXY does not name <address> as a proxy, ...
```
If that address is your proxy, every client behind it is sharing one bucket:
add it. If it is a client talking to the app directly, the header was ignored,
as it should be.
A value Express cannot parse stops the server at boot too.
**If you run behind a proxy, check after upgrading** that the app keys each
client on the client's own address. First, the line the server prints once it
is listening, which says what it trusts and whether that is the default:
```bash
docker compose -f docker-compose-release.yml logs app | grep 'Trusting proxies'
# ✔ Trusting proxies (the default): 127.0.0.1/32, ::1/128, 172.29.0.1/32
```
`(from TRUST_PROXY)` means your value was used. Make sure the list names the
address your proxy connects from.
Second, **from a machine outside**, through the proxy's public address, send
two failed sign-ins that each claim a different address, and compare how many
attempts the app says are left:
```bash
for claimed in 203.0.113.98 203.0.113.99; do
curl -s -o /dev/null -D - -H "X-Forwarded-For: $claimed" -H 'Content-Type: application/json' \
-d '{"username":"nobody","password":"x"}' https://codex.example.com/api/login \
| grep -i '^ratelimit-remaining'
done
```
- **One lower the second time** (say 19, then 18): both requests shared a
bucket, so the claimed addresses were ignored. Go on to the third check.
- **The same number twice**: each claimed address got its own bucket, so any
client can pick its key. The proxy passes the client's header through
(it sets none), or it appends and the address it appends is one
`TRUST_PROXY` trusts (an IPv6 client reaching a proxy container as the
gateway, above). Fix that first.
Third, sign in through the proxy in a browser and read the address the app
recorded for that session:
```bash
docker compose -f docker-compose-release.yml exec database \
sh -c 'mysql -u root -p"$MYSQL_ROOT_PASSWORD" "$MYSQL_DATABASE" -e \
"SELECT ip_address, last_active_at FROM sessions ORDER BY last_active_at DESC LIMIT 3"'
```
- **Your own address**: right.
- **The proxy's address**: the proxy is not trusted, so every user shares its
one bucket of 20 sign-in attempts per 15 minutes. List it in `TRUST_PROXY`
(the warning above names it too).
- **The Docker network's gateway** (`::ffff:172.29.0.1`): a proxy on the host
is trusted but hands on no client address, so every user shares the
gateway's bucket. If the second check passed, the proxy clears the header
without setting it; if it failed, the proxy sets none and your browser simply
sent none, which is the fatal case above.
---
## Health checks
Two unauthenticated endpoints, outside `/api` and ahead of every rate limiter.
Neither says anything about the install: no version, no counts, no names.
| Endpoint | Answers | Use it for |
|---|---|---|
| `GET /healthz` | `200 {"ok":true}` whenever the process is serving HTTP. It touches nothing, not even the database. | **Liveness.** Restart the container only when this fails. |
| `GET /readyz` | `200 {"ready":true}`, or `503 {"ready":false,"reason":"..."}` | **Readiness.** Route traffic to the instance, and call a deploy done, only on 200. |
The `reason` is one of four words, checked in this order:
| `reason` | Means | What to do |
|---|---|---|
| `shutting_down` | a stop signal arrived and the app is flushing and closing | nothing: it exits within ten seconds |
| `lock` | this process does not hold the instance lock (see [One process per database](#one-process-per-database)) | read the log: MySQL restarted and the lock is being taken back (it retries every second), or another process took it, in which case this one stops |
| `database` | `SELECT 1` failed or took longer than two seconds | check MySQL and the network |
| `migrations` | a file in `migrations/` has not been applied, the database has never been adopted, or `migrations/` is not mounted | run `npm run migrate` (see [Upgrades](#upgrades)); on a brand-new install, the one-time `--adopt-fresh-install` |
The published image carries a Docker `HEALTHCHECK` on `/readyz` (every 10 s,
3 s timeout, 20 s start period, 3 retries), so `docker ps` and
`docker inspect --format '{{.State.Health.Status}}' cloudcodex-app` show
`healthy` once the app is ready. Plain Compose does not restart an unhealthy
container; the status is for you and for whatever sits in front.
```bash
curl -fsS http://localhost:3000/readyz # exits non-zero on 503
```
Answering `/readyz` costs at most one `SELECT 1` and one read of
`schema_migrations` at a time, however many probes arrive together, but it is
still a database call anyone who can reach the port can make. If your proxy
forwards every path, consider not exposing `/healthz` and `/readyz` publicly;
your supervisor reaches the container port directly.
**Do not point a liveness probe at `/readyz`.** A database outage would then
restart a perfectly healthy app in a loop. Earlier releases of this document
suggested `GET /api/oauth/providers`, which reads no database and so proved
only that the process listened; use `/healthz` for that.
### Stopping cleanly
The image runs `node server.js` directly, so `docker stop` (SIGTERM) and
Ctrl-C (SIGINT) reach the app. It then answers `/readyz` with
`shutting_down`, stops taking connections, writes every open document's
not-yet-saved collaborative edits to the database, closes both WebSockets with
code 1001 (editors reconnect to the next process on their own), releases the
instance lock and the database pool, logs `stopped cleanly on SIGTERM` and
exits 0. It gives itself ten seconds and exits 1 past that. If a document's
final write fails, the last line is `stopped on SIGTERM with 1 document not
saved` instead (and the line before it names the document), so look for
`cleanly`, not just `stopped`. A stop signal that arrives during startup, before
the app is listening, ends it at once, and a second Ctrl-C does the same
mid-shutdown. Both production compose files set `stop_grace_period: 20s` so
Docker waits for it; if you run the image some other way, give it more than ten
seconds before a SIGKILL.
A SIGKILL, a crash or a power cut still loses up to the last three seconds of
live edits for documents nobody had saved, as before.
### One process per database
A Cloud Codex process keeps every open document's live state in memory, so
two processes on the same database would each keep their own copy and
overwrite each other's. At boot the app therefore takes a MySQL lock named for
its schema (`cloudcodex-instance:<database>`, or a SHA-256 digest of a schema
name longer than 44 characters, since MySQL caps lock names at 64) and holds it
on a connection of its own until it stops. A second process pointed at the same
database refuses to start and says which MySQL connection holds the lock:
```
✖ Another Cloud Codex process (MySQL connection 8) already serves this database.
Two processes would hold two different copies of every open document. Stop the other one,
or set C2_INSTANCE_LOCK=0 if you know exactly why you need both.
```
To find it, `SELECT * FROM performance_schema.processlist WHERE ID = 8;` on the
MySQL server. The lock goes when its process does, a `kill -9` included, so
there is nothing to clean up. Instances on **different** schemas of one MySQL
server never contend, and `npm run migrate` uses a lock of its own, so the
one-off migration container never collides with the running app.
If MySQL restarts under a running app, the lock goes with the connection:
`/readyz` answers `lock` while the app tries to take it back on a new
connection, once a second. If another process got it first (a duplicate that
was being refused and reconnected sooner), the app that lost it logs
`another process took the instance lock while this one had lost it`, stops
through the normal shutdown and exits 1; under `restart: unless-stopped` it
then comes back as an ordinary refusal naming the new holder, and you stop
whichever one you did not mean to run. `C2_INSTANCE_LOCK=0` turns the lock off entirely; it is an
escape for an operator who knows why, not a way to run replicas.