CHANGELOG.md

Rama main · réplica de lectura del repositorio original.

# Changelog

All notable changes to the NineDeploy project are documented in this file.

The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).

---

## [Unreleased]

### Added

- A separate manual macOS workflow records the complete configured server unit suite and its file/config manifest, alongside controlled CLI and workspace MCP checks (round 100).
- Manual macOS server audit workflow records native filesystem, SQLite and controlled Docker CLI regressions without starting a daemon or desktop GUI (round 89).
- **A read-only import plan for Coolify, Dokploy and CapRover.** `ninedeploy import plan <coolify|dokploy|caprover> --url …` reads the other platform's API from your machine and lists what NineDeploy would create:
  - services, with their repository, image or compose source, build pack, port, limits, domains and first volume;
  - environment variable names;
  - databases, with the matching engine.
  - Every option NineDeploy cannot carry, or that needs a person, is listed per item.
  - It creates nothing and changes nothing on the source. Applying a plan (step 2) and moving data (step 3) come later.
  - The credential comes from `NINEDEPLOY_IMPORT_TOKEN` / `NINEDEPLOY_IMPORT_PASSWORD` or a hidden prompt, never from the command line, and stays on your machine.
  - A plan holds variable names only, never a value, password, token or key, and every URL loses its `user:password@`.
  - The mappings were written against Coolify v4.7.0, Dokploy v0.31.0 and CapRover v1.16.0 and are tested with fixtures derived from their source. CapRover v1.16.0 was also checked against a live instance. That run found three gaps the fixtures missed, now fixed: a never-deployed app's placeholder image, stopped apps, and the source version. Coolify and Dokploy have not been checked live yet. See [docs/IMPORTING.md](docs/IMPORTING.md).
- **`ninedeploy import apply`: create what an import plan describes.** `import apply plan.json` shows what it would create; `--yes` creates it on the panel the CLI is signed in to. It reads the saved plan only, never the source.
  - It creates each service that has an image or a repository, with its port, replicas, limits and first volume. Databases are created empty, on the source's version when the source ran the same image.
  - It never deploys and never adds a domain. Adding a domain requests a certificate, and with the Cloudflare integration on it also writes DNS, both while DNS still points at the old platform. Domains move at the cutover (step 3).
  - Environment values come only from `--dotenv <service>=<file>`, through the panel's `.env` import. The flag is not `--env-file`, because Node reads that flag itself from anywhere on the command line.
  - A receipt next to the plan records each item's state and id, never a value. Running the command again repeats only what is not done, and a run that stopped halfway picks its items up again without duplicates.
  - Checked live: a plan of a real CapRover v1.16.0, applied to a disposable panel, then a resume after a simulated crash.
- **`ninedeploy import data`: move database contents from Coolify, Dokploy or CapRover.**
  - Without `--yes` it prints the order that loses no writes: lower the TTL, stop writes, take the final dumps, import, deploy and check health, then switch DNS and add the domains.
  - It also prints the dump command to run on the source host for each database. The command reads credentials from the database container's own environment, so no password appears in the output or the shell history.
  - `--dump <database>=<file> --yes` imports through the panel's chunked dump import with the file's sha256 declared, which the panel checks before restoring.
  - The receipt keeps the import id, hash and size. A stopped upload resumes the same import, and a database is never imported twice.
  - Checked live: a CapRover v1.16.0 PostgreSQL 16 app with 500 rows, dumped with the printed command, arrived with the same row count and checksum.
  - Plans now record where each database runs on the source (`runtime`). The live run showed that CapRover 1.16 names an app's Swarm service after the app; only legacy apps keep the `srv-captain--` prefix.
- **`ninedeploy import apply` attaches the new database where a copied connection string still names the old one.** A `--dotenv` file is read on your machine. A value such as `DATABASE_URL=postgres://…@srv-captain--pg:5432/…` that names a source database's host gets the new database attached to the service under that variable, and the attachment sets it at every deploy.
  - It attaches only where the panel's connection string can stand in exactly: `postgres://`, `mysql://` or `mariadb://` without `?options`, or `redis://` on database 0.
  - Every other match is listed with the reason, for a person to change: another scheme spelling, options, a Redis database index, Mongo, a host-only variable, or the old platform's own host (NineDeploy can still reach it, so the app would quietly keep using the old database).
  - Only variable, database and host names reach the output and the receipt. A rerun attaches nothing twice and counts an attachment made by hand. A variable another database already sets is refused. The panel attaches a database to a service under one variable, so when two variables name the same database (Prisma's `DATABASE_URL` and `DIRECT_URL`), the second is listed for a person instead of recorded as attached.
  - Checked live: after apply and a deploy, the container's `DATABASE_URL` named the new database, and the two listed variables kept their copied values.
- **`ninedeploy import plan heroku`: a plan from a Heroku app's `app.json` and `Procfile`.** It needs no credential and reads no Heroku API.
  - Each Procfile process becomes a service built from the repository with Nixpacks, with that process's line as its start command. Checked live on a panel: Nixpacks runs the command through a shell, so `$PORT` expands, and sets `PORT` to the service port.
  - Workers get the `running` health check and no domain. `release` is not mapped to the panel's pre-deploy hook, which runs in the panel's container: Nixpacks runs the Procfile's release line as a build step of each service, which the plan explains. `formation` quantities become replicas.
  - The `heroku-postgresql` and `heroku-redis` add-ons become databases, attached to every service as `DATABASE_URL` and `REDIS_URL`. `import apply` creates those attachments and sets the worker health check. `import data` prints the `heroku pg:backups` download for Postgres.
  - Config var values come from `heroku config --shell` through `--dotenv`. `scripts.postdeploy`, `heroku.yml`, custom buildpacks, other add-ons and dyno sizes are listed for a person.
  - Plans gain optional `settings`, `attachments` and a `heroku-app` runtime; they stay planVersion 1.
- **Service volume import, and `ninedeploy import data --volume`.** A gzipped tar of another host's volume can now be restored into a service's data volume through the panel (`POST /v1/services/:id/volume-imports`; SDK `volumeImports`).
  - The archive is uploaded in chunks with its sha256, which the panel checks before the volume is touched. A file that is not a gzipped tar is refused.
  - The import is refused while the service runs or deploys. A volume that already exists is first snapshotted as a `pre-import` volume backup.
  - It restores through the same staged swap as a volume backup restore, so owners, modes and symlinks are kept, and a bad archive leaves the volume as it was.
  - It is for operators, and for Docker services on the panel host with a volume mount. Swarm services, compose stacks, node services and services spread over several hosts are refused.
  - `import data` prints the command to run on the source host. A named volume is mounted read-only by name, which works after the app is scaled to 0. Then `--volume <service>=<file> --yes` imports it with a resumable receipt and never imports twice.
  - Checked live: a CapRover v1.16.0 app volume arrived in the deployed service with the same files, owners, modes, symlink and checksums.
  - New table `volume_imports` (migration 0124, additive). Stale uploads expire after 24 hours, and finished rows are pruned after 90 days.
- `ninedeploy import data` keeps names that came from the source platform out of the commands it prints unless they are plain Docker names. A name with shell syntax becomes a placeholder to fill in by hand, so pasting a command cannot run anything hidden in an app's name.
- **`ninedeploy import cutover`: add the imported domains once DNS points at NineDeploy.** Without `--yes` it reports, host by host, what is ready; with `--yes` it adds the domains that pass two checks.
  - The service is running and its latest deploy succeeded. The command never deploys.
  - Every A and AAAA of the hostname already resolves to this panel: a direct DNS query (`--resolver` picks the server, the output names it), compared with `--expect` or Settings → DNS records. A leftover record that still names the old host holds it back and is named. A host that fails waits, so no certificate is requested while DNS still points at the old platform.
  - The source's own generated hosts are never added; wildcard hosts are listed for a person. A domain already on the panel counts as done, and a pending ownership proof is verified once DNS passes (its TXT value is printed, never stored).
  - `--skip-dns-check <host>` covers proxied records, and is refused while the Cloudflare integration is on or unknown, because that integration adds a second A record next to the old one.
  - Checked live on a disposable panel: refused without an expected address, waited while the service was idle, added a host resolving to the panel (Traefik served it), held back one resolving elsewhere, held back a host whose AAAA was not expected until `--expect` named it, and added nothing on a rerun.

### Fixed

- **Dokploy import: a rate-limited API key no longer looks like a wrong one.** Checked live on Dokploy v0.31.0, a default API key is limited to 10 requests a day and was refused after 5 calls, with the same 401 as a wrong key. A plan makes 2 requests plus one per application, compose stack and database. The message now names both causes, and the docs say to create the key with rate limiting off. The same live run matched the plan to a seeded project field for field, kept as a test fixture. It also confirmed the Swarm service and volume names that the printed `pg_dump` and volume commands use: both ran on the Dokploy host.
- **Deploy logs no longer show secret values.** A build tool can echo them. Nixpacks, for one, fills a `$VAR` of a build command into its RUN step, and BuildKit prints that step, so a Procfile `release` line referencing `$DATABASE_URL` put the attached database's password in the deploy log, which every member of the service can read. Each deployment now masks its secret variables, attached connection strings (and their password on its own) and resolved vault references as `***` in every log line, on disk and live. Checked live: the same deploy that showed the password now shows `echo "***"`. Values shorter than 6 characters are not masked. The image itself still carries the value in its build history, as Nixpacks writes it; keep secrets out of build commands.
- Volume backup restores (and volume imports) no longer drop symlinks. The staged swap moved entries only when `[ -e ]` was true, which follows links. A link whose target was elsewhere, or not yet moved (`link.html -> index.html`), was silently left behind and deleted with the staging directory. Found by the volume import's live proof, and covered by a real-Docker test that fails without the fix.
- `ninedeploy import data` now prints a MySQL/MariaDB dump command without `--databases`. That flag writes `CREATE DATABASE` and `USE` into the dump, and the panel restores the file as written. For an admin the import reported success but put every row in a second database beside the managed `app` database, which the app's connection string names; anyone else was refused. Checked live with MySQL 8.4: the old form put 500 rows into `shopdb` and left `app` empty, while the new form put them into `app` with the same checksum. Dumps taken with the earlier command should be taken again.
- Swarm deployment regression tests now await database readiness and close their native database clients after every case (F1133).
- CloudWatch log drains retain captured batches when HTTP 200 acknowledges rejected log events, preserving accepted-event and rejection-boundary handling (F1126).
- Elasticsearch log drains send bulk NDJSON to the index bulk operation while preserving configured index, query parameters and authentication (F1124).
- Elasticsearch drains retain queued logs when a bulk acknowledgement is unreadable or malformed instead of recording delivery success (F1122).
- Elasticsearch drains recognize filtered per-item bulk errors even when the top-level error flag is omitted (F1123).
- API-only servers now keep query credentials out of missing-route logs when the dashboard has not been built (F1125).
- Failed OIDC discovery requests now release unused response bodies before returning the existing provider error (F1120).
- GitHub OAuth token, profile and email refusals now release unused HTTP bodies while preserving diagnostics and unverified-email fallback behavior (F1121).
- Teams and Resend success responses and Lark HTTP refusal responses now release unused bodies while retaining provider diagnostics and Lark application error handling (F1119).
- Basic webhook, chat and push notification deliveries now release unused HTTP response bodies on success and refusal, preserving Telegram diagnostic text and existing channel payloads (F1118).
- PagerDuty and Opsgenie deliveries now release unused success-response bodies before returning, while preserving provider refusal details and notification payloads (F1117).
- GitLab, Gitea and Bitbucket commit-status reports now cancel unused response bodies before returning, releasing HTTP resources on success and failure without affecting deployments (F1116).
- Git LFS builds now hydrate verified empty objects locally, avoiding a failed download or misleading credential error when no remote empty object is stored (F1115).
- Git LFS builds now reject conflicting sizes for the same object before downloading or changing checkout files, instead of silently accepting the last pointer declaration (F1114).
- Git LFS recognition now validates the complete basic-v1 pointer, preserving ordinary text with pointer-looking prefixes (F1112).
- Git LFS hydration now accepts native-compatible CRLF, omitted final newlines and outer Unicode whitespace for basic-v1 pointers (F1113).
- Workspace MCP readiness checks now remove token environment names case-insensitively before child startup, including Windows lowercase and mixed-case aliases (F1111).
- MCP workspace installs now link a tracked launcher before build output exists, and macOS setup verifies the command after build (F1110).
- Git LFS fetches now remove temporary objects on batch, stream, verification or checkout-copy failure before returning (F1109).
- Autoscaler regression fixtures now await SQLite readiness and release owned native clients after each test, with permanent ownership guards (F1108).
- Backup scheduler now refuses new dump admission after close, including callbacks resuming from pending reads and daily loops between databases (F1107).
- Backup policy cron publication now ignores superseded reads from saves, periodic refreshes and startup while preserving independent database changes (F1106).
- Backup drill history rows now include the database identity required by the SDK and already present on single-entry responses (F1105).
- Structural backup checkers now retain a UUID helper identity and attempt cleanup when Docker commits a create but its CLI reply is lost (F1103).
- Structural backup checker cleanup now also requests removal of helper-owned anonymous volumes (F1104).
- Manifest probes now verify SHA-256/SHA-512 against bounded response bytes before scheduling updates and skip unsupported hash algorithms (F1102).
- Manifest probes now reject malformed OCI content-digest metadata before it can seed an auto-update baseline or queue a deployment, preserving valid registered and opaque digest syntax (F1101).
- Watch parsing now rejects malformed registry host text before it can alter manifest probe URI path, query, or fragment semantics (F1099).
- Watch parsing now enforces the native 255-character repository-path limit with official Hub namespace normalization, while preserving registry and tag compatibility (F1100).
- Watch parsing now rejects malformed image repository components before dispatching registry manifest probes, including empty and dot path segments (F1097).
- Watch parsing now enforces Docker tag start syntax and the 128-character limit before dispatching a manifest probe (F1098).
- Image registry classification now recognizes uppercase single-label hosts accepted by Docker CLI, preserving their bound credentials and watch parsing instead of treating them as Hub namespaces (F1096).
- Explicit docker.io image credentials now use the default Hub login/logout operand so successful logout removes the canonical auth record instead of leaving it behind (F1095).
- Registry sessions now serialize per host credential store across different registries, preventing Docker CLI snapshot saves from dropping another live login (F1094).
- Cross-process lock identities now publish only after complete private preparation, so rejected initialization cannot leave a fresh public lock or delete a newer holder (F1093).
- Cross-process operation locks now complete short native identity writes before granting a holder, preserving recognition and cleanup by its release operation (F1092).
- Database-import chunks now complete short native writes before advancing upload metadata and fail without acknowledging a zero-progress write (F1091).
- Concurrent database-import chunk retries now preserve acknowledged bytes by serializing native writes and rechecking upload metadata before writing (F1090).
- A stale database-import refusal now returns a conflict without deleting staging owned by an admitted worker or emitting a false failure audit (F1089).
- CLI database import retries reconcile known server jobs, resume interrupted uploads and avoid restarting completed restores after lost responses; recorded job IDs cannot be rebound to a different dump (F1088).
- Bare :memory: SQLite transactions no longer borrow a nonexistent replacement connection; commit, rollback and foreign-key enforcement match file::memory: (F1087).
- Failed reset-link replacement rolls back deletion and preserves the previously delivered token on precommit failure (F1086).
- Reset-link issuance replaces pending tokens atomically through the managed transaction queue, so overlapping completed requests leave one pending token (F1085).
- Restricted-port sidecar inspection now treats only exact target-specific missing-container diagnostics as absence, preserving config and avoiding replacement on unknown Docker errors (F1084).
- Failed restricted-port sidecar removal now preserves its live config, propagates the original Docker error and avoids a false successful-removal log (F1083).
- Restricted-port sidecar sync and removal now run in order per service slug, so an older prepared sync cannot recreate the sidecar after a later removal or overwrite newer routes (F1082).
- Traffic collection now expires request buckets on advancing writes, bounding per-service history to the latest recorded 24-hour window even when no rate reader runs (F1081).
- Latest request-time calculations no longer overflow the call stack on large valid bucket histories, preserving scale-to-zero idle decisions during long-running traffic collection (F1080).
- Automatic image-update digest writes preserve completed opt-outs and replacement baselines across initial observation, refused update and already-admitted enqueue paths (F1079).
- Automatic image updates recheck opt-in, running state, image/digest and in-flight work atomically when enqueuing, preserving operator changes and avoiding competing duplicate deployments (F1078).
- Automatic image-update scheduler sweeps no longer overlap and enqueue the same image change twice while an earlier interval is still awaiting I/O (F1077).
- Restart-loop samples cannot stop replacement runtimes or interrupt deployments admitted while sampling; rejected stale trips no longer contaminate later runtime baselines (F1076).
- Restart-loop enforcement preserves newer operator policy changes during runtime sampling and does not recreate disabled limits or clear replacement history after an admitted stop (F1075).
- Shared-certificate renewal checks no longer overlap while reading settings, and pending checks cannot start a proxy render after server close (F1074).
- Pending runner jobs recheck queue expiry after asynchronous preparation and status/config reads, so an expired snapshot entry cannot consume a runner slot ahead of fresh work (F1073).
- **A PostgreSQL backup taken just after the database started no longer fails.** The official image initializes with a temporary server and then restarts, while the container already shows as running. A **Backup now** click, or a scheduled backup, in those first seconds failed with `pg_dump` exit 1. A failed dump now waits up to 20 seconds for the server to accept connections over TCP, and retries once. A backup that succeeds the first time sends the same commands as before. The panel's backup toast now shows the reason the server gave, not just "Backup failed".
- Runner job rechecks keep failed or unrecognized GitHub status reads pending without taking a slot; only confirmed queued jobs start and confirmed non-queued work is dropped (F1072).
- Self-hosted GitHub runner upkeep no longer applies a GitHub-hosted six-hour age cutoff to busy or unverifiable runners; confirmed-idle/exited cleanup and explicit operator stop remain available (F1071).
- Runner sweeps retain missing, malformed or unrecognized node/container states instead of treating them as confirmed non-running runners and automatically stopping them (F1070).
- Queued GitHub runner jobs revalidate current repository, public-repository and label policy before taking a slot, so previously accepted work cannot bypass later runner restrictions (F1069).
- GitHub runner batches stop admitting additional jobs when their configuration changes during preparation, retaining pending work for a fresh host/capacity pass (F1068).
- GitHub runner drains recheck pending-job identity after asynchronous preparation of earlier jobs or online status checks, so later workflow events cannot leave stale jobs eligible to start (F1067).
- Failed CI log file creation or append no longer consumes the chunk sequence; partial writes restore the prior file length and reservation so retries preserve accepted diagnostics (F1066).
- CI log writes serialize admission and file append per attempt within the panel process, preserving chunk order and acknowledging overlapping retries from current persisted sequence state (F1065).
- CI results that lose the job transition to cancellation return a conflict instead of success; cancellation closes the current attempt even when a competing result just finished its attempt write (F1064).

## [0.18.26] - 2026-11-09

### Added

- **The panel on its own domain, with HTTPS, from the installer.** `install.sh --domain panel.example.com --email you@example.com` serves the panel on that hostname, with a Let's Encrypt certificate from the first boot, and makes it the public URL.
  - Without the flags, a first install on a terminal asks for both. Re-runs and self-updates never ask, and an explicit flag replaces the value in `.env`.
  - Both values are checked strictly before anything runs, because bare metal sources `.env`.
  - The installer warns when the domain does not point at the server yet. It prints the setup link for the domain and for `http://<server-ip>:3000`, which works before the certificate exists.
  - The Docker compose file now passes `NINEDEPLOY_DOMAIN` to the panel. A compose file from an older pinned release does not, and the installer says so.
  - The first-run form offers to use the admin's email for Let's Encrypt when none is configured (`POST /v1/setup` takes `acmeEmail`; `GET /v1/auth/status` reports `acmeConfigured` before setup only). It never replaces an email that is already set.
  - A domain install still seeds free sslip.io service URLs: the installer records the server address in `NINEDEPLOY_PUBLIC_IP`.
- **A stable update channel.** Until now, the panel offered every release as soon as it was published. An operator can now choose **stable** under **Settings → System → Update channel**, or with `GET`/`PUT /v1/system/update-channel` (SDK `system.updateChannel`). Stable offers only the release the `stable` tag points at. Every release stays the default, so nothing changes until you switch.
  - A daily workflow (`.github/workflows/stable-channel.yml`) moves the `stable` tag to the newest release that has been out for 7 days.
  - A release whose GitHub release notes carry `<!-- stable: security -->` is promoted at once. A maintainer can also promote a tag by hand from the workflow. One marked `<!-- stable: hold -->` is never promoted.
  - The tag never moves backwards.
  - The installer takes the same channel: `install.sh --channel stable`. It refuses, with a clear message, while no stable release exists yet.
  - The installer also writes `NINEDEPLOY_UPDATE_CHANNEL=stable` into `.env`, so the panel follows the same channel until an operator picks one in Settings.
- **What a service can use, in one place.** `GET /v1/services/:id/capabilities` (SDK `services.capabilities`) answers, for nine features, whether each is available, limited, still being built, or refused where the service runs, with the reason. The features are maintenance mode, error pages, forward auth, restart limit, inline Dockerfile, file mounts, autoscaling, scale to zero and canary releases.
  - The routes now refuse through the same check, with the same messages as before.
  - The maintenance, restart-limit and forward-auth cards follow it. For example, the maintenance card no longer appears on a panel-host service that also runs on nodes, where saving was refused.
  - **Service → Settings** lists the limits with their reasons.
  - [docs/SUPPORT_MATRIX.md](docs/SUPPORT_MATRIX.md) is generated from it, so the docs cannot drift from the code. It also corrects one claim: an inline Dockerfile on a node service works only with **Build on: panel**.
- A native macOS development bootstrap (`node scripts/dev-macos.mjs --install`) and cross-machine handoff guide in `docs/MACOS_DEVELOPMENT.md`, including the separate desktop lockfile and the remaining DMG/Keychain acceptance checks.
- **An IP allow-list for the panel itself.** Turn it on under **Settings → Security → Panel access**, or with `PUT /v1/settings/panel-access` (SDK `settings.panelAccess`). It is off by default and operator only. When on, only the listed IPv4 or IPv6 addresses and CIDR ranges can sign in or use the panel with a session, over HTTP and WebSocket.
  - Integrations keep working from anywhere: git, registry and deploy webhooks, badges, status pages, CI runners, SCIM, error ingest, and the app traffic the panel serves for wake, forward auth and maintenance pages.
  - API and remote-MCP tokens are only restricted if you also turn on **Also restrict API and MCP tokens**.
  - A list that would exclude your own address is refused, and the card offers **Add my address**.
  - If you lock yourself out, set `NINEDEPLOY_PANEL_ACCESS=off` and restart the panel. Docker installs need the line `NINEDEPLOY_PANEL_ACCESS: ${NINEDEPLOY_PANEL_ACCESS:-on}` in their compose file. The installer writes it from this release; an older compose file does not pass the variable through. See [docs/TROUBLESHOOTING.md §14](docs/TROUBLESHOOTING.md#-14-locked-out-by-the-panel-ip-allow-list).

### Changed

- The Plugins catalog now points its PostgreSQL PITR, Sentry and Tailscale entries at the built-in features. Entries with no shipped equivalent say "Not available" instead of "Planned".

### Fixed

- CI runner revocation fences both lease creation and assignment; pending attempts are lost/requeued before publication, and stale claims cannot overwrite a newer runner's attempt (F1063).
- Concurrent CI claims reserve runner capacity inside the attempt insert; a losing request leaves its job queued without consuming an attempt or reviving cancelled work (F1062).
- CI cancellation closes the attempt current at cancellation time, and claims that lose assignment to cancellation retire their attempt instead of returning cancelled work or retaining runner capacity (F1061).
- **Three web panel faults the first browser journeys found.**
  - Every page tried to load Google Fonts, which the panel's own content security policy blocks. Each page load logged a console error, and loading the fonts would have sent every visitor to Google. The import is gone. The panel uses Inter and JetBrains Mono when they are installed, and the system fonts otherwise, which is what it already showed.
  - The sign-in page asked for the project list before anyone was signed in, and got a 401.
  - The Settings and service pages each had a second `<main>` element. A page should have only one; the Layout owns it.
- CI expiry sweeps recheck the persisted lease atomically before declaring an attempt lost, preserving heartbeat/restart renewals and counting only attempts actually lost (F1060).
- Scheduled deploys count CI-waiting rows toward the shared queue limit and reserve the last slot atomically, auditing concurrent losers as skipped while preserving the schedule trigger (F1059).
- Scheduled jobs keep their active crons while a refresh query waits or fails, preserving upcoming job deadlines until a successful reload replaces the schedule set (F1058).
- Image scan schedulers suppress critical-result callbacks after their application closes, preventing late audit alerts from a retained scan callback (F1057).
- **Advice that cut a panel off from its own domain.** Doctor, the README, the compose file and the panel IP allow-list docs said to bind the panel to `127.0.0.1` once it had an HTTPS domain. Traefik reaches the panel through the Docker gateway, so after that change the domain stopped answering. This was proven in a Linux Docker-in-Docker run: a loopback-bound panel is unreachable, `0.0.0.0` is reachable. The advice now is to keep port 3000 reachable from Docker and to block it from the internet with a firewall in front of the host. The README's first-boot steps also pointed at **Settings → System** for the ACME email. It lives under **Settings → Security**.
- GitOps result publication respects configuration removal/replacement, preserves cleared state and protects newer results; stale checks report a conflict and completed apply results remain available to their caller (F1056).
- Instance GitOps scheduler ticks no longer start new checks or publish late drift/failure alerts after their application closes (F1055).
- Active PITR retention now excludes competing pruning, recovery, base-backup and disable operations on the same archive; backup requests report a conflict instead of accepting work that cannot start (F1054).
- PITR recovery admission reserves capacity before asynchronous name/volume checks, keeping concurrent requests within the one-recovery limit and releasing capacity on admission failure (F1053).
- PITR maintenance reloads the database's retention policy before pruning, honoring retention increases committed while the archiver query or a base backup was running (F1052).
- Panel backup scheduler ticks waiting for settings now stop when their application closes, preventing a late backup from starting against closed resources (F1051).
- Panel backup retention now rechecks the selected destination and keep count before each deletion, preserving old recovery artifacts when settings change during an upload or prune (F1050).
- Loki log searches keep a reverse proxy's path prefix when deriving the query endpoint from a configured `/loki/api/v1/push` URL.
- Log delivery and Loki search apply header precedence without regard to name casing, avoiding duplicate authentication and Content-Type values from differently cased custom headers.
- Loki log searches preserve the drain's custom headers, including tenant IDs, and avoid duplicating an existing Bearer prefix, matching log delivery authentication.
- ZIP source uploads publish through a complete temporary file, so a partial write cannot corrupt an already accepted archive or expose a failed upload as deployable source.
- Failed `.tar.gz` source upload writes and final moves remove their owned temporary files while preserving an already published archive and its retention history.
- HTTP readiness probes tolerate unexpected informational replies such as `103 Early Hints` and wait for the final status instead of rejecting the workload early.
- Concurrent identical source archive uploads use separate temporary files, so one request cannot consume or remove another request's pending upload.
- Failed system imports remove newly installed database, key, routing and environment files when those paths did not exist before the import, preserving the original state during rollback.
- Inline Compose workspaces accept a repository directory configured at a filesystem root while continuing to reject invalid service IDs.
- S3 build-cache misses cancel unused HTTP response bodies, releasing transport resources while preserving the cache-miss result if cancellation fails.
- S3 multipart backup uploads fill short filesystem reads and abort on premature EOF instead of completing an object containing unread buffer bytes.
- Truncated encrypted backup envelopes are rejected before opening a plaintext output, preventing late empty staging files and leaked write descriptors when the authentication tag is incomplete.
- Backup replica retention rechecks the current destination list and copy limit before every remote delete. Removing a replica or raising its retained-copy count during a slow upload no longer lets the old policy delete recovery points afterward.
- Image cleanup no longer reports images removed or space reclaimed when both deletion and the follow-up inspection are unconfirmed; a daemon, permission or timeout error is kept separate from a missing image.
- Webhook replay detection keeps delivery IDs separate per provider and service, so one repository event can reach each configured service. Docker runtime patrols skip unreadable inspections instead of reporting a missing container and a false outage.
- **Cloning, exporting and importing a service no longer drops settings kept outside the service record.** These settings were silently lost: restart limit, custom error pages, CI pipeline, commit statuses, sticky sessions, maintenance mode, and per-domain forward auth and compose routes. Before, only the inline Dockerfile survived a clone, and an exported inline-Dockerfile service lost its Dockerfile.
  - Each value is checked with its own route's validation. Anything the new service cannot use is skipped and listed in the import response.
  - A clone starts outside maintenance mode.
  - Secrets (the registry push-webhook token and the error-ingest key) never travel; the import names them so you can create new ones.
  - Older bundles import as before. See [docs/DEPLOYMENTS.md](docs/DEPLOYMENTS.md#what-a-clone-and-a-service-bundle-carry-s3).
- **Deleting a service, database, domain, GitHub App or SSO provider now also deletes its stored settings.** Several features keep per-item state in the panel's settings table rather than in their own tables: restart limits, maintenance pages, forward auth, compose routes, registry-hook tokens, error-ingest keys, backup replica logs, runner configurations and SAML identity links. Until now this state was left behind after its owner was deleted. That included the hashed registry-hook token and the error-ingest key.
  - Leftover rows from earlier deletes, and rows of deleted workspaces, are removed by the hourly housekeeping run.
  - Cloud-VM links are kept, so a VM created through the panel can still be deleted from it.
  - IDs are never reused, so no new item ever picked up an old row. The restart-limit check stops walking deleted services every pass.

### Changed

- **Settings lookups by key prefix use the table's index.** These lookups run on every Traefik config render: maintenance pages, error pages, compose routes and forward auth. They now read a key range instead of `LIKE 'prefix%'`. `LIKE` could not use the index, and it treated the `_` in a prefix such as `restart_limit:` as a wildcard.

## [0.18.25] - 2026-11-08

### Added

- **Desktop installers on every GitHub Release (R6).** Each release now carries the desktop app for Windows (`NineDeploy-Setup-<version>.exe`), macOS (`NineDeploy-<version>.dmg` for Intel, `-arm64.dmg` for Apple silicon) and Linux (`.AppImage` and `.deb`). They are built from the release tag on each platform's own runner, unit-tested there and launched as installed, in parallel with the image smokes. `SHA256SUMS` and its Sigstore signature now cover them too. A missing or failed installer holds the release. They are not code-signed yet: Windows SmartScreen warns about an unknown publisher, and macOS opens the app only after you allow it under System Settings → Privacy & Security. Keeping the sign-in through the macOS Keychain is not verified yet. See [docs/DESKTOP.md](docs/DESKTOP.md).

### Changed

- The Windows installer is now named `NineDeploy-Setup-<version>.exe` (no spaces), so its checksum line names the release asset exactly.

### Fixed

- **Node wake catchers left behind are found and removed (R2).** Node sleep keeps a small wake catcher on each node, and catcher maintenance replaces it with a new one. A catcher could be left behind if the agent's reply to a preparation was lost, or if it came from an older image. One prepared on a node that no longer has a sleeping service could stay too. The panel now reads the node's catcher inventory before it prepares another one. Catchers it did not know about are recorded and retired, with the same ordered routing update, drain and immutable-ID check as any other retirement.
  - On a node with a wake key and no sleeping service, unused catchers are collected at most every five minutes, one node at a time. No replacement is created, and new sleep or wake requests on that node wait until collection finishes.
  - This needs agents that advertise the new `wake.catcher.inventory` operation. With an older agent, catcher maintenance is held until the agent is upgraded. Ordinary demand polling and wake work as before.
  - A maintenance job leased by another worker no longer takes one of the node poller's slots, so it cannot delay wakes on other nodes. An idle-state read that finishes after a newer one is ignored.
  - No SQL migration. The new `collecting` maintenance phase fits the existing column; an older panel, after a downgrade, refuses that phase instead of completing it.

## [0.18.24] - 2026-11-08

### Changed

- **Releases no longer repeat main CI's full test suite.** The release workflow re-ran typecheck, lint, build and every test on the tagged commit, about 39 of its roughly 95 minutes. When main CI's "Typecheck · Lint · Build · Test" job already passed on that exact commit, the release reuses that verdict and links the run in its summary. Otherwise it runs the checks as before: no CI run, a failed or cancelled one, one still running, or a tag on a commit that was never main's head. The database integration checks, the image smokes and signing are unchanged.
- **A deployment held for runtime recovery now says so on the Deploys tab.** When the outcome of a Swarm, primary-runtime or fan-out transition cannot be confirmed, the deployment is kept unsettled on purpose and retried every minute. Its row stayed "deploying" with nothing beside it, which looked like a deploy that never ends. The recovery cards were only under Settings. The Deploys tab of a Docker service now shows them too, with the reason, the next automatic check and, for operators, Retry recovery.
- **The release Swarm smoke reports the recovery journal when a deploy hangs.** A Swarm deploy that cannot confirm its outcome is held for recovery on purpose, which from outside looks like a deploy that never ends. The smoke's timeout report now starts with the service's Swarm recovery entry (phase, reason, attempts) and its newest deployments.

No migration and no configuration change.

## [0.18.23] - 2026-11-07

### Security

- **Every open dependency advisory is cleared.** `pnpm audit` reported 14 high, 31 moderate and 10 low advisories; it now reports none, for production and development dependencies alike. No migration and no configuration change.
  - **Server:** zod 4.6.7 (memory exhaustion while validating oversized constrained collections; also in `@ninedeploy/schemas` and `@ninedeploy/mcp`). fastify 5.12.8, with fast-uri pinned to 3.1.9 and 4.2.3 (scheme spoofing, WHATWG host confusion, scheme injection, path equivalence). @fastify/static 10.1.10, which brings glob 14.0.3. @fastify/rate-limit 11.2.2 and ip-address 10.7.4 or later (quadratic zone-identifier matching). nodemailer 10.3.1. follow-redirects 1.16.2 under PM2 (credentials kept off cross-origin redirects).
  - **MCP:** the MCP SDK 1.32.2, with body-parser 2.4.0, qs 6.16.2 and @hono/node-server 2.1.4 or later pinned. `/v1/mcp` uses none of their vulnerable paths (the SDK's express helpers are not imported, and `serveStatic` is not used), so these are defence in depth.
  - **Development and tests only** (not in the image): undici 8.12.3 under testcontainers and jsdom, @grpc/grpc-js 1.14.6, source-map-js 1.2.2, glob 14 and minimatch 11 under testcontainers' archiver, vite 8.3.6, vitest 5.0.5 and smol-toml 1.9.1.
  - PM2 stays on 7.0.4. Its 7.1 line replaces the file watcher, which is more than a security patch should carry.

### Changed

- The release smoke's Swarm checks bound each deploy wait at five minutes. When one hangs, they print the deploy log, the Swarm tasks and the panel log. v0.18.22's first release run hung there for 26 minutes and the step timed out with nothing to read.

## [0.18.22] - 2026-11-06

### Fixed

- **Swarm services: what each feature does, made true (R2).** [Deployments §7](docs/DEPLOYMENTS.md) now has a support matrix for Swarm services, checked on a real swarm. The cases that were silently wrong:
  - **Restart after Stop** left the service at 0 tasks while the panel showed it running. Restart now runs every task at the service's replica count.
  - **Status was never checked.** The runtime patrol skipped Swarm services, so a crash loop, an image no node could pull or a removed stack stayed "running". The patrol now checks a running Swarm service each minute. A service scaled to 0 outside the panel is scaled back, and one that no longer exists is marked down at once. One with no running task for five minutes is marked down too, with the service-down alert. Swarm still restarts its own tasks, and a manager that cannot answer is never a verdict.
  - **Log drains** never received Swarm logs, because they read a container named after the service, which does not exist. They now read each task through the manager, wherever it runs.
  - **The Logs tab** listed one task's lines, then the next task's. They are now in time order, newest kept.
  - **The `Container running` healthcheck** probed HTTP through Traefik, so a worker or a non-HTTP port failed its deploy. It now waits for the tasks only.
  - **The File Browser** answered a bare 500, and **the Manifest tab** showed the Swarm service object as an empty container. Both now open a task on the panel host, or name the node that runs it (422 `swarm_task_elsewhere`).
  - **Metrics** summed the tasks on the panel host and compared the sum with one task's limit, which gave false "raise the limit" hints. They now report one task's worth: the average.
  - **The dashboard** marked a 4xx health path through Traefik as down, unlike the deploy. This also affected containers on hosts that cannot route bridge IPs.
  - **A volume attachment** was accepted, then its redeploy failed and left the volume and the attachment behind. It is now refused (422 `swarm_unsupported`) before anything is created.
  - **The Settings tab** no longer offers HTTP or TCP readiness to a Swarm service, which the server refuses. The autoscaling notice no longer suggests Swarm scales by itself.

## [0.18.21] - 2026-11-05

### Added

- **Restore verification for MySQL, MariaDB, MongoDB, ClickHouse, Meilisearch, RabbitMQ, Redis, Valkey, KeyDB and Dragonfly backups (R3).** **Verify restore** and `backups drill --restore-query … --expect …` now also restore a backup of these engines into an isolated container of the database's own engine image and run your query there.
  - **Isolation:** the same as PostgreSQL. No network, all capabilities dropped, a read-only root, bounded tmpfs, and the container is removed afterwards.
  - **MySQL and MariaDB:** the whole `--all-databases` dump is restored, and the query runs in the `app` database. The expected output is the client's batch output without headers. Columns are separated by a tab, and NULL prints as `NULL`.
  - **MongoDB:** the archive is streamed into `mongorestore --stopOnError`, so no copy of it lands in the container. The query is a mongosh script, and the expected value is what mongosh prints for its last expression.
  - **ClickHouse:** the panel's `BACKUP` archive is restored with `RESTORE ALL`. The query runs in the `app` database, and the expected value is `clickhouse-client`'s TabSeparated output.
  - **Meilisearch:** the dump is imported as the container starts. The query is an API path, and the expected value is the JSON body a GET on it returns.
  - **RabbitMQ:** the definitions export is imported, and the restore counts as done only once every vhost, queue, exchange and binding it names exists. The query is one `rabbitmqctl list_…` command; its rows compare in any order. A definitions backup holds no messages.
  - **Redis, Valkey, KeyDB and Dragonfly:** the server loads the RDB backup as it starts, and an RDB the engine rejects is `failed`. The query is one command per line, and the expected value is what the command-line client prints.
- **Recovery duration of a restore verification.** The Backups tab shows how long the real restore took next to the verdict, for example "restored in 1m 05s". `backups drill --restore-query` prints it on a pass.
- **Point-in-time recovery for PostgreSQL (R3).** It is off by default; an upgrade changes nothing until an admin turns it on for a database (Backups tab, `ninedeploy databases pitr enable`, `PUT /v1/databases/:id/pitr`).
  - **Archiving:** the database restarts once with WAL archiving into a second volume on the panel host. At most a minute of committed work waits in an unarchived segment.
  - **Base backups and retention:** the panel takes a base backup a day (and on demand) and keeps them for the retention, 1–35 days, default 7.
  - **Recovering:** recover to any second inside the window, or to the latest archived point, into a **new** database. The source is never touched. The history records what PostgreSQL replayed to and how long it took.
  - **Refusals:** a missing or damaged segment, or a target the archive cannot reach yet, is refused with a reason instead of recovering silently to an earlier point.
  - **Limits:** panel-host databases only. Disabling deletes the archive. Restoring the volumes of a database with it on is refused.

### Fixed

- **MongoDB databases start on Linux 6.19 through 7.0.13.**
  - **Problem:** on these kernels MongoDB 8.0 refuses to start ("Linux kernel versions 6.19 and newer has a known incompatibility", SERVER-121912). The cause is the `mongo:8` image's own `GLIBC_TUNABLES=glibc.pthread.rseq=0`, which turns on TCMalloc's per-CPU caches; they crash on these kernels.
  - **Fix:** MongoDB containers now set `glibc.pthread.rseq=1`, which starts on every kernel. TCMalloc falls back to per-thread caches, as in MongoDB 7. This applies to databases on the panel host and on nodes.
  - **Existing databases:** an existing MongoDB database picks this up the next time it is started, with Stop and then Start. Restart keeps the container's old settings.
- **A restore verification whose database engine stopped is `unverifiable`, not `failed`.** This applies to every engine. Examples are a lost connection to the server, or a helper that is no longer running.
  - MariaDB aborts outright when its scratch space fills up. The helper's own log is now checked for that, so the verdict names the capacity limit.
  - MySQL's "The table … is full" is also recognized as the capacity limit.

## [0.18.20] - 2026-11-04

### Added

- **Restore verification for PostgreSQL backups (R3).** It restores a backup into an isolated PostgreSQL container on the panel host and runs your query there. The check passes only when psql's output equals the expected value exactly.
  - **Where to start it:** the Backups tab (**Verify restore**), the CLI (`ninedeploy backups drill <db> <backup> --restore-query <sql> --expect <value> [--no-wait]`), or the API (`POST /v1/databases/:id/backups/drill` with `restore: { query, expected }`).
  - **Isolation:** the container has no network, no published ports and no host mounts. All capabilities are dropped, the root filesystem is read-only, and storage is bounded tmpfs. The container is removed afterwards.
  - **What is never touched or kept:** the live database is never touched. The query and the expected value are not stored, logged or audited.
  - **Permissions:** the probe reveals the backup's contents, so it needs the `admin` role on the database, the same as a download. The structural drill stays at `member`.
  - **Asynchronous:** the request answers `202` with `status: "running"`, and clients poll `GET /v1/databases/:id/drills/:drillId`. The SDK adds `databases.drill(id, drillId)`, and the CLI waits for the verdict. One verification runs at a time per panel; another start gets `409 restore_verification_busy`.
  - **Node databases:** the panel pulls the PostgreSQL engine image on the panel host first when it is missing there.

### Fixed

- A panel that crashed or restarted during a restore verification left two things behind:
  - a second decrypted copy of the backup in its temp directory, which no cleanup matched;
  - the helper container, holding the dump and the restored data.

  The copy is gone, because the dump is now streamed straight into the helper. Housekeeping now removes helpers older than an hour.
- Running out of the helper's bounds no longer reports the backup as `failed`. This covers the 1 GiB scratch, the memory limit, a dump over 512 MiB, and an engine image that cannot be pulled. The verdict is now `unverifiable`; `failed` means the restore or the data itself failed.

### Verification

- Hosted real-PostgreSQL restore acceptance passed on the fixed adapter (run 41840272687). It covered plain and custom dumps, a wrong expected value, invalid SQL, a missing extension, cleanup and source preservation.
- The release user-journey smoke now runs the public flow on the published image. It takes a fresh backup of imported rows, checks that `count(*)=3` passes and that a wrong expectation fails, checks that a concurrent start gets 409, and checks that no helper is left behind.
- Local results:
  - full server suite: 13,354 tests passed, with the coverage gates met;
  - web: 2,770 passed;
  - SDK: 100% coverage;
  - CLI: 1,057 passed.

## [0.18.19] - 2026-11-04

### Fixed

- Prepare fixed-host-port Docker candidates before stopping the serving runtime. A creation failure leaves the previous application serving; a successful preparation hands off ports immediately before candidate start.
- Preserve the serving runtime when cancellation arrives during candidate preparation. Cleanup removes only the interrupted candidate.
- Apply the guarded handoff to multiline environment deployments through Compose create/start, preserving decoded environment values and named volumes.

### Deployment behavior

- Application startup with direct host ports remains sequential. This patch does not enable persistent TCP/UDP ingress or remove the startup interruption of all-in-one stateful applications.

### Verification

- The actual builder, Docker and Traefik acceptance preserved 131 successful HTTP responses across injected creation-acknowledgement failures in plain and multiline environment cases. The stopped candidate, serving old runtime, prepared successful start and persistent-volume marker were checked.
- 200 local tests across seven covering suites passed, including four assertion-based before/after regressions. Full hosted CI and primary-network readiness passed before release preparation.

## [0.18.18] - 2026-11-04

### Fixed

- A sleeping node service could not be woken by visitors after one of its domains was edited. Adding a header, turning SSL on or off, or any other domain change made every wake demand fail, and visitors saw "Waking up" until someone pressed Wake or redeployed. Changing a health setting (path, check, command or timeout) had the same effect. Wake now goes through after either kind of edit. Sleep admission already binds the routes to the idle evidence, and readiness uses the current health settings. Services that went to sleep on 0.18.17 still wake after the upgrade (R2).
- On a panel with several nodes, a wake could stall when other nodes were unreachable. The proxy update now goes to the waking service's own node first, ahead of the rest of the fleet. Before, nodes were updated in id order, and two unreachable nodes ahead of it could use up the 30 s publication window, so the wake was refused on every retry. Deploys and autoscaling also update their own node first (R2).
- A node service's wake could stay unfinished while two other nodes were unreachable, even after the waking node had already loaded the app route. The next routing update then sent visitors back to the "Waking up" page, and every retry failed the same way. Wake now completes once the waking node confirms the route (R2).
- Changing the CPU or memory limits of a sleeping node service left it unable to wake on visits: the waking page stayed up and the status said "waking" until someone pressed Wake. While a node service sleeps, or is going to sleep or waking, new limits are now saved and apply on the next deploy (R2).
- With `NINEDEPLOY_NODE_SLEEP_AUTOMATION=off`, a node sleep interrupted before its containers stopped (for example by a panel restart) left the running service's domains on the waking page with no way back: Wake answered "not asleep" and the panel showed no Wake button. Wake now accepts that state and the panel offers it (R2).
- Terminal & exec explains why it closes when the panel is opened at an address other than its configured one. Before, the terminal only showed "Access was refused: the session was revoked, or this page is not a panel origin." Now it names the page's address and says to use the address in `NINEDEPLOY_PUBLIC_URL`, or to add this address there or to `NINEDEPLOY_CORS_ORIGINS`.
- Port mappings under Service → Network showed only the starting container port of a saved range. They now show the complete range, a typed starting port expands to the range on blur, and an explicit range that does not match the host range is refused. The field is wider so both ends stay visible.

## [0.18.17] - 2026-11-03

### Added

- Automatic node sleep after continuously verified idle traffic, guarded demand wake through immutable runtime identities, readiness-held route restoration and panel restart recovery. Requires the node traffic capability and configured sleep policy; the automation flag can pause automatic actions. Public status reports observation, held and waking states without exposing execution credentials.
- Real single-manager Swarm acceptance for Docker migration, pinned artifact promotion after source HEAD advances, panel restart recovery on the original deployment and recorded rollback. Require this cohort and public manual/automatic node wake plus lost-routing-acknowledgement recovery on the pushed release image before promotion.
- Internal PostgreSQL restore-verification foundation using isolated networkless helpers, strict SQL/archive restore, known-data probes and confirmed cleanup. Public restore verification controls, other-engine verification and PITR remain unfinished; ordinary backup checks still validate dump structure.

### Changed

- A compose stack's domain now picks its service in its own row under Service → Network → Domains, next to the domain's other settings. The separate card under Service → Settings is gone. Save appears only once the choice differs from the stored one (G2.1).
- The autoscale status (`GET /v1/services/:id/autoscale`) lists `supportedMetrics`, the metrics that act where the service runs. A node service set to `rps` now shows why it holds, with no value, instead of "waiting for node autoscale evaluation". The panel marks a metric that does not act as "not available here", and the CLI lists the metrics under "Metrics here" and warns when `autoscale set` stores one that does not act (R2).
- Node services can autoscale on requests per second (`rps`). The node agent must run with `NINEDEPLOY_AGENT_PROXY_TRAFFIC=on` and serve a routed domain. The autoscaler counts the requests the node proxy served to the replicas in a 10 s window every 30 s. A lost measurement holds the current count; it is never read as zero. Before, a node service set to `rps` never scaled (R2).

### Fixed

- Route manual node wake through durable remote Start rather than starting a same-name panel container. Report pending lifecycle work in the API, SDK, CLI and panel until readiness and routing settle.
- Preserve the healthy application when a pre-Stop routing acknowledgement is lost, recover staged catcher retirement through ordered barriers, and refuse stale routing writes after retirement. Serialize managed database transactions across users of the same database handle.
- Clean partially fetched remote drill files after a failed download. Restore-verification history retains coarse verdicts and timings without persisting SQL probes, expected values or restored results.
- A service cannot move to a node or gain a fan-out node while its maintenance mode, custom error pages or panel sign-in are on. Node proxies do not route what the panel serves. After a move, the setting stayed stored where its card no longer showed, and came back on a move back to the panel host. A protected domain went dark on the node (G2.2, G2.3).
- Maintenance mode and custom error pages are refused for a service that also runs on fan-out nodes. Those nodes kept serving the app while the panel host showed the maintenance page (G2.2).
- A deploy that builds on a node or a build server is refused up front when the service has an inline Dockerfile, with a reason that names Build on: panel. The file is written only into the panel's checkout. The node cloned the repository again and the build failed on a missing `.ninedeploy.Dockerfile`, and the docs claimed node builds worked (G1.1).
- Cloning a service now copies its inline Dockerfile. The copy built the repository's own Dockerfile instead, or had nothing to build for a "Dockerfile only" service (G1.1).
- The restart-loop limit can no longer be turned on for a node service or a Swarm service, and its card is hidden there. The panel never samples those containers, so the saved limit never acted. A service with the limit on cannot move to a node (G1.3).
- Accepting a domain transfer clears the domain's compose route and panel sign-in. The route still named a service of the old stack, so the recipient's stack sent the domain to its own service of that name, or to nothing (G2.1, G2.3).
- Node serving-traffic receipts are pruned by housekeeping. After 7 days a receipt is deleted unless it is the newest for its node source, which still refuses a replayed interval. Before, the table grew without bound (R2).

### Security

- The Hub now says when an app is open before it is set up. The first visitor may become its admin, it may start with a publicly known login, anyone may be able to sign up, or it may have no login at all. 174 templates carry the new `openFirstRun` field, and the deploy wizard and the template detail show the warning. The Hub list marks those cards "open until set up". It is stronger when the app also has the Docker socket.
- The Coolify Portainer, Authentik and Glances templates are withdrawn from the bundled Hub. Each combines host Docker access with an open first run. Stacks already installed from them keep working ([docs/TEMPLATE_MIRRORS.md](docs/TEMPLATE_MIRRORS.md) §4).

## [0.18.16] - 2026-11-01

### Fixed

- Resolve a local Docker container's IPv4 address from its configured primary network, by name or full network ID. Preserve legacy default/bridge compatibility. Multiple attached networks previously produced concatenated addresses, and a detached primary could silently fall back to a private endpoint.
- Bind strict HTTP and TCP readiness to that selected primary address and require it to remain unchanged after the probe. A listener reachable only on a private network must not pass when Traefik cannot reach the application's primary endpoint. The command readiness port gate inherits the corrected TCP behavior; explicit running-mode policy remains unchanged.
- Require real Docker/Traefik primary-network acceptance in candidate CI and the published-image release matrix. The isolated regression compares the unchanged 0.18.15 image with the candidate across single/multiple networks, primary detachment/restoration and a private-only listener. Proxy errors injected by the negative controls must correspond to refused readiness; they are not successful serving responses.

### Added

- 7 more Dokploy blueprints in the bundled Hub (G4.7): Cut, Flatnotes (TOTP), HortusFox, Kokoro Web, MAZANOKE, Omni-Tools and Rustrak. Their upstream files use `latest`; each image is pinned here to a release at least two weeks old. Each started in an isolated smoke run, and asked without credentials for its admin pages, APIs and sign-up, answered with a login, a 401 or 403, or (MAZANOKE, Omni-Tools) is a static page with no server-side data.

## [0.18.15] - 2026-11-01

- Carries the readiness, remote MCP/OAuth, CI runner and runtime increments documented under the unpublished 0.18.14 candidate below.
- Start collecting at least 20 real traffic outcomes before submitting a canary smoke deployment. A fast owner-opt-out refusal could previously finish before the collector sampled enough requests, failing the release's minimum-evidence assertion. Keep the minimum sample and zero-unexpected-error requirements; retain early errors and the complete synthetic traffic trace.
- Require the public canary lifecycle smoke in normal candidate-image CI, in addition to the final published-image release matrix.

## [0.18.14] - 2026-11-01

Unpublished candidate: its Git tag and signed registry image are retained. Release workflow 41254561302 passed the full source gate and six smoke cohorts, then withheld GitHub/npm publication after the public canary smoke's insufficient-traffic assertion.

### Fixed

- Respect native Docker health before default HTTP, strict HTTP and TCP readiness can admit a replacement. Probe a non-loopback container address so a localhost-only listener cannot pass routed readiness. Reject command and running verdicts completed after their original deadline.
- After a command healthcheck succeeds, also wait for the configured TCP application port before switching traffic. Portless workers retain command-only readiness. Running mode retains its explicitly selected policy, including UDP workloads. The isolated delayed-listener panel/Traefik acceptance passed all five modes; this is not a guarantee for every production failure timing.
- Keep panel session updates usable if the desktop credential bridge throws synchronously.

### Runtime scope

- Prepare immutable node serving and spare replicas, collect fresh node autoscaling metrics and fence scaling against deployment, lifecycle and database topology changes. Public memory autoscaling with PostgreSQL passed its isolated 1→2→1 acceptance. RPS scaling, node sleep/wake, broader fan-out and the full R2 runtime acceptance remain unfinished; this release does not complete R2 or claim 0.19.0 readiness.
- Rehearse upgrades from the latest published image in CI and require continuous normal-deploy traffic across five healthcheck modes in the published-image release matrix.

### Added

- 10 more Dokploy blueprints in the bundled Hub (G4.7): 9Router, Dolibarr, Graylog, Kimai, Meilisearch, Miniflux, OpenObserve, Ponos, PrivatePaste and Typesense. Each started in an isolated smoke run and was then asked, without credentials, for its admin pages, APIs, install and sign-up pages; each set its admin login or API key from a generated value. YOURLS started too but stays out (its install page needs a manual step without a login).

- CI jobs on your own runners (R5; [docs/CI_RUNNERS.md](docs/CI_RUNNERS.md)).
  - A service's pipeline is an image and a script. Operators enrol runners, which pull jobs over HTTPS with their own token (`node apps/server/dist/ciRunner.js` in the panel image).
  - Each job runs in a throwaway container with the source the panel checked out (no `.git`, so no credential) and only the service variables the pipeline names. The container gets no Docker socket and no runner or lease token.
  - Jobs and attempts are kept in the database (migration 0117). A lease decides which attempt counts: a lost runner's job is requeued, its late result is refused, and its leftover containers are removed when it starts again. A panel restart renews live leases.
  - Approval, runner labels and capacity are supported. Logs arrive as numbered chunks and are kept for 30 days.
  - `runOnPush` runs a job for every push to the service branch (skip markers apply; one job per commit).
  - `gateDeploys` holds every deploy of the service and of its PR previews in `awaiting_ci` until a job at the deploy's commit passes. The deploy then builds the commit the job pinned; a failed or cancelled job fails it. A job for a pull request from a fork waits for an admin, and a fork preview deploy without a commit fails rather than run whatever the branch holds later. Runner-gated deploys never take the GitHub workflow gate's polling slots. Migration 0118 links a job to its deploy.
  - The panel has a **CI** tab on each service (pipeline, jobs, attempts, followed logs) and **Settings → CI runners**. The SDK has `client.ci`; the CLI has `ninedeploy ci` (runners, pipeline, set, jobs, run `--follow`, log, approve, cancel).
  - The attempt log route now answers JSON (`{ data, size, done }`) instead of text with headers.
  - Each job runs on a bridge network of its own, so it cannot reach other containers on the runner machine or another job. `NINEDEPLOY_RUNNER_NETWORK=none` takes the network away entirely. A starting runner removes job networks left empty for over ten minutes, even under an old token.
- A desktop client (R6, [docs/DESKTOP.md](docs/DESKTOP.md)). `desktop/` builds unsigned installers for Windows (NSIS), Debian (deb), Linux (AppImage) and macOS (dmg).
  - You add a panel by its address, which must answer `/health`, and open it in a sandboxed window with its own storage partition.
  - The panel keeps its refresh token in the OS credential store (DPAPI, Keychain, libsecret), so a sign-in survives closing the window. Without a real store nothing is written.
  - Other sites open in the system browser, apart from an SSO sign-in started on the panel.
  - Removing an instance revokes only that computer's session on the panel.
  - The panel's web UI picks the bridge up by itself; in a browser nothing changes.
  - The project keeps its own lockfile, so Electron stays out of the panel's dependency tree. A new Desktop workflow tests, packages and launches it on all three platforms.
- Remote MCP over HTTP (R4, first part). The panel serves its MCP tools at `/v1/mcp` over the Streamable HTTP transport, for clients that send an API token as `Authorization: Bearer` (for example `claude mcp add --transport http`). Each tool call runs as that token through the API's own authorization, so scopes, roles and revocation apply per call. Off by default: Settings → Security → Remote MCP, or `PUT /v1/mcp/policy`; an optional switch limits every caller to read-only tools. Clients that sign in instead of sending a header use OAuth 2.1 as the MCP authorization spec describes: discovery documents, dynamic client registration, a consent page in the dashboard (read only or full access), PKCE, rotating refresh tokens (a reused refresh token ends its grant) and revocation. Each grant shows under Settings → Security → API tokens, where deleting it disconnects the client; its tokens work only on /v1/mcp. No migration.
- The image scan deploy gate now covers Compose stacks on the panel host (G5.1). The stack's images are scanned after pull and build and before the running stack is taken down, so a blocked deploy leaves the previous stack serving. Build-only services are scanned by the image name Compose gives them. No migration.
- The restart-loop limit now covers PM2 services too (G1.3, Service → Settings → Restart-loop limit). It counts PM2's restarts per process, leaving out the ones the panel asked for, and stops the service as the Stop button does. PM2's own `max_restarts` only catches crashes within a second of starting; an app that crashes later looped on until now. No migration.
- Bare-metal installs on openSUSE, SLES and Arch now set up Node themselves (G4.1). On openSUSE and SLES it comes from NodeSource's RPM repository through zypper; every package signature is checked against NodeSource's key, as on dnf hosts. On Arch, Manjaro and EndeavourOS it comes from the distribution's LTS package of that line (`nodejs-lts-krypton` for Node 24) with `npm`, without upgrading anything else. A Node package of another line already installed on Arch must be removed first.
- Container options on Swarm services (ROADMAP item 10). Init, the open-files limit, added capabilities and shared memory now apply to each Swarm task: the first three as Swarm's own task fields, shared memory as a sized tmpfs at `/dev/shm`. GPUs have no Swarm equivalent and stay off; the deploy log says so. A Swarm service with these options set ignored them before; its next deploy applies them. No migration.

## [0.18.13] - 2026-10-30

### Fixed

- Probe a node service's port before promotion when no explicit readiness mode is configured. Explicit HTTP, TCP, command and running modes retain their selected policy. A TCP probe confirms a listening port; configure strict HTTP readiness to require a successful application response.
- Bind node readiness and primary promotion to recorded container identities, recheck container state, and refuse missing sealed agent capabilities before mutation. Older agents that cannot provide the required checks must be updated before the affected deployment proceeds.
- Preserve the existing fan-out generation when creation, routing or cleanup is unconfirmed. Persist candidate creation identities, hold ambiguous outcomes for review and reconcile only confirmed ownership.
- Use the claimed recovery state and recheck retry eligibility after concurrent recovery attempts. Do not report a target without a serving runtime as running.

### In progress

- Node autoscaling contains preparation and recovery infrastructure but is not yet available end to end. It remains disabled for node services until deployment integration, fresh identity-bound metrics and real-system acceptance are complete.

## [0.18.12] - 2026-10-30

### Added

- Nodes on a tailnet (ROADMAP item 11). SSH bootstrap can join a node to a Tailscale or Headscale tailnet (Servers → Add server → Join a Tailscale tailnet; `privateNetwork` on `POST /v1/servers/ssh-bootstrap`). The agent is then published on the node's tailnet address only, and the node is registered under it. The auth key travels on the SSH session's stdin and is not stored. The panel host must be on the same tailnet. No migration.
- openSUSE Leap, Tumbleweed, SLES, Arch, Manjaro and EndeavourOS in `install.sh` and in node SSH bootstrap and cloud-init (G4.1). Docker comes from the distribution's own `docker`, `docker-compose` and `docker-buildx` packages, enabled with systemd; on SLES the Containers Module must be active. The installer does not install Node there: the bare-metal mode needs it already present. On Arch nothing else is upgraded.
- Disk in the usage report (ROADMAP item 9). Once an hour the panel records the size of each service's volumes, on the panel host and on nodes, and Monitoring → Usage and cost shows it in GB-hours. No migration.
- File mounts on Swarm (ROADMAP item 10). A Swarm service's files (Service → Volumes → Files) are mounted read-only in every task, each one as a Swarm secret whose name carries a keyed version of its content; an edited file rolls the tasks, and the secrets no deploy uses any more are removed, keeping the rollback target's. Swarm services with files used to be refused at deploy time. No migration.
- PgBouncer for a postgres database on a node (ROADMAP item 10, agent capability `db.pgbouncer`, agent 0.18.12). The node's agent runs the same sidecar as the panel host, on the database's own bridge, with no port of the node published; attached services connect to `nd-pgb-<slug>`. `NINEDEPLOY_AGENT_PGBOUNCER=off` switches it off on a node. No migration.

### Fixed

- Apply resource limits to the owned replicas of a running node service, including fan-out targets, after checking agent capabilities and runtime ownership. Older agents and unconfirmed or partial updates report that another deploy is needed; clearing a limit still takes effect on the next deploy. Node agents need the `docker.limits` capability.
- Prevent a node service's resource-limit update from targeting a same-named container on the panel host, and verify every local replica before reporting a live update.
- Explain when an initial Stop or recovery is held because a node agent needs an update; recheck placement during recovery.
- Protect upgrade-smoke cleanup with per-run ownership labels and immutable container identities.

## [0.18.11] - 2026-10-30

### Added

- Public access to a database on a node (ROADMAP item 10, agent capability `db.publicAccess`, agent 0.18.11). The node's agent runs the same allow-listed TCP sidecar as the panel host, on the database's own bridge, and clients connect to the node's address. TLS stays end to end (`tlsMode: none`); termination remains panel-host only. `NINEDEPLOY_AGENT_PUBLIC_DATABASES=off` switches it off on a node. No migration.
- Usage and cost (ROADMAP item 9), under Monitoring → Usage and cost (operators). Shows CPU core-hours, memory GiB-hours, egress and requests per service, project and workspace over 24 hours, or 7, 30 or 90 days. Each server's monthly cost is split among its services by their share of its CPU and memory. Read from the existing resource history and traffic analytics. Routes `GET /v1/usage`, `GET`/`PUT /v1/usage/costs`, SDK `usage`. No migration.
- Nodes created as Vultr instances (G4.3, after Hetzner and DigitalOcean). Servers → Vultr takes an API key and creates a tagged instance whose user data installs Docker and starts the agent. The node is rekeyed as soon as the agent answers, exactly like a Hetzner VM. Routes `/v1/cloud/vultr/*` (operator), SDK `cloud.vultr`. `providerServerId` in the cloud server list is a string for Vultr nodes (Vultr names instances by UUID). No migration.

### Fixed

- Initial node deployments can be stopped and followed by another deployment without leaving the node's runtime journal permanently fenced. The panel waits for the Stop handoff before claiming its successor, keeps stale deployment IDs fenced, and can recover a Stop completed by an older panel. Upgrade the node agent with the panel before using the new handoff.
- Database deletion retains a record when node public-access proxy removal is unconfirmed, allowing the removal to be retried.

## [0.18.10] - 2026-10-29

Includes the previously unpublished 0.18.5–0.18.9 changes. Existing tags are preserved.

### Added

- Commit statuses on GitLab, Gitea and Bitbucket Cloud (ROADMAP item 8). A service that clones through a token source of one of those types can mark each deploy on its commit (`ninedeploy/<slug>`: running, then success, failure or cancelled). Turn it on under Service → Settings, or with `PUT /v1/services/:id/commit-status` (service admin). The token goes only to the repository's own host, fork previews never report, and a failed status never fails a deploy. No migration.
- Error-rate and latency alerts (ROADMAP item 7). Two new alert metrics, `error_rate` (percent of requests answered 5xx) and `latency_p95` (ms), read the last 5 minutes of a service's traffic analytics. They are per service, need traffic analytics on, and skip a service with fewer than 20 requests in the window. A `.ninedeploy` manifest declares them as the alert triggers `highErrorRate` (`thresholdPct`) and `slowResponses` (`thresholdMs`). No migration.
- Nodes created as DigitalOcean droplets (G4.3, after Hetzner). Servers → DigitalOcean takes a personal access token and creates a tagged droplet whose user data installs Docker and starts the agent. The node is rekeyed as soon as the agent answers, exactly like a Hetzner VM. Routes `/v1/cloud/digitalocean/*` (operator), SDK `cloud.digitalocean`. No migration.
- Node Dockerfile builds take a build stage and build arguments (agent capability `docker.buildOptions`, agent 0.18.10). A stage used to be refused on a node and build arguments were left out. The values travel only inside a sealed request and are masked in the build output. An older agent keeps the old behaviour, and the log names the version to install. `NINEDEPLOY_AGENT_BUILDS=off` switches it off with the other node builds.
- AI alert investigations (ROADMAP item 6), opt-in under Settings → AI assistant. When a service's alert rule fires, the panel sends the service's newest runtime logs (sanitized) and the alert to the configured provider, and shows the answer under the rule in Monitoring. At most 12 an hour; a protected lane is never investigated; notifications never carry the logs or the answer. No migration.

### Fixed

- Stop during a node deployment now fences delayed container creation, stops the selected and candidate generations, and retires only the interrupted candidates. A later deployment can start normally. Updated node agents advertise the required capabilities; older agents keep the handoff pending until upgraded.
- `ninedeploy tunnel` kept nothing a client sent before the target was connected, so a client that speaks first (psql, an HTTP request) waited forever. This was certain on nodes and possible on the panel host. Node database tunnels were also refused as not running.

## [0.18.9] - 2026-10-29

Includes the previously unpublished 0.18.6–0.18.8 changes below. Existing Git tags remain unchanged.

### Fixed

- Bound Docker readiness retry waits to the remaining deadline, including when a container has not yet acquired an address.
- Allow empty encrypted backups to be read while still checking their authentication tag.
- Remove a node upload's temporary file when its stream closes before the file finishes opening.

## [0.18.8] - 2026-10-29

This release includes the previously unpublished 0.18.6 and 0.18.7 changes described below, along with subsequent verification and fixes. The v0.18.7 Git tag remains unchanged.

### Fixed

- Clear stale GitOps apply results after a new check or configuration update, and allow removal of a GitOps configuration.
- Refuse malformed Dokploy service definitions instead of treating them as safe to import.
- Strengthen release verification with additional tests for node tunnels, runners, image scans, Compose volumes, service moves and panel workflows.

## [0.18.7] - 2026-10-29

### Added

- GitHub Actions runners can run on a node (G4.5). Pick the node under the App's **Run on**. The node's owner must opt in with `NINEDEPLOY_AGENT_GHA_RUNNERS=on` (agent 0.18.7, `gha.runner`). The agent applies the panel host's firewall isolation to its own host before every start and receives the just-in-time configuration only sealed.
- `ninedeploy tunnel service` and `ninedeploy tunnel db` reach docker services and databases on nodes (G5.8). The panel asks the node agent, in a sealed request, for a single-use channel encrypted per tunnel. The agent connects from inside the container's network namespace through a helper with no capabilities. Needs agent 0.18.7 (`tunnel.tcp`); `NINEDEPLOY_AGENT_TUNNELS=off` turns it off.
- The image scan deploy gate covers docker services on nodes (G5.1). The node agent scans the new container before it gets traffic; a node that cannot scan records why and does not block.
- Compose stacks move to another host with their data (G3.1). Every named volume compose made for the stack is copied to the target, with compose's labels, before the stack is deployed there. Stacks with a fixed volume name are refused. Nodes need agent 0.18.7 (`volume.compose`; `NINEDEPLOY_AGENT_VOLUME_COMPOSE=off` turns it off).
- Image scanning covers Swarm services (G5.1). The scan runs on the panel host, the manager, against the image the service spec runs, on the schedule and on request. If that image is not present on the panel host, the result says so.
- Compose stacks on nodes also skip the build when the commit and the resolved compose model are unchanged (G2.4). The resolved model is hashed on the panel and never logged. Older agents build as before.
- Image scanning covers docker services on nodes (G5.1).
  - **How:** the node agent runs the same `docker save | trivy` with its own cache volume and answers the parsed summary, which the panel validates before storing.
  - **When:** on the schedule and on request; an unchanged image is not scanned again until the result is stale.
  - **Agent:** needs the new capability `image.scan` (0.18.7); the node owner turns it off with `NINEDEPLOY_AGENT_IMAGE_SCAN=off`. No deploy gate on nodes yet.
- 3 Dokploy blueprints in the bundled Hub, each smoke-run in isolation and reviewed first (G4.7): AList, Healthchecks and Dagu. Entries that started but are not safe to expose by default stay out: mutable `latest` tags, known default logins, open registration, auth off, or a first visitor becoming admin. The registry holds 308 templates. The converter refuses host access: bind mounts, builds, env files, isolation-loosening keys (TEMPLATE_MIRRORS.md).
- Backups and restores of ClickHouse, Meilisearch and RabbitMQ databases on nodes (G1.8). The agent runs the panel host's routines against its local Docker and streams the result through the sealed channel.
  - **Agent:** needs the new capability `db.dump.extra` (0.18.7); an older agent gets 422 `node_agent_outdated`.
  - **Meilisearch:** the restore reads the image and data volume from the container, never from the request.
  - **Proof:** a dump, index deletion and restore against a real Meilisearch v1.53.
- Image scanning covers Compose stacks on the panel host (G5.1). Every distinct image of a running stack is scanned on the schedule and on request (`POST /v1/services/:id/image-scan`; Scan now on the Deploys tab once the stack has a result). The result is merged into one, and each finding names the compose services whose image has it. A stack has no deploy gate: `compose up` starts it at once.
- `ninedeploy tunnel service <id> [--target <port>] [--replica <n>]` (G5.8): a container port of a panel-host docker service on a local port, over the same WebSocket bridge as the database tunnel.
  - **Access:** the service admin role (it reaches the container past its domain protections), re-checked every minute.
  - **Limits:** 32 tunnels per user together with database tunnels; audited with byte counts.
  - **Route:** `GET /v1/services/:id/tunnel`.
- The service Network tab and the Domains page are translated too, so the whole panel is available in Turkish (G4.6).
- Settings → Integrations → Create a DNS record (G1.5): the domain-presets route by hand. It creates the A, AAAA or CNAME record for a hostname through the configured DNS provider (admins).
- Sentry-compatible error ingest (G5.4). Under Service → Activity → Errors, a service gets a DSN (`https://<key>@<panel>/<serviceId>`) for any Sentry SDK.
  - **Grouping:** events are grouped into issues by fingerprint: the exception type and its in-app frames, or the SDK's own `fingerprint`.
  - **Deploy link:** each issue names the deploy it first appeared under. With `release` set to the commit SHA, that is the deploy of that commit; otherwise it is the live deploy.
  - **Alerts:** new issues are audited as `alert.error_new`. A resolved issue that happens again reopens and is audited as `alert.error_regressed`.
  - **Ingest:** `/api/<serviceId>/envelope/` and `/store/` accept plain, gzip, deflate and br bodies up to 1 MiB. Limits: 600 requests a minute per client and 300 events a minute per service; past them SDKs get 429 with `X-Sentry-Rate-Limits`.
  - **Storage:** the key is stored hashed. Events are kept 30 days, and issues not seen for 90 days are removed.
  - **Who sees what:** issue titles and event payloads (exception messages, request headers, user data) are for members and up.
  - **Bounds (a browser DSN is public):** each issue keeps its first 5 and latest 15 events (32 KiB each), and retries are stored once. Past 500 unresolved issues (or 1,500 of any status), new ones fold into an overflow issue, which always alerts. Other alerts are limited to 10 per service per hour, then one `alert.error_burst`. Alerts never repeat the sender's text.
  - **Migration:** `0105_error_ingest` adds the `error_issues` and `error_events` tables and changes nothing else.

  See OBSERVABILITY.md → *Error tracking*.
- Instance GitOps (G5.3). Under Settings → GitOps, an operator points the panel at a YAML file in a repository that describes the instance: projects, their services, domains and project grants.
  - **Drift:** the panel reads the file on an interval or on request and lists what is only in the file, only in the panel, or different.
  - **Alerts:** a new drift is audited as `alert.gitops_drift`, and an unreadable file as `alert.gitops_failed`.
  - **Apply:** creates and updates through the ordinary API with the operator's rights. It never deletes.
  - **Export:** the panel can be exported as the file, to commit edits made in the UI.
  - **Rules:** the file is size-limited, secret-scanned and strictly validated.
  - **API:** routes under `/v1/gitops`; SDK `gitops`. No migration. See docs/INSTANCE_GITOPS.md.
- Move a service with its data (G3.1). Under Service → Settings → Move with data, an operator moves a stopped docker service between the panel host and nodes together with its `nd-svc-<slug>-data` volume.
  - **Steps:** the host change is the plain `PATCH /v1/services/:id { serverId }` with all its checks; then the volume is archived on the source, created and restored on the target (node transfers use the sealed, checksummed stream channel), and the service is deployed there.
  - **Limits:** the service must be stopped, not deploying, and have no extra volume attachments; Swarm, compose and PM2 services are refused. The source volume is kept until it is removed.
  - **API:** `POST /v1/services/:id/move` answers 202; `GET /v1/services/:id/move` shows the step or the error. SDK: `services.move`. No migration.
- SAML sign-in that works with real identity providers (G3.6). It is SP-initiated, verified with `@node-saml/node-saml` 5.1.0 (exclusive c14n, enveloped signatures).
  - **Binding:** responses are bound to the panel's own AuthnRequest and accepted once.
  - **Accounts:** like OIDC, an identity signs in only the account it was linked to from Settings → Account, within the provider's email domains.
  - **Where:** set up under Settings → SSO → SAML identity providers; the button appears on the sign-in page. The routes are `/v1/auth/saml/*`.
  - **Change:** `SAML_UNAVAILABLE` is lifted for new providers.

  See SECURITY_SSO.md → *SAML sign-in*.
- A domain per compose service (G2.1). Under Service → Settings → Domains per compose service, a domain of a compose stack routes to another service of the stack, on its port, instead of the main one.
  - **Variables:** the next deploy sets `SERVICE_FQDN_<SERVICE>` and `SERVICE_URL_<SERVICE>` for it.
  - **API:** `GET/PUT /v1/domains/:id/compose-route`. There is no migration.

  See TRAEFIK_INGRESS.md → *A domain per compose service*.
- Inline Dockerfile (G1.1). Under Service → Settings → Inline Dockerfile, a docker service builds from a Dockerfile pasted in the panel instead of the repository's.
  - **How:** it is written into the base directory as `.ninedeploy.Dockerfile` and built with the `dockerfile` pack.
  - **Dockerfile only:** a service with no repository and no image builds it from an empty context.
  - **API:** `GET/PUT /v1/services/:id/inline-dockerfile`. There is no migration.

  See DEPLOYMENTS.md → *Inline Dockerfile*.
- More log-drain targets (G3.4): Elasticsearch/OpenSearch (`_bulk`), Splunk HEC, Better Stack, New Relic, Axiom and AWS CloudWatch Logs (SigV4 `PutLogEvents`, creating the group and stream when missing).
  - **Delivery:** they use the existing shipper and queue.
  - **Validation:** a URL that does not name the index (Elasticsearch) or the group and stream (CloudWatch) is refused when the drain is saved. There is no migration.

  See OBSERVABILITY.md → *Log drain targets*.
- Panel sign-in in front of any app (forward auth, G2.3). Under Service → Settings → Panel sign-in, a domain can require visitors to sign in on the panel and hold a chosen role on the service. Traefik's `forwardAuth` asks the panel, so no extra container runs.
  - **Flow:** a one-time code on the app's domain, bound by a state cookie, sets a 12-hour session cookie.
  - **Revocation:** Sign out everywhere and role changes end access.
  - **API:** `GET/PUT /v1/domains/:id/forward-auth` and `POST /v1/forward-auth/grant`. There is no migration.

  See TRAEFIK_INGRESS.md → *Panel sign-in in front of an app*.
- Maintenance mode and custom error pages (G2.2). Under Service → Settings → Maintenance mode, for services on the panel host.
  - **Maintenance:** a service's domains answer a 503 maintenance page served by the panel. Addresses on a bypass list still reach the app.
  - **Error pages:** custom 502/503/504 HTML is shown through Traefik's `errors` middleware.
  - **API:** `GET/PUT /v1/services/:id/maintenance` and `/v1/services/:id/error-pages`. There is no migration.

  See TRAEFIK_INGRESS.md → *Maintenance mode and custom error pages*.
- Backup replica destinations (G3.3). Under Database → Backups → Backup replicas, an operator can name up to three more destinations for a database's scheduled dumps.
  - **How:** each replica receives the same encrypted dump after the main upload and keeps its own number of copies. A failing replica is shown on the card and never fails the backup.
  - **API:** `GET/PUT /v1/databases/:id/backup-replicas`. There is no migration.

  See DATABASES_BACKUPS.md → *Replica destinations*.
- Workspace-enforced 2FA (G1.4). A workspace admin can require every member to sign in with TOTP, a recovery code or a passkey.
  - **Effect:** any other session is held to an enrolment screen, and every other route answers 403 `two_factor_required`. Members set up a factor if they have none, then sign in again.
  - **SSO:** sign-ins through listed OIDC/SAML providers can be exempted. GitHub never is.
  - **API tokens:** existing ones keep working.
  - **API:** `GET/PUT /v1/workspaces/:id/two-factor` and `GET /v1/auth/2fa/requirement`. There is no migration.

  See SECURITY_SSO.md.
- Registry push webhooks (G1.7). Under Settings → Registry webhook, an image service gets a secret URL for Docker Hub, GitHub Packages, Harbor, a `registry:2` or a CI step.
  - **What it does:** a push of the service's own image and tag queues a deploy as the member who made the hook. Other pushes are ignored.
  - **Storage:** the URL is shown once and stored as a hash. The routes are `GET/POST/DELETE /v1/services/:id/registry-hook` and `POST /v1/registry-hooks/:serviceId/:token`. There is no migration.

  See DEPLOYMENTS.md → *Registry webhook*.
- Restart-loop limit (G1.3). Under Settings → Restart-loop limit, a docker service on the panel host is stopped when its containers restart N times within M minutes.
  - **How it stops:** the same way as the Stop button. It is audited as `service.restart_limit`, which notifies. A deploy in progress is left alone.
  - **Off by default:** set it with `PUT /v1/services/:id/restart-limit`. There is no migration.

  See DEPLOYMENTS.md → *Restart-loop limit*.
- Compose deploys skip the build when nothing that goes into it changed (G2.4).
  - **What counts as unchanged:** the same commit and the same resolved compose model (build contexts, Dockerfiles, args and environment) as the last build. The images are then reused; a missing image is still built.
  - **Building every time:** `NINEDEPLOY_COMPOSE_BUILD_SKIP=off` builds on every deploy, for stacks that need fresh base images.

  See DEPLOYMENTS.md → *Docker Compose Stacks*.
- Panel certificates shared with node proxies (G5.7). With `PUT /v1/traefik/certificates/shared` `{ enabled: true }`, a node proxy gets the panel's own ACME certificates (including a DNS-01 wildcard) for the hosts it routes.
  - **Why:** a domain served by several ingress nodes then has one certificate everywhere, and a wildcard is issued once rather than once per node.
  - **How the node behaves:** checked on Traefik 3, the node serves the shared certificate and skips ACME for the hosts it covers. An uploaded certificate still wins.
  - **Renewals:** they reach the nodes within the hour.
  - **Off by default:** it copies private keys to those nodes. There is no migration.

  See TRAEFIK_INGRESS.md → *Panel certificates shared with node proxies*.
- Pending changes tray (G5.2). Under Settings → Pending changes, a service can stage its edits instead of applying them one by one.
  - **Review:** the tray shows each change with its value now and after.
  - **Apply:** **Apply and deploy** (or **Apply only**) applies them together through the normal service PATCH with your own credentials, so every existing check applies. A deploy into an approval-gated lane waits for review. A refused apply keeps the tray.
  - **API, audit and storage:** `/v1/services/:id/pending` and `services.pending.*`. Each step is audited. There is no migration.

  See DEPLOYMENTS.md → *Pending changes*.
- Scheduled minimums and right-sizing hints (G5.6).
  - **Scheduled minimums:** an autoscaled service can raise its minimum in weekly time windows (`schedules` with days, from, to and a minimum, in an IANA `timezone`), set under Settings → Autoscaling. Windows may run past midnight. The highest floor in force wins, capped at the maximum, and the scaling event names the window.
  - **Right-sizing hints:** Settings → Right-sizing suggests setting, raising or lowering the memory and CPU limits from up to a week of metric history (95th percentile and peak). It is advisory, and needs at least 12 hours of history. `GET /v1/services/:id/sizing` returns the same.
  - **No migration:** schedules are part of the stored autoscale configuration.

  See DEPLOYMENTS.md → *Scheduled minimums* and *Right-sizing hints*.
- `ninedeploy tunnel db <id>` (G5.8). It offers a panel-host database on a local port through the panel, so the database port stays closed to the internet.
  - **How it works:** each local connection travels over its own WebSocket (`GET /v1/databases/:id/tunnel`) to the database container, byte for byte. Checked against a real PostgreSQL 15: SCRAM login, a 2 MiB row in and out, parallel connections.
  - **Access:** it needs the database admin role, the same as its credentials. Scoped API tokens must be allowed to write databases. Access is re-checked every minute.
  - **Limits and audit:** idle tunnels close after 30 minutes, and one user can hold at most 32. Tunnels are audited as `database.tunnel.open` and `database.tunnel.close`, with byte counts.
  - **UI:** the database page shows the command.
  - **Not yet:** databases on nodes.

  See DATABASES_BACKUPS.md → *Tunnels through the panel*.
- Image vulnerability scanning with Trivy (G5.1). It is off by default; an operator turns it on under Settings → Image scanning.
  - **During a deploy:** each docker deploy on the panel host has its new image scanned before it gets traffic.
  - **The gate:** it reports only, or blocks critical (or critical and high) findings. A blocked deploy fails before the switch, so the previous release keeps serving. By default the gate counts only findings that have a fix.
  - **Rescans:** running images are rescanned every 24 hours, configurable. New fixable critical findings are audited as `image_scan.critical`.
  - **Where results show:** each service shows its latest scan on its Deploys tab.
  - **How Trivy runs:** in a throwaway container that reads the image from `docker save`, without the Docker socket. It is pinned to `aquasec/trivy:0.69.3` by digest, which predates the malicious 0.69.4 to 0.69.6 releases (CVE-2026-33634). `NINEDEPLOY_TRIVY_IMAGE` overrides it.
  - **Failures:** a scan that cannot run never blocks a deploy.
  - **Not scanned yet:** services on nodes, Swarm and Compose.
  - **No migration.**

  See DEPLOYMENTS.md → *Image vulnerability scanning*.
- Deploy freezes (G5.5). An operator can hold deploys under Settings → Deploy freeze, or with `/v1/deploy-freezes`.
  - **Scope:** the whole instance, a workspace (with its environments), a project or an environment. Each freeze has a reason and an optional end.
  - **How it holds:** the queue checks freezes when it claims a deploy, so every trigger waits as `queued`: the UI, the API, git pushes, deploy hooks, schedules, image polls and rollbacks. The deploy log says why.
  - **When it ends:** when the freeze ends or is lifted, the held deploys start. Nothing is dropped.
  - **Audit and storage:** changes are audited as `deploy.freeze` and `deploy.unfreeze`. There is no migration; the freezes are stored in a settings row.

  See DEPLOYMENTS.md → *Deploy freezes*.
- Create nodes on Hetzner Cloud (G4.3). On the Servers page, an operator saves a Hetzner Cloud API token, then picks a location, server type, image and SSH keys.
  - **What the panel does:** it creates the VM and registers it as a node. The VM's cloud-init installs Docker for its distribution, using the same script as the SSH bootstrap, and starts the NineDeploy agent. The node turns online once the agent has been rekeyed (below).
  - **Rekey after first contact:** the first agent key's hash is in the VM's user data, which any container on the node can read from the metadata service. Right after the agent first answers, the panel replaces that key through the new `agent.rekey` op (agent 0.18.6). The new key is stored before the agent hears of it, and a sweep every 2 minutes finishes a rekey that a panel restart cut short. Until then the node shows as *rekeying*, **Test** and **Approve** are refused, and the agent itself (started with `NINEDEPLOY_AGENT_REKEY_REQUIRED=1`) runs nothing but `agent.ping` and `agent.rekey`. One rekey runs per node at a time, and a pending key is sent again rather than replaced. The agent ignores a rekeyed key that was made for a different `NINEDEPLOY_AGENT_TOKEN`.
  - **Key rotation:** the Hetzner token and the pending rekey keys are now re-encrypted with the other secrets when the master key rotates.
  - **Deleting:** **Delete** destroys the VM and removes the node. It is refused while the node still hosts services, databases or Swarm work.
  - **The token:** it is checked before it is stored, stored encrypted, and never shown back.
  - **No migration:** the token and each node's link to its VM are settings rows.
  - **Not tested against Hetzner:** the Hetzner API was only exercised through a fake in tests; no real server was created.

  See MULTI_NODE.md → *Creating nodes on Hetzner Cloud*.

- More Linux distributions (G4.1). `install.sh` now sets up a panel host on these distributions, not only Debian and Ubuntu:
  - **RHEL, Rocky, Alma, CentOS Stream, Oracle Linux and Fedora:**
    - Docker and Node come from Docker's and NodeSource's signed RPM repositories.
    - The packages that conflict with Docker are removed first: podman-docker, buildah and runc.
  - **Alpine:**
    - Docker comes from Alpine's own package and is enabled with OpenRC.
    - A host without systemd gets the Docker mode by default; asking for `--bare-metal` there is refused, because the bare-metal panel is a systemd service.
    - The installer also works with BusyBox's `flock`, which has no `-w`, and in a root shell without `sudo`. Before this, both made it stop with a misleading "lock held" error.

  The SSH bootstrap of a node installs Docker the same way, per distribution:
  - Rocky and Alma, which `get.docker.com` refuses, are now covered.
  - The script is sent base64-encoded through `sh`, so the node's login shell never parses it.

  Testing:
  - Detection was checked against Debian, Ubuntu, Rocky, Alma, CentOS Stream, Oracle Linux, Fedora, Alpine, Amazon Linux and openSUSE images. Amazon Linux and SUSE are reported as unknown.
  - The installer's Docker and Node installs were run in Rocky 9, Fedora 41 and Alpine 3.20 containers.
  - A full Docker-mode install was run on Alpine; it ended with a healthy panel.

  See README → Requirements.

- OS security updates from the panel (G4.2). For the panel host (Settings → System) and each node (Servers), operators can:
  - see the pending package updates, security ones first;
  - see whether the host is waiting for a reboot;
  - turn automatic **security** updates on or off.

  How each package manager is handled:
  - **apt (Debian, Ubuntu):** uses unattended-upgrades, with its automatic reboot turned off explicitly.
  - **dnf (Fedora, RHEL, Rocky, Alma):** uses dnf-automatic, with `upgrade_type = security`.
  - **Other package managers:** reported as unsupported.

  NineDeploy never reboots a host.

  How it runs:
  - Every action is a fixed script, run on the host through the same `nsenter` helper as host shells.
  - On a node, it goes through the new sealed agent op `host.packages` (capability `host.packages`, agent 0.18.6). The node's owner can turn it off with `NINEDEPLOY_AGENT_HOST_PACKAGES=off`.

  The scripts were run against real Debian 12, Ubuntu 24.04, Rocky 9 and Fedora 41 images, and through the nsenter helper on a Docker host. No migration. See NODE_OPERATIONS.md.

- Self-hosted GitHub Actions runners (G4.5). A GitHub App can now run Actions jobs on the panel host.
  - **How it works:** on a queued `workflow_job` whose `runs-on` names one of the App's runner labels, the panel registers a just-in-time runner and starts one `actions-runner` container for it. That container takes one job and is removed.
  - **Setup:** Sources → GitHub App → *Self-hosted Actions runners*. The section shows what the App still lacks. New Apps can request the event and permission from the setup form:
    - an organization App needs `organization_self_hosted_runners: write`;
    - a personal App needs `administration: write`.
  - **Isolation:**
    - no Docker socket;
    - its own network, firewalled off from the host itself and from private, cloud-metadata and CGNAT ranges, with internet access open. Without these rules no runner starts, unless `NINEDEPLOY_GHA_RUNNERS_UNISOLATED=1` is set;
    - CPU, memory and pid limits.
  - **Which repositories:** public repositories are refused unless allowed, and an optional repository list narrows it further.
    - GitHub enforces this as well: an organization runner is registered in an App-owned runner group, open only to the allowed repositories. Without that group, a job from another repository could take the runner.
    - Custom runner groups need GitHub Team or Enterprise; on another plan, organization runners do not start.
    - A personal installation registers repository runners, which serve only their own repository.
  - **Limits and queue:** jobs above the per-App limit wait and start as slots free up. Runners that never pick up a job are removed after 10 minutes.
  - **Image:** pinned by digest (`actions-runner` 2.338.2).

  No migration: the settings live in the settings table. See GITHUB_APP.md §7.1.

- Turkish interface (G4.6). The web panel now has a translation layer and a Turkish locale.
  - **Choosing a language:** pick it under Settings → Appearance → Language, or with the switch under the sign-in form. The choice is kept in the browser. Without one, a Turkish browser opens in Turkish.
  - **Translated:** every page except the service Network tab and the Domains page, about 4,580 strings. Server error messages, API values, product names and help topics stay English.
  - **Catalogue check:** a test fails on any untranslated string, unused entry or placeholder mismatch.

  No new dependency and no migration. See TRANSLATIONS.md.

### Changed

- The dashboard banner for a single unhealthy service now reads "1 service needs attention".

## [0.18.6] - 2026-10-27

### Fixed

- Pull the published candidate image on every independent release smoke runner before a smoke inspects it, and check the complete secret-provider list in upgrade and user-journey smokes.
- Keep local Docker retirement journals under the writable panel data directory, preventing permission errors from blocking later deployments and canary rollouts.

### Added

- Paketo and Heroku buildpacks (G4.4). Two new build packs, `paketo` and `heroku`, build with Cloud Native Buildpacks through the `pack` CLI.
  - **Pinning:** the Dockerfile and `install.sh` install `pack` 0.40.9 checksum-verified, as they do Nixpacks and Railpack. Each builder image is pinned by digest. Paketo's base builder covers amd64 only; Heroku's covers amd64 and arm64.
  - **Panel host only:** a service that a node builds is refused when it is saved, at deploy and in build placement. It never quietly falls back to a Dockerfile build. Choose **Build on: panel** to ship the image to the node.
  - **Build env:** it goes through a private env file, never argv, and is not written into the image.
  - **Untrusted builder:** builds run with `--trust-builder=false`, so the repository's own build code never shares a container with the Docker socket. `pack` would otherwise trust the Paketo and Heroku builders, run one container with the socket mounted, and let a member's repository take over the host.
  - **Caches:** each service keeps one pair of cache volumes, removed with the service. `pack`'s default would have left a new pair per deploy.
  - **Platform:** builds target the panel host's platform.
  - **Containerd image store:** Docker's containerd image store (the default for new Docker 29 installs) breaks the Heroku builder whenever the build has env. The deploy log names the fix.

  Both builds were run for real against the pinned builders with Docker 29: a Node app built with Paketo, served its page, and its build env was absent from the image config. Heroku built on this host only without build env; with env it hit the containerd fault. No migration. See PRIVATE_REPO_GUIDE.md §6.

- Dokploy template converter (G4.7). `scripts/buildDokployMirror.ts` turns a checkout of the Dokploy blueprint catalog (MIT) into a NineDeploy template bundle, next to the Coolify mirror.
  - **What it converts:** each blueprint's `.env` is folded into the compose file. Its generators become the matching `SERVICE_*` magic tokens: `${password:N}`, `${base64:N}`, `${hash:N}`, `${domain}`, `${username}` and `${email}`.
  - **What it skips:** any blueprint without a faithful equivalent, with the reason: file mounts, more than one routed service, `${uuid}` and `${jwt}`, among others.
  - **Results:** 383 of today's 586 blueprints convert. In a first smoke run, n8n, Umami and NocoDB started. The upstream Ghost blueprint did not: it has no `depends_on`, so Ghost restarts until MySQL is ready.
  - **Catalog:** nothing joins the bundled catalog until it passes the smoke run.

  The compose safety checks are now shared by both converters. `smol-toml` is a new dev dependency, used only by the build script. No migration. See TEMPLATE_MIRRORS.md.

- More secret managers (G3.2). New reference forms, configured under Settings → Integrations → *Secret managers*:
  - `${{ssm:/name}}` reads AWS Parameter Store with the existing AWS credentials.
  - `${{gcp:secret[@version]}}` reads GCP Secret Manager with a service account key.
  - `${{azure:name}}` reads Azure Key Vault with an app registration.
  - `${{op:vault/item/field}}` reads 1Password Connect.
  - `${{exec:name}}` runs an operator-defined command and uses its stdout. This covers any manager that has a command-line client, such as Bitwarden, Doppler, `op`, `pass` or sops.

  All forms except `op` also take `#jsonKey`. Exec runs without a shell and without the panel's environment. It stays off unless the panel was started with `NINEDEPLOY_SECRET_EXEC=1`.

  Like Vault and AWS references, the new forms stay literal (with a deploy-log warning) until their provider is configured, and the per-workspace allowlist governs them. `ninedeploy secrets providers test|delete` accept every kind. No migration. See SECRET_MANAGERS.md §3a.

- Automatic DNS records in Amazon Route 53, OVHcloud, Porkbun and Hetzner DNS (G3.5). The new card is under Settings → Integrations → *DNS records: Route 53, OVH, Porkbun, Hetzner*. Saving credentials there makes that provider the active one: a domain gets its A, AAAA or CNAME record when it goes live, and the record is deleted with the domain. **List zones** checks the credentials. Records that already exist are never changed.
  - Route 53 skips private hosted zones and needs no AWS SDK.
  - OVH refreshes the zone after each change.
  - Hetzner uses the Cloud API, since the old DNS API has been retired.

  The credentials are stored encrypted and re-encrypted by master-key rotation. Saving the Cloudflare card switched off no longer turns off a different active provider. No migration. See TRAEFIK_INGRESS.md §2.

- Five more notification channels (G3.7), all in the notification wizard:
  - **Mattermost** (incoming webhook) and **Google Chat** (space webhook; only `https://chat.googleapis.com` URLs are accepted).
  - **Matrix**: posts to a room with a bot account's access token. Each event has a fixed transaction id, so a retried delivery never posts twice.
  - **PagerDuty** (Events API v2) and **Opsgenie** (Alert API), US or EU region. These two receive only problems: fired alerts, a monitor or service going down, a failed deploy, a missed backup. The matching recovery, or the service's next successful deploy, closes the incident it opened. Every other event is skipped and never pages anyone. A channel test opens a test incident and closes it right away.

  `alert.fired` and `alert.recovered` events now carry the rule id, so recovery closes the right incident. No migration.

- Backups for ClickHouse, Meilisearch and RabbitMQ (G1.8): the three engines that answered "backup not supported" now have manual and scheduled backups, off-site copies, restore and the **Check** drill on the panel host. ClickHouse backs up every user database with `BACKUP … TO File` and restores each under a temporary name before promotion. Previous copies are retained for recovery on a failed promotion; a multi-database restore is not atomic and may require manual recovery. Meilisearch takes a dump; a restore stops the database and imports the dump with a one-off container into a new path, swapping it in only once the import is healthy. RabbitMQ backs up its definitions (users, vhosts, queues, exchanges, bindings, policies), not queued messages; a restore waits until everything in the file exists. Each flow was run against the real images. Node databases of these engines still refuse backups, now with a message that says why. No migration. See DATABASES_BACKUPS.md §10.

- Email templates in the web UI (G1.5): workspace admins can edit, preview and reset the workspace's email templates on the Workspaces page. Account emails such as the password reset are shown as fixed. Until now this needed `ninedeploy email-templates` or the API.

- Outgoing IP per project in the web UI (G1.5): Settings → Security has an operator-only *Outgoing IP per project* card that lists the egress rules and sets or removes one, picking the project and one of the panel host's IPv4 addresses. Traffic from that project's containers then leaves through that address (iptables SNAT). Until now this needed `ninedeploy egress` or the API.

- Firebase push (FCM) channels in the web UI (G1.5): the notification wizard creates an FCM channel from a device registration token and a Firebase service account JSON, checking that the JSON has the fields the server needs before continuing. Until now only `ninedeploy notifications create-fcm` could create one.

- Shipped-log search in the web UI (G1.5): a service's Activity Logs tab can search the lines its Loki log drain received, over a window from 15 minutes to 7 days, including lines of containers that no longer exist. Until now only `ninedeploy logs search` and the API could do this.

- Backup checks in the web UI (G1.5): each backup in a database's Backups tab has a **Check** button that runs the existing integrity drill, and a *Last check* column with the newest verdict (`passed`, `failed` or `unverifiable`). Until now drills ran only from the CLI and API.

- PgBouncer in the web UI (G1.5): a PostgreSQL database on the panel host gets a *PgBouncer connection pooling* card showing the sidecar's state, port, pool mode and the pooled connection URL to copy. Instance operators can enable it (with an optional port) or disable it there; until now this was only possible through the CLI, SDK and API.

- Self-hosted GitLab sources (G1.2): a `gitlab` source can now have a base URL, including a relative path such as `https://example.com/gitlab`, with the same https and private-address rules as Gitea. The project list, the branch picker and the live token test use that instance; without a base URL a source still means gitlab.com. GitLab branches are now actually listed, including projects in nested groups; until now they always fell back to `main`/`master`. Available in the web UI, `ninedeploy sources add --base-url` and the API. No migration.

- R2 development: persist plain-Docker node deployment authority in a private journal atomically with actual admission. Preserve queued predecessors, retain the original owner across configuration-snapshot rewrites, refuse old ownership after Stop or a newer deployment, and retain node identity across removal. Migration 0104 preserves existing work and deployment API payloads. Physical prepare/network/retire and scale/wake integration remain open.

- R2 development: accept a durable node Stop before the first deployment has selected a runtime. Cancel queued, building, deploying, CI-waiting and approval-waiting work atomically with runtime ownership, fence every accepted node, capture immutable managed generations and confirm routing withdrawal and physical Stop before completion. Nullable Stop anchors retain existing journal data and command revision high-water marks through migration 0103. The new protocol requires capable 0.19.0 agents; production deploy/scale/wake integration and full R2 acceptance remain open.

### Fixed

- Docs: ROADMAP.md no longer lists SAML as shipped. Creating a SAML provider has been refused since r354; OIDC and SCIM are the supported SSO paths.

- R2 development: fence superseded node deployment continuations after delayed successful or rejected agent replies. Keep newer owners and moved services intact, recheck SQL publication/failure writes at execution, retain unresolved effects for the current owner and suppress obsolete outcome callbacks. An in-flight SQL response cannot reverse an already observed ownership loss. Ordered physical mutation integration remains open.

- Ignore stale pipeline invocations for non-claimable deployment rows before build, routing, hooks or lifecycle outcomes; stop continuations that lose the conditional queued/building claim. Preserve eligible worker handoffs, explicit Stop cleanup and log-writer ownership.

- Clear an additional node's pending deployment status after its generation-independent Stop completes. Lost routing confirmation retains the captured targets for retry, while a newer runtime selection prevents the old Stop from resuming.
- Offer Stop during the first plain-Docker node deployment, show the pending verification state, refresh cancelled deployment history and hide Start while no node runtime has been selected.

## [0.18.5] - 2026-10-27

Unpublished candidate: its signed image and Git tag are preserved. A smoke runner did not pull the candidate image before inspecting it, and the user-journey smoke expected an outdated secret-provider list; 0.18.6 corrects both release gates.

## [0.18.4] - 2026-10-26

Supersedes the unpublished 0.18.3 candidate while preserving its Git tag and version image. Includes the deployment safety and node recovery maintenance package documented below; remaining R2 coordination still targets 0.19.0.

### Fixed

- Match the public lifecycle smoke owner-disabled refusal to the actual 0.18.3 agent capability floor, including subsequent maintenance releases. The previous 0.19.0-only test boundary held 0.18.3 publication after its upgrade, canary and node lifecycle checks passed. Regression tests compare the smoke policy with the production capability contract.

## [0.18.3] - 2026-10-26

Unpublished maintenance candidate for deployment safety, durable node lifecycle recovery and confirmed replica routing. The remaining R2 feature work targets 0.19.0.

### Added

- Persistent per-service runtime ownership for node lifecycle commands, with migration of pending commands and deleted-history revision protection. Superseded lifecycle continuations and retries lose authority. This foundation keeps existing agent replay identities; deployment creation, scale and wake integration remain pending.
- Agent capability `docker.lifecycle.generations` for one durable Stop across retained deployment generations. Each immutable target carries its own expected deployment label; Start keeps its single-generation contract. Explicit Stop batches are bounded at thirty targets, while existing requests retain their ten-target limit. The existing lifecycle owner switch disables both capabilities. Panel coordination for unresolved transitions remains R2 work.

### Fixed

- Drain already-admitted managed node creation under the same persistent lock as lifecycle effects, covering runSpec create/network/Start and runEnv. A timed-out Docker child now retains ownership until close, including SIGKILL escalation; an acknowledged Stop cannot be overtaken by that still-running child. Delayed request admission and complete handoff coordination remain separate R2 work.
- Bind ordered Start, Stop and Restart commands to the accepted primary and additional-node placements. Capture immutable container IDs per node, verify each node's generation and lifecycle receipt, require all serving targets to pass readiness, and confirm routing on every participant before completion. Offline nodes and uncertain receipts keep the original command pending; newer commands and changed placements supersede stale continuations.
- Render one backend for each additional-node target, matching the container actually created there, instead of copying the primary node's replica count into nonexistent target replica addresses.
- Add a two-daemon, sealed-agent and Traefik lifecycle smoke to the release gate, covering readiness refusal, offline recovery, lost effect/routing responses and a newer Stop.

## [0.18.2] - 2026-10-25

Supersedes the unpublished v0.18.0 and v0.18.1 candidates. Acceptance gates held publication; their existing Git tags and the v0.18.0 image were preserved.

### Added

- Guarded single-replica node canary for plain Docker services with explicit HTTP readiness, domain routing and no published host ports. Validate the selected agent before candidate startup, verify the stable traffic source before splitting, and keep measured promotion, cancellation, Stop and durable recovery authoritative. Panel and CLI distinguish a promotion request from completion.
- Sealed, owner-opt-in node proxy traffic reservations (`proxy.traffic`, agent target 0.18.2), with immutable runtime/source receipts, expected loaded HTTP/TLS routing-generation checks, request-sequence coverage and a final drain. Raw access logs stay on the node. Eligible node canary workers use this internal transport; update panel and agent together and enable observations explicitly on the node.
- Node database client terminals and private Web Studio for PostgreSQL, MySQL, MariaDB, Redis, Valkey, KeyDB and Dragonfly. Studio uses sealed operations and an encrypted relay, publishes no node port, follows the operator's live session/API grant, and confirms helper cleanup before database lifecycle changes. Migration 0100 stores private Studio identity receipts; update both panel and agent.
- Database-scoped resource telemetry, including database-only nodes, with explicit pending/unavailable/stale/stopped states. The SDK refuses an older panel that ignores the database filter instead of displaying another resource's metrics.
- Durable routed primary Docker transition recovery on the panel host and nodes: migration 0098 journals immutable replica identities before traffic changes, protects interrupted candidates from boot cleanup, and repeats readiness and loaded-routing acknowledgement before obsolete-generation removal. Settings/API/SDK expose pending reasons and safe retries; Stop/cancellation and newer owners remain authoritative.
- Recorded Docker artifact promotion and rollback can fan out the same immutable image to configured additional nodes without checkout or rebuild. Each target uses its current runtime environment and existing readiness/routing/recovery guards; transfer or identity failure preserves its previous runtime and is reported separately from the primary.
- Durable routed fan-out recovery after a panel restart, with leased transition records, fresh readiness/routing checks and cleanup by immutable container ID. Settings shows pending recovery, its reason and retry time; operators can retry through the panel or API/SDK without stealing a live deployment lease. Additive migration 0097 and a sealed `docker.recovery` agent capability (target 0.18.2) are required.
- Opt-in 1–90 day CPU/memory history with weighted rollups and longer service charts; authenticated Prometheus text export and OTLP/HTTP JSON gauges with encrypted collector settings. Configure through Monitoring, the API/SDK or CLI. Existing installs retain one day with export disabled.
- Durable encrypted per-destination captured-log queues and cursors, independent retry/backoff, bounded backpressure, destination-binding checks and master-key rotation. Panel and CLI expose backlog, scheduled retry and explicit discard. Source capture remains bounded and delivery is at least once. Additive migration 0096 creates the history/queue tables. See [Observability](docs/OBSERVABILITY.md) for limits and upgrade details.

### Fixed

- Admit bounded agent control traffic for readiness and measured canaries without exhausting the former shared 120/minute request bucket. Keep a separate 120/minute rejection response budget, enforce it for concurrent invalid requests, retain the 1,000/minute operation ceiling and preserve sealed authentication/replay checks.
- Close and await encrypted node volume/database backup writers before removing failed partial files. Integrity failures that arrive during a pending file open no longer leave orphan `.part` files.
- Wait for primary-node retirement to settle before changing the next readiness case in the release smoke. Preserve the existing HTTP failure, previous-runtime, metric-history and log-delivery assertions.
- Preserve sealed routing-generation confirmation during ordinary node proxy refreshes while a node canary split is active. Refuse disabled confirmation support, plaintext transport and stale receipts.
- Verify immutable node candidate groups through a sealed, bounded readiness batch when supported by the agent. Preserve three consecutive samples for every replica, recheck Docker health/restarts after application probes, keep older agents on their negotiated path, and refuse protocol loss without downgrade.
- Handle images without Docker HEALTHCHECK in batched readiness instead of failing on a missing State.Health field. Add a real Docker smoke for images with and without HEALTHCHECK and wire it into the release workflow.
- Bind primary canary traffic to the durable transition lease and record confirmed rollback routing before candidate retirement. The worker uses a shared local/node source adapter with immutable journal identities, mandatory node HTTP/TLS acknowledgement and no panel-traffic fallback.
- Fence canary continuations and parent cleanup by an opaque run owner, including retries of the same deployment. Late routing, source, readiness and close replies cannot overwrite a newer canary split. Check deployment ownership before creating canary state.
- Scope canary rendering to its primary host and stable runtime, pass explicit observation step context, and make the internal node-mode promotion wait for the current measured step to pass. The node worker rechecks workload and live agent requirements before candidate startup.
- Preserve initial readiness and stable-sample checks while giving periodic canary verification up to ten seconds, capped by the configured service deadline. Reuse a probe that already completed after the step boundary instead of burning the remote observation reserve with a duplicate check.

- Judge measured local canary steps from independent, bounded access-log windows and a final drain. Refuse unavailable/incomplete observations or unreadable analytics settings, verify the selected service and proxy/log source, coordinate rotation, and preserve a newer canary owner's routing after late replies.
- Refuse plaintext fallback for every `proxy.traffic.*` operation and preserve completed node-observation receipts for bounded retries after their active-window deadline.
- Invalidate a node observation when its routing/source check fails, discard private routing content on window closure, and verify the proxy did not restart during routing confirmation.
- Honor bounded HTTP 429 Retry-After responses on sealed agent calls within the original request deadline, preserving nonce/replay protection and cancellation. Remote readiness inspection, probes and diagnostics share the remaining budget and refuse late success. Packaged Docker healthchecks use the configured panel or agent endpoint for their role.
- Propagate canary leaf health to Traefik's weighted parent and probe single-container releases when HTTP-strict readiness is configured. Preserve accepted health query strings and percent encoding for canary and ordinary replica backends.
- Preserve Swarm's current and rollback placement while preloading a new image. Select an unused one of three fixed per-service slots across all service specs, use the full image identity, retain legacy constraints and refuse unconfirmed/exhausted placement instead of evicting serving tasks during preparation.
- Preserve explicit Stop during deploy readiness and omit stopped services from proxy routes. Pending primary recovery blocks same-service deploy claims, autoscaling and sleep while leaving other services deployable.
- Honor explicit HTTP/TCP readiness, resolved source ports and configured deadlines on fan-out targets; refuse missing readiness capabilities before starting a replacement. Routed fan-out targets require sealed routing acknowledgement before retiring the old runtime. Confirm rollback before removing a candidate, retain both generations on uncertainty, and guard target updates against replacement, deletion and service stop. Routed targets with published host ports are refused before retirement because they cannot overlap safely.
- Keep deployments in `deploying` with no completion timestamp until routing is acknowledged. Supersede the previous deployment only after success; preserve cancellation during the transition. Include this phase in worker claims, sleep/stop guards, webhook deduplication, diagnostics and failure recovery.

- Confirm local routed deploy/canary/autoscale transitions against the actual HTTP and TLS handler generation and enabled routing definitions before retirement. Routed primary-node deploys require a sealed agent advertising `proxy.routingAck` (agent target 0.18.2) and validate its generation receipt. Keep potentially routed containers alive when a write response, transition or rollback cannot be confirmed. Canary observation windows now begin after routing acknowledgement. The restricted internal probe is excluded from application routing tables; node canary and autoscaling remain separate work.
- Hold autoscaling on incomplete, invalid or stale CPU/memory samples and restart the load window after an unreadable sample. Guard serving-count changes against deployment/configuration races; restore the count and keep containers running when a proxy write fails. Drain active scaling operations during shutdown. Node and Swarm autoscaling remain unsupported.
- Keep autoscaling status errors visible with retry and highlight holding reasons. Resource telemetry now distinguishes missing measurements from real zero usage, labels historical fallback readings, and leaves empty charts empty.

### CI

- Give the serialized coverage job 30 minutes: the 20-minute limit canceled the run after all 8,929 server tests passed, before web coverage and the remaining guards finished. Test commands, coverage thresholds and required checks are unchanged.

## [0.17.1] - 2026-10-22

> HTTP readiness during deploys, recoverable Docker/Compose releases, required CI workflows and the October Node security baseline.

- Keep ordinary Docker HTTP readiness checks as HTTP after the direct-container-IP grace period. The sibling probe and exposed-port recovery now require a valid 200–499 response on the configured health path, rather than accepting an open TCP socket while the app still returns 502/503. Do not follow redirects to another host during direct readiness checks. Explicit strict HTTP/TCP/command modes retain their own contracts.

- Record ordinary node-built Docker artifacts with the actual running image ID and buildpack conventions. Retain a verified copy on the panel before saving the record, so promotion and rollback survive origin-node removal. Deployment details expose the historical build node. Optional capture failures keep legacy deployment behavior; configured registry archival remains required before traffic changes.

- Promote a panel-origin Docker artifact to the panel or one node without rebuilding. Use sealed verified stream transfer or an optional registry archive pinned by manifest digest and checked against the original image ID. Archives survive loss of the panel image and source history; credentials are re-resolved against their operator-approved host binding. Source builds configured with a push registry use the existing secret-export guard and fail before traffic changes if archival fails. Deployment cards show the archive reference; node runtime port conventions and candidate image checks are preserved. Swarm/fan-out and host deploy hooks remain unsupported.
- Durable artifact copies pin the original image/commit, retain target runtime configuration and still pass target CI/lane approval and protected-source secret authority. UI confirmation names the image and explains baked build-time values; the SDK/API retain commit promotion as the default. Additive migration 0095 leaves existing deployments unchanged. Retained artifact history protects image IDs from NineDeploy cleanup; missing unarchived images fail closed.

- Required GitHub Actions workflows per service: opt-in exact workflow paths, a durable `awaiting_ci` queue state, immutable commit pinning before lane review/build, run/attempt receipts and cancellation. Missing checks or API/permission errors never pass; completed non-success checks fail the candidate. Polling holds no build slots and resumes after restart. Additive migration 0094 defaults all existing links to no required workflows. New App manifests request Actions read permission; existing Apps need that permission before enabling the gate.
- Respect outbound request cancellation while DNS is still resolving, so stopping CI polling cannot wait indefinitely on a stalled lookup. A late DNS answer never starts the cancelled request.

- Require the October 21, 2026 Node security baseline (22.23.4+, 24.22.1+ or 26.12.1+ within supported majors). Pin Docker/CI to 26.12.1; check the runtime before release checks, and recheck the standalone installer's actual Node executable after upgrading. Preserve existing supported majors; a Node 26 security update must not switch the host to Node 24.
- Check recorded Compose image availability in recovery metadata and again before queueing a restore. Distinguish missing images from Docker connection failures, hide host error details, and recheck eligibility when opening confirmation.
- Keep image IDs referenced by retained Compose release records out of all NineDeploy image-cleanup paths, including dangling and automatic cleanup. Unreadable records stop image cleanup. Inventory/CLI and deploy recovery explain protection and when it ends.

- Recorded panel-host inline Compose releases: encrypted resolved configuration/environment and actual image IDs, copied into rollback jobs; restore without current YAML/env, pulls or rebuilds. Recovery eligibility and limits are shown in Deploys. Named volume data stays current; records require retained local images and a supported self-contained model. See [Deployments](docs/DEPLOYMENTS.md#recorded-compose-releases-0171).
- Additive migration 0093 stores encrypted release records separately from deployment history and marks copied restores. Key rotation includes the new encrypted store; release promotion adds a real-Docker Compose restore smoke.
- Fix in-memory database transactions trying to reserve a second connection for a PRAGMA. Commit/rollback and foreign-key enforcement are regression-tested.
- Remove the stale feature-version prefix from `deploys diagnose --help`.

## [0.17.0] - 2026-10-20

> Application readiness, node resource history and log drains, and AI that proposes a fix.

### Upgrade notes

- **No SQL migration.** AI features remain off without a configured provider. Node resource history and configured log drains become available with the updated panel/agent; readiness remains opt-in.
- Existing healthcheck behaviour is unchanged. HTTP and TCP readiness are explicit opt-in modes for plain Docker services. Node readiness and log drains require a 0.17.0 agent with sealed transport; older agents keep their existing deploy behaviour.
- `POST /v1/ai/services/:id/deploys/:depId/diagnose` still answers `diagnosis` and `model`. It may now add `parts` and `proposal`, which older clients ignore.

### Added

- **A diagnosis that proposes a change.** The model sees the service's current build settings and answers in a structured form:
  - `parts`: the root cause, the quoted evidence and the steps;
  - `proposal.settings`: port, health path, build pack, base directory, install, build or start command, or Dockerfile path;
  - `proposal.files`: up to five exact find/replace edits to repository files, each with its reason.
- **Validation.** Every part is checked against a strict schema. An invalid part is dropped on its own, and a value equal to the current one is removed. Lifecycle hooks can never be proposed, since they are host-privileged for non-operators. A prose answer still works: it is the diagnosis text, with no proposal.
- **Apply and redeploy.** The diagnosis card shows each proposed setting next to its current value. Applying uses `PATCH /v1/services/:id`, so all its rules apply, and then starts a deploy. File edits are shown as a patch to copy; the panel never writes to the repository.
- **Clients.** CLI: `ninedeploy deploys diagnose <service> <deployId> [--apply] [--json]`. SDK: `ai.diagnose` returns `AiDiagnosisResult`. See [docs/AI.md](docs/AI.md#a-proposed-fix-0170).
- **HTTP/TCP readiness.** Plain Docker services on the panel host or a node can require an HTTP 2xx/3xx response, or a TCP connection, before taking traffic. HTTP readiness never falls back to an open TCP port. A bounded helper shares the workload's network namespace, publishes no host port, and is removed after the attempt. Unsupported nodes are refused before deploy side effects.
- **Node service resource history.** CPU and memory samples from node agents are stored and evaluated by the existing alert rules. Cache identity includes the node, serving replicas are summed, and malformed/missing samples stay unknown. Node collection has bounded concurrency and cancellation; the Local Host monitoring view stays local.
- **Node log drains.** Primary Docker/Compose runtimes and their named replicas can send logs through the sealed agent to configured drains. Reads and output are bounded. This remains a best-effort in-memory drain; restarts or excess volume can lose history. See [docs/NODE_OPERATIONS.md](docs/NODE_OPERATIONS.md) for modes, capabilities and limits.
- **Feature availability.** Servers and the Docker deployment wizard show which features the selected host supports. Healthcheck choices are available in service settings and displayed by the CLI.

### Fixed

- **AI context redaction.** Current build commands and health paths now use the same credential sanitizer as logs before reaching the configured model. Proposed changes cannot replace credential-bearing settings or introduce redaction placeholders. A captured-provider-request regression verifies that inline URL passwords and tokens stay on the panel while ordinary settings remain useful.
- **Dependency hardening.** PM2's `follow-redirects` is pinned to 1.16.1 for cross-origin credential stripping ([array-form headers](https://github.com/advisories/GHSA-q3jh-wrc6-jgq8), [sensitive-header matching](https://github.com/advisories/GHSA-8r9p-f939-6h3c)). Hono is at least 4.13.11 for [static-path double decoding](https://github.com/advisories/GHSA-5r4p-p66f-jhc7); the MCP server remains stdio-only. Upstream has no fixed `braces` release for [nested-pattern stack exhaustion](https://github.com/advisories/GHSA-vfj7-8cjw-p6xm), so the package carries a local patch: parse nesting and caller-supplied ASTs are limited to 256 levels before recursive walkers. Ordinary patterns and the supported boundary are regression-tested through PM2's actual dependency. The raw audit still reports the upstream 3.0.3 advisory; it is not suppressed.
- Failed log delivery retains its cursor and is retried. A destination that already accepted the batch may receive duplicates while another destination recovers.
- Node container names no longer contaminate panel-host service statistics; service history remains scoped by service access.
- The registry drift guard passes Git's tag glob without a shell, so POSIX expansion cannot silently skip the baseline. Release checks fetch the full tag history needed by that guard.
- Release promotion also requires a real two-daemon node readiness/history/log-drain smoke on the pushed image, alongside the existing upgrade and user-journey smokes.
- The scale-to-zero route test mocks the stop command and asserts its arguments, instead of depending on the developer's Docker daemon.

## [0.16.15] - 2026-10-20

> Claude as an AI provider, and AI that reads a running service's logs.

### Upgrade notes

- **No migration.** The AI configuration gains an optional `provider`. A configuration saved before has none and keeps working as an OpenAI-compatible endpoint, unchanged.
- `GET /v1/ai/config` reports `provider`. `PUT /v1/ai/config` without `provider` means `openai` and still needs `baseUrl`.

### Added

- **Claude as a provider.** `provider: 'anthropic'` on `PUT /v1/ai/config` (Settings → AI assistant, which now has a provider selector).
  - It calls the Anthropic Messages API (`POST <base>/v1/messages`, `x-api-key`, `anthropic-version: 2023-06-01`).
  - `baseUrl` is optional and defaults to `https://api.anthropic.com`.
  - Every AI feature uses it: failed-deploy diagnosis, manifest suggestions and the log analysis below. The provider code is shared (`lib/aiProvider.ts`), so the upstream body is still never relayed in an error.
- **Runtime-log analysis.** `POST /v1/ai/services/:id/logs/analyze` with `{ tail? }` (20–5000, default 300).
  - It reads the service's newest runtime log lines through the same reader as `GET /v1/services/:id/logs` (PM2, Swarm, nodes, the panel host), and applies the diagnosis sanitizer and its 16 KB cap.
  - It answers the model's analysis (health, problems with the log lines quoted, next steps), the model, and how many lines were sent.
  - Gates: `member` on the service, protected lanes, the AI spend rule, 10 calls a minute. It is audited as `ai.analyze_logs`.
  - Web: an **Analyze** button on the runtime logs card when a provider is configured. CLI: `ninedeploy logs analyze <service> [--tail] [--json]`. SDK: `ai.analyzeLogs`.
- **Docs:** [docs/AI.md](docs/AI.md) describes the providers, who may use them, what leaves the panel, and each feature.

## [0.16.14] - 2026-10-20

> Pull requests from forks get a preview, held for approval and cloned with no credential.

### Upgrade notes

- **One additive migration (0092: `services.preview_forks`, default off, and `services.preview_from_fork`, default false).** Nothing is turned on: a fork's pull request is refused as before, and every existing preview reads as one from the repository itself.
- **GitLab and Bitbucket:** a merge request or pull request from a fork used to be read as coming from the target repository. Its preview cloned the target at the fork's branch name, which failed or built the wrong code. It is now recognised as a fork, so it is refused unless the service turns fork previews on. A merge request from the repository itself is read as before.

### Added

- **Previews of pull requests from forks.** `previewForks: true` on a service (Settings → Previews; `PATCH /v1/services/:id`; CLI `ninedeploy previews forks <service> on|off`) lets a fork's pull request create a preview:
  - **Approval:** every deploy of a fork preview waits as `awaiting_approval`, whatever the lane says, until an admin approves or rejects it. In a lane, the lane's admins decide; outside a lane, `admin` on the service is enough. A newer push replaces the waiting deploy.
  - **No credential:** the fork is cloned anonymously. Neither the service's Git credential nor a GitHub App token is resolved for it, on the panel or on a node, so only a public fork builds.
  - **Less inherited:** the parent's non-secret variables and non-secret preview-only values; no secrets of any kind, no database copies.
  - **Limits:** docker services without the Docker socket only (`fork_preview_unsupported_type` otherwise), and a fork on the repository's own git host (`fork_on_another_host` otherwise). The GitHub App tells a fork by repository id.
  - **Clients:** the preview row carries `previewFromFork: true`. The web switch is in the preview settings, and `ninedeploy previews show` shows both. See [docs/DEPLOYMENTS.md](docs/DEPLOYMENTS.md#pull-requests-from-forks-01614).

### Fixed

- **GitLab and Bitbucket previews clone the source repository.** The merge request's `source` project and the pull request's `source.repository`, falling back to the target as before.
- **Preview settings at create time.** A service created with `previewDatabaseCopy` (or the new `previewForks`) in its create request now stores it; the create route accepted the field but ignored it.

## [0.16.13] - 2026-10-20

> Protected lanes and deploy approvals: production secrets stay with admins, and production deploys wait for one.

### Upgrade notes

- **One additive migration (0091), nothing to turn on, nothing that worked is refused.**
  - On `environments`: `protected` and `require_approval`, both default off.
  - On `deployments`: `reviewed_by` and `reviewed_at`, both nullable; `reviewed_by` is `ON DELETE SET NULL`.
  - No lane is protected or gated after the upgrade, and no deploy waits.
- A new deployment status, `awaiting_approval`, appears only on a deploy into a lane whose admin turned approvals on.
  - It is kept out of `GET /v1/services/queue`, so clients that parse that list strictly are unaffected.
  - It is listed at `GET /v1/services/approvals` instead.

### Added

- **Protected lanes.** `protected: true` on a lane (`PATCH /v1/environments/:id`, admin; Services → Lanes) keeps the following for the lane's admins:
  - the variable values of the lane's services and of their PR previews: a member reads `''` with `hidden: true`, can still write values, but cannot export them;
  - deploy logs (stream and download), runtime logs, log search, AI diagnosis and mounted files: a member is refused with `403 protected_lane`;
  - the lane itself: a member cannot clone a protected service or move a service out of a protected or gated lane.
- **Deploy approvals.** With `requireApproval: true` on a lane:
  - **What waits:** every deploy that is not an admin's own, from a webhook, a schedule, an image update, a deploy hook URL or a member. It waits as `awaiting_approval`; the worker holds it at claim time, so every trigger is covered.
  - **Deciding:** `POST /v1/services/:id/deploys/:depId/approve` queues it; `/reject` cancels it. The author can withdraw it with cancel. The deploy list carries `reviewedBy` and `reviewedAt`.
  - **One waiting deploy per service:** a newer waiting deploy replaces the older one, and a deploy that starts drops the older waiting ones, so approving never ships older code.
  - **Kept and announced:** waiting deploys are kept by the history sweep and by webhook replay dedup. They send a `deploy.awaiting_approval` notification, and decisions are audited as `deploy.approve` and `deploy.reject`.
- **Lane admin.** An operator, an admin or owner seat in the lane's workspace, or an admin grant covering the service. Being the service's creator is not enough.
- **Clients.**
  - **Web:** the two switches in Services → Lanes; an approval card on the service's Deploys tab and a "Waiting for approval" list on the Deploys page; hidden values and refused logs say why.
  - **CLI:** `ninedeploy lanes list|protect|require-approval` and `ninedeploy approvals list|approve|reject`.
  - **SDK:** `environments.update`, and `deploys.approvals`, `deploys.approve`, `deploys.reject`.
  - See [docs/DEPLOYMENTS.md](docs/DEPLOYMENTS.md#protected-lanes-and-deploy-approvals-01613).

### Not yet

- PR previews from forks are still refused, as before. Building them behind an approval, with a clone that never sends the parent's credentials to the fork, is next on the roadmap.

## [0.16.12] - 2026-10-20

> A database copy per PR preview: realistic data in every preview, production never touched.

### Upgrade notes

- **One additive migration (0090: nullable `databases.preview_copy_of` and `databases.preview_service_id`, both `ON DELETE SET NULL`, and `services.preview_database_copy`, default off), nothing to turn on, nothing that worked is refused.** No preview gets a copy until a parent service turns it on.

### Added

- **A database copy per preview.** With `previewDatabaseCopy: true` on a parent service (Settings → Previews), each PR preview gets its own copy of every database attached to the parent.
  - **When and from what:** on the preview's first deploy, from the parent database's latest completed backup (fetched from remote storage when the local file is gone), or a dump taken then when there is none.
  - **The copy:** a managed database named `<database> · PR #<n>`, with its own container, volume and credentials, and the parent's limits. It is attached under the parent's variable name, so the preview's `DATABASE_URL` points at it.
  - **Lifecycle:** later deploys of the PR keep it. It is removed with the preview (container, volume, backups) and swept hourly once orphaned (for example when the parent service is deleted). It is left out of the daily built-in backups.
  - **Engines:** PostgreSQL and the Redis family on the panel host. MySQL, MariaDB and MongoDB dumps carry the server's own users, so they are not copied yet; neither are node databases. The deploy log names what is left out.
  - **Failure:** a copy that cannot be made fails the preview's deploy; it never falls back to production.
- **Clients.** The web app has the switch in the preview settings and a "Preview copy" badge on a preview's attached databases. The CLI has `ninedeploy previews show|database-copies`. See [docs/DEPLOYMENTS.md](docs/DEPLOYMENTS.md).

## [0.16.11] - 2026-10-20

> Scale to zero: an idle service sleeps, and its next request wakes it.

### Upgrade notes

- **One additive migration (0089: nullable `services.sleep_after_minutes` and `services.asleep_at`), nothing to turn on, nothing that worked is refused.** No service sleeps after the upgrade. A Docker install whose panel is published on loopback (the default before 0.16) is reached through the `ninedeploy` network instead. The panel container joins it as `ninedeploy-panel`, but only when a service is first about to sleep, never at upgrade.

### Added

- **Scale to zero** for docker services on the panel host.
  - **Setting:** `sleepAfterMinutes` (5–1440) on `PATCH /v1/services/:id`.
  - **Going to sleep:** once the service's domains have had no request for that long, they are routed to the panel's wake endpoint (`/_nd/wake`, after the domain's basic auth and allow list) and its containers stop.
  - **Waking:** the next request starts the containers, waits for the healthcheck and routes the domains back.
    - A GET or HEAD gets `307` to the same URL.
    - Other methods get `503` with `Retry-After`.
    - A wake slower than 25 s answers a self-reloading "Waking up" page.
  - **Requirements:** traffic analytics on (requests are counted from the traffic log) and a routed domain.
  - **Audit:** events `service.sleep`, `service.wake` (with the trigger) and `service.sleep_now`.
- **By hand.** `POST /v1/services/:id/sleep` and `POST /v1/services/:id/wake`. Start and restart wake a sleeping service; stop and a deploy end the sleep. The web app shows an Asleep badge with Wake now and Sleep now, and a Sleep when idle card in Settings. The CLI has `ninedeploy sleep show|set|off|now|wake`, and the SDK `client.services.sleep` and `client.services.wake`. See [docs/DEPLOYMENTS.md](docs/DEPLOYMENTS.md).
- **A service that could not wake never sleeps.** Before anything sleeps, the panel checks that a request from inside the Traefik container reaches it. When it does not, the sleep is refused with the reason.

### Changed

- **The runtime reconcile and the autoscaler** leave a sleeping service alone.

## [0.16.10] - 2026-10-20

> Autoscaling: containers follow the load, without a redeploy.

### Upgrade notes

- **One additive migration (0088: a nullable `services.autoscale` column), nothing to turn on, nothing that worked is refused.** Every existing service keeps running `replicas` containers. The autoscaler looks only at services with an autoscale configuration, and its windows are held in memory, so they restart after a panel restart.

### Added

- **Autoscaling** for docker services on the panel host. Set it with `autoscale` on `PATCH /v1/services/:id`:
  - `minReplicas` 1–10, default 1, and `maxReplicas` up to 10;
  - `metric`: `cpu` (percent of the CPU limit, or of one core), `memory` (percent of the memory limit) or `rps` (requests per second per container, from traffic analytics);
  - `target`, the value per container to keep near;
  - `scaleUpSeconds` (default 60) and `scaleDownSeconds` (default 300).

  Every 30 s the needed count is the current count times value ÷ target, with a 10 % tolerance. Scaling up waits the up window and takes the smallest count asked for in it; scaling down waits the down window and takes the largest. Each deploy creates every container up to the maximum and stops the spares. Scaling starts a spare and routes it after it passes the healthcheck, or routes containers away and then stops them. Changes are audited as `service.autoscale`.
- **Autoscale status.** `GET /v1/services/:id/autoscale` shows the containers serving, the last value, the count asked for, why it holds (a deploy or canary in progress, no traffic analytics for `rps`, a node or Swarm service, no spare yet) and the last change. The web app has an Autoscaling card in Settings (Replicas shows "managed by autoscaling") and a status card on the Overview. The CLI has `ninedeploy autoscale show|set|off`, and the SDK `client.services.autoscale`. See [docs/DEPLOYMENTS.md](docs/DEPLOYMENTS.md).

### Changed

- **The runtime reconcile** revives only the containers of an autoscaled service that serve, never its stopped spares.

## [0.16.9] - 2026-10-20

> Canary releases: a new version takes the traffic step by step, and backs out on its own.

### Upgrade notes

- **One additive migration (0087: a nullable `services.canary` column), nothing to turn on, nothing that worked is refused.** Every existing service has no canary configuration, so every deploy switches all traffic at once as before. A canary in progress is held in memory: a panel restart during one fails that deploy, like any interrupted deploy, and routes everything to the previous release.

### Added

- **Canary releases** for docker services on the panel host (`canary` on `PATCH /v1/services/:id`: `steps` rising 1–99 %, up to 5, default `10, 50`; `stepSeconds` 30–3600, default 120; `maxErrorRate` default 5 %; `maxP95Ms` optional; `minRequests` default 20). After the healthcheck, Traefik splits each routed domain between the previous release and the new one with a weighted service. Each step probes the new release's healthcheck every 15 s. With traffic analytics on, the step is then judged on the new release's 5xx share and p95, read per release from the access log. A failed step routes everything back, stops the new container and fails the deploy (`Canary aborted: at 10%, …`). Sticky sessions keep a visitor on one release.
- **Canary control.** `GET /v1/services/:id/canary` (step, share, both releases' requests, 5xx rate and p95) and `POST /v1/services/:id/deploys/:depId/canary/promote`; cancelling the deploy aborts the canary. The service page shows a live panel with Promote and Abort, the deploy view a Canary stage, and Settings a Canary releases card. CLI `ninedeploy canary show|set|off|promote`, SDK `client.deploys.canary` and `client.deploys.promoteCanary`. See [docs/DEPLOYMENTS.md](docs/DEPLOYMENTS.md).
- **Skipped with the reason in the log** for rollbacks, services on a node or on Swarm, services that publish host ports, and services with no previous release or no routed domain.

## [0.16.8] - 2026-10-20

> Uptime monitors and public status pages, built in.

### Upgrade notes

- **One additive migration (0086: `uptime_monitors`, `uptime_checks`, `uptime_daily`, `status_pages`), nothing to turn on, nothing that worked is refused.** No check runs and no page is served until an operator creates a monitor or publishes a page. Checks are kept 7 days and daily rollups 400 days, pruned by housekeeping.

### Added

- **Uptime monitors.** `http` monitors check the status code (default `200-399`, lists and ranges), an optional keyword in the first 1 MiB of the body, redirects (up to 5) and the TLS certificate's expiry. `tcp` monitors check that `host:port` opens. Checks run from the panel host every 30 s–1 h, with a 1–60 s timeout, at most 20 at once. A monitor goes down after 1–10 failures in a row and comes up on one success. `GET/POST /v1/monitors`, `GET/PATCH/DELETE /v1/monitors/:id`, `GET /v1/monitors/:id/checks`, `POST /v1/monitors/:id/check`. A monitor on a service is readable by everyone who can see that service; changes are operator-only.
- **Down/up alerts.** `alert.monitor_down` (with the reason) and `alert.monitor_up` (with the downtime) go through notification channels, the service's `alert` subscriptions and the plugin bus as `alert.triggered`. A monitor that stays down does not repeat.
- **Public status pages.** `/status/<slug>` (HTML, no scripts or external assets, light and dark) and `GET /v1/status/<slug>` (JSON), both cached for 30 s. A page shows names, state, 90 daily bars and optionally response times, never targets or errors. An unpublished page answers 404, like an unknown slug. Pages are managed with `/v1/status-pages`, operator-only.
- **Clients.** CLI `ninedeploy monitors list|add|show|check|pause|resume|rm` and `ninedeploy status-pages list|add|publish|unpublish|notice|rm`; SDK `client.monitors` and `client.statusPages`; MCP `list_monitors`. See [docs/UPTIME.md](docs/UPTIME.md).

## [0.16.7] - 2026-10-19

> Uploaded source reaches nodes.

### Upgrade notes

- **No migrations, nothing to turn on, nothing that worked is refused.** A node service built from uploaded source was refused at upload and failed at deploy before 0.16.7. Its node needs agent 0.15.2 or newer (`stream`, `image.manage`) to receive the image; an older agent is refused when the deploy is queued, with the update named.

### Added

- **Uploaded source on nodes.** A docker service built from an uploaded archive and pinned to a node, with no build placement set, is built on the panel (which keeps the archive) and its image is shipped to the node, as with Build on: panel and Git LFS services since 0.16.5. `POST /v1/services/:id/source` takes uploads for node services; `?deploy=true` (the default) checks the node before anything is stored. A placement on the node itself or on a build server is refused (`400 remote_deploy_unsupported`), with Build on: panel named.
- **Fan-out of uploaded source.** Fan-out targets of an uploaded-source service receive the primary's image; they never rebuild from the archive. Before, they were skipped silently.

## [0.16.6] - 2026-10-19

> Container options reach nodes and fan-out targets.

### Upgrade notes

- **No migrations, nothing to turn on, nothing refused.** A node service with container options deployed without them before 0.16.6; on an agent older than 0.16.6 it still does, and the deploy log now names the update. Update a node's agent to 0.16.6 to have them applied.
- **New node switch:** `NINEDEPLOY_AGENT_RUNTIME_OPTIONS=off` on the agent refuses container options on that node (GPUs and added capabilities are the node owner's call). A service there deploys without them, with a log line naming the switch.

### Added

- **Container options on nodes and fan-out targets** (capability `docker.runtimeOptions`, agent 0.16.6). GPUs, shared memory, the init process, the open-files limit and the allowed extra capabilities travel on `docker.runSpec`'s new `runtimeOptions` field; the agent checks them against the panel's own strict schema and builds the same flags as the panel host. GPUs need the NVIDIA container toolkit on the node. Swarm services still start without them.

### Fixed

- **The API reference and the SDK** still described file mounts as panel-host only and Git LFS as building on the panel host only; they now say what 0.16.5 does.

## [0.16.5] - 2026-10-19

> File mounts reach nodes and fan-out targets, and a node service with Git LFS deploys by building on the panel.

### Upgrade notes

- **No migrations, nothing to turn on.** Services deploy exactly as before unless they have file mounts or Git LFS on a node, which were refused until now.
- **Node agents:** a node service (or a fan-out target) with file mounts needs agent 0.16.5 (capability `docker.fileMounts`). On an older agent the deploy is refused when it is queued, naming the update; nothing is written to the node. Services without files deploy on any agent as before.
- **Git LFS on a node** with no build placement now builds on the panel and ships the image, so the node needs `stream` and `image.manage` (agent 0.15.2+); this is checked when the deploy is queued.

### Added

- **File mounts on nodes and fan-out targets.** The agent writes each file under its home (`/var/lib/ninedeploy-agent/.agent-files/<slug>/d<deployment>/`, directories 0700, the three newest deployments kept), one request per file over the sealed transport, and `docker.runSpec` mounts it read-only through its new `fileMounts` field. Only files the agent wrote there are mounted, and only when they exist, so Docker never mounts an empty directory in their place. Deleting the service removes the copies on its nodes.
- **Git LFS on a node service.** With Build on left unset, the build runs on the panel host, where the LFS files are fetched and checked, and the image is shipped to the node (as with Build on: panel). The node never clones, so a PAT or deploy key stays on the panel. A placement chosen on the service is kept; building on the node itself or on a build server is still refused, since a fresh clone there holds pointer files.

### Fixed

- **Deleting a panel-host service left its file mount copies** under `service-files/<slug>/` unless it had an allow-list port proxy: the removal sat inside that branch.
- **A file mount path containing `:` was accepted**, though `docker run -v` splits on it and the deploy could never mount it. It is now refused when the file is saved.

## [0.16.4] - 2026-10-19

> Docker labels and the healthcheck command reach nodes, fan-out targets and Swarm, and the whole panel was gone over in dark and light themes, on desktop and on a phone.

### Upgrade notes

- **No migrations, nothing to turn on.** Services keep exactly the labels and healthcheck they had.
- **Node agents:** a node service (or a fan-out target) that uses custom Docker labels or the healthcheck command needs agent 0.16.4 (capabilities `docker.userLabels` and `docker.healthCmd`). On an older agent the deploy is refused when it is queued, naming the update; nothing starts without them. Services that use neither deploy on any agent as before.

### Added

- **Custom Docker labels on nodes, fan-out targets and Swarm.** `docker.runSpec` takes `userLabels`, validated like the panel's (`ninedeploy.*`, `com.docker.*` and `traefik.*` stay reserved); Swarm stacks put them on the service's containers.
- **The healthcheck command on nodes, fan-out targets and Swarm.** There it runs as the container's own Docker healthcheck (`--health-cmd`, every 5s, 10s timeout, 3 retries; `CMD-SHELL` in a stack), and the deploy waits for the container to report healthy.
- **Motion and art, at a modest scale:** animated SVG illustrations for empty states, a short rise when a page opens, staggered card grids, shimmering skeletons, and pulsing dots for running and in-progress statuses. All of it stops under `prefers-reduced-motion`.

### Changed

- **Phones:** below 768px the side menu is a drawer (it used to leave ~170px for the page), the service tabs scroll in one row with the active tab in view, Settings has a section picker, and the theme toggle and ⌘K hint step aside.
- **Topology:** networks are laid out in a grid above the services column and only those a service or database is on are drawn; the rest sit behind a **+N more networks** tile that expands them. The duplicate React Flow zoom controls are gone (the page has its own).
- **Field hints sit under the control** instead of beside the label, and button and badge labels never wrap.

### Fixed

- **Fields stretched past the width they were given.** Input, Select and Textarea joined their classes without merging, so the default `w-full`/`h-10` beat a caller's width: the domain row's path box took the whole row and pushed Add out of the card, and the environment KEY box squeezed the value. A caller's width, height, padding, colour and size now win (Button and Card too; icon buttons had shrunk to a dot). The domain row wraps on a phone; the env row splits 2:3.
- **Light theme:** about 40 dark-tuned surfaces that stayed dark, native date and colour pickers drawn dark, the deploy pipeline and the topology/architecture views ignoring the theme, and unreadable text around the terminal.
- **The server Docker page counted 0 images.** `docker system df` prints the count as `TotalCount`; the panel read `Total`, which current Docker CLIs no longer print (still read for older daemons).
- **Settings → Labels hid instance-wide labels** while a workspace was selected.
- Layout fixes on Monitoring limits, service cards, API tokens, Activity, Workspaces, Hub cards, the service overview and the database detail page; duplicate React keys on the Docker page.

## [0.16.3] - 2026-10-19

> Recovery codes for 2FA, an API tokens page, and more of what Coolify and Dokploy offer per service: file mounts, a healthcheck command, Docker labels, Git LFS and searchable runtime logs.

### Upgrade notes

- **Nothing changes until someone turns it on.** Every new setting is off or empty on an upgraded panel; node agents need no update (a node on an older agent returns the last 300 runtime-log lines whatever is asked).
- **Five additive migrations.** `0081_recovery_codes` adds the empty `user_recovery_codes` table. Accounts with 2FA sign in exactly as before; they have no recovery codes until they create a set under Settings → Account. `0082_service_files` adds the empty `service_files` table, so every container starts with exactly the mounts it had. `0083_health_command` adds the nullable `services.health_command`; every service keeps its healthcheck. `0084_docker_labels` adds the nullable `services.docker_labels`; containers keep exactly the labels they had. `0085_git_lfs` adds `services.git_lfs`, off for every service.

### Added

- **Two-factor recovery codes**: turning 2FA on hands out ten single-use codes, shown once to copy or download. At sign-in, one stands in for the authenticator code (web and CLI). Settings → Account shows how many are left and creates a new set (password and a current code). The help text already promised them; the panel had none.
- **Search and download for runtime logs** (Service → Runtime logs), like Coolify's and Dokploy's log viewers: filter to the lines that contain a text (case-insensitive, matches highlighted), load the last 300, 1000 or 5000 lines, pause the live refresh, and download what is loaded. API `GET /v1/services/:id/logs?tail=N` (1 to 5000, default 300 as before); SDK `services.logs(id, { tail })`. A node on an agent older than 0.16.3 returns the last 300 lines.
- **CLI:** `ninedeploy services logs <id> --tail N --grep text`.
- **File mounts** (Service → Volumes → Files), like Coolify's and Dokploy's: text files (an nginx.conf, a config.yaml) kept encrypted by the panel and mounted read-only at a path in the container; up to 20 per service, 256 KiB each, applied on the next deploy. Docker services on the panel host; a node, Swarm or compose service with files is refused rather than started without them. API `/v1/services/:id/files`; SDK `services.files`.
- **Healthcheck command** (Service → Settings → Healthcheck → Command inside the container): the deploy runs a command such as `pg_isready` or `curl -fsS localhost:3000/up` with `sh -c` in the new container until it exits 0, logging failed attempts. Docker services on the panel host; API `healthCheck: 'command'` with `healthCommand`.
- **Docker labels** (Service → Settings → Docker labels): your own `key=value` labels on the container and its replicas, for host tools that read them; `ninedeploy.*`, `com.docker.*` and `traefik.*` are reserved. Docker services on the panel host; API `dockerLabels`.
- **Git LFS** (Service → Settings → Git LFS, off by default): each deploy replaces LFS pointer files with their content, fetched from the repository's own HTTPS LFS server with the service's credential; `.lfsconfig` is ignored, every download and redirect passes the egress check, and each object is verified by SHA-256 and size. Builds on the panel host; API `gitLfs`.
- **API tokens page** (Settings → Security → API tokens): create a token for the CLI, CI or an MCP client with a name, an access level (read only, read & write, or full access for operators) and a lifetime; it is shown once. The list shows each token's access, last use and expiry, flags legacy unrestricted and expired tokens, and revokes after a confirmation. The API existed; the docs pointed to this page before it did.

### Changed

- **The top bar is two rows**: the page title and the actions (New, search, activity, help) above, the workspace / project / label filters below, so the filters no longer push the actions off the bar.

### Fixed

- **`ninedeploy services logs` printed the oldest 50 of the last 300 lines.** It now prints the newest lines.

## [0.16.2] - 2026-10-18

> Redeploy from CI with a deploy hook, and build one stage of a multi-stage Dockerfile.

### Upgrade notes

- **Nothing changes until someone turns it on.** Node agents need no update; a node build with a build stage set is refused with a clear message.
- **Two additive migrations.** `0079_deploy_hooks` adds three columns to `services` (a null token, a null creator and `deploy_hook_operator = false`) and a unique index over the token. No service has a hook until a member turns one on. `0080_build_target` adds the nullable `build_configs.build_target`, so every existing service builds its Dockerfile's last stage as before.

### Added

- **Deploy hook** (Service → Settings → Deploy hook), like Coolify's and Dokploy's deploy webhooks: a secret URL that redeploys the service when a CI job POSTs to it (`curl -fsS -X POST …`). It deploys as the member who made it, re-checking their access on every call, and never with more operator authority than that request had. Only that member sees the URL; another member sees the hook is on and can rotate it to make it theirs. Rotate it or turn it off from the same card. API: `GET`/`POST`/`DELETE /v1/services/:id/deploy-hook`, public `POST /v1/deploy-hooks/:token`; SDK `services.deployHook`; CLI `ninedeploy services deploy-hook <id> [--enable|--disable]`.
- **Build stage** (Service → Settings → Build → Build stage): build one stage of a multi-stage Dockerfile (`docker build --target`), like Coolify's and Dokploy's build target. API `build.buildTarget`. A node build with a stage set is refused before cloning, because the agent would build the last stage.

### Fixed

- **Fan-out nodes rebuilt a Dockerfile without its build arguments.** A service with build arguments (or a build stage) that deploys to extra nodes now ships the panel's image to them instead of a rebuild that would differ.

## [0.16.1] - 2026-10-18

> First deploy, the Coolify way: a **New** button on every page, working URLs without DNS work and a setup checklist. Port forwarding grows up: UDP, port ranges, a dedicated IP, allow lists, several IPs per server and HTTP/3. Also: deploy from an uploaded folder, Dockerfile build arguments, container options, status badges and redirect rules.

### Upgrade notes

- **Nothing changes until someone turns it on.** Every new setting defaults to the 0.16.0 behaviour; Traefik is not recreated by the upgrade.
- **Node agents:** a node keeps working on its current agent. Port mappings on a node-pinned service need the node's agent at 0.16.1; an older one is told to update when such a service deploys.
- **Five additive migrations.** `0078_host_ips` adds the empty `host_ips` table; `0077_port_mappings` adds the nullable `services.port_mappings` and `services.health_check`; `0076_domain_redirects` adds the nullable `domains.redirects`; `0075_status_badge` adds the nullable `services.badge_token` and its unique index. `0074_build_and_runtime_options` adds three nullable columns, `build_configs.build_args`, `services.health_timeout_seconds` and `services.runtime_options`. No row is touched, so every existing service builds and runs with exactly the `docker build` / `docker run` line it had and keeps the 300 s healthcheck deadline. Rolling back to 0.16.0 ignores the columns. The other new settings are rows in the existing `settings` table.
- **Automatic URLs stay off on an upgraded panel.** Without an `auto_url_mode` setting nothing changes, and `NINEDEPLOY_WILDCARD_DOMAIN` keeps working exactly as before and wins over the new setting. Only a fresh install turns on sslip.io URLs, and only when the installer recorded a public IPv4 for the panel.
- **The Settings → Security "Wildcard Domain" card is gone.** It showed the DNS-01 certificate apex as if it gave services URLs, which only the environment variable ever did. The apex itself is unchanged and still set in the DNS challenge form.

### Added

- **New resource picker (`/new`).** One page to start anything, like Coolify's New Resource: applications (Git repository, GitHub App, Docker image, Dockerfile, static site, and for operators Docker Compose and PM2), the 11 database engines, and the one-click app catalog, with search (`/` focuses it) and category tabs. The top bar, the rail and the Deploy menu all lead here; ⌘K adds "Deploy a Git repository", "Deploy a Docker image", "Deploy a Compose stack" and "Create a database". `?deploy=<kind>` on any page opens the wizard pre-set (`git`, `github-app`, `image`, `compose`, `dockerfile`, `static`, `pm2`, or `new`).
- **Automatic URLs** (Settings → Security). Every service gets its own URL when it deploys: a free `<slug>.<server-ip>.sslip.io` test URL with no DNS to set up, or `<slug>.<your base domain>` behind a wildcard DNS record. A node-pinned service resolves to the node's address. The names are reserved per service, so another tenant cannot claim one. API: `GET`/`PUT /v1/settings/auto-url`, `POST /v1/settings/auto-url/detect-ip` (asks a public echo service, only when clicked).
- **Generate URL** on a service's Network tab (`POST /v1/services/:id/domains/auto`) adds the automatic URL now instead of on the next deploy.
- **Getting started checklist** on the Dashboard for the instance operator (`GET /v1/onboarding`, `PUT /v1/onboarding/dismissed`): a URL for services, a Git provider, the first service and its first live deploy, then the optional own domain, HTTPS, notifications and backups. Every step is read from the panel's own state; hiding it leaves a one-line link to bring it back.
- **Deploy Wizard:** asks for the build pack up front (Auto, Nixpacks, Railpack, Dockerfile with its path, Static with its output directory), the project and environment the service lands in, and an optional domain, which is added before the first deploy.
- **Deploy from anywhere:** an empty Dashboard opens with a "Deploy your first app" card (your code, a template, or the Next.js demo); `?deploy=new` is a deep link to the wizard.
- **Dockerfile build arguments** (Service → Settings → Build). Pick which of the service's variables a Dockerfile build receives as `--build-arg`, like Coolify's build variables; the service's variables are offered as chips. Opt-in by name: a service gets none until you list some. The values travel as argv and are masked in every log line; a secret among them is flagged, because build arguments stay readable in `docker history`, and a build that pushes its image to a registry refuses one. Nixpacks and Railpack builds already saw every variable. API: `build.buildArgs` on create, PATCH and the service response. A Dockerfile build on a remote node does not pass them yet and says so in the log.
- **Deploy without Git: uploaded source** (like Dokploy's Drop and Railway's `up`). New → **Upload source** takes a `.zip` or `.tar.gz` of a project folder, creates a docker service and deploys it; the service's Deploys tab takes new versions. From a terminal: `ninedeploy deploy up ./my-app --create my-app` (or `--service <id|slug>`), which packs the folder without `.git`, `node_modules`, `.env` and build caches. The build is the usual one: a Dockerfile wins, otherwise Nixpacks. A single top-level folder in the archive becomes the root. The last five uploads are kept, and redeploying an older deployment rebuilds the upload it came from. API: `POST /v1/services/:id/source` (raw body, up to 100 MiB, `?deploy=false` to only store it) and `GET /v1/services/:id/source`; SDK `services.uploadSource` / `services.sourceUploads`.
  - The archive is checked completely before it is kept: regular files and directories only (no symlinks, hardlinks or devices), relative paths without `..`, at most 50 000 entries and 1 GiB extracted, no encrypted or zip64 zips. Authentication and the service checks run before the body is read.
  - Uploads build on the panel host; a service pinned to a remote node is refused. Deleting the service deletes its uploads.
  - A docker service created with neither a repository nor an image used to fail its deploy at `git clone ''`; it now says to upload its source.
- **Port mappings: UDP, port ranges and a dedicated IP** (Network tab → Port Mappings), like Coolify's port mappings and Dokploy's ports. A docker service publishes extra host ports straight to its container, outside Traefik: one port or a range (`27015-27030`), TCP or UDP, on every address or on one IP of the host (`-p 203.0.113.7:53:53/udp`). Up to 20 rules and 1000 ports per service. Overlaps within the service and ports another panel-host service or a database's public access holds are refused with the owner named; ports below 1024 need an operator, and the panel, SSH, 80 and 443 stay reserved. The Host IP field offers the panel host's own addresses, and an IP the host does not have is refused when saved (`GET /v1/services/:id/host-addresses`). A service with host ports redeploys stop-then-start, and replicas do not publish them. A node-pinned service publishes them on its node through the agent's new `docker.ports` capability (agent v0.16.1; an older agent is told to update before anything reaches the node, and the agent itself refuses SSH, 80, 443 and its own port). Swarm services are refused. CLI: `ninedeploy services ports <id> [mappings...]`. API: `portMappings` on `PATCH /v1/services/:id` and the service response.
- **Allow lists on TCP port mappings.** A single TCP port mapping can list the addresses or CIDR ranges allowed in (`5432@10.0.0.0/8,203.0.113.4`, `allowFrom` in the API). The port is then published by a small locked-down Traefik proxy on the service's bridge (`nd-portproxy-<slug>`) behind an `ipAllowList`, not by Docker's `-p`, which a host firewall such as ufw cannot restrict. The proxy follows every deploy and is removed with the service or the last allow list. TCP single ports on the panel host only; `/0` is refused.
- **"Container running" healthcheck** (Service → Settings → Healthcheck) for docker services with no HTTP or TCP port to probe, such as UDP game servers, VoIP and queue workers: the deploy waits for the container to stay up for 10 s without a restart instead of probing a port it would never reach. The Port Mappings card offers it when only UDP is published. API: `healthCheck: 'http' | 'running'`.
- **Several IPs on one server** (Settings → Security → IP addresses, operator). The host's addresses are read live; an operator labels them, marks the **primary** one (named in the Deploy Wizard's DNS hint and offered as the sslip.io address for automatic URLs) and **dedicates** one to a docker service, after which no other service may publish ports on it. The Host IP field of a port mapping shows the labels and hides addresses dedicated to other services. API: `GET /v1/host-ips`, `GET /v1/host-ips/primary`, `PUT`/`DELETE /v1/host-ips/:ip`; SDK `hostIps`; CLI `ninedeploy ips list|set|forget`.
- **HTTP/3 switch** (Traefik → Overview, operator). Traefik always declared HTTP/3 on 443, but 443/udp was never published, so QUIC never reached it. Turning the switch on recreates Traefik with `-p 443:443/udp`; a failed recreate restores the previous container and answers 502. Off by default: the container and its fingerprint are exactly 0.16.0's, so upgrading does not recreate Traefik. API: `GET`/`PUT /v1/traefik/http3`; SDK `traefik.http3`.
- **Redirect rules per domain** (Network tab → a domain's Security panel → Redirects), like Dokploy's Redirects: a regular expression over the full URL → a target (`${1}` inserts the first group), temporary or permanent, up to 10 per domain, applied in order before auth, headers and rate limits. A "Moved path" preset fills in the common case. Traefik compiles them with Go RE2, so lookaround and backreferences are refused when saved, and a stored rule RE2 would refuse is skipped instead of taking the whole dynamic config down. API: `redirects` on `PATCH /v1/services/:id/domains/:domainId` and in the domain list.
- **Deploy status badge** (Service → Settings → Status badge): a public SVG for a README that reads running, deploying, failed, stopped or not deployed and nothing else about the service; `?label=` changes the "deploy" label. It is off until a member turns it on; the token in its URL is the only credential, can be rotated (the old URL then shows "not found") and turned off. API: `GET`/`POST`/`DELETE /v1/services/:id/badge`, public `GET /v1/badges/<token>.svg`; SDK `services.badge`.
- **Container options** (Service → Settings, operator only): GPUs (`--gpus all` or a count), shared memory (`--shm-size`, for browsers, Puppeteer and PyTorch), an init process (`--init`), the open-file limit (`--ulimit nofile`) and extra capabilities from a short allowlist (`NET_ADMIN`, `NET_RAW`, `SYS_PTRACE`, `SYS_NICE`, `SYS_RESOURCE`, `IPC_LOCK`, `SYS_TIME`). Structured fields, never free-form flags, so privileged mode, host mounts or the host network cannot be asked for, and `SYS_ADMIN`-class capabilities are not offered. Members see what is set and can still redeploy; a member's clone and a PR preview start without them. Applied on the panel host, in both the `docker run` path and the compose bridge; a remote node or Swarm deploy says in its log that they were not applied. API: `runtimeOptions` on `PATCH /v1/services/:id` and the service response.
- **Healthcheck timeout per service** (Service → Settings). A deploy waited a fixed 5 minutes for the new container; a slow first boot (migrations, model downloads) can now get up to an hour, and a quick one can fail faster. The failure message names the limit and where to raise it. PR previews keep the parent's value. API: `healthTimeoutSeconds` (10–3600, `null` = 300).
- **Shared variables UI.** Projects → Shared variables (`/projects?env=<id>`) edits the project-wide variables every service in the project receives. The API already had them, but nothing in the panel used it. A service's Environment tab lists what it inherits, with a link to the project and an "overridden" mark where its own variable wins.
- **Path routing in the panel.** The Network tab's domain form takes an optional path (`/api`), so several services can share one host. Pasting `https://app.example.com/api` fills both fields. The API has accepted `path` all along.
- SDK: `settings.autoUrl`, `domains.generate`, `onboarding`.

### Fixed

- **Two services could publish the same direct host port.** The second one was saved and its first deploy failed with Docker's "port is already allocated". Saving or creating it is now refused with the owner named (409), compared per host: the panel host, or the same node.
- `PATCH /v1/services/:id` with only a `build` object (no service field) answered 500 "No values to set".
- A PR preview of a static-pack service lost the parent's output directory and SPA setting, so it built with the defaults.
- The add-domain form showed "Could not add the domain" for every refusal; it now shows the server's reason (taken, outside the zone, reserved).

## [0.16.0] - 2026-10-17

> A fresh install answers on `http://<server-ip>:3000` straight away, behind a one-time setup token. The npm packages ship with every release again.

### Upgrade notes

- **No migration** (0073 is still the latest).
- **Your panel's bind does not change on upgrade.** Docker installs whose `.env` has no `NINEDEPLOY_BIND` stay on `127.0.0.1`; bare metal keeps its `NINEDEPLOY_HOST`. Only a fresh install (no `.env` yet) gets `0.0.0.0`, and only when the release it installs is 0.16.0 or newer, so the installer on `main` never exposes an older pinned release without the token gate.
- **The setup token only matters while the instance has no users.** A server that already has an admin never asks for it. A production server that was installed but never set up writes `DATA_DIR/setup-token` and logs the token at boot. Bare metal: `journalctl -u ninedeploy`. Docker: `docker compose logs ninedeploy`.
- **The master key loads as before.** It is now created atomically, and a symlink is read only when it points at a regular file. A key with characters after its 64 hex digits, which used to be accepted silently, still loads and now logs a warning. Rewrite it to exactly 64 hex characters when convenient.
- **Behaviour changes to know about:**
  - A redis or valkey backup whose `SAVE` does not answer OK now fails instead of copying a possibly stale dump.
  - Saving a Vault or AWS secret provider with a new address or endpoint drops the stored credential, so enter it again.
  - Creating a volume on a node now happens when it is attached, so attaching fails while the node is offline or runs an agent without `volume.manage`.
  - A system export started during an import answers 409.
  - The agent refuses to start when it cannot write its replay cache.
- **Rolling back to 0.15.8 is supported.** There is no schema change. A pinned host-key list written by 0.16 starts with a `#` generation line, which 0.15.8 skips.

### Added

- **Panel reachable after install.** Fresh installs publish the panel on `0.0.0.0` (Docker: `NINEDEPLOY_BIND=0.0.0.0` in `.env`; bare metal: `NINEDEPLOY_HOST`). They set `NINEDEPLOY_PUBLIC_URL` to the detected server address (`NINEDEPLOY_PUBLIC_IP` overrides the detection) and print that URL. They also warn when an active ufw blocks the port. `--private` (or `NINEDEPLOY_PRIVATE=1`) keeps the panel on `127.0.0.1` and prints the SSH-tunnel command.
- **First-run setup token.** The first account on an empty instance becomes the instance operator, so creating it now needs `NINEDEPLOY_SETUP_TOKEN`. The installer generates it, and the installer's `…/login#setup-token=…` link fills it in. The token is in the URL fragment, so it never reaches a request log. The gate covers `POST /v1/setup`, the first `/v1/auth/register` and the first SSO auto-enrolment. Without the variable, a production server writes `DATA_DIR/setup-token` and logs it at boot; development keeps the open bootstrap. `GET /v1/auth/status` reports `setupTokenRequired`. The CLI asks for the token, takes `--setup-token` or `NINEDEPLOY_SETUP_TOKEN`, and reads it from a local panel container on its own.
- **npm packages ship with each release.** A new `publish-npm` job publishes `ninedeploy`, `@ninedeploy/sdk`, `@ninedeploy/schemas`, `@ninedeploy/mcp` and `@ninedeploy/plugin-sdk` with provenance, after the release is promoted. An older tag is published as `previous`, so `latest` never moves backwards.

### Fixed

- Concurrency:
  - Access-grant holds are serialised per workspace.
  - A stale branding read can no longer overwrite the cache after a save.
  - Duplicate PR-preview creation no longer returns 500.
  - System export, import and panel backup exclude each other.
  - Node volume creation is serialised per name.
  - A failed restore setup no longer leaves the backup lock held.
- Backups:
  - Redis and valkey check the `SAVE` reply.
  - Dump file names carry a unique suffix.
  - A lowered preview cap evicts enough previews to fit.
- Host-key reset writes a generation marker, so a deploy that started before the reset cannot pin the old keys again.
- Smaller fixes:
  - Dashboard probes have a deadline.
  - Stale image-transfer archives are swept.
  - The terminal keeps output that arrives after the process exits.
  - GitHub App URLs starting with `HTTP:` are refused like `http:`.
  - `ninedeploy panel-backup` decryption no longer destroys an existing output file on a wrong passphrase.
- Kernel, plugin, sandbox and background logs go through the panel's structured logger.

## [0.15.8] - 2026-10-10

> Optional pinning of a Git host's SSH keys for deploy-key sources, and `deploys watch` that exits non-zero after a failed deploy.

### Upgrade notes

- **One small migration, 0073.** It adds two columns to `sources` (a flag that defaults to off, and a nullable text). No row is rewritten, so the upgrade is instant and a rollback to 0.15.7 boots on the migrated file.
- **Nothing changes for an existing source.** Pinning is off for every one of them, which keeps today's behaviour: each deploy accepts the host key it is shown and refuses one that changes during that deploy.
- **`ninedeploy deploys watch` now exits 1 when the deploy ended `failed` or `cancelled`.** Until now a clean end of the log stream exited 0 whatever the outcome, so `deploys watch && next-step` carried on after a failed deploy. If a script relied on the old exit code, check the status itself. When the status cannot be read the command exits 0 as before.
- **Rolling back to 0.15.7 is supported.** Pinning stops being enforced (clones are back to the per-job policy) and the pinned keys stay in the table for when you upgrade again. `docs/ROLLBACK.md` has the note.
- **Verified before release:** in-place upgrades from 0.15.7 and from 0.10.45, a rehearsed rollback to 0.15.7, the user-journey smoke (which now also drives the pinning setting), and a clone against a real OpenSSH server: the first clone learns the key, a second one with the same key passes, a server that then presents another key is refused with OpenSSH's own "host identification has changed" message, a source that does not pin still accepts it as before, and after a reset the new key is learnt and pinned.

### Added

- **Pin a Git host's SSH keys (opt-in, per source).** Sources → *Pin git host keys*, or `ninedeploy sources pin-host-keys <id> on|off`. The first clone after switching it on learns the host keys (only after the clone succeeded, which needs the deploy key); every later clone refuses a host that presents a different key, and the deploy log says `pinned host key changed` with what to do. Switching it on needs nothing extra. Switching it off and resetting the pinned keys (`sources reset-host-keys <id>`) let a changed key through, so both need an interactive session and your password (or a sign-in from the last 10 minutes, as for every step-up), and are audited. The panel shows fingerprints only (`sources host-keys <id>`), never the keys. It applies to clones the panel makes with the SSH deploy key; a clone on a remote node keeps its per-job policy.

### Fixed

- **`deploys watch` exited 0 after a failed or cancelled deploy** (see the upgrade notes).

## [0.15.7] - 2026-10-10

> 175 more one-click templates (130 → 305), every one of them started in an isolated container before it was listed, and password-reset and invitation emails through Resend.

### Upgrade notes

- No migration (0072 is still the latest). The catalog is data that ships with the panel; nothing about an existing service changes.
- **Services you already deployed are untouched.** A template is only read when you deploy from it, and a stack that is already running keeps the compose file it was created with.
- **New templates need Docker Compose on the host**, like every compose template before them, and are for operators only (compose stacks always were). Five of them mount the Docker socket or run privileged (Portainer, Glances, Authentik, OneDev and Autobase); the Hub says so on each.
- **Rolling back to 0.15.6 is supported.** Stacks deployed from the new templates keep running. 0.15.6 does not know those template ids, so it cannot show them in the Hub; redeploying such a stack from its service page still works.
- **Verified before release:** in-place upgrades from 0.15.6 and from 0.10.45, a rehearsed rollback to 0.15.6, and the user-journey smoke, which now also lists the catalog and deploys one converted stack through the panel's own compose pipeline.

### Added

- **175 templates from the Coolify catalog**, each marked verified in the Hub: Actual Budget, AnythingLLM, Authentik, Docmost, Immich, Karakeep, Kimai, Listmonk, Logto, Matrix Synapse, Miniflux, Paperless, Vikunja and many more. Each was started with `docker compose` in an isolated project and had to answer on its routed port. Categories were folded into the ones the Hub already had, so "Ai" and "AI" are one filter.
- **Password-reset and invitation emails can go through a Resend channel.** They used the SMTP channel only, so an install with just a Resend channel sent nothing. SMTP is still used first when both exist, so nothing changes for installs that already work; the message goes to the account's own address, never to the channel's shared recipients.

### Fixed

- **The Coolify mirror converter produced stacks Docker Compose refuses or runs wrongly.** It now declares the named volumes Coolify creates implicitly ("refers to undefined volume"), drops the Coolify-only `exclude_from_hc` and `is_directory` keys ("additional properties not allowed"), shows the resolved tag for images written `image:${TAG:-x}`, skips stacks that rely on a `COOLIFY_*` variable, and gives a bare `$VAR` the default written for `${VAR:-x}` elsewhere in the file (the mix that crash-looped umami). The converter is a tool for operators who build their own template source; its output for existing bundles is unchanged.
- The template smoke now generates `SERVICE_*` values with the panel's own resolver (a `HEX_` token is hex, a URL token is a URL), tears down a failed stack at once, and can test a candidate bundle before it is added to the catalog.

## [0.15.6] - 2026-10-10

> Two new database engines (KeyDB and Dragonfly), two new notification channels (Microsoft Teams and Resend), and a fix for secrets leaking into pushed Nixpacks images.

### Upgrade notes

- No migration (0072 is still the latest). Database engines and notification channel types are plain text columns.
- **One deploy can now be refused that was not before.** A service that builds with **Nixpacks**, has a **push registry** set (Service → Settings → Build), and has any secret variable, attached-database credential or vault reference now fails its next deploy with a message naming the variables. Nixpacks bakes its build environment into the image config, so pushing the image would let anyone with pull access read those values. The running version keeps serving. To deploy again, switch the service to a Dockerfile or Railpack, remove those variables from the build, or clear the push registry. Services that don't push to a registry, and Dockerfile and Railpack services, are unchanged.
- **KeyDB and Dragonfly databases on nodes need node agent v0.15.6.** Creating one on an older agent answers "update the node agent". Everything else in this release is on the panel.
- **Rolling back to 0.15.5 is supported, with two things to know.** If a KeyDB or Dragonfly database exists, the 0.15.5 database list answers an error for instance operators and its scheduled backups fail; delete or stop those databases before rolling back. If a Teams or Resend channel exists, 0.15.5 lists it but cannot send to it, and logs a failed delivery for every event until it is removed. `docs/ROLLBACK.md` has the details.
- **Verified before release:** in-place upgrades from 0.15.5 and from 0.10.45, a rehearsed rollback to 0.15.5, and the user-journey smoke, which now creates a KeyDB and a Dragonfly database for real, enforces the password, takes a backup, runs the drill, restores, and deletes it.

### Added

- **KeyDB and Dragonfly databases.** Redis-protocol engines that work like Redis in the panel: a generated password, `redis://` connection strings, backup and restore, RDB import, the backup drill, public access with TLS, the terminal and Studio, attachment to services, and databases on nodes.
  - **KeyDB:** `eqalpha/keydb`, with the architecture-specific tag (`x86_64_v6.3.4` or `arm64_v6.3.4`).
  - **Dragonfly:** `ghcr.io/dragonflydb/dragonfly` v1.34.2, started so that it saves the standard Redis dump file. Its dump is zero-padded, and the drill checks it with the image's own `redis-check-rdb`.
  - Redis and Valkey commands are unchanged.
- **Notification channels:** Microsoft Teams (a Workflows or legacy Office webhook, sent as an Adaptive Card) and Resend (the transactional email API). Secrets are encrypted at rest like the other channels, and the test button works for both.

### Fixed

- **Secrets in pushed Nixpacks images** (see the upgrade note above).
- **The backup drill reported valid backups as failed** on Docker installs. The drill copies the decrypted backup into a throwaway checker container, and the checker, running as root with every capability dropped, could not read a file owned by the panel's user. The container now keeps one read-only capability (`DAC_READ_SEARCH`) and still has no network. Found while testing KeyDB; the helper serves every engine, so check any drill that failed on a backup you know is good.
- The multi-node docs now say that a Nixpacks build on a build server leaves a copy of its build environment in the image on that server until the next deploy.

## [0.15.5] - 2026-10-10

> Remote nodes, part 4: every multi-node feature is now in the web panel, the CLI, the SDK and MCP. The whole series is verified on two real Docker hosts.

### Upgrade notes

- No migration (0072 from 0.15.2 already has every column this release uses).
- **Update node agents to v0.15.5** to get the clone fixes and the Railpack check below. Everything else in this release is on the panel and works with the agent versions that 0.15.2–0.15.4 required.
- **A successful "Test connection" on a node now refreshes what the panel knows about its agent at once.** Before, it took up to 5 minutes after an agent update.
- Rolling back to 0.15.4 is supported; this release adds no data the older release cannot read. `docs/ROLLBACK.md` now has one consolidated multi-node runbook for rolling back to 0.15.3, 0.15.2 or 0.15.1.
- **Verified before release:**
  - in-place upgrades from 0.15.4 and from 0.10.45, and a rehearsed rollback to 0.15.4;
  - the new two-host smoke: a fresh install, an upgrade from 0.15.1 with an old agent and then an updated one, a rollback to 0.15.1 with the new agent left running, and Nixpacks and Railpack builds on the node.

### Added

- **Web panel:**
  - **Servers:** each node shows its agent version and features, with an "update the agent" hint naming what is missing. It also has a build-server toggle with concurrency, Swarm join and leave (with the node opt-in hint), and its hosted database count.
  - **Service → Settings:** build placement (where it runs, the panel, or a build server), a push registry, image-transfer history, an "Orchestrator: Swarm" switch with the server's refusal shown inline, and Swarm tasks.
  - **Databases:** choose the host when creating one. Node databases show a node badge and their reachability. Studio, PgBouncer and public access are disabled on them, with the reason.
  - **Volumes:** a host switcher. **Sources:** "Allow on nodes", behind a password re-check.
  - **Settings → Swarm:** init, enable, the node list and the firewall guide.
  - **Deploy wizard:** operators can deploy a new docker or compose service to a node again. Nodes that cannot run it are disabled with the reason.
  - Seven new help topics.
- **CLI:**
  - `servers list|roles`;
  - `services placement|transfers|swarm`;
  - `swarm status|init|enable|disable|join|leave`;
  - `volumes create` and `--server` on volume commands;
  - `databases create --server`;
  - `sources allow-on-nodes`.

  Step-up actions prompt for your password.
- **SDK:** placement, image transfers, Swarm, server roles, node volumes and `allowOnNodes`. Existing signatures are unchanged and new response fields are optional.
- **MCP:** read-only `get_service_placement`, `list_image_transfers`, `get_service_swarm` and `get_swarm_status` (operator).
- **API:** `GET /v1/services/:id` includes `placement`. `GET /v1/servers` includes `swarmNodeId` and `swarmRole`. The OpenAPI document describes the node fields, and the schema lists every Swarm and placement error code.
- **Docs:** a new `docs/MULTI_NODE.md`. It covers enrolment, agent updates, a capability table with each feature's minimum agent version, node switch and refusal, build placement, private clones, node volumes and databases, and the Swarm firewall ports. `.env.example` documents every node-agent switch.
- **Smoke:** `scripts/smoke-multinode.mjs` (`pnpm smoke:multinode`) runs a panel and a node agent on two Docker hosts, in fresh, upgrade and rollback modes. `--with-builds` adds Nixpacks and Railpack on the node.

### Fixed

- A labelled volume attached to a node service was also created on the panel host.
- Attaching an existing volume by name to a node service checked the panel host. A volume that existed only on the node was refused, and a panel-host volume with the same name was accepted.
- **Node clones from git servers without shallow-clone support** (plain HTTP hosting, some self-hosted servers) failed, while the panel's own clone worked. The agent now retries once with a full clone or fetch. Needs agent v0.15.5.
- Every node deploy printed "network with name ninedeploy already exists".
- **Railpack on a node whose Docker uses the classic image store** now stops before building, with the fix in the deploy log: turn on the containerd image store, set `NINEDEPLOY_AGENT_BUILDKIT_HOST`, or build on the panel or a build server. Before, it failed inside BuildKit with "mergeop has been disabled". Needs agent v0.15.5.
- **Rollback runbook:** the Swarm clean-up now disconnects Traefik before removing a service's overlay network.

## [0.15.4] - 2026-10-09

> Remote nodes, part 3: Docker Swarm as an opt-in orchestrator. Nothing changes until an operator turns it on.

### Upgrade notes

- No migration (0072 from 0.15.2 already has the columns this release uses).
- **Swarm is off until an operator enables it.** The upgrade does not put Docker in Swarm mode or move any service onto a stack. Services with no orchestrator set, or set to `container`, deploy exactly as before.
- **Joining a node to Swarm needs node agent v0.15.4, and the node's owner must opt in.** Set `NINEDEPLOY_AGENT_SWARM_MANAGER=<panel advertise address>:2377` in the agent's environment and restart the agent. The agent joins only that manager, and never while `NINEDEPLOY_AGENT_DOCKER_SOCKET=off`. Older agents keep working for everything else and refuse join and leave with "update the node agent".
- **Swarm runs NineDeploy services only on nodes joined through the panel** (labelled `nd.member=1`). A node that joined the swarm another way gets no NineDeploy task.
- **Node owners:** a join is also refused while TLS Docker variables (`DOCKER_TLS_VERIFY`, `DOCKER_TLS`, `DOCKER_CERT_PATH`, an `https://` `DOCKER_HOST`) or a non-default Docker context are set on the node.
- **A server that is a Swarm member cannot be deleted** (409 `server_swarm_member`). Make it leave first.
- **Firewall:** Swarm overlay networks are always encrypted (IPsec). Between all Swarm hosts, allow 2377/tcp, 7946/tcp and udp, 4789/udp, and **ESP (IP protocol 50)**, restricted to the cluster's own hosts. Encrypted overlays do not work on Windows nodes.
- **Deploy-key clones on the panel now check SSH host keys.** They used to accept any host key; now a key that changes within one checkout is refused. Keys are not kept between deploys, so a server that rotates its host key does not block the next deploy.
- **Rolling back to 0.15.3 is supported.** Swarm services keep serving until Traefik restarts or the service is redeployed. The redeploy runs as plain containers and leaves the stack running without a route. `docs/ROLLBACK.md` lists the clean-up commands.
- Verified before release: in-place upgrades from 0.15.3 and from 0.10.45 leave Docker out of Swarm mode, and a rehearsed rollback to 0.15.3.

### Added

- **Docker Swarm (opt-in per service).**
  - **Turning it on:** an operator initialises Swarm on the panel host (an interactive session and a password re-check are required) and enables it. Nodes join and leave through their agents; leave drains the node first. Set a service's orchestrator to `swarm` and deploy.
  - **How a service runs:** one stack per service, applied with `docker stack deploy` on every deploy, so env, replicas and limits all take effect. Updates are rolling and start-first. If the panel's probe through Traefik fails, the panel rolls the service back.
  - **Images:** built on the panel or a build server. With a push registry every node pulls by digest. Without one, the image is copied to each joined node, and a node that cannot receive it gets no tasks.
  - **Security:** Traefik is the only way in; no port is published on the Swarm ingress mesh. Env reaches Swarm through a temporary 0600 file, never a command line.
  - **Join tokens:** never stored, logged, returned or put on a command line, and rotated after every join and leave.
  - **Node checks:** a node is checked on the manager before it is linked. It must have exactly the id it reported, be a worker, be at the server's address (by IP, or the DNS of its host name), and not be linked to any other server. `GET /v1/swarm` warns about a node still in the swarm after its owner turned the Docker socket switch off.
  - **Registry logins:** each deploy writes its own temporary Docker config with only that service's registry auth, and never runs `docker login`, so credentials never reach a shared config or the host's credential store.
  - **Management port:** `swarm init` binds it to the advertise address when that is a local interface; otherwise the response says to firewall 2377/tcp.
  - **Existing networks:** an existing `nd-swarm-<slug>` network that is unencrypted, or not an overlay, is refused instead of reused.
  - **Resource limits:** Swarm tasks get no memory-swap cap and no CPU shares; a stack file cannot express them.
  - **Refused on Swarm:** compose, PM2, a service pinned to a node, volumes, the Docker socket, a published host port, managed databases, fan-out targets, and env values spanning several lines. A PR preview of a Swarm service runs as a normal container.
  - **Day-to-day:** logs, rolling restart, stop/start (scale to 0 and back), the terminal (a task on the panel host) and stats all work for Swarm services.

### Fixed

- **The old, unreachable Swarm driver is replaced.** Its redeploys changed only the image. It put env on the command line and published ports past Traefik. It wrote state to a directory a Docker install cannot write. It never applied rotated secrets. Images built locally could not reach other nodes.
- Deploy-key clones on the panel host no longer accept any SSH host key.

## [0.15.3] - 2026-10-09

> Remote nodes, part 2: a build server, image transfer between hosts, and managed databases on nodes. Single-host installs keep working as before, apart from the two fixes below.

### Upgrade notes

- No migration (0072 from 0.15.2 already holds every column this release uses).
- **Managed databases on nodes need node agent v0.15.3.** Building on the panel or a build server and shipping the image needs v0.15.2. Re-run a node's bootstrap from the Servers page to update it. Older agents keep working as before, and anything new is refused with "update the node agent".
- **Two fixes reach existing services on their next deploy:**
  - **Attached databases are now connected to their service's network.** Previously only databases added through a template were, so a database attached any other way could not be reached by name from the app.
  - **PgBouncer:** a second sidecar no longer fails on port 6432. From the next time PgBouncer is enabled, only a non-default port is published on the host; running sidecars keep their binding until they are re-enabled.
- **Fan-out (copies of a service on other nodes):**
  - Copies of a Nixpacks, Railpack or static service now receive the primary's image instead of being rebuilt from a Dockerfile.
  - A copy that needs a command override, the Docker socket or volumes now needs node agent v0.15.2. Before, it started without them.
  - Deploy hooks still run on the primary only, and the deploy log now says so for each copy.
- Attaching a panel-host database to a service on a node now answers 409 instead of creating an attachment that could never deploy.
- **Rolling back to 0.15.2 is supported.** A database placed on a node keeps running on its node. 0.15.2 cannot manage it, refuses every action on it, and never starts a copy on the panel host.
- Verified before release: in-place upgrades from 0.15.2 and from 0.10.45, and a rehearsed rollback to 0.15.2. Panel-host databases send byte-identical Docker commands to v0.15.0 for 151 operations across all nine engines.

### Added

- **Build placement.** A service can build where it runs (the default, unchanged), on the panel, or on a build server, and then ship the image to its node. Building on the panel keeps private-repository credentials off your nodes.
  - **Image transfer:** by default the image goes through the panel over the sealed agent channel, checked end to end. A service can push and pull through a registry instead (operators bind the registry).
  - **Failures:** a failed build or transfer fails the deploy before anything switches over, so the running version keeps serving.
  - **Build servers:** mark a node as a build server and set how many builds it runs at once. Transfers are listed per service and per deployment.
- **Managed databases on nodes.** Operators can place a new database on a node. It runs on its own network, and services on the same node can attach to it.
  - **What works there:** start, stop, restart, limits, logs, credentials, backups and backup schedules, restore, dump import and the database shell.
  - **Refused for now:** Studio, PgBouncer, public access, connections across hosts, and moving a database between hosts.
  - **Server deletion:** a server that hosts databases cannot be deleted.

### Fixed

- Databases attached by hand were never connected to their service's network.
- A second PgBouncer sidecar failed because every sidecar published port 6432.
- Fan-out copies silently dropped a service's command override, Docker socket and volumes.
- Fan-out rebuilt Nixpacks, Railpack and static services from a Dockerfile.

## [0.15.2] - 2026-10-09

> Remote nodes, part 1: builds, private clones and volumes on nodes. Everything new is off until a node runs the updated agent; nothing changes for single-host installs.

### Upgrade notes

- One additive migration, `0072_multi_node`: a new `image_transfers` table and nullable or defaulted columns on databases, services, servers, sources, backups and deployments. No existing row changes behaviour.
- **The new node features need node agent v0.15.2.** Re-run a node's bootstrap from the Servers page to update it. An older agent keeps working exactly as before. Anything that needs the new agent is refused with "update the node agent to v0.15.2", and nothing else is sent to it.
- **The runtime image now includes `openssh-client`** (about 5–6 MB). Without it, deploy-key clones could not run in Docker installs or on nodes.
- **Railpack on nodes:** after the node agent is updated, a Railpack service placed on a node builds with Railpack itself instead of the Dockerfile stopgap. The resulting image can differ.
- **Scheduled backups of a service on a node** now back up its volume on that node. Before, they skipped it or copied a stale volume of the same name from the panel host.
- **Private repositories on nodes:** a source's personal access token or deploy key is sent to a node only after you turn on "Allow on nodes" for that source (this asks for your password). With the updated agent, a personal access token on a node needs that agent too, so the node owner's switch (`NINEDEPLOY_AGENT_STATIC_CREDENTIALS=off`) can refuse it. GitHub App sources are unaffected. Building on the panel and shipping the image (coming next) keeps credentials off nodes entirely.
- **Rolling back to 0.15.1 is supported.** The new columns and table are ignored. Node volumes and backups stay where they are. Services that use the new node features are refused by 0.15.1 as before.
- Verified before release: in-place upgrades from 0.15.1 and from 0.10.45, a rehearsed rollback to 0.15.1, and compatibility tests in both directions between panel and agent versions. The node features themselves are covered by unit tests. A two-host smoke arrives with the next part of this series.

### Added

- **Builds on nodes.** Nixpacks and Railpack run on the node. The panel works out the build pack on its own checkout with the same rules as a local build, so services that already build on a node send the same build inputs.
- **Private clones on nodes.** A personal access token or deploy key is sent only for one job, over the sealed agent transport. A deploy key is kept in memory-backed storage on the node and removed afterwards, even when the job fails. The node checks the Git host's SSH key: it records the key on first use and refuses a different one later.
- **Volumes on nodes.**
  - Create, list, back up and restore volumes on a node. Backups are encrypted like the panel's; the node's data never touches the panel's disk unencrypted.
  - Docker services on a node can use volume attachments, a command override and the Docker socket.
  - An attachment shared with a service or database on another host is refused.
- **Node agent.**
  - A sealed, checksum-verified data channel for images and volumes.
  - A structured run operation that accepts a validated spec, never a raw command line.
  - Image archives are checked before and after `docker load`, so an archive cannot replace the proxy's or NineDeploy's own image tags.
  - The Servers API shows each node's agent version and features.

### Fixed

- Compose stacks on a node with attached volumes failed at `compose up` ("external volume not found"). Missing service volumes are now created first.
- With a node selected, the volume file manager opened the panel host's volume of the same name. It now refuses.
- Deploy-key clones failed in Docker installs because the image had no SSH client.

## [0.15.1] - 2026-10-09

> Security patch: patched releases of two dependencies of the MCP server.

### Upgrade notes

- No migration and no configuration change. Upgrading from 0.15.0 changes nothing except the bundled dependencies.

### Security

- `@modelcontextprotocol/sdk` is now 1.32.1. Versions before 1.31.0 had an OAuth client that could send credentials to an authorization server an attacker steered it to (GHSA-6qxp-vccf-f47h, high).
- `proxy-addr` is now 2.0.8. Earlier versions could be fooled by IPv4-mapped addresses when deciding a client's IP (critical). It came in through the MCP SDK's HTTP server.
- NineDeploy's MCP server talks over stdio and uses neither code path, so these were not reachable in practice. Both now resolve to fixed releases anyway.
- One audit finding remains: `braces`, which pm2 depends on. No fixed release exists yet.

### Changed

- MCP: a tool whose inputs are all optional now runs when called with no arguments, as the MCP specification allows. Tools with a required input still refuse such a call.

## [0.15.0] - 2026-10-09

> Operations and API: browser and CLI terminals with a real TTY (also on remote nodes), opt-in traffic analytics, an OpenAPI 3.1 document for every route with generated MCP tools, and project- and environment-level access grants. See [docs/TERMINALS.md](docs/TERMINALS.md), [docs/TRAEFIK_INGRESS.md](docs/TRAEFIK_INGRESS.md), [docs/WORKSPACES_RBAC.md](docs/WORKSPACES_RBAC.md) and [docs/AI_MCP_CLI.md](docs/AI_MCP_CLI.md).

### Upgrade notes

- One additive migration, `0071_operations_api`: three new tables (`terminal_sessions`, `traffic_rollups`, `access_grants`). No existing row is changed.
- **Traefik is not touched by the upgrade.** Traffic analytics is off by default, and with it off Traefik's configuration is byte-identical to 0.14. Turning analytics on recreates Traefik once. On the json-file/local Docker log drivers, Traefik's own container log starts rotating (20 MB × 3) the next time the proxy is recreated for any reason.
- **Permissions are unchanged.** With no access grants, every access decision is identical to 0.14 (checked over 4839 caller × resource × action cases). Grants only ever raise access.
- **Terminals:** container shells are available to operators. Host shells (a root shell on the panel host or a node) are off until an operator enables them under Settings → Security, and each one needs a browser session and a password re-check. `NINEDEPLOY_HOST_TERMINAL=off` forbids them on the panel; `NINEDEPLOY_AGENT_HOST_TERMINAL=off` forbids them on a node.
- **Node terminals need node agent v0.15.0** (re-run the node's bootstrap from the Servers page). Older agents get a clear "update the agent" refusal; nothing else about them changes.
- **The web terminal in Docker installs gets a real TTY.** The runtime image never had python3, so the old shell always ran in pipe mode.
- WebSocket frames are capped at 1 MiB.
- **Rolling back to 0.14 is supported;** see [docs/ROLLBACK.md](docs/ROLLBACK.md). The new tables are ignored, so guests and raised roles lose their grant access (never the reverse). If analytics was on, 0.14 recreates Traefik without the access log; `<data>/traffic-logs` is left behind.
- Verified before release:
  - in-place upgrades from 0.14.0 and from 0.10.45, with every new feature exercised against a real Traefik and Docker;
  - an upgrade that leaves Traefik untouched;
  - a rehearsed rollback to 0.14.0.

### Added

- **Terminals.** Shells into services, databases and managed containers, on the panel host and on remote nodes.
  - A real TTY with resize, short-lived single-use tickets, idle and maximum-duration limits, and per-panel and per-user caps.
  - Every session is recorded (who, target, duration, bytes, IP) and audited at start and end. Operators can list and terminate sessions. Nothing typed or printed is stored.
  - Node sessions run over an encrypted, authenticated agent channel.
  - Available in the panel, as `ninedeploy terminal <service|db|container|host>` and through the SDK's `connectTerminal`.
- **Traffic analytics (opt-in).** Requests, status classes and latency per domain and service, read from a Traefik access log that keeps no client IP, path, query or headers. Rolled up per minute (48 h) and per hour (30 days by default). Turn it on and see the instance summary on the Traefik page; each service's Overview shows its own chart.
- **OpenAPI 3.1.** `GET /v1/openapi.json` (sign-in required) describes every route (435 operations), built from the same zod contracts the server validates with. A test fails if a route has no description or its access level disagrees with the authorization matrix.
- **MCP.** The server gains `search_api` and 16 read-only tools generated from the spec, including traffic summaries, access grants and terminal sessions. There are still no write tools.
- **Access grants.** A workspace admin can raise a user's role on one project, one environment, or a project+environment pair. A grant can also give a user without a workspace seat (a guest) access to just those resources.
  - Guests never get workspace-level rights and cannot create services or databases.
  - A grant only covers services in its own workspace.
  - Grants can be suspended and reinstated. An IdP (SCIM) suspension can only be lifted by the IdP.
  - Manage them in the Workspaces page and a project's Access view, with `ninedeploy access grants …`, or at `/v1/workspaces/:wid/access-grants`.
- **SDK and CLI:** `terminals`, `traffic`, `accessGrants`, `access` and `api.get`; `ninedeploy terminal|terminals|traffic|access`.

### Fixed

- The web terminal in Docker installs ran without a TTY (no python3 in the runtime image).
- The old service shell socket recorded only a start audit entry. It now records the session's end, duration and client too, and operators can terminate it.
- Terminal resize was never sent to the server.
- WebSocket frames up to 100 MiB were buffered before authentication. They are now capped at 1 MiB, and a plain subprotocol is preferred over echoing the bearer token.
- Eight places read a workspace seat directly instead of going through the access check. They now share one access path.

## [0.14.0] - 2026-10-08

> Network and data access: public database access, editable Traefik configuration with your own certificates, database dump import, and HashiCorp Vault / OpenBao and AWS Secrets Manager. See [docs/TRAEFIK_INGRESS.md](docs/TRAEFIK_INGRESS.md), [docs/DATABASES_BACKUPS.md](docs/DATABASES_BACKUPS.md) and [docs/SECRET_MANAGERS.md](docs/SECRET_MANAGERS.md).

### Upgrade notes

- One additive migration, `0070_network_data_access`: four new tables (`database_public_access`, `tls_certificates`, `database_imports`, `secret_providers`). No existing row is changed.
- **Traefik is recreated once** on the first 0.14 boot. Traefik now reads its dynamic configuration from a directory (`/etc/traefik/dynamic`). The current routes are copied there before the recreate, so sites keep serving. If the recreate cannot happen (for example the image pull fails), the old container keeps running and route changes are mirrored to the old file until it succeeds.
- **Remote nodes start serving their routes.** Since remote servers were introduced, a node's Traefik read a file the agent never wrote, so it loaded no routes. Nodes recreate their proxy once after the upgrade and then serve the routes the panel sends.
- **Certificate-expiry alerts start working.** Expiry dates of Let's Encrypt certificates were never read (Traefik stores them base64-encoded), so the `cert-expiry` alert could not fire. After the upgrade it fires for certificates that are really about to expire.
- Everything else is opt-in. No database is exposed, no custom config or certificate exists, and no secret provider is configured until an operator sets one up.
- `${{vault:…}}` and `${{aws:…}}` references stay literal while their provider is not configured, exactly as before. PR previews now withhold them.
- **Rolling back to 0.13** is supported; see [docs/ROLLBACK.md](docs/ROLLBACK.md).
  - Traefik goes back to the single file.
  - Delete the leftover `<data>/traefik/dynamic/` directory: it holds uploaded private keys.
  - Remove public-database sidecars with `docker rm -f $(docker ps -aq --filter label=ninedeploy.public-db)`.
  - Containers receive the literal text of `vault:` and `aws:` references.
- Verified before release:
  - in-place upgrades from 0.13.0 and from 0.10.45;
  - every new feature exercised on the upgraded panel with a real Traefik;
  - a rehearsed rollback to 0.13.0.

### Added

- **Public database access.** An operator can publish a managed postgres, mysql, mariadb, redis, valkey or mongo database on one host port.
  - It runs through its own Traefik TCP sidecar (`nd-dbpub-<slug>`); the main Traefik is never touched.
  - An IP allow-list is required (at most 100 ranges, never `/0`).
  - TLS termination with an uploaded certificate is optional; it is not available for mysql/mariadb.
  - Database credentials gain `publicConnectionString`.
  - Off by default. The connection uses the database's root credentials, so create a limited user.
- **Custom Traefik dynamic config.** Operators can add their own routers, middlewares and services.
  - Every name must start with `custom-`, so it can't collide with the panel's own routes.
  - Each save is validated, then tried in a throwaway Traefik container before it is applied. If the live Traefik rejects it, the panel reverts to the last good version.
- **Uploaded TLS certificates.** Operators can upload PEM certificates and keys; the key is stored encrypted. A domain fully covered by an uploaded certificate stops requesting a Let's Encrypt certificate. Uploaded certificates feed the certificate inventory and the expiry alerts.
- **Database dump import.**
  - Sources: resumable 8 MiB chunked uploads (database admin), or an object from a backup destination (operator).
  - Formats: postgres (custom or plain SQL), mysql/mariadb (SQL), mongo (archive) and redis/valkey (RDB), each optionally gzipped.
  - A `pre-import` safety backup is taken first, and a check afterwards confirms the panel can still sign in.
- **Secret managers.** Env values can reference HashiCorp Vault / OpenBao KV v2 (`${{vault:path#field}}`) and AWS Secrets Manager (`${{aws:id}}` or `${{aws:id#key}}`).
  - Configure them under Settings → Integrations. Infisical and Doppler keep working as before.
  - Every call goes through the egress guard.
- **Surfaces.**
  - SDK, with `importFile` for chunked, resumable uploads.
  - CLI: `ninedeploy databases public-access|import|imports`, `proxy config`, `certificates custom` and `secrets providers`.
  - MCP read-only tools `get_database_public_access` and `list_database_imports`.

### Fixed

- Remote-node proxies loaded no routes: the static config pointed at `dynamic.yml`, while the agent writes `dynamic/ninedeploy.yml`.
- Let's Encrypt certificate expiry was never read, so the `cert-expiry` alert never fired, and the certificate inventory always showed the issuer "Let's Encrypt" and the source `acme.json`.

## [0.13.0] - 2026-10-08

> GitHub App integration: short-lived installation tokens instead of long-lived PATs, App webhooks, commit statuses and PR comments, private clones on remote nodes, and Gitea repository listing. See [docs/GITHUB_APP.md](docs/GITHUB_APP.md).

### Upgrade notes

- One additive migration, `0069_github_app`: four new tables (`github_apps`, `github_app_installations`, `service_github_links`, `github_pr_comments`) and a nullable `sources.base_url` column. No existing row is changed.
- Nothing changes until an operator registers a GitHub App. PAT, deploy-key and webhook services deploy exactly as before, and their per-service webhooks keep working.
- Moving an existing service to the App is explicit and reversible: **migrate** links it while keeping its PAT and webhook, **finalize** detaches them, and **revert** goes back.
- Deploying an App repository to a **remote node** needs the node agent v0.13.0 or newer over the sealed transport, and git 2.31 or newer on the node. An older agent gets a refusal telling you to update it; nothing is minted. The check runs when the deploy is queued.
- Deleting an App in the panel also removes its installation rows and service links. The generated `github_app` sources stay, and their clones then fail closed until you attach another credential.
- Rolling back to 0.12 is safe: the new tables and column are ignored, migrated-but-not-finalized services keep their PAT and webhook, App-only services fail closed on private repositories (no secret is exposed), and `/v1/hooks/github-app/*` answers 404. Details in docs/GITHUB_APP.md §9.
- Verified before release: in-place upgrades from 0.12.0 and from 0.10.45. After the upgrade a pre-existing GitHub PAT source still lists and tests, and the App API, Gitea base URL and manifest guards work on the upgraded panel.

### Added

- **GitHub App setup.**
  - One-click setup through GitHub's manifest flow, for a personal account or an organization (Sources → GitHub Apps).
  - Manual entry for GitHub Enterprise Server or an existing App. The webhook is registered automatically.
  - Maintenance: install, sync installations, rotate the private key or webhook secret, re-point the webhook after a domain change.
  - The private key, webhook secret and client secret are encrypted at rest and covered by master-key rotation; no route returns them.
- **Installations as sources.** Each installation becomes a `github_app` source. Clones use a repository-scoped, read-only installation token that lives in memory for at most an hour. A token is only ever sent to the App's own GitHub host.
- **App webhooks.** Pushes and pull requests arrive at `POST /v1/hooks/github-app/<key>`, signature-checked with replay protection, and are routed by repository id, so a renamed or transferred repository keeps deploying and the service's URL is corrected. A service linked to the App skips its old per-service webhook; while the installation is suspended the old webhook takes over again.
- **Service links.** The Deploy Wizard links a new service to the App repository it picked; existing services can migrate, finalize and revert from the service's Settings → GitHub card, the API (`/v1/services/:id/github`), SDK `services.github` or CLI `ninedeploy services github`.
- **Commit statuses and PR comments.** Opt-in per service: a `ninedeploy/<slug>` commit status for each deploy, and one preview comment per pull request, edited in place with the URL and outcome. Feedback never fails a deploy.
- **Private clones on remote nodes.** For App repositories, the panel mints a per-job token, sends it sealed to the agent (capability `git.credential`), which passes it to git only through the environment, and revokes it when the job ends.
- **Gitea base URL.** Gitea sources take a base URL; the token test and the Deploy Wizard's repository and branch listing then work against it.
- **Surfaces.** SDK `githubApps` and `services.github`; CLI `ninedeploy github-app list|show|add-manual|sync|rotate-key|remove`, `services github` and `sources add --base-url`; MCP tool `list_github_installations`.

### Fixed

- `/v1/hooks/<non-numeric id>` answers 404 instead of 500.
- After a session expired, the login redirect now keeps the query string, so returning from GitHub's setup pages still completes the setup.

## [0.12.0] - 2026-10-08

> Backups, preview environments and disk alerts — the first release of the [roadmap](docs/ROADMAP.md).

### Upgrade notes

- Two additive migrations: `0067_database_backup_policies` and `0068_preview_env_vars`. Both are new tables only. No existing row is changed, and a rollback to 0.11.x ignores both tables.
- Nothing changes until you configure it:
  - A database without a backup policy keeps the daily backup with 7 kept.
  - Panel self-backup is off.
  - No alert rules are created.
  - PR previews get exactly the environment they got before.
- The panel now pings approved remote nodes every 30 seconds. SSH-bootstrapped nodes therefore stay "online" on the Servers page while they are reachable. They used to show offline about 5 minutes after bootstrap.
- Verified before release: in-place upgrades from 0.11.2 and from 0.10.45, plus the feature checks above on the upgraded panel.

### Added

- **Per-database backup policy.**
  - Settings: a cron schedule, local retention (1–365), optional remote retention, and a destination or local-only.
  - Where to configure it: the database's Backups tab, `GET`/`PUT /v1/databases/:id/backup-policy`, SDK `backups.getPolicy`/`setPolicy`, or CLI `ninedeploy databases backup-policy get|set`.
  - Retention never prunes the newest good dump or the newest remote copy.
  - Missed-backup alerts follow each policy's own interval.
  - Choosing a specific destination is operator-only.
- **Panel self-backup.**
  - What is backed up: the panel database (a consistent snapshot), master key, `.env` and Traefik config.
  - It is written to a backup destination on a schedule, encrypted with a recovery passphrase the operator keeps, and old copies are pruned by keep-newest-N retention.
  - Runs never overlap. Every outcome is audited, so a failure triggers notifications.
  - Where to use it: the Settings → Panel backup section, `/v1/system/panel-backup`, SDK `system.panelBackup`, or CLI `ninedeploy system panel-backup status|set|now|list|decrypt`.
  - Restore: from Settings after typing the file name to confirm, or offline with `decrypt`.
  - See `docs/PANEL_BACKUP.md`.
- **Preview-only environment.** A git-backed service can hold environment values that only its PR previews receive. Previews still never receive the service's secrets.
  - Where to set them: Env tab → Preview deployments, `/v1/services/:id/env/preview`, SDK `previewEnv`, or CLI `ninedeploy env list|set|rm --preview`.
  - Precedence: project env < the preview's own copy < the preview-only set < attached databases.
  - Existing previews pick up the set on their next deploy.
- **Alert metrics `disk` and `server_offline`.**
  - `disk` is the worst disk-usage percentage across the panel host, Docker's data root and the remote nodes.
  - `server_offline` is the number of minutes a remote node has gone unseen.
  - Both are host-wide. Firing and recovery notify through the existing channels.

### Changed

- System export and import share one archive implementation (`lib/systemArchive.ts`) with panel backup. The API and the archive format are unchanged.
- An alert rule PATCH now validates the merged rule. An existing `cert-expiry` rule can no longer be scoped to a service.

## [0.11.2] - 2026-10-08

> Security release: a failed submodule clone could put the Git source's access token into deploy logs, the audit log and notifications.

### Upgrade notes

- **Rotate Git source tokens that may have leaked.** Before this release, a deploy whose repository had a submodule the token could not clone wrote the token into:
  - the deploy log (live stream and `logs/<id>.log`);
  - the `deploy.failed` audit entry;
  - every notification sent from that entry: Slack, Telegram, webhook, email, the activity feed and plugins.

  If such a deploy ever failed on your instance, revoke the token at the provider and update the source. Existing log files and audit entries are not rewritten.
- No database migrations or new configuration fields.
- The OIDC provider `defaultRole` field is deprecated. The API still accepts and returns it, but it has had no effect since 0.11.0.

### Security

- A failed clone's git output is now redacted before it is logged or stored. git rebuilds a submodule's URL from the tokenized origin and prints it on failure; that text was passed through unchanged (F1013).

### Fixed

- A failed deploy clone names its reason in the deploy log and the failure message: authentication failed, HTTP 403, not found or no access, could not resolve/connect, TLS/host-key failure, or missing branch. It adds advice for GitHub tokens and says when a submodule's repository needs access too (F1012).
- The Deploy Wizard shows why a repository list is short: the classic token lacks `repo`, the list was capped, a later page failed, or the provider returned an error. The SDK gains `sources.reposWithDiagnostics` (F1010).
- Dashboards served from a separate origin can read that diagnostic, because CORS now exposes `x-nd-source-error` (F1016).
- `ninedeploy sources test` prints the token type, scopes and warnings. Provider-supplied text is stripped of terminal control sequences (F1011).

### Deprecated

- OIDC provider `defaultRole`. OIDC users always own their personal workspace; add them to team workspaces with invitations or SCIM. The SSO settings form no longer shows the field (F1014).

## [0.11.1] - 2026-10-08

> Private repositories: complete repository lists, clone errors that say why, and a token check.

### Upgrade notes

- No database migrations or new configuration fields.
- `POST /v1/insights` and `POST /v1/services/:id/insights/refresh` answer 400 `repo_unreachable` or `branch_not_found` when the repository cannot be cloned. The first answered 500 and the refresh route answered 404 before.
- `GET /v1/sources/:id/test` returns `tokenKind`, `scopes` and `warnings` for GitHub sources. These fields are additive.

### Fixed

- Git source repository lists stopped at the provider's first page (100 repositories, most recently updated first), so older private repositories never appeared. Lists now follow GitHub `Link`, GitLab `X-Next-Page` and Bitbucket `next` pagination up to 10 pages (1000 repositories). Later pages reuse the fixed API URL, so the token is never sent to a provider-supplied URL, and truncation is reported.
- Analysing a repository the selected credential cannot clone answered "Internal Server Error". The answer now names a reason: authentication failed, HTTP 403 (permission denied), not found or no access, could not resolve/connect, or TLS/host-key verification failed. For GitHub tokens it adds advice: classic tokens need the `repo` scope, and fine-grained tokens need the repository selected, Contents: Read-only, and organization approval or SSO authorization. git's output, URL credentials and the token never appear in the message.

### Added

- Sources: a "Test token" button shows the login, the GitHub token type, a classic token's scopes, and warnings when private repositories cannot be listed or cloned.
- The Deploy Wizard explains, next to the repository picker, why a repository may be missing from a GitHub source.

## [0.11.0] - 2026-10-07

> The evidence audit release: over 300 defects, each reproduced on the previous code before it was fixed, then verified and, where it mattered, pinned by a regression test.

### Upgrade notes

- No database migrations or new configuration fields.
- **API changes:**
  - `POST /v1/build-cache/store` answers HTTP 400 `{ error }` for a missing key, malformed digest or unknown cache (was `200 { ok: false }`). It also accepts an optional `ref` (`<repo>@sha256:<64 hex>`).
  - Compose services and compose templates refuse `publishedPort`, `cpuShares`, `memLimitMb`, `cpuLimitMilli` and image overrides with 400. Nothing ever applied them; the compose file controls these.
  - Deleting a backup whose off-site copy cannot be removed answers 502 `remote_delete_failed` instead of orphaning the remote dump.
  - Deleting a project that still holds databases answers 409.
  - A deactivated account cannot be added to a workspace or given ownership (409 `user_deactivated`, SCIM 403).
- **CLI:**
  - `ninedeploy deploys watch` exits 1 on any close other than a clean end, including when its 30-minute wait limit is reached. Ctrl-C still exits 0.
  - The source-token environment variable is `NINEDEPLOY_SOURCE_TOKEN`.
- **One-shot repairs on first boot:**
  - Team services and databases still owned by a deactivated user move to the workspace owner (`ownership_repair_deactivated_owner_r976`).
  - SSO personal workspaces get their owner seat back.
  - Both write audit entries and never block startup.
- **Behaviour:**
  - The build cache now stores a pullable image reference (new marker and registry-manifest fields).
  - An FCM channel whose device token is rejected as dead is deactivated.
  - Docker inspects that time out (10 s) fail instead of falling through to a pull.
  - A pre-0.3.0 cleartext invite link answers 404.
  - `alerts: []` in a manifest clears the alert rules that manifest created.
  - Preset values are stored encrypted.

### Security

- Enabling the host firewall could lock the operator out. ufw rejected most generated rules, including the SSH/HTTP/HTTPS safety rules. A failed safety rule was ignored, and the panel port was never allowed.
- Tenant-supplied header and basic-auth values were evaluated as Traefik Go templates, which could leak environment secrets. A covering wildcard router could outrank an exact host and skip basic auth or IP allow-lists.
- Templates could deploy onto another tenant's attached volume. A member could claim another tenant's future PR-preview hostnames, or the www/apex half of an active redirect. A pending domain transfer outlived its initiator's admin seat.
- An HTTPS token stayed in `.git/config` of relative submodules and was copied into images. Repository framework detection had a ReDoS on member-controlled files.
- `logs search` printed tenant container output raw, so terminal escape sequences (clipboard writes, title changes) reached the operator's terminal.
- Okta's pathless SCIM `PATCH active:false` was ignored, leaving the user active. The demo seed gave every tenant owner owner rights on the operator's demo service.
- Audit events for service lifecycle and manual deploys carry the correct service and deployment ids. A service name can no longer spoof another id to webhook-out consumers. Plugins receive only an allow-listed part of audit metadata.

### Fixed

- **Data safety:**
  - Backup retention could delete the last good backup after a failed upload.
  - Restore checked only the primary container while replicas kept running on the volume.
  - Prune, delete and the doctor treated database-adopted volumes as ownerless and could destroy stopped database data.
- **DNS:**
  - A Namecheap reply without a host list could wipe the whole zone on the next write.
  - Concurrent record changes in one zone overwrote each other.
  - IPv6 targets were sent as CNAME records; they are now AAAA.
- **Deploys:**
  - A template service with a second attached database took its database settings from the wrong one.
  - Compose deploys picked the wrong main container.
  - A pipeline crash left deployments `building` for up to 45 minutes.
  - Scheduled 6-field crons fired every second.
  - A second database could take an env alias already in use.
- **Runtime:**
  - About 25 Docker calls that could hang a deploy, a health check or a request for 30 minutes are now bounded.
  - A user-stopped service is no longer restarted by the background reconcile.
  - The deploy-log WebSocket closes once a deploy settles.
  - Deleted services' networks no longer exhaust Docker's address pools.
  - Deleted PM2 services no longer come back at boot.
- **Build cache:**
  - Cache hits now reach `docker buildx --cache-from` on the inline, registry and S3 backends.
  - Malformed digests are refused instead of overwriting a valid entry.
- **Plugins and kernel:**
  - Uninstalling a refused plugin, or a purge with wildcard characters, could wipe a built-in plugin's settings and secrets.
  - Plugins reusing a menu item id took over each other's sidebar entries.
  - The sticky-IP egress SNAT never took effect.
  - Template deploys trigger the manifest generator again.
- **Tenancy:**
  - Deactivated users' team services move to the workspace owner, and their seats no longer keep resources.
  - SSO sign-ups always own their personal workspace.
- **Passkeys:** sign-in works, and passkeys enrolled on earlier versions migrate to the canonical credential id on first use.
- **Self-update:** a second update start during the in-flight check could launch a second root installer.
- **Web:**
  - The deploy wizard and Network tab no longer offer settings compose services ignore.
  - Plugin Reload/Enable buttons match what the server allows.
  - The project delete dialog describes the database rule correctly.

### Tests

- Permanent regressions cover the race, ownership, High and Critical fixes: reconcile and stop races, concurrent passkey migration, domain transfer accept, seat hand-over, audit entity formats, deploy-log close codes and the boot repairs.

## [0.10.45] - 2026-10-05

> Deployment reliability and shared-contract fixes from the evidence-led audit.

### Upgrade notes

- No database migrations or new configuration fields.
- Existing source builds on remote nodes retain the agent requirements introduced in earlier releases.

### Fixed

- Deployment queues, filesystem locks and lifecycle cleanup handle failure paths consistently. Stale auto-update probes no longer update a service whose deployment settings changed while the probe was running.
- Community template creation uses exclusive writes, preserving an existing template when concurrent installs select the same name. Explicit replacement remains available.
- Agent connection caching follows the current node host and port. Node proxy sync errors identify the failed operation stage even when a transport error omits its name.
- Log timestamps preserve fractional nanoseconds. Log search, runtime log selection, deployment history and scheduled-job fixtures include the preceding audit repairs.
- Manifest import/export preserves explicit `redirectWww: false`; service creation retains `healthPath`. SDK manifests and errors, CLI configuration and terminal formatting include the audited contract fixes.
- Volume inventory continues past an attachment whose service is absent to find a later valid link. S3 canonical query names and values encode `/`, while object-key separators remain literal.
- Label workspace filters no longer truncate fractional or partial values into another workspace ID.
- Dashboard deploy-log selection resets old buffers, and WebSocket URLs retain configured API path prefixes. SSO fragment parsing recognizes the exact token key.
- Deleted workspace/project/label chips are removed even when the authoritative catalog becomes empty. Unsupported saved theme and accent values fall back to supported defaults.
- A completed update timer can dismiss only its own success banner. A later successful update retains its full display window.
- Includes the earlier local audit fixes for DNS provider responses, session expiry and rotation, backup parsing, image references and shared utility boundaries.

### Tests

- Permanent regressions cover concurrent template creation, stale auto-update completion, session rotation, deploy-log selection and successive panel-update timers.
- Shell-based installer and update fixtures use the native Git Bash installation on Windows rather than a WSL launcher.

## [0.10.44] - 2026-10-03

> Ownership follow-up to 0.10.43 (r710–r711): team services whose creator was
> removed by SCIM before 0.10.43 deployed without their project's shared
> environment; the first boot of this release hands them to the workspace
> owner. Repository analysis now needs the same seat as creating a service.

### Upgrade notes

- **One-time ownership hand-over on first boot (r710).** Services and managed databases in a workspace whose creator no longer holds a seat there (SCIM suspend or deprovision before 0.10.43) now belong to the workspace owner — what 0.10.43 does for every new removal. The change is recorded in the audit log (`ownership.backfill`) and the panel log. Untouched: personal (untagged) resources, operator-owned resources, creators still seated in any workspace the resource lives in, and services shared across workspaces with different owners (logged; reassign them by hand). It runs once; reinstating a user later restores the seat, not the ownership.
- **Repository analysis needs a member seat (r711).** `POST /v1/insights` (the Deploy Wizard's analysis step) answers 403 for accounts with no seat or only viewer seats, the same rule as creating a service. Operators and members are unaffected.
- No migrations.

### Fixed

- **Team services lost their project's shared env after a SCIM removal (r710).** The deploy pipeline injects a project's shared env only when the service owner is seated in the project's workspace; a creator removed by SCIM before 0.10.43 stayed the owner without a seat, so redeploys came up without those variables and without an error.

### Security

- **Repository analysis was open to any signed-in account (r711).** A seatless or viewer-only account could have the panel clone arbitrary public repositories (bounded and rate-limited, but outside the role model every other write follows).

### Docs

- README → Verifying a release: image signatures need cosign v3, or v2.6+ with `--new-bundle-format`; older cosign reports "no signatures found".

## [0.10.43] - 2026-10-02

> The verified-release release (r690–r703): releases are signed and their
> assets checksummed, installs verify what they download, and every API route
> is now held to an authorization matrix that fails the build when a new route
> is not classified.

### Upgrade notes

- **Creators who lost their seat lose access (r694).** A service or database creator counts as its owner only while they hold a seat in a workspace it belongs to. Someone removed from the workspace (for example by SCIM) now gets "not found"; re-seating them restores access. Personal resources outside any workspace stay their creator's. Workspace owners always hold a seat, so they are unaffected.
- **SCIM removal hands ownership over (r695).** Suspending or deprovisioning a user through SCIM transfers what they created to the workspace owner, as the panel's own member-removal already did. Reinstating the user restores the seat, not the ownership.
- **Installs verify downloads (r702).** From this release on, the installer downloads the release's own source archive and checks it against the signed `SHA256SUMS`; when `cosign` (v2.4+) is installed it also verifies the signature. A mismatch stops the install and the previous service keeps running. Hosts that cannot reach Sigstore can set `NINEDEPLOY_SKIP_SIGNATURE_VERIFY=1` (the checksum is still enforced). Older tags install as before, with a warning.
- **Self-update verifies the installer (r703).** Takes effect from the first update a 0.10.43 panel starts.
- Revoking something that is already gone, or that is not yours, now answers 404 instead of 200 (r691).
- No migrations.

### Security

- **Seatless creators kept owner access (r694, high).** Whoever created a team service or database kept full access after losing their seat — secret env export, database credentials, backup downloads, deploy and delete (82 routes).
- **Cross-tenant ids on service create left half-created services (r692);** the backup-drill route revealed whether another tenant's backup id exists (r693); owner-scoped deletes answered 200 for ids outside the caller's scope (r691).
- **Authorization matrix (r690).** A test boots the real API on a real database, enumerates every route (361), and checks each against two tenants and every role: no cross-tenant access, no write or outside call on refusal, the role floor holds, no secret in a viewer's response. A new route without a classification fails the build.

### Release integrity

- **Signed images (r700).** Every published image is signed with Sigstore keyless signing (GitHub OIDC, no stored key) and carries build provenance. The exact `cosign verify`, `cosign verify-blob` and `gh attestation verify` commands are in README → Verifying a release.
- **Checksummed, signed release assets (r701).** Each release carries `ninedeploy-vX.Y.Z.tar.gz`, `install.sh`, `SHA256SUMS` and its signature bundle. The release is created as a draft, its assets are verified the way installers check them, and only then published — `releases/latest` never points at a release without checksums.
- What this guarantees: the archive, installer and image are byte-for-byte what this repository's release workflow produced. What it does not: the trust root is still GitHub and Sigstore.

## [0.10.42] - 2026-10-02

> The second-pass audit release (r630–r680): ingress, managed data, PR previews
> and remote nodes. Two bugs could freeze routing for the whole instance, PR
> previews still received production credentials, and managed Redis/Valkey
> have never started since 0.2.2. All fixed, each with a regression test proven
> to fail on 0.10.41, and every refusal says what to do.

### Upgrade notes — read before updating

- **Update your node agents (r660).** Agents older than 0.10.42 are refused for remote **source** builds (repository Dockerfile, repository compose, source fan-out) — the error names the node. Image deploys and inline compose stacks keep working. To update a node: on that node run `docker rm -f ninedeploy-agent`, then run the agent command shown for it on the Servers page (it carries the panel's version). Every older panel keeps working with the new agent.
- **SSH host keys are now pinned (r661).** The first SSH test or bootstrap after updating trusts and records the node's host key (shown as a warning). A reinstalled host then fails until you enter its new fingerprint on the Servers page.
- **Managed Redis/Valkey start now (r644).** Databases created before stayed in `error`; start them from the database page (or delete and recreate).
- **PR previews lose production credentials (r651).** They no longer receive project-shared secrets, vault references, the parent's databases, or a manifest `database:` attachment; each preview's deploy log lists what was withheld. Give a preview what it needs explicitly: set env vars on the preview service, or attach a non-production database to it.
- **Manifest restrictions (r650, r652).** A `.ninedeploy` `database:` section only attaches databases the service owner administers; `notifications:` only applies to operator-owned services (existing subscriptions on other services stop delivering).
- **Volume changes on privileged services (r640).** Attaching, updating, detaching or repairing a volume on a compose, static-pack, lifecycle-hook or docker-socket service now needs the same rights as deploying it; members get a 403 naming why.
- **Domains (r630–r637).**
  - Hostnames are validated on input; stored rows the proxy would have rewritten are skipped (and audited) instead of routed.
  - Sticky sessions and plaintext basic-auth entries now actually work (both were silently broken).
  - SSL domains redirect plain HTTP to HTTPS.
  - Pending domains older than 30 days are removed.
  - Non-operators are capped at 50 own-zone domains per service and 30 additions per hour (configurable via `PUT /v1/settings/domain-policy`; operators exempt).
- **Template databases (r641, r642).** A new dependency database never adopts a retained volume its owner cannot prove is theirs; it gets the next free name (`-db-2` …) with a deploy-log note.
- **Remote backups (r645).** An off-site copy that should be encrypted but is not is refused on restore. Pre-0.10.3 plaintext volume snapshots keep restoring.
- **Dockerfile with a base directory (r666).** When a Dockerfile exists both at the repository path and under the base directory, the same file as before is built and the deploy log says how to switch.
- No migrations.

### Security

- **Hostname check vs render mismatch (r630).** A member could store a hostname the checks treated as one host and the proxy rendered as a catch-all — routing every tenant's automatic domain to their own container.
- **PR previews inherited production credentials (r651)** through project secrets, vault references and manifest database attachments — reachable by anyone who can push a branch and open a PR.
- **Manifest attached databases the owner could only view (r650)**, handing their password to the container.
- **Volume routes queued redeploys without the deploy privilege check (r640)** — a member could get a privileged operator-owned stack redeployed from a repository they push to. One enqueue helper now serves every user-triggered deploy, pinned by a wiring test.
- **Template databases could adopt a deleted tenant's retained volume (r641)**; dependency slugs could be squatted (r642).
- **Node agents followed symlinked build paths (r660)** in a workspace shared by every tenant on the node — and wrote workspace files through a symlinked `.tmp` as root. Agents now resolve every build path component by component.
- **SSH bootstrap did not verify host keys (r661)** while carrying the agent's sealing key.
- Also: a viewer seat sufficed to stack routes on another service's host (r632); the `www` companion skipped claim checks (r633); pending rows squatted hostnames (r635); basic auth was stored and shown in plaintext (r636); PgBouncer status showed members the database password (r643); `--requirepass` leaked into logs (r644); remote restores accepted unencrypted objects (r645); manifests could subscribe operator channels (r652); webhook templates allowed JSON injection and Discord `@everyone` (r653); log search returned a reused slug's previous owner's logs (r654); sealed agent requests without a nonce, and replays across an agent restart (r668).

### Fixed

- **Routing froze for the whole instance** when any service enabled sticky sessions (Traefik has no `sticky` middleware) (r631) or ran more than one replica (mis-indented `healthCheck`) (r638). A golden test now validates every middleware the panel can emit.
- **Managed Redis/Valkey never started (r644):** `--requirepass` was passed before the image. The user-journey smoke now creates a managed Postgres and Redis on every release (r680).
- **Deleting a database left its off-site dumps (r646);** concurrent restores of one remote backup clobbered each other (r647); volume names could collide with another service's primary volume (r648); the volume list ran a helper container per attachment per request (r649).
- Bundle import is transactional and claim-checked (r656); repository analysis clones are shallow and bounded (r657); webhook-out no longer follows redirects and `/ai/config` hides the base URL from non-operators (r658); Doctor flags a plaintext `http://` template source (r655).
- Node checkouts are removed on delete/move (r662); fan-out targets are torn down on service delete (r662); IPv6-safe host normalisation (r663); ssh errors no longer echo the remote command (r664); terminal and container routes refuse node-pinned services with a clear message (r665); bounded command capture and log reads (r667).

## [0.10.41] - 2026-10-02

> **Security release — update now.** A second-pass audit found that a signed-in
> user could make the panel read a file on the panel host through a symlink
> committed to a repository and get its content back — any file the panel
> process can read: its environment (`NINEDEPLOY_JWT_SECRET`), `master.key`,
> the panel database, and on bare-metal installs (which run as root) system
> files. Every release before 0.10.41 is affected.

### What to do after updating

1. Open **Doctor**. A critical finding **"Repository analysis stored data that is not a Node version"** means stored analyses captured host content: follow its rotation steps.
2. **If people you do not fully trust have accounts on your panel, rotate secrets even when Doctor shows nothing** — the on-demand analysis route returned the content without storing it, so it leaves no trace in the database:
   - set a new `NINEDEPLOY_JWT_SECRET` and restart (every session signs out);
   - add a new master key to `NINEDEPLOY_MASTER_KEYS` and run `ninedeploy system rotate-keys`;
   - rotate registry, git, S3, DNS and vault credentials stored in the panel, and the SSH keys of the panel host if it is a bare-metal install;
   - have users change their passwords if the panel database may have been read.
   The panel's request log (`journalctl -u ninedeploy`, or `docker logs` in docker mode) records each `POST /v1/insights` call with its client address, for as long as your log retention keeps it; reviewing those narrows this down.

### Security

- **Symlinked repository files were followed on the panel host (r620).** Repository analysis and the `.ninedeploy` loader now read only regular files inside the checkout (no symlinks, FIFOs or devices; `O_NOFOLLOW`; bounded reads). A detected Node version must look like one — anything else is dropped, and values stored by earlier releases are no longer returned by the API and are flagged by Doctor (`repo_insights_leak`).
- **Workspace email overrides could send arbitrary mail through the instance (r621)** — a regression in 0.10.40, where they were first wired into sending. Overrides now apply only to workspaces an instance operator owns and must keep `{{acceptUrl}}`; elsewhere the built-in text is sent, exactly as before 0.10.40.
- **Git never runs hooks from a checkout (r622)** — `core.hooksPath=/dev/null` on every clone, fetch and submodule update.

### Upgrade notes

- No migrations. Stored analyses are left in place (Doctor needs them to tell you what happened); re-run analysis or redeploy to rewrite them.
- A `.ninedeploy` that is a symlink now fails with a message naming it instead of being followed — commit the file itself.
- Custom invitation emails on workspaces not owned by an instance operator stop being sent (the built-in text goes out); saving one there answers 400 with the reason.


## [0.10.40] - 2026-10-02

> The first published build of the 0.10.38 changes — read the 0.10.38
> section below, including its upgrade notes. Neither the 0.10.38 nor the
> 0.10.39 tag reached a panel: the smoke-gated release pipeline (r581)
> stopped each one before any image was pushed, `:latest` moved or a GitHub
> Release was created.

### Fixed

- **Railpack on arm64 (r584).** Railpack names its arm64 Linux binary `arm64-unknown-linux-musl`, not `aarch64-…` (Nixpacks' spelling). The arm64 image build 404'd, and `install.sh` on arm64 bare-metal hosts has silently never installed railpack since it was introduced. Both archives re-verified against Railpack's published checksums.

## [0.10.39] - 2026-10-02 — not published, superseded by 0.10.40

> The first published build of the 0.10.38 changes. The 0.10.38 tag never
> reached a panel: its release checks failed on a test that only broke on
> Node 26, and the smoke-gated pipeline (r581) pushed no image, moved no
> `:latest` and created no GitHub Release. Read the 0.10.38 section below —
> including its upgrade notes — for everything this release contains.

### Fixed

- The r605 pinned-dispatcher test built its Agent from the Node 22/24 undici slot and failed on Node 26 (CI's Node). It now uses the same `bundledAgentClass()` lookup `guardedFetch` uses. The production code was already correct, verified on Node 22.13, 24 and 26.

## [0.10.38] - 2026-10-02 — not published, superseded by 0.10.39

> The proven-upgrade release (r580–r610): releases are now only offered to
> installed servers after the published image has been upgraded onto from the
> previous release and walked through the user journey — and the follow-ups the
> 0.10.37 audit deliberately left out.

### Upgrade notes — read before updating

- **Workspace invitations (r604).** For non-operators, adding an email that is not yet a member always sends an invitation, even when an account exists — the colleague accepts it from the link. Instance operators still add registered accounts instantly.
- **Sandbox plugins on Node < 25 (r600).** Already-installed sandbox plugins keep loading; installing a new one (or reinstalling) is refused unless you upgrade Node to ≥ 25 or set `NINEDEPLOY_ALLOW_SANDBOX_NETWORK=1`. Doctor lists sandbox plugins that have open network on this Node.
- **Email templates (r610).** Workspace invitation overrides you saved earlier are now actually sent. Without an override, emails are byte-for-byte what 0.10.37 sent. Password-reset overrides are refused (a workspace admin must not rewrite another user's reset email); stored ones are ignored and can be deleted.
- **Railpack (r582).** Ships in the container image now. Railpack builds need a BuildKit daemon: set `BUILDKIT_HOST` (e.g. `docker-container://buildkit` next to a `moby/buildkit` container). Without it, saving or deploying a railpack service is refused with that fix named — such builds never worked before.
- **Compose redeploys (r590)** build before taking the old stack down, so the previous stack keeps serving during the build and a failed build leaves it running.
- No migrations, no breaking API changes; new response fields are additive.

### Release process

- **Publication is gated on end-to-end smokes (r581).** The release workflow pushes `:vX.Y.Z` only, then runs `smoke:upgrade` (previous published release → this tag, on one data volume) and `smoke:user-journey` against the pushed image; `:latest` and the GitHub Release — which is how panels and the installer discover updates — are created only after both pass.
- `pnpm smoke:upgrade` (r580): boots the previous release on a data volume, seeds an operator, a deployed service with a domain, an orphaned project secret, and a deploy hard-killed mid-build; boots the new release on the same volume and proves sign-in, data, running apps, interrupted-deploy recovery, a fresh green deploy and the migration journal. Proven locally for 0.10.35 → 0.10.37 and 0.10.36 → 0.10.37.

### Security

- **DNS-rebinding window closed (r605).** `guardedFetch` connects only to the addresses it vetted (pinned lookup on Node's bundled undici Agent — verified on Node 22.13, 24 and 26), keeping SNI and the Host header.
- **Email enumeration through member-add and invitations (r604)** — registered and unregistered emails now get the same response.
- **Sandbox network on Node < 25 (r600)** — new installs refused unless knowingly allowed; plugin list shows `networkRestricted`.
- **Vault references at write time (r601)** on template deploys and bundle import.
- **Preview patterns from `.ninedeploy` manifests are validated (r602)** — and the manifest's documented `{n}` placeholder is finally understood (it was never substituted).
- **Passkeys after a password change (r603)** — the response and Account page say how many passkeys remain, with a link to review them.
- **Password-reset emails cannot be rewritten by a workspace admin (r610)**; subject lines cannot inject headers.

### Fixed

- **Workspace email templates were never sent (r610)** — invitations from both invite paths now render the workspace's override; a wiring guard pins every template to a sender.
- **Compose deploys took the stack down before building (r590).**
- **Node-pinned compose services are health-patrolled (r591).**
- **Fan-out source builds pull private base images with the bound registry credential (r592).**
- **A container started by a deploy interrupted by a panel restart is removed at boot (r593)** — containers now carry `ninedeploy.deployment` / `ninedeploy.service` labels.
- **Job runs left `running` by a crash are closed (r594).**
- **Railpack never worked (r582)**: it requires `BUILDKIT_HOST`, which nothing provided and the build env stripped; now passed through and checked up front.
- Manifest Creator suggests a preview pattern the panel accepts (r602); README test counts corrected (r583).

## [0.10.37] - 2026-10-02

> The tenancy release (r500–r576): a whole-repo deep audit found that several
> per-instance resources (the vault, registry credentials, preview hosts, SCIM,
> the plugin event bus) were reachable across tenants. All fixed, each with a
> regression test proven to fail on 0.10.35, and every behaviour change ships
> with an upgrade path so a 0.10.35 panel updates in place.

### Upgrade notes — read before updating

- **Vault allowlist (r510).** On first boot the allowlist is seeded from current use (workspaces of non-operator services/projects that already reference the vault), audited as `settings.vault_allowlist_seeded` and logged. Working deploys keep working; review the list in Settings → Integrations → Vault.
- **Registry credentials (r512).** Each registry source is bound to the hosts its services pull from today (audited as `source.registry_hosts_seeded`). A credential is no longer sent to any other host.
- **Preview patterns (r511).** Stored patterns without both `{{pr}}` and `{{slug}}` keep deploying previews but provision no domain (`previewDomainSkipped: pattern_requires_pr_and_slug`).
- **Passkeys / TOTP (r502)** ask for your password (or a sign-in from the last 10 minutes) when enrolling.
- **SSO (r505).** A tab loaded before the update will be asked to sign in again once.
- **SCIM (r501).** Pushing an account that is not already a member of the token's workspace now answers 409 — invite it first.
- **AI (r508).** Non-operators only get AI through a seat in an operator-owned workspace.
- **Studios (r560)** open in a new tab; a studio tab open during the update asks you to reopen it once.
- **Node-pinned services (r522, r523).** Services with pre/post-deploy or pre-stop hooks, and exec jobs on PM2/node services, are refused with the fix named (clear the hook, or unpin the service) — they used to run on the panel host.
- **Railpack on container installs (r520)** is refused up front; it used to fail mid-build.
- **Sandbox plugins (r530, r531)** that emit events outside `plugin.<id>.*` keep running but those emits are dropped; a previously installed plugin whose id equals a built-in is shown as errored.
- **Daily backups (r528)** now notify channels subscribed to backup events.
- **Migration 0066** deletes orphaned project secrets and builds six indexes once at start.
- **Docker installs (r571).** The image is pinned to the installed tag via `NINEDEPLOY_IMAGE_TAG` in `.env`; upgrade by re-running the installer (a bare `compose pull` re-pulls the same release).
- **Doctor (r575)** warns on bare-metal installs that listen on `0.0.0.0` over plain HTTP. The default bind is unchanged; set `NINEDEPLOY_HOST=127.0.0.1` once the panel is behind HTTPS.

### Security

- **Instance vault readable by any member (r510).** `${{infisical|doppler:KEY}}` resolved with the operator's vault token for every service. Now only operator-owned services and allowlisted workspaces resolve references; non-operators writing a reference elsewhere get a 403.
- **Preview-domain takeover (r511).** A member-editable preview pattern could render another tenant's host and steal its traffic. Patterns must contain `{{pr}}` and `{{slug}}`, and the rendered host goes through the own-zone claim check.
- **Registry credential exfiltration (r512).** Changing `image` pointed `docker login` and the auto-update probe at any host. Credentials are now bound to registry hosts, and members cannot repoint such a service to another registry.
- **Auto-update skipped the owner-privilege check (r513)**; its registry probe was a blind SSRF for member-owned services (r514).
- **Login lockout never tripped (r500).** Per-IP locks zeroed the counters the account tier sums. Rewritten with a per-account window; IPv6 bucketed by /64.
- **SCIM cross-tenant deactivation (r501).** A workspace SCIM token could adopt any account by email and deactivate it instance-wide.
- **Step-up for durable credentials (r502)**, **immediate session revocation (r503)**, **constant-time unknown-email login (r504)**.
- **Login-CSRF via URL fragment (r505).** Fragment tokens are only accepted on `/auth/callback` with a per-tab nonce; `returnTo` is restricted to same-origin paths; the SSO button honours a split API origin.
- **Prototype poisoning (r506)**: the raw-body JSON parser now uses Fastify's protected parser (malformed JSON is a 400, not a 500).
- **OIDC email-domain restriction (r507)** implemented as documented. **AI spend gate (r508)** can no longer be passed by creating your own workspace.
- **Plugin sandbox (r530–r534).** Forged kernel events (e.g. `audit.recorded` → real DNS record deletion) are blocked: plugins emit only `plugin.<id>.*` and the bus tags plugin origin. Built-in id collisions refused; per-call hook ids; secrets redacted at the IPC boundary. On Node ≥ 25 sandboxes have no network; on Node 22/24 they still do (documented).
- **Studio isolation (r560).** Studios opened in a same-origin iframe could read the panel's session tokens through `parent`. They now open in a `noopener` tab with `frame-ancestors 'none'`, and the studio cookie is bound to the user and their token version.

### Fixed

- **Railpack double build (r520)**; railpack refused on container installs, which ship no railpack CLI.
- **Node deploys reported success while the node's proxy kept pointing at the removed container (r521).**
- **Deploy hooks and exec jobs on node-pinned services ran on the panel host (r522, r523).**
- **A restart mid-deploy blocked local deploys for up to 45 minutes (r524)**; interrupted deploys are failed at boot.
- **Node-pinned services were never health-patrolled (r525)**; an unreachable node raises `alert.node_unreachable` instead of "service down".
- **Remote builds over 5 minutes died as `fetch failed` (r526)** — long agent ops now use `node:http`; stop grace periods reach the node; fan-out logs into the registry before building.
- **Disabling or reloading a plugin erased its saved configuration (r527).**
- **Scheduled backups never fired `backup.completed` (r528).**
- **Deleting a user deleted every workspace they owned (r540)**; ownership now transfers to the acting operator (or `transferTo`).
- **Project-scoped secrets survived project deletion (r541)**; migration 0066 removes existing orphans.
- **Remote copies of scheduled backups were never pruned (r542)**; **backups left `running` by a crash (r543)**; **one failing housekeeping step skipped the rest (r544)**; **hourly full-table scans (r545)**; **a failing migration could stay half-applied (r546)**.
- **Web:** env/settings forms no longer save over data that failed to load (r562); full deploy-log download (r561); SDK-backed calls replace raw fetches (r563); accessible palette and events drawer (r564); robust log reconnect de-dup (r565); one shared, typed plugin-menu query (r566).
- **CLI/SDK:** token refresh and base-URL sub-paths for raw calls (r550); logout revokes only its own session (r551); SDK types match the server (r552); dead schema removed (r553); prune flags validated (r554); unused MCP dependency dropped (r555); published source maps resolve (r556); new `networks.members`, `traefik.config/version/update` (r557).

### Installer, release and docs

- `latest_tag` prefers the published release over a freshly pushed tag, and its warnings no longer leak into the captured tag (r571). Docker installs pin the image tag, keep the previous compose/`.env` for a real rollback, and wait up to `NINEDEPLOY_HEALTH_TIMEOUT` (300 s) for migrations; the bare-metal health gate probes `NINEDEPLOY_HOST` (r570).
- Self-update answers 409 `deploys_in_flight` while a deploy is building, unless `force: true` (r572).
- `bump-version.js` validates before writing; `:latest` only moves forward; per-tag release concurrency (r573).
- apache2/nginx are only stopped on fresh installs that need :80/:443; Nixpacks checksums keyed by version; provenance comment no longer overclaims (r574).
- Doctor finding `panel_plaintext_exposure`; README/ARCHITECTURE counts corrected (r575). CI exercises the real `install.sh` helpers and runs a non-blocking `pnpm audit` (r576).

## [0.10.36] - 2026-10-02

> The bounded-backstop release (r480): the audit of the rejection backstop
> found its own memory hole and two polish items. The security surface is now
> bounded, honest in shape, and self-documenting.

### Fixed

- **The rejection map grew without bound (P2).** Every distinct source IP that ever produced one 400/401/403 left a permanent entry — freed only when that exact IP returned after its window drained. Ordinary background internet scanning adds entries monotonically on any public panel, and a single IPv6 /64 can mint unlimited source IPs at one request each: ~5k distinct-IP requests/s for an hour ≈ 18M entries ≈ 4–5 GB. **Prune-on-write** past a 10,000-key soft cap: entries whose newest rejection has aged out go first (currently-engaged IPs survive the sweep), then oldest-inserted keys as the hard ceiling. Bounded to a few MB. Churn regression test included (12,000 one-shot IPs through the production path).

- **The backstop 429 used the Fastify-default body shape (P3).** The web client parses the app error envelope (`{ error: { code, message } }`) — with the default shape it rendered the generic "Request failed with status 429" at exactly the flood moment this release targets. The refusal now carries the envelope (code `rate_limited`, the real retry countdown in the message) plus the same `retry-after` header as before.

### Hardened

- The user-journey smoke **fails fast** when it cannot derive the current version from `version.ts` (it previously fell back to `v0.0.0`, burned the whole DinD bring-up, and died at image pull with a misleading error).

- Changelog wording corrected: the backstop counts **all** pre-limiter and in-handler 400/401/403 deaths, not only guard rejections — same ceiling, same never-legitimate threshold, now stated as the code behaves.


## [0.10.35] - 2026-10-02

> The backstop release (r479): the fresh-eyes audit of the owner-throttle fix
> found the fix's own hole — requests that die at a privilege guard had
> escaped rate limiting entirely.

### Fixed

- **Guard-rejections were unmetered (P2, regression from r478 found by audit).** The principal limiter is appended to route-level preHandlers, but the module guards (`requireAdmin`/`requireOperator`, ~25 modules) are **instance-level** preHandlers — they run FIRST, and a thrown 403 short-circuits the lifecycle before the limiter ever fires. A valid low-privilege credential (member session, read-only API token) could hammer operator-gated routes unmetered: three DB round trips of auth per request, unbounded RPS, DB-pool exhaustion. The same blind spot covered 401s (bad tokens — a pre-existing hole, now closed too) and parse-failed 400s. **Backstop:** responses that die with 400/401/403 are counted per IP in the same 1000/min window; past the cap the IP is refused outright with the same retry-after shape. Successful traffic is never counted at IP level (the principal limiter owns it), so the owner-throttle guarantee is intact — reaching the backstop takes a thousand guard-rejections in a minute, which is never legitimate traffic. Regression test: an instance-level guard 403ing before the route-level limiter still ends in a 429.

- **`/v1/about` pooled the operator into the anonymous bucket (P3).** Its optional auth resolved in the handler — after the limiter had already keyed by IP. Optional auth now runs at onRequest (invalid tokens still swallowed: the public subset serves, never a 401), so the limiter sees the principal here too.

- **The r478 WebSocket trade-off note was factually wrong (P3).** `@fastify/websocket` hijacks the socket in the route handler, not at onRequest — upgrades DO traverse the preHandler limiter (IP-keyed). The behavior was safer than documented; comment and changelog corrected to say what actually happens.

### Hardened

- The user-journey smoke now derives its default image from the repo version instead of a hardcoded tag — a bare run could prove the journey green on a two-releases-old artifact, exactly the drift class the script exists to catch.


## [0.10.34] - 2026-10-01

> The owner-throttle release (r478): reported live by the owner — the panel
> rate-limited its own operator on the dashboard. The limiter now buckets by
> PRINCIPAL, not by IP.

### Fixed

- **An authenticated operator could be throttled by their own panel (P1, reported live).** The global limiter bucketed by IP only — behind a proxy or NAT, every client (browser tabs, the CLI, uptime monitors, anonymous probing) shared one 1000/min bucket, and a busy neighbor could lock the owner out of their own dashboard ("Rate limit exceeded, retry in 23 seconds"). The limiter now runs **after authentication** and buckets by principal:
  - authenticated requests key as `user:<id>` — the dashboard's own polling can never be crowded out, and a runaway script with a valid token is still capped per account;
  - unauthenticated requests (login brute-force, `/setup`, the public webhook receiver, agent `/announce`) keep the per-IP bucket — and the tight per-route ceilings on exactly those surfaces (20/min login, 10/min setup, 60/min hooks) are untouched, so the brute-force posture is unchanged;
  - trade-off, stated openly: WebSocket upgrades hijack the socket at onRequest and no longer pass the limiter — every WS endpoint authenticates in the subprotocol and revalidates every 60 s, and connection floods are a proxy-level (Traefik) concern, not an application-rate one.

  Verified by test: two authenticated users from the SAME IP draw independent buckets while anonymous traffic draws a third, and per-route ceilings still fire.


## [0.10.33] - 2026-10-01

> The journey release (r477): the product now proves its own core loop on the
> published image — and the very first run of that journey caught a real bug
> no unit test had: a bodyless JSON-declared request answered 500.

### Added

- **`pnpm smoke:user-journey`** — the user-journey smoke. Spins the validated DinD topology (privileged docker:28-dind sidecar; the panel image under test gets plain `DOCKER_HOST=tcp://…`) and drives the product's core loop against the PUBLISHED artifact: register the first admin → create a docker-image service → trigger a deploy and wait for green (deployment rows use `running` as the success terminal — the events layer translates to `success`) → read logs → route a domain → delete the service → full teardown with diagnostics (panel log tail) on any failure. Unit and route tests cover each hop; this proves they still compose on the image a user actually pulls. It joins the release drill.

### Fixed

- **A bodyless `application/json` request answered 500 (found by the journey's first run).** The rawBody plugin — which overrides Fastify's JSON parser to capture exact bytes for webhook HMAC verification — called `JSON.parse` on the raw buffer with no empty-body guard, so `JSON.parse('')` threw into a 500. Real clients send exactly this shape (fetch wrappers that always set content-type on DELETEs — our own smoke among them); Fastify's own parser treats the empty body as `{}`. The override now mirrors that default. Regression test added.

### Security

- **basic-ftp pinned to 6.2.1** (scoped override): <=6.2.0 has a quadratic-time CPU DoS in `Client.list()`'s Unix LIST parser (2026-10-01 advisory); it rides `apps/server > pm2 > proxy-agent > pac-proxy-agent > get-uri`. Prod audit clean again.

### Chain & docs polish (the r475–r476 delta audit: 0 P1 / 0 P2 / 4 P3)

- `ninedeploy backups restore` documented with BOTH required args (`<databaseId> <backupId>`).
- The repo-wide createClient `token:` guard now also sweeps `apps/web/src` — the in-product help surface this project rewrites most often was the one surface unchecked.
- **The release scripts share one package list** (`scripts/lib/package-list.mjs`, imported by bump-version and tag-release) with a test pinning it against the real workspace globs — the two hardcoded 10-entry copies could drift, and an 11th package would have shipped at a stale version with every gate green.
- Webhook help no longer promises a branch picker the panel UI does not have (the Environment tab matches the service branch; override via CLI), and the Sources help topic points at the real flow instead of a Sources-page wizard that does not exist.


## [0.10.32] - 2026-10-01

> The truth-in-docs release (r476): the documentation sweep that had never
> been done, plus the audit of the two chain-guard releases. The docs were
> not merely stale — they actively lied, in four P1 ways, and two of them
> were fresh copies of a bug this project had already fixed once.

### Fixed — documentation that lied

- **Two more copies of the unauthenticated-SDK bug (P1).** r472 found `docs/AI_MCP_CLI.md` teaching `createClient({ token: … })` — an option that does not exist (the SDK takes `getToken`), silently ignored, so a copied example 401s on every call. The sweep found the SAME example in the **root README** and in the **website's docs page**. Both fixed — and a repo-wide test now greps every doc-bearing surface (README, docs/, website, package READMEs) for the `token:` pattern so this bug family cannot ship a third generation.
- **The README taught a nonexistent CLI command (P1)** — `ninedeploy watch 12 480`; the real command is `deploys watch`. And the **website's CLI cheatsheet was written for an imaginary CLI (P1)**: six of ten lines broken — `services deploy my-app` style operands where the CLI takes numeric IDs, `sources deploy-key` (real: `sources keygen`), `webhooks create` (real: `webhooks add <serviceId> [branch]`), plus a mislabeled `system dashboard`. Rewritten against the actual command surface.
- **The in-product help taught the pre-hardening privilege model (P2)** — "anyone who is owner or admin in at least one workspace is an operator": precisely the self-promotion path migration 0038 eliminated. The help now states the operator flag is per-instance (granted on the People page, never inherited), and member-visible topics no longer point at operator-only surfaces (Sources/Volumes/Activity/Docker/Notifications are marked operators-only; the webhook tip points at the real location — the service's Environment tab, not Sources; the agent-transport claim now says sealed envelopes, not "plain HTTP").
- **Docs accuracy wave (P2/P3):** the microkernel doc no longer teaches the r473-removed `rollback` option or the nonexistent `ctx.events.on` member; the website plugin example uses the real `ctx.tapHook` and the real install sources (marketplace|sandbox — not "NPM packages, Git repos, local directories", which r440 removed); TROUBLESHOOTING's diagnostic command works (`services logs <id>`); the backups doc drops the nonexistent "MCP restore tool"; QUICKSTART names BOTH dev commands (`pnpm dev` for the API, `pnpm dev:web` for :5173); every API-token pointer says Settings → **Security**; install.sh no longer cites a nonexistent docs page. Counts refreshed against reality: 38 MCP tools (was 35, in seven places), 52 tables (was 41), 67 route modules (was 48), 130 templates across twenty categories (was 89/ten), test badge aligned with its own table.

### Fixed — release chain (the r474/r475 diff audit: 0 P1, 1 P2, 3 P3)

- **`release:tag` certified the wrong tree when given a commit (P2).** Its compile/test gates run against the working tree, but provenance reads the commitish — tagging an older SHA ran the gates on HEAD while proving the old commit (false pass), and tagging during a mid-bump HEAD false-blocked. The gate now refuses unless commitish == HEAD: check out what you mean.
- Provenance checks **all ten package.jsons**, not just the root (the workspace packages are what `packages:publish` ships). An unresolvable commitish now dies with a `✗` diagnostic instead of a Node stack trace.
- **bump-version.js treats install-command files as critical**: a silent pattern miss on the `--version` occurrences in About/QUICKSTART/README used to pin every copier to the previous release while the script claimed success.


## [0.10.31] - 2026-10-01

> The chain-guard release (r475): the release chain audited itself. The last
> never-audited surface produced this round's only P1 — and the guards it
> shipped are the ones yesterday's v0.10.30 escape proved were missing.

### Security / Release integrity

- **Version provenance is verified before anything is built or pushed (P1).** Tags are created by hand (`git tag vX <commit>`), and nothing in the chain compared the tag's VALUE to what the checkout contains — a typo or a stale SHA would pass every check (it IS a green commit), push a mislabeled image whose panel reports a different VERSION, silently roll `:latest` backward, and create a GitHub Release the CHANGELOG asset contradicts. `release-publish.yml` now asserts **tag == package.json == version.ts** immediately after checkout, before install/build/push; `release-workflows.test.ts` pins the step's existence, its content and its position before `pnpm install`.

### Fixed

- **The local gate now means what CI's gate means (P2).** `pnpm release:check` was a single `turbo run typecheck lint build test` — turbo has no dependency edges between those task names, so the phases could interleave (and a warm tree could satisfy `server#test` from a stale `apps/web/dist`). CI's verify job has always run the four invocations sequentially; `release:check` now mirrors it: `pnpm typecheck && pnpm lint && pnpm build && pnpm turbo run test --concurrency=1`. This is exactly the gap the 0.10.30 unparseable-`version.ts` escape slipped through locally.

- **bump-version.js refuses to lie (P2).** A pattern miss on the load-bearing `VERSION` literal printed a warning and exited 0 — ten package.jsons saying 0.10.31 while the panel kept reporting 0.10.30, every gate green. That miss is now fatal, and the script closes by asserting every package.json and the `VERSION` literal actually landed on the new version before printing its success banner.

### Added

- **`pnpm release:tag` — the local tag gate.** Wraps `git tag` with the checks the drill previously held as discipline: clean working tree; the tag does not already exist (re-tagging stays an explicit delete-and-recreate); **tag == package.json == version.ts == CHANGELOG[0]** read from the exact commitish via `git show` (not the working tree); no unfilled stub survives; and the server typechecks — the precise step an unescaped-apostrophe changelog fails. Used for this very release.

### Verified sound (for the record)

CI's structure held under the audit: build-before-checks, checks-before-push, `needs`-gated edge publish, least-privilege permissions, no `pull_request_target`/fork-secret exposure, strict tag-shape validation; bump-version.js is idempotent with no flag bypass of the >=1.0.0 owner policy. The parked dependency majors stay parked **with evidence**: drizzle-kit 0.31.11 still depends on the deprecated `@esbuild-kit/esm-loader` (our patch remains required; the bump buys nothing), and js-yaml 4.3.2 carries no advisory while pm2 pins the 4.x line.


## [0.10.30] - 2026-10-01

> The clean-sweep release (r474): the fresh-eyes audit of 0.10.29 came back
> **0 P1 / 0 P2** — the notification encryption, the SDK-honouring bootstraps
> and the member gating all held. What ships here are the three P3s it did
> find, plus the two debris items the r473 audit had noted and deferred.

### Fixed

- **The SDK type for the masked notification config (P3).** `listChannels` still promised `configJson: string | null` without the new `hasConfig` field — so a consumer read `null` on a webhook/FCM channel and had no way to distinguish "no config" from "masked". Worse, a read-modify-write flow that rebuilt `configJson` from that null and PATCHed it back would have **silently destroyed the stored HMAC secret / FCM service account** (PATCH treats any non-empty string as the full replacement). The type now carries `hasConfig: boolean` and documents that non-Discord blobs are write-only: send the full config or omit the field.

- **Empty webhook branches fall back again (P3).** r473's charset rule turned `{"branch": ""}` (or whitespace-only) into a 400; the route's historical behaviour was to treat it as absent and use the service's own branch. The schema now normalizes empty/whitespace to `undefined` before validation — the git charset rule still applies to non-empty values.

- **A malformed plugin definition no longer disguises itself as an init timeout (P3).** r473 forwards the object a plugin's code returns verbatim — unlike the install manifest, that object passed no schema. A `configSchema: 5` or a menu item with a numeric id threw inside the READY handler, the message wrapper swallowed it, and the only symptom was a ten-second "timed out during initialization" that hid the cause and stalled plugin load. The READY handler now validates the payload's shape, skips the malformed entries with a warning naming the plugin, and completes the handshake (tested: garbage + one well-formed menu item → plugin loads, the good item registers, the warning fires).

### Hygiene

- `dockerVolumeName`'s uppercase acceptance is documented as **deliberate**: docker itself accepts uppercase volume names, and rejecting them would break attach flows against pre-existing out-of-band volumes (the managed `nd-svc-`/`nd-db-` prefixes are lowercased by construction).
- The `webhooks.events` column is marked **dead** in the schema (defaulted `["push"]`, never written otherwise, never branched on): kept physically — dropping needs a table-rebuild migration for zero value — but flagged so nobody builds on it.


## [0.10.29] - 2026-10-01

> The contract-honest SDK + secrets-at-rest release (r473): the auditors-audit
> of 0.10.28 plus the first deep pass over plugin-sdk, db and schemas. One
> P1 (the plugin SDK promised a contract the sandbox never honoured), one
> secrets-at-rest P2, and the last member-reachable 403 holes in the UI.

### Security

- **Notification channel configs are encrypted at rest (P2).** `notification_channels.config_json` carries real credentials — the webhook channel's HMAC signing `secret` and the FCM **service-account private key** — and sat in a plaintext column outside both the encrypted-at-rest model the schema file itself claims and the key-rotation sweep (a row only kept working across rotations because it was never encrypted). It now follows the r435 SSO pattern exactly: envelope-encrypted on write (POST and PATCH), a best-effort boot normalization rewrites pre-r473 cleartext rows in place, the rotation registry carries the column (plaintext survivors skipped, never fed to `reencrypt`), the dispatch paths and the test route read it through a tolerant `channelConfigOf`, and `GET /channels` answers `hasConfig` with the blob exposed only for the **Discord** shape — the one that is secret-free (username/avatar/title/color) and the only one the UI edits. Webhook/FCM configs are write-only over the API now.

### Fixed

- **The plugin SDK's declared contract is now the runtime's actual contract (P1).** `configSchema`, `menuItems` and `dependencies` declared on the object a plugin's code RETURNS — the thing `definePlugin` validates — were silently dropped: both sandbox bootstraps posted only the install-request manifest at READY, so an author's Settings fields and menu entries never registered and `ctx.config.getSecret` returned null forever unless an operator hand-wrote the key. Both bootstraps (process and worker, kept in behavioural sync) now prefer the returned definition, manifest as fallback. The logger's promised varargs are stringified through to the panel log. And the phantom half of the SDK was deleted rather than left as a trap: `start`/`stop` lifecycle hooks (never called), per-tap `rollback`/`timeoutMs`/`id` options (silently dropped — the host applies its own 5 s budget and rejection rollback), and `ctx.registerMenuItem` (a hard no-op). The flagship example was rewritten to the real sandbox code shape — an async function BODY (`plugin-body.txt`, no import/export, `ctx` is the only panel surface) — and the package ships a README documenting the two halves of a plugin. The unused `@ninedeploy/schemas` dependency no longer rides into every author install.

- **The last member-reachable 403 holes in wizards and modals (P2).** The Deploy wizard (member-reachable from Services and Hub) fired the operator-only `sources` + `servers` listings on every open — doubled by the app-wide retry — and silently rendered "Public / none"; the Attach-Volume modal's inventory query made its DEFAULT tab dead for members (they now start on "Create New", the tab hidden); the Database wizard's retained-volume reuse and the service Danger tab's volume hint each fired a swallowed 403 (the option/hint simply doesn't exist for members now). The Volumes tab's remaining fabricated member numbers ("Host Storage: 0 B", "0 active database volume(s)", an empty "Attached Database Volumes" list for a member with attached DBs) are gone — unknowable values render an honest `—`, the DB-volumes section is operator-only, members' attached databases stay visible where they always were (Architecture tab). The events drawer's "View Full Audit Ledger" CTA and Topology's "Manage All Volumes" link no longer lead members to pages that can only refuse them.

- **Schema boundary honesty (P3).** `webhookCreate.branch` now carries the same git charset rule as the service's own branch field (whitespace-tolerant — trimmed before validation, exactly as the route always did) instead of accepting any string; the dead, laxer twin `createWebhook` in service.ts (used by no route) was deleted before someone "hardened" the wrong copy — the SDK now types `webhooks.create` off the live schema. `createDomain.path` aligned with the manifest's `route.path` rule instead of relying on a downstream strip. Env-var values capped at the same 32 KB the Hub path applies. And the agent `docker run` line shell-quotes its env values — a panel URL legitimately containing `&` used to background the pasted command and mangle everything after it (plain values stay byte-identical).

### Verified sound (for the record)

db's migration chain (data-copying rebuilds, sequence preservation, index coverage for the hot query paths — checked against the real query patterns) and schemas' regexes (anchored, linear, transforms that never throw into a 500) both survived the deep pass; the remaining P3s (dockerVolumeName's `/i` flag contradicting its comment, the dead `webhooks.events` column) are noted, not shipped.


## [0.10.28] - 2026-10-01

> The member posture release (r472): first deep audits of the two surfaces
> that had never had one — the MCP package and the web frontend. Both came
> back structurally sound at the core (0 P1s anywhere); what shipped is the
> retrofit: the operator gating that moved server-side in r469/r470 finally
> reached the UI.

### Web — members stop hitting walls that were never theirs

- **Six operator-only routes left member-visible (P2).** The sidebar (and command palette) offered `/volumes`, `/activity`, `/docker`, `/sources`, `/servers` and `/users` to members although every one is operator-only server-side. Each rendered a 403 error card that read as breakage; worst cases: **Activity** had no error branch at all — a refused load fell into the empty-state arm and claimed "No activity recorded" while `refetchInterval` re-fired the 403 every 5 seconds; **Servers** rendered the entire operator console (capacity cards, the auto-join command banner, the add-server wizard) around its error card. All six are now `operatorOnly` in the sidebar and palette, and the pages carry honest "operators only" one-liners for direct URL access — Activity additionally gained a real error card (with retry, interval-stopping on error).
- **Eleven operator-only Settings sections showed member error states (P2).** `/v1/settings` and its satellites (firewall, log drains, notifications, resources, storage, migration, OIDC) are operator-only; members saw the sections render with failed lookups — the Firewall one worst of all, where a refused status probe rendered the g