RELEASING.md
Rama main · réplica de lectura del repositorio original.
# Releasing Artifact Keeper
Runbook for cutting a release from `main`. Maintenance releases from
`release/X.Y.x` branches follow the same sequence; the only difference is
that fixes reach the branch by cherry-pick from `main` first (see
"Release Branch Strategy" in [CLAUDE.md](CLAUDE.md) and the
release-branch-gate workflow).
Throughout, `X.Y.Z` is the version being released and the git tag is
`vX.Y.Z` (Docker tags drop the `v`).
**The shape of a cut (#3769):** prep PR → `Release Candidate` dispatch →
green → `Release Promote` dispatch. Nothing gets a permanent name — the git
tag `vX.Y.Z`, the image `:X.Y.Z`, `:latest`, `:X.Y` — until the full gate has
passed on that exact commit. The candidate tests the commit's `sha-<sha>`
images and records a certification; the promote applies `:X.Y.Z` to the
certified digests and creates the tag **last**. You never push a stable tag
by hand any more — and you cannot usefully: the promote creates every stable
tag with `GITHUB_TOKEN`, which fires no `push` event, so a *push* of a
`vX.Y.Z` is by construction a hand push and is refused outright by both
`docker-publish.yml` and `release.yml` before anything is built, whether or
not the commit is certified. (A certified commit is no exception: the push run
would build fresh bytes and name them `:X.Y.Z`, bytes the gate never saw.)
Since #3823 a `workflow_dispatch` at a stable tag is refused the same way
unless `github.actor` is `github-actions[bot]` — the identity a dispatch made
with `GITHUB_TOKEN` from inside `release-promote.yml` carries, and one a human
dispatching from the Actions UI cannot choose. Two exceptions, both deliberate:
a `docker-publish.yml` dispatch **with** `promote_version` is allowed for
anyone, because it cannot build (it re-applies an existing digest and the
digest guard refuses anything else) and it is the documented recovery below;
and `github.actor` survives a re-run, so re-running the promote's own dispatch
still works.
State the limit precisely: **what is refused is building, dispatching and
releasing a stable tag — not creating one.** Ruleset 19144026 has no
`creation` rule, so a `vX.Y.Z` can still be created by anyone with write
access; it just cannot be made to do anything. If you have hand-tagged a
certified commit, dispatch the promote on it — it accepts an existing tag that
names the same commit.
## Cut sequence
1. **Confirm scope and health.** The milestone for `X.Y.Z` has no open
issues you still intend to ship, and `main` is green (CI Complete on the
release candidate commit).
Then run the **release preflight** and get a `READY` before you tag:
```bash
scripts/ci/release-preflight.sh # local (needs an authenticated gh)
# or, from the Actions UI: run the "Release Preflight" workflow
```
It asserts main is actually releasable — `.trivyignore` accounts for
every active `release/*` branch's suppressions, live or via a
`# RETIRED:` tombstone (the drift that stalled v1.7.0-rc.1 at Security
Scan, #3039; the tombstone mechanism is #3309), the version set is
consistent, Docker Publish for the exact commit being tagged (not
merely the latest run, #3338) cleanly published its manifest, no
component pinned by a checked-in `VERSION` file would try to republish an
exact tag that already exists with different content (the collision that
killed the v1.7.2 tag), and the pending CHANGELOG section and the commit
range `<previous stable tag>..HEAD` describe the same work in both
directions (#3537). A `NOT READY` (exit 1) means fix main first;
tagging over it costs a full re-cut cycle. An exit 2 is `INFRA` — a check
that could not be measured, which is neither a pass nor a failure. If a
check has already found a blocking problem and a *later* check cannot be
measured, you get exit 1, not exit 2 (#3538): the blocking problem is
definite and retrying will not remove it, so "retryable" would be the wrong
instruction. The transcript says which checks did not run, so re-run the
preflight for a full verdict once you have fixed them.
**This step is now enforced, not advisory (#3538).** `release.yml`'s
`preflight-evidence` job refuses to proceed unless a green run of the
**Release Preflight workflow** exists for the exact commit the tag points
at. A local `scripts/ci/release-preflight.sh` run is still useful, but it
leaves no evidence — only the workflow does, as a
`release-preflight-<sha>` artifact carrying the sha it actually checked
out. **The Release Candidate (step 6) dispatches the preflight on the
commit for you and requires its evidence**, so in the normal flow this
step is "run it early to find out sooner", not a separate chore. So:
- Run **Release Preflight** from the Actions UI (or
`gh workflow run release-preflight.yml --ref <branch>`) on the branch
whose tip is the commit you are about to tag, and wait for it to go
green. Cutting a maintenance release? Dispatch it **on that branch** —
the workflow audits the ref it was dispatched on, and says which one in
its verdict line.
- A preflight that is still running does **not** satisfy the gate. That is
deliberate: v1.7.7 was tagged 18 minutes after a preflight reported
`verdict does not exist yet`, and the run it was waiting on concluded
`failure` 19 seconds after the tag push.
- If the branch moves after the preflight goes green, the preflight no
longer applies — tag the commit it audited, or run it again.
**Break glass.** The gate predicts whether the chain will stall; it does
not certify the bytes, so it is override-able (the gates that *do* certify
bytes — `resolve-candidate-digest`, `release-gate`,
`verify-images-published` — are not). The override is a trailer in the
annotated tag's own message, and it requires a real reason:
```bash
git tag -a v1.8.2 -m "Release 1.8.2
Preflight-Override: cut off-branch at 635496d0, which has no branch to
dispatch the workflow on; preflight run locally against that exact tree,
transcript on #3538"
```
It lives there rather than in a workflow input or a repository variable
because the tag object is immutable (ruleset 19144026), is scoped to
exactly one version, cannot be added after the fact, and is readable
forever with `git show v1.8.2`. A trailer with no reason (or under 20
characters of one) is **refused**, not honoured. Every honoured override is
printed as a workflow warning and written to the release run's summary,
together with the verdict it overrode.
Check 5 reads the issue reference each entry leads with, so **every
CHANGELOG bullet must name the issue it closes** — `- **Summary**
(#NNNN). prose` — and every merged PR in the range must be named by some
pending entry. The pending entries are the unassembled fragments in
`changes/unreleased/` plus anything under `## [Unreleased]`; check 5
renders the fragments exactly as step 3 will. `### Sponsors` and
`### Thank You` bullets are credits, not entries, and are not reconciled.
Four commit shapes are exempt, and nothing else is. Each is a **subject
pattern and a path set** — a subject alone exempts nothing, because a
subject is free text:
| shape | subject | may touch only |
|---|---|---|
| release prep | `chore(release): …` | `Cargo.toml`, `Cargo.lock`, `**/openapi.rs`, `CHANGELOG.md`, `changes/unreleased/**`, `.github/release-notes/**`, `docker/*/VERSION` |
| changelog-only | `docs(changelog): …` | `CHANGELOG.md`, `changes/unreleased/**`, `.github/release-notes/**` |
| dependency bump | `chore…: bump …` | dependency manifests and lockfiles, the CI paths below, and the changelog set (`CHANGELOG.md`, `changes/unreleased/**`, `.github/release-notes/**`) |
| CI/workflow-only | *any* | `.github/workflows/**`, `.github/actions/**`, `.github/scripts/**`, `scripts/ci/**` |
This is the **same** set the release-branch gate applies as its path C
(`scripts/ci/release-commit-exemptions.sh`, shared by both, #3829), so a
commit that satisfies one gate satisfies the other. Two consequences worth
knowing before a cut:
- A cherry-picked tooling forward-port does **not** need retitling. `git
cherry-pick -x` of a `feat(ci): …` commit keeps that subject, which used
to make check 5 demand a CHANGELOG entry the change did not deserve and
the branch gate refuse it; the CI/workflow-only rule now covers it by its
paths, whatever the subject says. Keep the original subject and the `-x`
trailer.
- A dependency bump that also edits shipped source — a Dockerfile base
image, backend code — is **not** a bump for either gate's purposes. It
changes bytes a user receives, so it owes a CHANGELOG bullet and a trip
through `main` like any other change.
When a commit is refused, each gate names the rule it failed and, for a
near miss, the path that put it outside the set.
Check 6 refuses curated notes that are not finished: if
`.github/release-notes/<version>.md` exists for the version `Cargo.toml`
names, any uppercase `TODO` or `DRAFT` word in it is blocking. Notes may be
drafted on `main` before the cut (1.11.0's were, #4468) and their open
items are usually HTML comments, which GitHub does not render, so a
forgotten draft would otherwise publish with the gaps invisible. Resolve or
delete every marker in the prep PR.
**Cut checklist, beyond the preflight.** Before dispatching the candidate,
also confirm the secrets the post-release jobs need (step 8): the
`DISCORD_RELEASE_WEBHOOK` webhook is live (rotate it if the last
announcement failed with 404), and `AWS_AMI_BUILDER_ROLE_ARN` names a role
whose OIDC trust admits this repository. Neither is visible to the
preflight, and a missing one turns a published release's announcement or
AMI job red.
2. **Bump the version set.** The version is displayed or pinned in several
decoupled places; a partial bump ships a stale version string. Update
all of them in one PR (or one PR per repo):
- `Cargo.toml` (workspace `version`, this repo) and the regenerated
`Cargo.lock`
- `backend/src/api/openapi.rs` (hardcoded `version = "..."` in the
OpenAPI info block, this repo)
- `package.json` `version` in artifact-keeper-web
- `charts/artifact-keeper/Chart.yaml` `version` and `appVersion` in
artifact-keeper-iac
3. **REQUIRED: assemble the CHANGELOG.** Entries do not live in
`CHANGELOG.md` between releases: every PR adds one fragment file under
`changes/unreleased/` (format in [changes/README.md](changes/README.md)),
so no two PRs edit the same file and a merge no longer forces every other
open PR to rebase and re-run CI. In the release prep PR, after the version
bump (step 2):
```bash
scripts/release/assemble-changelog.sh --check # preview the section; changes nothing
scripts/release/assemble-changelog.sh # write it
git add CHANGELOG.md changes/unreleased
```
It renders every fragment into a new `## [X.Y.Z] - <date>` section
directly below `## [Unreleased]` (X.Y.Z from `Cargo.toml` unless given;
sections in Keep a Changelog order, entries ordered by the number in the
file name), merges in any bullets still written under `## [Unreleased]`
(PRs opened before fragments existed), carries any other `### ` heading
there through verbatim, leaves `## [Unreleased]` holding only its pointer
line, and deletes the fragments it rendered. Then add the Sponsors and
Thank You recognition sections to the new section per the "Changelog and
Release Notes" policy in [CLAUDE.md](CLAUDE.md) (`scripts/release/sponsors.sh`
prints the Sponsors block from the active GitHub sponsorships, and
`scripts/release/sponsors.sh --check .github/release-notes/<version>.md`
confirms the notes name every public sponsor), and edit the order or
wording if the release wants it — the assembler writes a first draft of
the section, not a verdict on it.
If PRs merge after the prep and before the tag, their fragments are not in
the section; preflight check 5 says so. Fold them in with
`scripts/release/assemble-changelog.sh --append` in a
`docs(changelog):` commit (exempt, see the table above). That commit is
its own PR, opened after the fragment is on `main`: the CHANGELOG
placement gate (`check-changelog-placement.sh`) recognises a fold-in by
the fragment the diff deletes and the section's version having no tag
yet, so a fix PR that adds its fragment and folds it in the same diff is
refused as a stranded bullet. Without
`--append` the assembler refuses a version that already has a section,
because a `Cargo.toml` nobody bumped names the version that already
shipped; `--append` itself is refused once the tag exists.
`## [Unreleased]` stays the first `## [` heading, which the assembler
guarantees. That is not cosmetic: before fragments, a promotion that
renamed the heading without opening a fresh one let every PR branch cut
before it merge into the *renamed, already-released* section — no
conflict, no warning. That is how 30 entries of 1.8.0 work ended up filed
under `[1.7.5]` (#3433). `scripts/ci/check-changelog-unreleased.sh` runs
in CI's shell-tests job and fails if the first `## [` heading is anything
other than `## [Unreleased]`; it also validates every fragment. A
fragment cannot land in a released section at all, which is the shape
`check-changelog-placement.sh` (#3797) exists for; that gate still covers
the transition-window bullets.
This step is enforced, not advisory: the release gate's
`version-set-integrity` check (artifact-keeper-test) and the
`verify-images-published` job in `release.yml` both assert that
`CHANGELOG.md` contains a non-empty `## [X.Y.Z]` section for the
version being released. A release with no CHANGELOG entry for the
version will fail the gate and the GitHub Release will not publish
(it stays a draft). Land the assembled section on `main` before tagging.
4. **Pre-tag verification.** This is what step 6 does, against the exact
images the release will ship, with a certification at the end. A manual
Release Gate Rehearsal (`release-gate-rehearsal.yml`) is still available
for poking at one image or one suite, but it certifies nothing.
5. **REQUIRED for backports: check the versioned component source sets.**
Before tagging, confirm no change since the previous tag touches a
versioned component's source set unless that component's `VERSION` is
bumped in the same change. Today that is `docker/scanner-adapter/**` and
`docker/Dockerfile.scanner-adapter`; the general rule is "any directory
under `docker/` with a `VERSION` file, plus its sibling Dockerfile".
```bash
git ls-files 'docker/*/VERSION' # the component list
git diff --stat vX.Y.Z-1..HEAD -- \
docker/scanner-adapter docker/Dockerfile.scanner-adapter
```
Any output means either bump `docker/scanner-adapter/VERSION` or drop the
change. The publish gate treats **any** edit under the source set as a
source change and refuses to republish an existing exact version tag — a
comment-only line counts. `v1.7.5` died on exactly that: a comment carried
along "for tidiness" in the #3424 backport made the tag's Docker Publish
fail, and the ruleset forbids deleting or moving the tag, so the version
was burned (#3429). `v1.7.2` died the same way (#3340). When backporting,
restrict the cherry-pick to files that are functionally required.
Preflight check 4 (step 1) asserts this against the registry, so a `READY`
already covers it — this step is the one to run when you are assembling a
backport, before you get as far as preflight.
6. **Dispatch the Release Candidate on the prep commit.** Once the prep PR
(steps 2–3, plus the curated notes file) is the tip of `main`. You need
not wait for its Docker Publish run: the candidate's `publish` job waits
for that run to succeed (up to 130 minutes) and only then dispatches the
preflight, whose check 3 audits the same run. Until #4372 the preflight
started beside that wait, so a candidate dispatched right after the prep
PR merged failed on "Docker Publish for HEAD is still 'pending'" (run
36862392025, v1.10.2).
```bash
gh workflow run release-candidate.yml --repo artifact-keeper/artifact-keeper \
--ref main # -f sha=<40-hex> to pick a commit that is not the tip
```
`--ref main` is required, not a convention: the certification is verified
pinned to the candidate workflow *on main* (an exact `--cert-identity` of
`release-candidate.yml@refs/heads/main`, plus `--source-ref refs/heads/main`),
so a run from any other ref is refused at once rather than an hour later.
The commit need not be the tip. The preflight is dispatched on `main` with
`ref=<sha>`, so the run's `head_sha` is main's tip, not the commit; the
evidence gate therefore also finds preflight runs by the
`release-preflight-<sha>` artifact they left, which names the tree that
was actually audited. A merge landing on `main` while the candidate runs
does not invalidate it.
It reads the version from `Cargo.toml` at that commit (there is no
version input, on purpose), refuses the commit unless it is on `main`,
carries the workflows this flow dispatches (a commit whose `release.yml`
has no `workflow_dispatch` trigger would end in an immutable tag nothing
can run on), has no `vX.Y.Z` tag and no Release, has a green Docker
Publish run, and passes the bookkeeping and CHANGELOG assertions; it
dispatches the **Release Preflight** on the commit and requires its READY
evidence; it
runs the stable-only publish checks that a `-rc` tag never exercised
(#3773) — the exact-tag digest guard for backend and openscap on both
registries, the scanner-adapter exact-version decision on both registries,
and the adoption gate (the published `sha-<sha>` images must record this
commit as their source); and then it runs the **same reusable Release
Gate** `release.yml` used to run, against the `sha-<sha>` images Docker
Publish already built for the commit, pinned by digest. Nothing is
rebuilt and nothing gets a name.
The candidate also warms the CLI binary build caches on `main`: its
`warm-cli-caches` job dispatches `release-cli-caches.yml` on `main`, which
compiles main's tip for the five release targets and saves the
`rust-cache` entries the tag run's `build-binaries` restores (#4927); let
that run finish before the promote. The key follows `Cargo.lock` and the
workspace manifests rather than the commit, with package versions
normalised, so a tag with the same dependencies as main's tip logs
`full match: true`. One whose dependencies differ logs `full match: false`
and recompiles only the difference. That is the normal case for a patch
from a `release/X.Y.x` line, whose tag runs the line's own `release.yml`,
which restores anything only once that copy carries the
`shared-key: cli-<target>` `Cache Cargo` step. `No cache found.` means the
caches were evicted (7-day GitHub retention) or the run never finished, and
the build will be cold.
On green it records a **certification**: a signed attestation
(`actions/attest`, Sigstore via GitHub OIDC) on each image's digest whose
predicate names the commit, the version, the run and every digest.
The scanner-adapter is certified at the digest the release ships
(#4076): this commit's `sha-<sha>` rebuild when its
`docker/scanner-adapter/VERSION` is new, but the already-published
`:<adapter VERSION>` digest when that version stays (published from an
earlier commit with unchanged sources). The predicate records which
(`scanner_adapter_decision`: `new`/`stays`, the owning revision, the
tag), the verifier anchors the adapter at that tag, and the promote's tag
message and summary print the certified adapter digest and decision.
Until artifact-keeper-test#380 lands (tracked here as #4439), the
adapter's attestation is **provenance of what ships, not proof that the
gate exercised those bytes**: the Release Gate takes no adapter input and deploys the chart's
pinned adapter, for "new" and "stays" alike. The candidate exposes the
value to pass (`needs.images.outputs.adapter_image_ref`) for when it can.
**Maintenance lines need the #4076 verifier before their next cut.**
`release.yml` runs the *tagged* commit's copy of
`scripts/ci/assert-candidate-certified.sh`, and a copy older than #4076
cannot read a "stays" certification: it would refuse the release after
the tag already exists. So the candidate (and, again, the promote, before
anything is named) refuses to certify "stays" on a commit whose verifier
predates #4076, with an error naming the line. To cut a patch on a
`release/X.Y.x` whose adapter version is already published, first backport
#4076 to that line (`scripts/ci/assert-candidate-certified.sh`, its
self-test, and main's current `release-candidate.yml`, which the line
needs byte-identical anyway), then certify the backport commit. A
`release-candidate-<sha>` artifact and a `release-candidate/certified`
status on the commit are written for humans; the release path verifies
the attestation (`scripts/ci/assert-candidate-certified.sh`, pinned to
the candidate workflow's identity), nothing else.
**If it fails, fix the cause and dispatch a new candidate.** Do not
"Re-run failed jobs" on a failed gate: GitHub replays the gate's `deploy`
outputs instead of re-executing them, so every suite targets a namespace
the first attempt's teardown already deleted and dies on "backend not
ready" (#3774; how v1.9.0's re-runs failed). The workflow refuses a
re-run of the gate and says so. Nothing was named, so a failed candidate
costs a gate run and nothing else — the version number is intact.
**Fallback: a prerelease tag.** `vX.Y.Z-rc.N` still works as it always
did (the ruleset excludes `v*-rc*`, `v*-beta*`, `v*-alpha*` from
immutability, so it is deletable and re-cuttable) and still runs the gate
inside `release.yml`. It is the old path, kept for a rehearsal that needs
a real tag; it certifies nothing and it does not run the stable-only
checks. Cut stable releases with the candidate — including patch releases
from a maintenance branch, which now have their own candidate path (see
"Patch releases from a `release/X.Y.x` branch").
7. **Dispatch the Release Promote.** With the candidate green:
```bash
gh workflow run release-promote.yml --repo artifact-keeper/artifact-keeper \
--ref main # -f sha=<the certified commit> if it is not the tip
```
In order: it verifies the certification for the commit and that the
registry still serves the certified digests for `sha-<sha>` (the scanner
adapter at its exact `:<adapter VERSION>` tag when its version stays, and
then that the commit's own verifier can read that certification); re-checks the
preflight evidence; applies `:X.Y.Z` to the certified digests through
Docker Publish's PROMOTE mode (`promote_version=X.Y.Z
promote_source_sha=<sha>` — no rebuild, the digest-aware guard still runs,
the same signing and verification steps run, and the scanner-adapter's
exact tag is applied to *its* certified digest with the adapter VERSION
read from the promoted commit); asserts `:X.Y.Z` now resolves to the
certified digests; **creates the annotated `vX.Y.Z` tag last**, on the
commit, with `GITHUB_TOKEN`; dispatches `docker-publish.yml` on the tag
(a no-op re-apply plus `verify-published`, and the successful publish run
`release.yml` requires for the tag); and dispatches `release.yml` on the
tag. A ref created by `GITHUB_TOKEN` fires no `push` event, which is why
both are dispatched explicitly — and why no PAT, App or ruleset bypass is
involved: ruleset 19144026 restricts updates and deletions of `v*`, not
creation. The promote is idempotent: if it dies after the tag exists,
dispatch it again on the same commit.
**After the `:X.Y.Z` step, the version belongs to these digests.** The
merge jobs of that Docker Publish run write `:X.Y.Z` on both registries in
parallel, so a promote that fails there may have applied the exact tag for
some images already. Re-dispatching is safe (same digests; the digest
guard passes), but a candidate for a *different* commit at the same
version will now be refused by the exact-tag guard: to abandon the commit
after that point, bump `Cargo.toml` and certify the new commit as the next
version.
8. **Watch `release.yml` on the tag.** It requires the preflight evidence
and the certification for the commit, resolves `:X.Y.Z` and asserts it is
the certified digest, **skips the gate** (it already passed on these exact
bytes; running it again after the tag exists is the failure mode the
candidate removes), builds and signs the binaries, verifies the images on
both registries plus the CHANGELOG entry, creates the GitHub Release, and
then `promote-floating-tags` dispatches `docker-publish.yml` with
`promote_version=X.Y.Z -f promote_floating=true`. That run rebuilds
nothing: it re-points `:latest` and `:X.Y` (and the scanner-adapter's own
floating tags, #3770) at the manifest-list digest `:X.Y.Z` already names,
and the job then asserts that both tags, on both registries, resolve to
the digest the gate tested.
Until the GitHub Release exists, `:latest` and `:X.Y` still point at the
PREVIOUS release, and that is correct. A failure in a post-gate job
(signing, the Release object, the floating promotion) is a genuine
re-run: every step there is idempotent, and the tag is already correct.
If it fails, the release is published and correct but `:latest` has not
moved. Re-run the job, or dispatch it by hand:
```bash
gh workflow run docker-publish.yml --repo artifact-keeper/artifact-keeper \
--ref vX.Y.Z -f promote_version=X.Y.Z -f promote_floating=true
```
A backport moves only its series alias: promoting `1.7.9` while `1.8.2`
is the newest release advances `:1.7` and leaves `:latest` alone.
**What runs after the Release is published.** The Release is created
with `GITHUB_TOKEN`, and an event caused by `GITHUB_TOKEN` never starts
another workflow, so nothing that listens for `release: published` runs
for it (#3789, #3896, #3897). `release.yml` therefore dispatches each
post-release workflow itself, on the tag, once the `release` job has
succeeded, and follows each run to a verdict with
`scripts/ci/follow-dispatched-run.sh`. A red follow job means the release
IS published and the images are correct; only that side effect is
missing. Re-run the job, or dispatch the workflow by hand:
| `release.yml` job | dispatches | runs for | by hand |
|---|---|---|---|
| `sync-openapi-spec` | `sync-openapi-spec.yml` | every release | `gh workflow run sync-openapi-spec.yml --ref vX.Y.Z -f dry_run=false` |
| `release-announce` | `release-announce.yml` (Discord `#announcements`, secret `DISCORD_RELEASE_WEBHOOK`) | every release; a prerelease gets the amber "pre-release" embed | `gh workflow run release-announce.yml --ref vX.Y.Z -f tag=vX.Y.Z` |
| `ami-build` | `ami-build.yml` (Packer, AWS OIDC role secret `AWS_AMI_BUILDER_ROLE_ARN`) | **stable releases only**: an AMI costs money and an `-rc.N` AMI has no consumer | `gh workflow run ami-build.yml --ref vX.Y.Z -f version=X.Y.Z` |
For a tag whose own workflow files predate the dispatch inputs (anything
released before this chain landed), dispatch on `main` instead, e.g.
`--ref main -f tag=v1.9.0`. Both inputs are validated: the version or tag
must be well-formed, must name a **published** GitHub Release (a draft or
a missing Release is refused), and, when dispatched on a version tag, must
name that same tag. Each announcement run posts again; there is no
de-duplication, so re-run it only if the post did not happen. All three
keep their `release: published` trigger for a Release published by hand,
whose event does fire (`ami-build.yml` skips a hand-published prerelease
there too). No PAT or App token is involved: `workflow_dispatch` is the
documented exception to the rule above, and the dispatching jobs use
`GITHUB_TOKEN` with `actions: write`.
`scripts/ci/check-release-downstream-dispatch.sh` (Shell Tests) fails CI
if any dispatch, follow, manual trigger or prerelease rule is dropped.
Re-running a failed `ami-build` job (or `ami-build.yml` itself) builds a
**second** AMI for the same version; there is no de-duplication, so check
the AMI build run's summary for an AMI ID before re-running it.
**Release prerequisites for these jobs.** They fail until two secrets are
in place:
- `DISCORD_RELEASE_WEBHOOK` must be a live webhook. The last two announce
runs (v1.7.0, v1.7.3) got HTTP 404 from Discord, so it needs rotating
(channel → Integrations → Webhooks → new webhook → update the secret).
- `AWS_AMI_BUILDER_ROLE_ARN` (repo or org secret) must name an IAM role
whose OIDC trust policy admits this repository's runs on `refs/tags/v*`
(the dispatch runs on the tag) and on `refs/heads/main` (manual runs for
older tags). No AMI build has ever succeeded. Until one does, the
`ami-build` job in `release.yml` is `continue-on-error: true`: a red AMI
build shows as a failed job without failing the release run. Remove that
line (TODO in the job) once a tagged build succeeds.
9. **Post-release checks.** Confirm the GitHub Release is published (not
draft), release notes are the curated per-version body (see "Release-notes
style" below), not the raw auto-generated PR list, `:latest` and `:X.Y`
moved (stable releases only), and the demo
or any pinned environments are updated intentionally (see
"Infrastructure & Cost Rules" in CLAUDE.md).
Then verify the release the way an operator would, from a clean directory —
the `release` job already did this before publishing, but running it from
outside proves the assets a downloader actually receives are the ones that
were verified:
```bash
REPO_DIR="$PWD" # your artifact-keeper checkout
IDENTITY="$("$REPO_DIR"/scripts/ci/release-identity-regexp.sh \
artifact-keeper/artifact-keeper release.yml refs/tags/vX.Y.Z)"
cd "$(mktemp -d)"
gh release download vX.Y.Z --repo artifact-keeper/artifact-keeper
cosign verify-blob \
--bundle checksums.txt.cosign.bundle \
--certificate-identity-regexp "$IDENTITY" \
--certificate-oidc-issuer https://token.actions.githubusercontent.com \
checksums.txt
sha256sum -c checksums.txt
gh attestation verify artifact-keeper-linux-amd64.tar.gz \
--repo artifact-keeper/artifact-keeper \
--signer-workflow artifact-keeper/artifact-keeper/.github/workflows/release.yml
```
`scripts/ci/release-identity-regexp.sh` is the same script the release job
uses to build its pin, so this checks the release against exactly the
identity the job required — not a hand-typed approximation of it.
`SECURITY.md` carries the same commands written out for a user who does
not have the repository checked out; keep the two in step.
## Creating a release line
Cutting `release/1.10.x` — the branch itself, not a release from it — is the
one step in this document that a `git push` cannot do. The supported procedure
is:
```bash
# the Release Candidate run that certified v1.10.0 (its run id, from the
# Actions UI or the certification record's candidate_run_id)
RUN=<id>
# check every precondition and print the plan; changes nothing
scripts/release/create-release-line.sh --dry-run --tests-from-run "$RUN" 1.10
# then do it (needs gh authenticated as a repository admin, with push access
# to artifact-keeper-test)
scripts/release/create-release-line.sh --tests-from-run "$RUN" 1.10
```
Paste the audit summary it prints into the release PR, or into the issue that
asked for the line. An **existing** line needs none of this — only creating
one, and deleting one, do.
**A line is created in both repositories (#4372).** The Release Gate is
`artifact-keeper-test`'s `release-gate.yml@main`, and every job in it runs
that repository's `scripts/switch-tests-ref.sh` (artifact-keeper-test#420):
when `artifact-keeper-test` has a `release/X.Y.x` for the candidate's
version, the suite runs from that branch, otherwise from `main`. Without it,
a maintenance candidate is gated by `main`'s suite, which keeps gaining
companion tests for features the line never receives; that failed ten suites
on the v1.10.2 candidate (#4350). So the script also creates
`release/X.Y.x` in `artifact-keeper-test`, at the suite commit that
certified the line's first release:
- `--tests-from-run <id>`: the successful Release Candidate run that
certified `vX.Y.0`. The commit is the `release-gate.yml` sha GitHub recorded
for that run (`referenced_workflows` in the run's API record). The gate's
jobs check the suite out at `main` when each starts, so a suite merge
landing mid-run can make them differ; if that happened, pass the commit the
gate jobs' checkout step logged with `--tests-sha`.
- `--tests-sha <40-hex>`: that commit, given directly. It must be on
`artifact-keeper-test`'s `main`.
`artifact-keeper-test` has no ruleset on `release/*`, so its branch is a plain
ref creation after the backend window has closed, and nothing has to be
dispatched there: the gate reads the branch each time a candidate runs. It is
idempotent. A test branch that already exists is left where it is, because its
tip legitimately moves (see "Patch releases" below). Running the script for a
line whose backend branch already exists, with a `--tests-*` option, creates
only the test branch and never touches the ruleset; that is how a failed
test-branch creation is retried and how an older line gets one.
### Why the push is refused, and why that is not a bug
Ruleset 20038606 ("Release branch protection") governs
`refs/heads/release/[0-9]*.[0-9]*.x` with `pull_request`,
`required_status_checks` — `✅ CI Complete` and `Verify commits trace back to
main`, both pinned to app 15368, with `do_not_enforce_on_create: false` —
`non_fast_forward`, `deletion`, and **no bypass actors**. A required status
check cannot exist on a ref that does not exist yet, so the very push that
would create the line is rejected:
```
remote: error: GH013: Repository rule violations found for refs/heads/release/1.10.x.
remote: - Required status check "Verify commits trace back to main" is expected.
```
#3798 tried both ways out of that and rejected both:
- **Waiving the check on create** (`do_not_enforce_on_create: true`) does
unblock it — but `required_status_checks` was the *only* rule that blocked
creation; `pull_request` never did. Waiving it therefore left ref creation
under `release/*` **entirely unguarded**: any write-access user could push
`release/<anything>.x` at any commit, with no pull request, no CI and no
ancestry check. That is load-bearing rather than untidy — "The signing
identity never widens" below is safe precisely because an untrusted actor
cannot create a `release/*` ref at all. It was applied, demonstrated to be a
hole, and reverted.
- **A standing bypass actor** permanently weakens a branch class that is today
*stronger* than `main` (no bypass at all, a pull request required) in order
to make a once-or-twice-a-year act convenient.
Creating a release line is a rare privileged act, and it should be deliberate
and auditable. So the toggle stays — and the script exists so that it is the
same toggle every time, rather than a hand-typed improvisation during a cut.
### What the script does
1. Refuses unless `gh` is authenticated as an **admin** of the repository and
`git fetch origin` succeeds — every ancestry answer below is read from
local refs, so a stale `origin/main` would make them lies rather than
checks.
2. Derives the branch name as `release/<X.Y>.x` and refuses any other shape
(see the next subsection), and refuses if it already exists.
3. Resolves the root commit: by default the commit tagged `v<X.Y>.0`, and if
that tag does not exist it demands an explicit sha. Either way the sha must
be a full 40 characters, reachable from `origin/main`
(`git merge-base --is-ancestor`), and its `Cargo.toml` `version` must be on
the `X.Y` line — `Verify commits trace back to main` enforces that property
for every *later* commit on the branch, and nothing but this enforces it
for the root one.
4. Snapshots the ruleset to `release-line-<X.Y>-ruleset-before.json`, PUTs
**only** `{"enforcement":"evaluate"}`, creates the ref through the API, and
puts `enforcement` back to `active` **from an `EXIT` trap** — so a failed
ref creation, a `die`, or a Ctrl-C cannot leave `release/*` protection
switched off. It then snapshots again and diffs the two field by field,
failing loudly on anything that differs beyond `enforcement` and
`updated_at`: "I put enforcement back" is a weaker claim than "nothing else
moved during the window".
5. Verifies the outcome rather than assuming it: `gh api
repos/…/rules/branches/release%2F<X.Y>.x` must report all four rules
(`pull_request`, `required_status_checks`, `non_fast_forward`, `deletion`),
and `git ls-remote` must show the branch at the sha it was asked for.
6. Creates `release/<X.Y>.x` in `artifact-keeper-test` (above), resolved and
checked with the other preconditions before anything is relaxed: push
access, a successful Release Candidate run or an explicit sha, and that
commit on the suite's `main`. Then it reads the branch back.
7. Prints the audit summary — actor, UTC window, sha, branch, how many
seconds the ruleset spent in `evaluate`, and the test branch.
`scripts/ci/test-create-release-line.sh` (in the `shell-tests` job) pins the
refusals, the "`--dry-run` issues no PUT/POST" property, the restore-on-
failure trap and the test-branch cases (created at the run's gate commit,
left alone when it exists, created alone for an existing line), because none
of them is exercised by a normal CI run.
### `release/1.9.1-prep` is not a release line
The ruleset's include pattern was narrowed to
`refs/heads/release/[0-9]*.[0-9]*.x` after a working branch named
`release/1.9.1-prep` was caught by the full ruleset — its first push
succeeded, then `non_fast_forward` refused every force-push, so it could not
be rebased and the PR had to be reopened from a differently named branch
(#3798).
Two consequences, and they point in opposite directions:
- **Working branches may live under `release/`** — `release/1.9.1-prep`,
`release/notes-fix` — and are governed like any other topic branch.
- **Do not name one `release/<digits>.<digits>.x`.** That shape *is* the
protected pattern: you would be unable to force-push it, unable to delete it
without the toggle below, and it would read as a release line to everything
that derives a branch name from a version (`resolve-certified-ref.sh`, and
the table in "Branch protection, for reference").
### Deleting a release line
Same toggle, for the same reason: the `deletion` rule also has no bypass
actor, so retiring `release/1.6.x` needs an admin to put the ruleset in
`evaluate` for the length of one call. That is the deliberate trade-off — an
accidentally deleted release line is far worse than an occasional announced
toggle. If retirement ever becomes routine, add a narrow bypass actor for the
maintainer role rather than reaching for enforcement.
## Patch releases from a `release/X.Y.x` branch
A patch release (1.9.1, 1.7.6) is cut from its maintenance branch, not from
`main`. The candidate-then-promote flow above is the same flow, dispatched the
same way: **both workflows are dispatched on `main`**, and you name the commit.
```bash
# 0. the branch exists and carries the flow
# Ruleset 20038606 has no `creation` rule; what refuses a new
# release/X.Y.x ref is its required-status-check rule with
# do_not_enforce_on_create: false and no bypass actor, so the push is
# rejected with "Required status check ... is expected". Cutting a NEW
# line therefore means an admin toggling that ruleset off for as long as
# the push takes -- deliberate, and worth announcing, because during the
# toggle creation is unguarded for everyone with write access. Do that
# through scripts/release/create-release-line.sh, which checks every
# precondition before it relaxes anything, restores enforcement from a
# trap and prints an audit line to paste here -- see "Creating a release
# line" above. An existing line needs no toggle.
# The branch must carry scripts/ci/resolve-certified-ref.sh,
# assert-candidate-certified.sh, follow-dispatched-run.sh and a
# dispatchable release.yml / promotable docker-publish.yml; the candidate
# refuses the commit by name if any is missing.
# It also needs the matching release/1.9.x in artifact-keeper-test, which
# the script creates; the gate runs the line's suite from it (#4372).
# 1. land the fixes on main first, then cherry-pick onto release/1.9.x
# (Release Branch Gate enforces this). In the same cycle, cherry-pick the
# fixes' companion tests onto artifact-keeper-test's release/1.9.x, or the
# line's gate does not exercise the line's own fixes
# (artifact-keeper-test#424).
# 2. the release prep commit on release/1.9.x sets Cargo.toml to 1.9.1 and
# runs scripts/release/assemble-changelog.sh, exactly as on main (the
# cherry-picked fixes carry their changes/unreleased/ fragments with them)
# 3. certify — ON MAIN, naming the maintenance commit
gh workflow run release-candidate.yml --repo artifact-keeper/artifact-keeper \
--ref main -f sha=<the release/1.9.x commit>
# 4. promote — ON MAIN, same commit
gh workflow run release-promote.yml --repo artifact-keeper/artifact-keeper \
--ref main -f sha=<the certified commit>
```
There is no `--ref release/1.9.x` step, and that is the security property.
### The signing identity never widens
A certification is trusted because of one thing: the Sigstore certificate's
SubjectAlternativeName, pinned byte-for-byte to
`...release-candidate.yml@refs/heads/main`. A maintenance release does **not**
relax that pin. `release-candidate.yml` is dispatched on `main` and certifies
the maintenance commit from there, so main's copy of the workflow remains the
only thing that can produce an accepted certification. A copy of that file on
a `release/*` branch — including one an attacker managed to create — signs
with an identity nothing accepts, exactly as a copy on any topic branch does.
The alternative was to widen the pin to
`...release-candidate.yml@refs/heads/release/1.9.x`; certifying from main is
strictly stronger, because the attack of creating a `release/*` ref, putting
an edited workflow on it and satisfying an exact pin has no identity to reach
for at all, rather than being bounded by argument.
Nothing in the workflow depended on the dispatched ref for this to work: the
reusable Release Gate is an absolute cross-repository reference
(`artifact-keeper-test/.../release-gate.yml@main`) that takes the images by
digest, and no job needs the certified commit on disk (see "What runs from the
certified commit" below — the answer is nothing).
### Which commits main may certify
That is the second, independent control, and it lives in one place:
`scripts/ci/resolve-certified-ref.sh`. Three consumers ask **main's copy** of
it, and they are the ones that decide: the candidate, `release-promote.yml`,
and `docker-publish.yml`'s certified-candidate promote (dispatched `--ref
main`). `release.yml` on the tag asks it too, but runs the **tagged commit's**
copy — see "What runs from the certified commit" below. The commit must be an ancestor of `main`; failing
that, an ancestor of `refs/heads/release/<X>.<Y>.x`, where X and Y are read
from `Cargo.toml` **at that commit**, its existence checked case-sensitively
against the git-refs API. The line is therefore a function of the commit's own
content and the repository's refs; no dispatch input names it. The predicate
records the resolved line as `certified_ref`, and the verifier asks the
repository the same question again at promote time — a disagreement is never
promoted.
For a commit on a maintenance line the candidate also **blesses the content**
of `release-candidate.yml` once, at certification time: it must be
byte-identical (same git blob id) to the copy `main` carries then. The blob is
recorded in the signed predicate, and every later consumer compares against
**that record**, never against main's tip — otherwise any merge to main that
touched the workflow would strand an already-certified commit permanently, and
recovery would mean a new commit and a full gate re-run. Not to the copy at the merge base — the merge
base is a function of the commit's own ancestry, so whoever chooses the
commit's parent would be choosing which historical copy gets blessed,
including one from before a guard existed, permanently. Main's current copy is
the only copy nobody but main can choose, and cherry-picking it forward was
already the documented remedy. So: if the candidate refuses with "not main's
current copy", do **not** edit the workflow on the branch — cherry-pick main's
copy across and dispatch a new candidate. (Nothing is demanded, explicitly,
for a commit on `main`, where comparing main to itself decides nothing.)
The blessing is no longer load-bearing for the identity; it backstops the
window in which an admin has the release ruleset toggled off to create a
line.
### What runs from the certified commit
Nothing, in the candidate. Every `actions/checkout` in `release-candidate.yml`
is `refs/heads/main`, so every script it executes is main's reviewed code —
including in `certify`, the job that holds `id-token: write` and
`attestations: write`. That matters more than it looks: those steps share the
job's OIDC environment, so a single checkout of the certified commit would
hand a maintenance commit arbitrary shell with the signing identity in scope,
and the certification the identity pin exists to make unforgeable would be
forgeable after all — attempt 2's attack relocated from the workflow file to
the scripts beside it.
The certified commit therefore enters only as **named data**:
`CHANGELOG.md` and `.github/release-notes/` are overlaid into the bookkeeping
job's worktree (`git checkout <sha> -- …`, read, never executed),
`docker/scanner-adapter/VERSION` and `Cargo.toml` are read as git objects or
over the API, and the images themselves are pinned by digest. `resolve`'s
`need` checks still assert that `scripts/ci/*` exist at the commit — that is
about what the *promote* runs later at the tag, not about this run. A self-test
(`scripts/ci/test-resolve-certified-ref.sh`) pins all of this structurally, so
a future checkout of the certified sha fails CI.
The CLI build cache warm-up (`release-cli-caches.yml`, #4927) is not given the
certified commit at all. Its caches land in main's scope and the tag run
restores them into the binaries it signs, and `cargo build` executes the tree's
build scripts and proc macros in a run that can write any key in that scope,
whatever its `permissions:` say. So it has no input and no checkout `ref:`: a
dispatch on `main` builds main's tip, the same code and the same cache scope as
`ci.yml` on a push to main. A self-test
(`scripts/ci/test-resolve-certified-ref.sh`) fails CI if that changes.
**Downstream of the tag is a different story, deliberately.** The promote
dispatches `release.yml` and `docker-publish.yml` on the tag, which runs the
tagged commit's copies — the maintenance line's, now that a stable tag can
name one. `release.yml`'s re-verification of the certification is therefore
defence in depth that cannot be trusted above the line; the authoritative
verification is the one `release-promote.yml` performs from main's tree
*before* the tag exists.
Do not overstate that, and state exactly what each guard evaluates. Both
workflows refuse a stable tag **push** (`github.event_name == 'push'`), and
since #3823 both also refuse a stable-tag **`workflow_dispatch`** whose
`github.actor` is not `github-actions[bot]`:
| event on `refs/tags/vX.Y.Z` | `docker-publish.yml` | `release.yml` |
|---|---|---|
| `push` (necessarily a hand push) | refused | refused |
| `workflow_dispatch` by a person, no `promote_version` | refused (#3823) | refused (#3823) |
| `workflow_dispatch` by a person, **with** `promote_version` | allowed — build-free; the digest guard permits only re-applying the digest `:X.Y.Z` already names, and a floating move additionally needs a published, newest-in-line Release | n/a (no inputs) |
| `workflow_dispatch` by `release-promote.yml` / `release.yml` (`GITHUB_TOKEN` ⇒ actor `github-actions[bot]`) | allowed | allowed |
| re-run of one of those by a person | allowed — `github.actor` stays the original initiator; a re-run creates nothing new | allowed |
What that does **not** do is stop the tag from existing. Ruleset 19144026
restricts `update`, `deletion` and `non_fast_forward` on `refs/tags/v*` but has
**no `creation` rule** (tracked as #4071 for 1.11.0), so a stable tag can still
be created by hand — it simply cannot be built, dispatched or released
afterwards. It now carries **no bypass actors at all** (`bypass_actors: []`,
2026-09-20): an emergency tag edit is an admin toggling the ruleset's
enforcement off for as long as the edit takes, which is deliberate and
auditable, rather than a standing per-user exemption nothing records.
The guard is a control on *people*: a workflow already in the repository that
dispatches either of these with `GITHUB_TOKEN` carries the automation identity
by definition, which is why branch protection on the workflow files (below)
still matters. Release assets are separately verified
with `--signer-workflow`, which carries no `@ref`. Treat every tag-side check
as corroboration, never as the proof; the promote is the proof.
### Branch protection, for reference
Relevant because it decides who can put a commit on a release line at all
(ruleset 20038606 vs `branches/main/protection`), not because the
certification depends on it:
| write path | `main` | `refs/heads/release/X.Y.x` |
|---|---|---|
| direct push | **refused** — a pull request is required (`required_pull_request_reviews`, 0 approvals) | **refused** — a pull request is required |
| force push / rewrite | blocked (`allow_force_pushes: false`) | blocked (`non_fast_forward`) |
| deletion | blocked (`allow_deletions: false`) | blocked (`deletion`) |
| ref creation | n/a (it exists) | **refused** — by the required-status-check rule (`do_not_enforce_on_create: false`, no bypass actor); there is no separate `creation` rule. Use `scripts/release/create-release-line.sh` ("Creating a release line") |
| required checks | 3 | 2, incl. `Verify commits trace back to main` |
| required approvals | none configured | none configured |
| admin bypass | **no** — `enforce_admins: true` | **no** — `bypass_actors: []` |
Every required context on both branches is pinned to the GitHub Actions app
(`app_id` / `integration_id` 15368) — `🧪 Backend Unit Tests`, `✅ CI Complete`
and `🦀 Check Rust` on `main`, `✅ CI Complete` and `Verify commits trace back
to main` on `release/*` — so a same-named status posted by any other app or by
a token does not satisfy them.
One thing to fix, and it is bigger than the release flow: on **both**
branches, a pull request's required checks are defined by workflows the pull
request itself can edit, and neither branch requires an approval. The real
floor under a release is therefore *write access*, not review. Raising
`required_approving_review_count` above 0 on ruleset 20038606, and requiring
code-owner review on the gate files themselves, would raise it; both are
deferred to #4071.
## Supply chain: what the release job signs, and what it refuses
Release binaries used to ship with a `<name>.sha256` beside them and nothing
else, which is a corruption check routinely read as an authenticity check: it
is served from the same place, by the same authority, as the artifact it
describes. Since #3558 the `release` job does four things between "Collect
release assets" and "Create Release", **in this order**:
1. writes one `checksums.txt` covering every asset (the per-file `.sha256`
files stay, and are themselves listed in it);
2. signs `checksums.txt` with **cosign keyless** — GitHub OIDC to Fulcio to
Rekor, no key material to hold or rotate — producing
`checksums.txt.cosign.bundle`;
3. attaches **build provenance** to every archive, SBOM and the manifest with
`actions/attest-build-provenance`;
4. **verifies all of it** with `scripts/ci/verify-release-assets.sh` before
anything is published.
The order is the point, and it is not stylistic. `assert-release-absent.sh`
refuses to run against a tag that already has a release and
`softprops/action-gh-release` runs with `overwrite_files: false`, so a
*partial* publish is not recoverable by re-running — it burns an immutable
version number. Everything that can fail therefore fails while the job has
produced nothing public, and the recovery is "re-run the job".
The verification refuses, among other things: an asset missing from
`checksums.txt` (an unlisted asset is an unsigned asset), digests that do not
match the bytes on disk, a missing signature, a valid signature carrying the
**wrong certificate identity**, a certificate-identity pattern permissive
enough to accept one, provenance signed by a different workflow, and a missing
`.cdx.json`. It separates a Sigstore or GitHub API outage (exit 2, INFRA,
"retry") from a genuine mismatch (exit 1, BLOCKED, "something replaced your
bytes"); both stop the release. Every one of those legs is exercised offline
by `scripts/ci/test-verify-release-assets.sh` in CI's shell-tests job, because
this code path runs once per release and cannot be rehearsed without cutting a
tag.
Binaries are built with `cargo auditable`, so the dependency graph is embedded
in the shipped executable and a `.cdx.json` CycloneDX SBOM ships per target.
`scripts/ci/assert-binary-sbom.sh` asserts the embedding actually happened,
which is necessary rather than belt-and-braces: `cargo audit bin` **exits 0**
on a binary with no embedded data, printing a warning and falling back to a
partial list scraped from panic messages. Without an assertion on its output,
dropping `auditable` from the build command leaves CI green and every shipped
binary hollow.
**Do not add `continue-on-error` to any of these steps.**
`scripts/ci/check-supply-chain-soft-fail.sh` blocks it, for the reason #2824
exists: a soft-failed cosign step let 1.6.1 ship unsigned and a human found it
later.
## Rolling `:latest` back
There is no separate rollback path, because there is nothing to roll back in
the normal failure case: if any gate fails, `:latest` and `:X.Y` were never
moved and still name the previous release. The only visible consequence is
that after a failed cut the registry looks like the release did not happen —
`:X.Y.Z` exists and is immutable, but the floating tags are unchanged.
To move `:latest` back off a release that DID publish and then turned out to
be bad:
1. Delete the bad GitHub Release, or mark it as a prerelease. This is the
deliberate act; the tag movement follows from it.
2. Dispatch the promotion for the version you want:
```bash
gh workflow run docker-publish.yml --repo artifact-keeper/artifact-keeper \
--ref vX.Y.Z -f promote_version=X.Y.Z -f promote_floating=true
```
`.github/scripts/floating-tag-plan.sh` reads the published-release set, so once
the bad release is gone the previous version is the newest one and the floating
tags are allowed to move to it. While the bad release is still published, the
same command is refused — floating tags never move backwards by accident.
Do NOT delete container tags to roll back. `:X.Y.Z` is immutable and names real,
scanned bytes; deleting it breaks every chart that pins it.
## Policy summary
- Every release documents itself: no `vX.Y.Z` tag without a non-empty
`## [X.Y.Z]` section in `CHANGELOG.md`. Enforced by
`version-set-integrity` (artifact-keeper-test release gate) and
`release.yml` `verify-images-published`.
- The GitHub Release body is a curated high-level paraphrase of the
version's `## [X.Y.Z]` CHANGELOG section (see "Release-notes style"),
NOT the raw auto-generated PR list; recognition sections (Sponsors,
Thank You) follow the CLAUDE.md policy. For a stable release
`.github/release-notes/<version>.md` is REQUIRED on the ref being
released — `generate_release_notes` is a prerelease-only fallback, and a
stable tag without the file is refused (#3537).
- Every CHANGELOG entry is a fragment in `changes/unreleased/` until the
release prep assembles it (`scripts/release/assemble-changelog.sh`).
Fragments are validated by `check-changelog-unreleased.sh` (whole
directory) and `check-changelog-placement.sh` (the PR's own); bullets
still added under `## [Unreleased]` are accepted for the transition.
- Every CHANGELOG entry names the issue it closes, and every merged PR
since the previous stable tag is named by some pending entry. Enforced
in both directions by `release-preflight.sh` check 5 (#3537).
- Prerelease tags (`-rc.N`, `-beta.N`) are exempt from the CHANGELOG
entry requirement; final releases are not.
- `CHANGELOG.md` always has an open `## [Unreleased]` as its first `## [`
heading. Enforced by `scripts/ci/check-changelog-unreleased.sh` in CI's
shell-tests job (#3433).
- A stable release is the promotion of a **certified commit**, and which
branch may have certified it is **derived** from the commit
(`scripts/ci/resolve-certified-ref.sh`), never passed in: `main`, or the
`release/X.Y.x` its own `Cargo.toml` names, and every script the candidate
executes comes from `main`, never from the certified commit. The Release
Candidate is always
dispatched on `main`, including for a maintenance release, so the signing
identity is always `...release-candidate.yml@refs/heads/main` and never
widens; `release-candidate.yml` at the certified commit must additionally be
byte-identical to a copy that lives on `main`. See "Patch releases from a
`release/X.Y.x` branch".
- Floating tags (`:latest`, `:X.Y`) are applied ONLY after the release gate
and the GitHub Release, by a build-free promotion that re-points them at the
digest `:X.Y.Z` already names. A floating tag may only name a version with a
published, non-draft, non-prerelease GitHub Release, and only if that version
is the newest in the line the tag represents. Enforced by
`.github/scripts/floating-tag-plan.sh` and pinned by
`scripts/ci/check-floating-tag-promotion.sh` in CI's shell-tests job.
Prereleases never take a floating tag. The scanner-adapter's own
`:latest` / `:X.Y` / `:X` follow the same rule and the same moment (#3770):
its "published set" is the `docker/scanner-adapter/VERSION` each published
release ships, so a normal build writes no adapter floating tag (its exact
version, `sha-*` and the branch `dev` tags only), and a base-image errata
rebuild reaches the chart-pinned `:1` through a VERSION bump and a release.
The promote resolves the adapter version from `v<promote_version>` itself,
so the dispatch ref does not matter.
- Release binaries are signed and verified before publication, never after:
one `checksums.txt` over every asset, a cosign keyless signature over it, and
build provenance on the assets — all three VERIFIED by
`scripts/ci/verify-release-assets.sh` before `Create Release` runs, because
`overwrite_files: false` plus `assert-release-absent.sh` make a partial
publish unrecoverable. The certificate identity is pinned to this repository,
this workflow file and this tag; a permissive pattern is refused by the gate
rather than passed to cosign (#3558).
- Binaries embed their dependency graph (`cargo auditable`) and ship a
per-target CycloneDX SBOM, and the embedding is ASSERTED at build time —
`cargo audit bin` exits 0 on a binary that has none (#3558).
- No change reaches a tag that touches a versioned component's source set
(`docker/*/VERSION` and its sibling Dockerfile) without bumping that
component's `VERSION` — step 5. Exact version tags are never republished,
and a tag that fails to publish is burned (#3429, #3340).
- Every `release/**` branch publishes images on push, same as `main`
(#3422). A maintenance-branch commit therefore has a Docker Publish run of
its own, which is what preflight check 3 resolves by `head_sha` (#3338);
no manual `workflow_dispatch` is needed before a cut.
- The release-branch gate accepts two shapes that cannot trace to `main` by
patch-id (#3422): a release-hygiene commit (path C — the four shapes in the
table under step 1, shared verbatim with release preflight check 5 since
#3829) and a narrowed backport (a `(cherry picked from commit <sha>)`
trailer naming a commit on `main`). Use `git cherry-pick -x` so the trailer
is written for you, and keep it when you resolve hunks away.
- **`release-process: approved` is a review marker, not a bypass** (#4070). It
used to skip the gate's job outright, and a skipped job still concludes
success — so the required `Verify commits trace back to main` context was
satisfiable by anyone who could open a PR against `release/*` and label
their own PR. The check now always runs; a labelled PR fails exactly as an
unlabelled one does, with a workflow warning saying the label did nothing.
Do not apply it to get a red gate green. A shape that genuinely belongs on a
release line belongs in `scripts/ci/release-commit-exemptions.sh`, where it
is reviewed code with a test beside it — everything else lands on `main`
first and is cherry-picked across.
- No permanent name until the full gate has passed on the exact commit
(#3769). A stable release is the promotion of a commit certified by
`release-candidate.yml`; `release-promote.yml` applies `:X.Y.Z` to the
certified digests and creates `vX.Y.Z` last. Both `release.yml` and
`docker-publish.yml` refuse every stable-tag *push* (the promote creates
the tag, and a token-created ref fires no push) and every stable-tag
*dispatch* that is not the automation's (#3823 — `github.actor` must be
`github-actions[bot]`; a `docker-publish.yml` promote dispatch, which cannot
build, is the one exception). Creating the tag is still unrestricted —
ruleset 19144026 has no `creation` rule. `release.yml` refuses a
dispatched stable tag whose commit has no certification
(`scripts/ci/assert-candidate-certified.sh`, a signed attestation pinned
to the candidate workflow's exact identity on `main`) and does not re-run
the gate on a certified tag; `docker-publish.yml`'s plain promote on the
tag can only re-apply an existing `:X.Y.Z` to its own digest. Prerelease tags (`v*-rc*` / `v*-beta*` / `v*-alpha*`) are
the fallback: excluded from tag immutability, re-cuttable, gated inside
`release.yml` as before, certifying nothing.
- Every check that runs only on a clean `refs/tags/v*` ref runs in the
candidate too, against the same inputs (#3773): bookkeeping, the CHANGELOG
entry, the preflight evidence, the exact-tag digest guard and the
scanner-adapter exact-version decision on both registries, and the image
adoption gate.
- A failed Release Gate is never re-run; the recovery is a new candidate
(#3774). "Re-run failed jobs" replays the gate's `deploy` outputs, so the
suites target a namespace that no longer exists. Both workflows refuse the
re-run and say why.
## Release-notes style
The GitHub Release body is **not** the raw auto-generated PR list.
`generate_release_notes` produces an unscoped PR dump -- and when
intermediate prereleases did not publish a Release object it reaches back
into prior minor lines -- which buries the value. Instead the body is a
curated **high-level paraphrase of THIS version's `## [X.Y.Z]` CHANGELOG
section**:
- Lead with the big-ticket epics / security themes so the value lands in
the first few lines.
- Keep the intro human-skimmable: group and paraphrase, do not reproduce
every PR.
- Point to the full `## [X.Y.Z]` CHANGELOG section for depth (both the
skimming reader and the auditing reader are served).
- Prepend the required `### Sponsors` and `### Thank You` recognition
sections (see CLAUDE.md "Changelog and Release Notes").
- Scope strictly to X.Y.Z -- only changes since the previous minor/patch,
never a multi-version diff.
Mechanics: author the body as `.github/release-notes/<version>.md` and
commit it in the same PR as the CHANGELOG assembly (step 3). `release.yml`'s
"Resolve release notes" step uses that file as the Release `body_path`. A
long release may land a draft earlier, with its open items marked `TODO`;
preflight check 6 refuses the cut until no `TODO` or `DRAFT` marker is left
in the notes for the version being released.
For a stable `vX.Y.Z` the file is **required**: with no curated file the
release-preflight job refuses the tag, and the "Resolve release notes" step
refuses it again an hour later. `generate_release_notes` survives only for
prereleases (`-rc.N`, `-beta.N`). That fallback used to apply to everything,
silently: v1.7.1 was cut with no curated file, the step logged "using GitHub
auto-generated notes", and the release published — and because auto-notes do
not promote the CHANGELOG, eight `[Unreleased]` entries that had already
shipped stayed under `[Unreleased]`, one of them telling operators to act
"before upgrading to 1.7.2" when they had been exposed for a week (#3318,
#3537).
The notes file is read from **the ref being released**, not from `main`.
`main` and `release/1.7.x` hold disjoint halves of the 1.7.x set — 1.7.6 and
1.7.8 live only on the release branch, which is where they belong. Put the
file on the branch you are cutting from.
`.github/release-notes/` must also hold no file for a version that has
neither a tag nor a Release. A stale notes file is what a human reads when
reconstructing what shipped, and it is one rename away from being published
as another version's body. Security hotfix releases use the tighter "am I
affected" table format instead.