CLAUDE.md
Rama main · réplica de lectura del repositorio original.
# CLAUDE.md
This file is the **single source of truth** for agent and contributor guidance in
this repository. It is written for Claude Code (claude.ai/code) but applies to
**every** coding agent — [`AGENTS.md`](./AGENTS.md) and
[`.github/copilot-instructions.md`](./.github/copilot-instructions.md) are thin
pointers back here so the guidance lives in exactly one place and never drifts.
Read whichever entry point your tool prefers; they all resolve to this document.
## Repo type
Helm charts + ArgoCD GitOps for the Camer Digital AI platform. **Not an application** — no build, no compile, no traditional test loop. The "verification cycle" is `helm template` (does it render?) + the relevant `helm lint` + CI's chart-render and security scans. Things deploy via ArgoCD reconciling these charts in the cluster.
> **Maintainer:** @stephane-segning. Use this handle in ADR `Deciders:` lines and any maintainer-attribution context. Don't substitute branch names.
> **Companion repos:** `home-os` (shared cluster infra ai-helm only *consumes* — cert-manager + ClusterIssuers, redis-ha, CNPG + Barman, Traefik, ESO; local `/Users/selast/dev/personal/home-os`) and `hetzner-k8s` (Terraform nodes/network/CNI/LB + the platform bootstrap; local `/Users/selast/dev/personal/hetzner-k8s`). The old "ai-gitops" deployment-state repo was never built under that name — it exists now as the PRIVATE **`adorsys-gis/ai-helm-values`** (ADR-0055/0056), which holds per-app workload values + image tags + the `environments/` deps overlays. **Chart logic + templates live here in `ai-helm`; everything that is *deployed* (values, tags, per-env CRs) lives in `ai-helm-values`** (see the "ai-helm ↔ ai-helm-values" section below + `ai-helm-values/README.md`). The root `ai-apps-v2` Application is pinned in **`home-os`** `charts/cd`.
**`inference-ops`** is the team's inference knowledge base (Diátaxis + ADRs +
immutable benchmark reports) — VRAM budgeting, quantization, engine selection,
the add/replace/roll-back/measure recipes and the inference runbooks live
there, NOT in ai-helm. ai-helm ADRs own the GitOps shape; inference-ops ADRs own
the inference decisions.
## Read these first when changing anything architectural
1. `docs/adr/README.md` — index of every Architecture Decision Record. ADRs are the source of truth for "why". Read the relevant ones before touching anything they cover.
2. `docs/architecture.md` — system map (ArgoCD topology, sync waves, auth flow, observability pipeline). Orients quickly.
3. `CONTRIBUTING.md` — conventions, commit style, ADR process.
4. `docs/migrations/2026-currency-audit.md` — every chart-pin / API-version freshness assessment, with a punch-list of follow-ups.
## ADRs are immutable once Accepted
Don't edit the decision body of an Accepted ADR. To change a decision, write a new ADR that supersedes the old one (status → `Superseded by ADR-NNNN`, add a one-paragraph header note explaining what changed; the original body stays). The index in `docs/adr/README.md` shows the supersession chain.
## Common commands
```bash
# Render a chart (the primary "does this still work?" check)
helm template <release-name> charts/<chart>/ [-n <ns>] [-f /tmp/values.yaml]
# Fetch chart dependencies (required after touching Chart.yaml)
helm dep build charts/<chart>/
# or to also update versions:
helm dep update charts/<chart>/
# Lint a chart (run before pushing changes to charts/)
helm lint charts/<chart>/
# Dashboard generator (Python — uv + ruff, NEVER pip/poetry)
cd tools/dashboards
uv sync # one-time, reads uv.lock
uv run dashboards build # regenerate every JSON
uv run dashboards check # CI guard: fail if committed JSON drifts
uv run ruff format . && uv run ruff check .
# After editing the dashboard .py source, you MUST run `dashboards build`
# and commit the regenerated JSON — CI fails otherwise.
# Match CI before pushing chart changes (.github/workflows/helm-lint.yaml). CI
# lints --strict ONLY for charts with their own templates/ dir, NON-strict for
# subchart-only charts (no templates/ — --strict false-trips on "templates/:
# directory not found"), and render-only LEAF charts (Chart.yaml annotation
# ai-helm.adorsys-gis.github.io/lint-mode: ci-values, e.g. ai-model) are
# lint+rendered via their ci/*-values.yaml fixtures, not default values. To
# check the whole repo at once, matching that logic:
for cf in charts/*/Chart.yaml; do c=$(dirname "$cf"); n=$(yq e .name "$cf")
[[ $(yq e '.type // "application"' "$cf") == library ]] && continue
sf=--strict; [[ -d "$c/templates" ]] || sf=
if [[ $(yq e '.annotations."ai-helm.adorsys-gis.github.io/lint-mode" // ""' "$cf") == ci-values ]]; then
for vf in "$c"/ci/*-values.yaml; do helm lint "$c" $sf -f "$vf" >/dev/null && helm template x "$c" -f "$vf" --dry-run >/dev/null || echo "FAIL: $n"; done
else helm lint "$c" $sf >/dev/null && helm template x "$c" --dry-run >/dev/null || echo "FAIL: $n"; fi; done
```
There is no `npm`, no `pytest`, no `cargo`, no `go build` in this repo. The dashboards Python project at `tools/dashboards/` is the only code that runs; everything else is YAML rendered by Helm. **CI gates** (`.github/workflows/`): `helm-lint` (per-chart `helm lint` + `helm template --dry-run`; `--strict` only for charts with own `templates/`, render-only leaves go through their `ci/*-values.yaml` fixtures — see the tool-quirks note below), `dashboards-drift` (`uv run dashboards check`), `security` (scans), `release-helm-charts` (non-strict lint + Trivy config scan + package on tag), `opencode` (the agent/CI rules — see `.github/workflows/opencode.yml`, the canonical agent rules; the stale `.opencode/README.md` is unrelated). Two more run on push-to-main (not PR gates): `publish-charts-oci` (ADR-0055 OCI publish) and `release-please` (ADR-0082 — `googleapis/release-please-action@v5`, config `release-please-config.json` + `.release-please-manifest.json`; grooms ONE Conventional-Commits changelog/version PR that owns only each chart's `MAJOR.MINOR` floor + `CHANGELOG.md` — publish still derives the deployed PATCH from commit-count, so the PATCH release-please writes into `Chart.yaml` is cosmetic; `bjw-common`/`bjw-template` are excluded because dependents exact-pin them).
## The orchestrator-plus-leaves pattern (used for `ai-models` and `librechart`)
When a chart's components have different lifecycles (sync waves, rollback granularity, per-component ArgoCD UI surface), split it:
```
charts/<thing>/ # orchestrator: emits ApplicationSet only
charts/<thing>-<componentA>/ # leaf: one Application worth of stuff
charts/<thing>-<componentB>/ # leaf
```
The orchestrator's `Chart.yaml` depends only on `common`. Its `templates/applicationset.yaml` emits ONE `ApplicationSet` whose List generator has one element per child. Children point at the sibling leaf charts via `path: charts/<thing>-<componentN>`.
See ADR-0012 (`ai-models`) and ADR-0014 (`librechart`) for the canonical examples + the rationale. **Don't reinvent this pattern** — copy from one of these when you need it again.
> **App-of-Apps variant (`observability` ADR-0020, `lightbridge`, `homepage`).** When an orchestrator's children are *fixed and heterogeneous* — e.g. local Helm charts + upstream charts that deploy as sources, with large inline valuesObjects — an ApplicationSet List generator handles it poorly (OCI-as-dep needs a `Chart.lock`; inline sources need controller-time goTemplate). These orchestrators instead render their child `Application` CRs directly via plain Helm (`templates/applications.yaml`, iterating `.Values.children`). Same controlPlane/destination rules; children support `depsOverlay` (cert/secret overlays) and an orchestrator-level `podSecurityEnforce`. `charts/observability` → the 10 observability components + a (disabled) `observability-secrets` child.
## Umbrella Applications + `environments/` overlays (ADR-0018)
> ⚠️ **Amended by [ADR-0055](docs/adr/0055-oci-charts-and-image-updater-writeback-to-values-repo.md):** the multi-source umbrella shape below is the foundation the continuous-delivery model builds on, but two things move. (1) An app opting into OCI mode (`chart:` set) gets Source A = the **OCI chart** (`argocd.chartRegistry` @ `argocd.chartVersionRange`) and Source B = the `$values` ref to the **private `ai-helm-values` repo** (image tags written-back). (2) The `environments/` overlays themselves **migrate into `ai-helm-values`** at the cutover — the deps Source then resolves via `argocd.depsRepoURL`/`depsTargetRevision` (default = this repo until flipped). See `docs/continuous-delivery.md`.
Flat leaf apps in `charts/apps/values.yaml` use a multi-source **umbrella** so a workload and its app-scoped prerequisites sync as one Application:
- **Source A — workload**: the Helm chart (`path: charts/<x>` or an upstream `chart:`).
- **Source B — app-scoped deps**: a **kustomize** overlay at `path: environments/<env>/deps/<app>` emitting the ingress `Certificate` and any per-app image-pull / session `ExternalSecret`. Kustomize is confined to these plain CRs — **never kustomize-over-Helm** (that needs the controller-wide `--enable-helm` flag).
- **Source C — `ref: values`** (optional): a `$values` ref so Source A can pull a per-env values file when a *workload* knob actually diverges between environments.
Per-env knobs (`clusterIssuer`, `secretStore`, `ingressClass`, `storageClass`, `domainBase`) live in `environments/<env>/cluster.yaml` (source of truth) and are patched into the dep CRs by the `environments/<env>/deps/<app>/` kustomize overlay (base under `environments/base/deps/<app>/`). Today only `environments/prod/` exists (Hetzner); a second env is a drop-in sibling directory.
Ownership split: umbrellas own **app-scoped** secrets/certs (referencing `ssegning-aws` by name). The store is never defined here. ⚠️ The wholesale-provisioner `secrets` Application (`ai-ops-secrets.git`) was **removed (2026-06-04)** — secrets are now **chart-owned** (in-chart ExternalSecrets + the `environments/<env>/deps/*` overlays). **App** secrets resolve against the consolidated `ssegning-aws` key **`ai/camer/digital/prod/env`** (one property each); **platform** secrets (S3, redis password) against `prod/meta/test-app`. LESSON: re-home a secret in-chart *before* retiring its provisioner — pruning `secrets` cascade-deleted the lightbridge secrets (incl. `lightbridge-opa-auth`) and caused a gateway outage.
**Workload values can also live in `ai-helm-values` (ADR-0056).** Set `valuesFromRepo: true` on a singular-`source:` app entry and the template injects `helm.valueFiles: [$values/environments/<env>/values/<app>.yaml]` (+ `ignoreMissingValueFiles`, + the `ref: values` source) — so the app's big `valuesObject` moves to a file in `ai-helm-values` instead of sitting inline in `charts/apps/values.yaml`. ⚠️ **Cut over values-repo-FIRST**: the file must exist on `ai-helm-values` `main` before the `charts/apps` change merges, or `ignoreMissingValueFiles` silently falls back to chart defaults (for `security-policies` that DROPS the gateway AuthConfig). A `render-check.yml` in `ai-helm-values` guards this (YAML-validates + `helm template`s the OCI charts). Migrated: **every flat `charts/apps` app that had a `valuesObject`** — `security-policies` (the ~405-line AuthConfig, now in the *private* repo), `eg`, `opencode-k8s-agent`, `apprise-api`, `aieg`, `core-gateway`, `lightbridge-repo-auth`, `mongodb-backup`, `converse-ui`. (`converse-ui`'s config was deep-merged into its existing image-updater-owned values file; `mongodb-backup`, a top-level-`chart:` OCI app, already had the `$values` ref injected so it needed no flag.) Don't leave a stale inline `valuesObject` behind — it would layer on top of the file. **The orchestrators are also done (ADR-0056 phase 3):** the `observability` (8 children incl. the Alloy River config) and `lightbridge` App-of-Apps templates gained the same per-child `valuesFromRepo` injection, and their child configs live in `ai-helm-values` too (e.g. `environments/prod/values/{alloy,mimir,loki,tempo,grafana,…,lightbridge-app}.yaml`). ⚠️ observability children shared YAML anchors (`*s3-endpoint` etc., a `global:` block) — separate `$values` files can't share anchors, so those were **inlined to literals** per file and the `global:` block removed; the Alloy River config moved free of any Go-template double-braces (it's `tpl`'d). `ai-models`/`librechart`/`mcps` have no inline `valuesObject` (leaf-chart config, ADR-0012/0014) — ⚠️ but for `ai-models` that was never the whole story, and **ADR-0126 finished the job**: its config wasn't inline in `charts/apps` because it *was* `charts/ai-models/values.yaml`, ~2,000 lines of backends, models, prices and plans shipped as chart defaults. That whole file is now `ai-helm-values environments/prod/values/models.yaml`, and the chart's defaults are an empty skeleton whose `requireCatalog` guard **hard-fails the render** rather than letting an absent file prune every model route. `librechart`/`mcps` are the same shape and remain candidates. **Every inline `valuesObject` with real config now lives in `ai-helm-values`.**
**How to attach deps:** add one field to the app entry — `depsOverlay: environments/<env>/deps/<app>`. `applications.yaml` folds it in as a Source pointing at the **`ai-helm-values`** repo (`argocd.depsRepoURL` @ `depsTargetRevision` = `main`, where `environments/` lives post-ADR-0055), keeping the workload's `source:`/`chart:` + `valuesObject` verbatim (no re-indenting big value blocks). Also drop the `cert-manager.io/cluster-issuer` annotation from that chart's ingress — the overlay `Certificate` now owns the TLS secret. Flat umbrella: `lightbridge-backend`. Dep-less infra/backends stay single-source. (`grafana` also consumes `environments/<env>/deps/<app>` cert overlays, but as a **child of an App-of-Apps orchestrator** — `grafana` under `observability` ADR-0020 — not as a flat umbrella. The orchestrator templates support `depsOverlay` on children too.) Also drop the `cert-manager.io/cluster-issuer` annotation from that chart's ingress — the overlay `Certificate` now owns the TLS secret. Flat umbrella: `lightbridge-backend`. Dep-less infra/backends stay single-source. (`grafana` also consumes `environments/<env>/deps/<app>` cert overlays, but as a **child of an App-of-Apps orchestrator** — `grafana` under `observability` ADR-0020 — not as a flat umbrella. The orchestrator templates support `depsOverlay` on children too.)
The umbrella needs **no ApplicationSet** — `applications.yaml` already passes `.sources` through. (The List/Matrix-generator conversion, old ADR-0006, is decoupled future work.) Orchestrators (`models`, `librechat`) are **not** wrapped — they're already ApplicationSets.
> ⚠️ The `applications.yaml` template's custom-`syncPolicy` branch previously omitted the `syncPolicy:` wrapper key (contents leaked into `destination:`), so ~13 apps that declared their own `syncPolicy` rendered with **no** `spec.syncPolicy` (manual sync, declared automation silently dropped). Fixed in the ADR-0018 work — those apps now get their declared `automated: {prune, selfHeal}`. Sanity-check sync behaviour on the live cluster after merge.
## ai-helm ↔ ai-helm-values: what lives where (ADR-0055 + ADR-0056)
The deployment-state repo older ADRs called "ai-gitops" exists now as the PRIVATE
**`adorsys-gis/ai-helm-values`** (ADR-0055). The split is **complete and is a hard
contract** — its full inventory + consumption model live in **`ai-helm-values/README.md`**;
the summary:
| Concern | `ai-helm` (this repo) | `ai-helm-values` (private) |
|---|---|---|
| Helm **chart logic / templates** | ✅ `charts/<x>/templates/`, the `charts/apps` umbrella + orchestrator templates | — |
| Chart **defaults** (structural only) | ✅ `charts/<x>/values.yaml` | — |
| **Workload config** (per-app `valuesObject`) | ❌ moved out (ADR-0056) | ✅ `environments/<env>/values/<app>.yaml` |
| **Image tags** | ❌ | ✅ same files, written back by argocd-image-updater |
| **The model catalog** (backends, models, prices, rate-limit plans) | ❌ moved out (ADR-0126) — `charts/ai-models/values.yaml` is an empty skeleton that refuses to render | ✅ `environments/prod/values/models.yaml` |
| **The GPU fleet catalog** (which model runs on which card, engine profiles) | ❌ moved out (ADR-0129) — `charts/inference/values.yaml` is an empty skeleton too | ✅ `environments/prod/values/inference.yaml` |
| **Model prices** | ❌ | ✅ machine-maintained — a 6-hourly job syncs them from the providers' APIs and commits to `main` (ADR-0127) |
| **Per-env deps overlays** (Certificate / ExternalSecret / CiliumNetworkPolicy) + `cluster.yaml` | ❌ moved out (ADR-0055) | ✅ `environments/<env>/deps/<app>/` |
| ArgoCD **root** `ai-apps-v2` | — | ❌ — pinned in **`home-os`** `charts/cd` |
**Rule of thumb:** if you're about to add or edit a Helm `valuesObject`, a model, a price, an image tag, or a
per-env CR in `ai-helm`, stop — it belongs in `ai-helm-values`. ai-helm holds *how to render*;
ai-helm-values holds *what is deployed*. ⚠️ Always cut over **values-repo-first**: the file
must exist on `ai-helm-values` `main` before the ai-helm chart change merges, or
`ignoreMissingValueFiles` silently falls back to chart defaults. Charts publish to OCI and
float; image tags + per-env overrides live in the values repo; the root stays in `home-os`.
**ADR-0010** proposed image-updater write-back (to the then-named "ai-gitops"); **ADR-0013**
deferred it; **ADR-0055** realised it as `ai-helm-values` (charts→OCI float + image-tag
write-back); **ADR-0056** extended its scope from image-tags to **all workload config** (every
flat `charts/apps` app + the `observability`/`lightbridge` orchestrator children).
## Deploys: CONTINUOUS DELIVERY (ADR-0055) — root tracks `main`, no release tag
> ✅ **Cutover complete (2026-06-22).** The whole fleet runs on
> [ADR-0055](docs/adr/0055-oci-charts-and-image-updater-writeback-to-values-repo.md)
> continuous delivery. `tools/release.sh`, `docs/releasing.md`, the `release-*`
> tag model, and this repo's `environments/` tree are **retired/removed**.
How deploys work now (runbook: `docs/continuous-delivery.md`):
- **Charts** publish to `oci://ghcr.io/adorsys-gis/charts` on merge to `main`
(auto-semver, `.github/workflows/publish-charts-oci.yml`). Every app sources its
chart from OCI on a **floating semver range** (`argocd.chartRegistry` @
`chartVersionRange`, full chart path in repoURL + `chart: "."`); the orchestrators
float their children the same way. A merge to `main` is a **live deploy** —
immutability deliberately abandoned.
- **Image tags + per-env values + the `environments/` overlays** live in the
PRIVATE **`adorsys-gis/ai-helm-values`** repo. argocd-image-updater writes new
cosign-gated image tags there (direct commit); ArgoCD reads it via each app's
`$values` and `depsOverlay` sources (`argocd.valuesRepoURL` / `depsRepoURL` @
`main`). ArgoCD reads it with the org-wide `github-app-creds--adorsys-gis`
repo-creds; the OCI registry needs no chart-pull cred (public, and the
registry must be a registered Helm-OCI repo only for *private* charts).
- **The root `ai-apps-v2`** tracks **`main`** (pinned in `home-os` `charts/cd`),
not a tag. **Rollback** = `git revert` in `ai-helm-values` (image tags / values)
and/or pin an app's `chartVersionRange` to a known-good version. There is no
immutable fleet snapshot.
- **External non-OCI sources** still pin their own SHAs/versions: `converse-frontends`
(aii-converse-ui), `lightbridge-repo-auth`, the upstream Lightbridge chart
(`adorsys-gis.github.io/lightbridge-authz`), `opencode-k8s-agent`, and upstream
Helm charts (aieg, eg, grafana, …). Bump those deliberately.
The historical **tag-based** model (ADR-0031, `release-YYYY.MM.DD` + `release.sh`
+ manual root repoint) is **superseded** — kept only as ADR history.
## ArgoCD destinations: two-tier — control objects in-cluster, workloads home-remote (ADR-0017)
**Two distinct "where"s.** ArgoCD `Application` / `ApplicationSet` CRs (control objects) must live on the cluster ArgoCD runs in — **in-cluster**, `argocd` namespace — or the controllers never watch them. The **workloads** those Applications deploy go to the registered cluster **`home-remote`**.
- **Workloads → `home-remote`.** Driven by `argocd.destination` (`name` / `server` / `allowInCluster`) in `charts/{apps,ai-models,librechart}/values.yaml`. Never use ArgoCD's built-in in-cluster handle (`name: in-cluster` / `server: https://kubernetes.default.svc`) for a workload — even if it resolves to the same physical cluster, it's a different ArgoCD destination. The helper `<chart>.argocd.destinationClusterRef` (each chart's `templates/_helpers.tpl`) **hard-fails the render** if a workload destination resolves to the in-cluster handle unless `allowInCluster: true`.
- **Control objects → local cluster / argocd.** In `charts/apps`, an app whose deployed content is itself a control object (an orchestrator emitting an ApplicationSet — `models`→`charts/ai-models`, `librechat`→`charts/librechart`) sets **`controlPlane: true`** on its entry. The template then targets `argocd.inClusterServer` (**`server: https://kubernetes.default.svc`** — the canonical local-cluster ref, used instead of the `name: in-cluster` handle which depends on that registration existing) / `argocd.controlPlaneNamespace` (`argocd`) and bypasses the guard. The orchestrators' ApplicationSet `template.spec.destination` (the **child** Applications) stays `home-remote`.
- **The root `ai-apps-v2` Application** (its `targetRevision` pinned in `home-os` `charts/cd/values.yaml` — there is no `ai-gitops`) deploys `charts/apps` and **must itself target in-cluster/argocd** so the generated Application CRs land where the controller watches.
- **`homeCluster: true` — the ONE sanctioned ADR-0017 exception (ADR-0022), now LEGACY-ONLY (ADR-0095).** A *workload* (not a control object) that must run on the cluster ArgoCD itself runs on — the `model-serving-*` apps needing the home RTX A2000 (all now `enabled: false`). They target `argocd.inClusterServer` but keep their own workload namespace (unlike `controlPlane`, which forces `argocd`), and the destination guard is called with `allowInCluster: true`. ⚠️ **New GPU workloads must NOT use this** — the Hetzner GPU nodes are on `home-remote`, so `charts/inference` children are ordinary workloads. The exception is scoped to the `admin@homeos` generation and retires with it. Don't add more `homeCluster` apps without an ADR.
Don't re-hardcode a cluster name in the templates (the old `lke560142-ctx` magic string is gone). The render-time guard is complemented out-of-band by the `ai` AppProject's `destinations:` allowlist. See ADR-0017.
**Project invariant: every Application/ApplicationSet from this repo lives in the `ai` AppProject.** `charts/apps` hardcodes `project: ai` (not a per-app value — so an app entry can't drift); the four orchestrators (`ai-models`, `librechart`, `observability`) set their children to `argocd.project` (= `ai`). There is intentionally **no per-app `project` override**. Verified live: all repo-sourced apps (flat + orchestrator children, old `ai-*` + new `aii-*`) are in `ai`. If you add an app, it inherits `ai` automatically — don't introduce a project knob.
## Hetzner cluster realities & recurring gotchas (2026 cutover)
Operational narrative + live verification: **`docs/migrations/2026-hetzner-cutover.md`** (read it before debugging the live cluster). The high-impact facts:
- **Two clusters, two kubeconfigs.** ArgoCD itself runs on a **separate** cluster — context **`admin@homeos`** (Talos, in `~/.kube/config`); all `Application`/`ApplicationSet` CRs live there in ns `argocd` (`kubectl --context admin@homeos -n argocd get applications`; trigger syncs via `argocd.argoproj.io/refresh=hard` or patching `.operation`). **Workloads** run on Hetzner k3s = `home-remote` — inspect with `KUBECONFIG=/Users/selast/dev/personal/hetzner-k8s/kubeconfig kubectl …`. The Hetzner kubeconfig has NO argo CRDs.
- **⚠️ `ai-*` vs `aii-*` = two DIFFERENT workload clusters, not a same-cluster collision.** Both app generations live on the same ArgoCD (`admin@homeos`) but deploy to different destinations: root **`ai-apps` @ HEAD → the other (pre-Hetzner) cluster** (the `ai-*` apps); root **`ai-apps-v2` → Hetzner `home-remote`** (the `aii-*` apps). They don't interact. **Only ever act on `aii-*` (Hetzner). NEVER touch `ai-*`** — a delete/sync hits the OTHER cluster (and the self-healing `ai-apps` root re-creates it anyway). Decommissioning the old `ai-*` gen is the maintainer's own exercise on that cluster.
- **CNI is Cilium + a default-deny-egress baseline.** Each app namespace (`apps`/`data`/`observability`/`platform`) carries a manual `allow-dns` NetworkPolicy (DNS-to-kube-system only, **not** ArgoCD-managed; defined in `hetzner-k8s`), so every pod is egress-deny-by-default. Any pod reaching the **API server** (operators: grafana-operator, kube-state-metrics) or **external object storage** (mimir/loki/tempo) crashloops until it gets an additive allow. ⚠️ A plain k8s `NetworkPolicy` `ipBlock` does **NOT** match on Cilium (node IPs carry `remote-node`/`host` identity). Use a **`CiliumNetworkPolicy`** with `toEntities: [kube-apiserver]` (API) or `toFQDNs: "*.your-objectstorage.com"` (S3), shipped via the app's deps overlay — see `environments/prod/deps/{grafana-operator,kube-state-metrics,observability-secrets}/`. Classic symptom: pod hangs ~32s then a 0-`initialDelay` liveness probe kills it → looks like a silent exit-2 CrashLoop. (`converse` has NO baseline → its pods egress freely.)
- **Object storage = Hetzner Object Storage**, NOT the old `s3.ssegning.me`. Endpoint `nbg1.your-objectstorage.com`, shared bucket **`ssegning-k8s-state`** (Keycloak CNPG backups + mimir/loki/tempo + LibreChat, each in its own folder/prefix). Creds in `ssegning-aws` at key `prod/meta/test-app`, props `s3_backup_cnpg_client_id` / `s3_backup_cnpg_secret`; region `us-east-1` works (Ceph-RGW). mimir `storage_prefix` is **alphanumeric-only** (no `/`) → folders like `mimirblocks`.
- **Hetzner LB targets WORKERS ONLY.** The 3 control-plane nodes are labelled `node.kubernetes.io/exclude-from-external-load-balancers` (durable now in `hetzner-k8s` `install-platform.sh`). LB Services need `load-balancer.hetzner.cloud/use-private-ip: "true"` (cp-1 has a stale providerID and is an unusable server-ref target). core-gateway's data-plane LB is at `46.225.38.138`.
- **nginx static-serve charts** (`librechat-opencode-wellknown`, `ai-models-info`): their custom `default.conf` MUST set `root /usr/share/nginx/html;` or `try_files` 404s. ⚠️ **A CONTENT change needs no pod restart** — this entry used to claim the content was a subPath-mounted ConfigMap requiring the pods to be deleted; that is **wrong for both charts** (verified 2026-09-01 by rendering each: every mount is a plain directory mount, zero `subPath`). A directory-mounted ConfigMap is synced in place by the kubelet, and nginx re-reads a static file from disk per request, so new content serves itself. Measured during the ADR-0135 opencode cutover: the well-known JSON changed on the live endpoint while the pods stayed **19 days old** — never restarted. Don't delete pods for a content change. (The pods DO roll on a chart-version bump, which is a Deployment-spec change — that is a different thing.) The restart caveat only ever applied to **`default.conf`**: nginx reads its own config at start, so a config-file change does need the pods replaced — and `kubectl rollout restart` is reverted by ArgoCD selfHeal, so delete the pods instead.
- **Restart-ordering trap:** the `ai-gateway-controller` mutating webhook is fail-closed and gates ALL `envoy-gateway-system` pod creation — after a cluster restart, anything stuck on `FailedCreate: no endpoints available for ... ai-gateway-controller` recovers by deleting the stuck ReplicaSet once the controller is up.
- **Mimir ring wedges if memberlist is blocked at startup.** `mimir-distributed` forms its ingester/store-gateway ring via memberlist gossip (pod↔pod `:7946`). The Cilium deny-egress baseline blocks that until an `allow-same-namespace` policy exists; if the pods start *before* the policy, they exhaust their join retries and never re-form → the distributor logs `DoBatch: InstancesCount <= 0` and **every** remote-write 500s → **Mimir silently stores nothing**. Guard in place (two layers): (1) `allow-same-namespace` for `observability` is **already durable + ArgoCD-managed** — it ships from THIS repo via the `observability-secrets` child (wave -3, before the wave -2 stores): `environments/base/deps/observability-secrets/allow-same-namespace.yaml`. (Do **not** also add it to the hetzner-k8s baseline — two ArgoCD apps owning one NetworkPolicy = sync ping-pong.) The recent wedge was an *ordering artifact* (the secrets app was enabled 9h after Mimir already started), not a missing policy. (2) Mimir `memberlist.rejoin_interval: 1m` (in `charts/observability` structuredConfig) self-heals the residual wave-ordering race — a late policy now heals the ring within a minute, no restart. If you ever still see `InstancesCount <= 0` (egress confirmed open), the operational unwedge is `kubectl -n observability rollout restart statefulset/mimir-ingester` then the distributor/queriers. (Audit: `docs/migrations/2026-06-07-observability-datasource-audit.md`.)
- **Stateless Grafana + grafana-operator: GrafanaFolders need a `resyncPeriod`.** Grafana runs stateless (emptyDir, ADR-0023), so every Grafana pod roll wipes operator-pushed folders/dashboards. A `GrafanaFolder` whose status is cached `ApplySuccessful` is NOT re-created in the fresh pod → `folderRef` dashboards fail `[400] folder not found` until the operator is restarted (`kubectl rollout restart deploy/grafana-operator`). Fix: set `resyncPeriod` on every folder in `charts/observability-dashboards` `folders[]` (dashboards already carry theirs) so folders self-heal after a restart. Symptom to recognise: a *new* folder's dashboards stick on "folder not found" right after a Grafana roll.
- **The Grafana LLM assistant (`grafana-llm-app`) was REMOVED (ADR-0076, supersedes ADR-0062) — don't re-add it without a new ADR.** It worked end-to-end through our gateway but `grafana-llm-app` is only an LLM *backend*: self-hosted OSS Grafana has no AI chat UI (the Assistant is Cloud-only), and the sole OSS chat front-end (`vikshana-graft-app`) is an unsigned third-party plugin we declined on security grounds. If a future consumer resurrects the backend, the two live-found gotchas are in ADR-0062's (superseded) body: the plugin needs `provider: custom` (NOT `openai`, which ignores `openAI.url` and calls api.openai.com), and Cilium egress must target the gateway's **backend** port `:10443` (the Service is `:443` but Cilium enforces post-DNAT on the pod's targetPort). ⚠️ **The `grafana-internal-ca` cert + its `/etc/ssl/certs/internal-gateway-ca.pem` mount were RETAINED** — ADR-0070 repurposed them so the Grafana `Redis` datasource can verify redis-ha's internal TLS. Don't delete them as "leftover LLM config". Future in-Grafana AI, if wanted: `mcp-grafana` (no unsigned plugin needed).
- **kube-state-metrics ServiceMonitor MUST set `honorLabels: true`.** ksm reports labels *about other* objects (`kube_pod_info{namespace="converse",…}`). Without honorLabels the scrape overwrites `namespace` with ksm's own namespace (`observability`) and moves the real one to `exported_namespace` → every Kubernetes dashboard's namespace/pod picker shows only `observability`. Set `prometheus.monitor.honorLabels: true` on the ksm child.
- **Grafana `dashboards:` (gnetId) with the new nested datasource format needs the LIST-form `datasource:`.** The grafana chart's string-form `datasource: Mimir` only rewrites the old single-line `"datasource": "...",`; dashboards using `"datasource": {"type":…,"uid":"${DS_PROMETHEUS}"}` keep the unresolved token → "Datasource ${DS_PROMETHEUS} was not found". Use `datasource: [{name: DS_PROMETHEUS, value: mimir}]` (value = the datasource **UID**, lowercase) to substitute the token directly. Input names vary per dashboard (`DS_METRICS`, `DS_EXPRESSION`→`__expr__`, etc.); template-variable-backed dashboards (k8s-views) don't need it. Mimir 429 "too many outstanding requests" on dense boards = bump `frontend.max_outstanding_per_tenant`. ArgoCD dashboards are pointless here (ArgoCD is on homeos, not home-remote). See `docs/migrations/2026-06-07-observability-datasource-audit.md` §7.
- **Grafana third-party datasource plugins need an explicit `plugins:` values-list entry, or the datasource silently "exists" but errors at query time.** Declaring `type: redis-datasource` (or any third-party type) in `datasources.yaml` provisions the datasource config but does NOT install the plugin binary — that's the upstream `grafana/grafana` chart's separate `plugins:` values key (renders into `GF_INSTALL_PLUGINS`, verified against the chart's own `templates/_pod.tpl` at the pinned version, `charts/observability/values.yaml`). Without it: Grafana logs `Could not find plugin definition for data source`, the panel shows `Datasource <uid> was not found` / "No data" — no provisioning error, nothing obviously wrong in the YAML. Bit the AI Gateway rate-limit quota dashboard's Redis census panel for every user for hours (confirmed broken since at least 2026-07-31T13:48Z across several real users) before being traced back to the missing plugin. Fix: add the plugin id to `plugins:` in `ai-helm-values environments/prod/values/grafana.yaml`; check `grafana.com/api/plugins/<id>` for `signatureType` first — unsigned plugins additionally need `GF_PLUGINS_ALLOW_LOADING_UNSIGNED_PLUGINS` (`redis-datasource` itself is Grafana-signed, `commercial`, so no override was needed there). See `docs/patterns/ratelimit-quota-observability.md`.
- **Grafana's `extractFields` transform: `format` must be `"regexp"` (not `"regex"`), and the pattern MUST be wrapped in `/…/` delimiters — two silent-failure traps, not one.** Verified against Grafana's own pinned-version source (`public/app/features/transformers/extractFields/{types,fieldExtractors}.ts` + `packages/grafana-data/src/text/string.ts`). (1) `"regex"` isn't a registered `FieldExtractorID` (valid: `json`\|`kvp`\|`auto`\|`regexp`\|`delimiter`) → fails the WHOLE transform with "Error transforming data: unknown extractor". (2) Even with `"regexp"` correct, `stringToJsRegex` only checks that the first character is `/` (`stringStartsAsRegEx`) — an unwrapped pattern is silently discarded and Grafana falls back to its own built-in default `/(?<NewField>.*)/`, so instead of an error the table just renders one "NewField" column holding the whole raw value, with your named capture groups never applied. Bit the rate-limit quota dashboard's Redis census panel (`_panel_live_census()`, `tools/dashboards/src/dashboards/envoy_ai_gateway/ratelimit_quota.py`) — both bugs pre-dated the panel's #532/#862 rewrites and were invisible until the plugin-install gotcha above was separately fixed, since the datasource error had been short-circuiting the transform pipeline before either could bite. See `docs/patterns/ratelimit-quota-observability.md`.
- **Alloy needs API-server egress AND OTLP ingress allows, or metrics+traces are empty.** Like kube-state-metrics / grafana-operator, the Alloy collector is default-deny-egress under the baseline, so without an additive allow its `prometheus.operator.servicemonitors`/`podmonitors` informers time out (`Get "https://10.43.0.1:443/api": i/o timeout`), node discovery + all ServiceMonitor scrapes return nothing, and Mimir holds only the static `up{job="apiserver"}`. Separately, `observability`'s `default-deny-ingress` drops cross-namespace OTLP, so Alloy's `:4317/:4318` receiver gets `accepted_spans=0` → Tempo empty. Fix shipped: `environments/{base,prod}/deps/alloy/` (wired via `depsOverlay` on the alloy child) — Cilium egress `toEntities: [kube-apiserver, cluster]` + DNS, and ingress on `:4317/:4318` `fromEntities: [cluster]`. Diagnose with Alloy's component-health API (`:12345/api/v0/web/components`) + its `/metrics` (`prometheus_remote_storage_samples_total`, `*receiver_accepted_spans_total`). Details: `docs/migrations/2026-06-07-observability-datasource-audit.md` §6.
- **Disabling Loki's canary REQUIRES disabling its helm test too.** The grafana/loki chart's `templates/validate.yaml` does `{{- if and (not .Values.lokiCanary.enabled) .Values.test.enabled }}{{- fail "Helm test requires the Loki Canary to be enabled" }}`, and `test.enabled` defaults to **true**. Setting `lokiCanary.enabled: false` without `test.enabled: false` makes the **whole loki render fail** → the ArgoCD `loki` app goes `ComparisonError`/`Unknown` and stops syncing (the running `loki-0` looks fine — it's a manifest-generation error, not a pod crash). Always pair the two.
- **metrics-server is the k3s-bundled one now — we don't ship our own (ADR-0054).** We *used* to deploy a GitOps `metrics-server` (`aii-metrics-server`, kubernetes-sigs chart), but it collided with k3s's bundled addon on the shared `Deployment`/`Service` name in `kube-system`: k3s re-applies its pod (label `k8s-app` only), our Service selector (`app.kubernetes.io/{name,instance}`) stopped matching → empty endpointslice → `v1beta1.metrics.k8s.io` `MissingEndpoints` ("no addresses with port name https") → `kubectl top`/HPA dead, on **every sync**. Resolved by **removing our app** (ADR-0054) so the healthy k3s addon re-owns the Service+APIService — there is **no `metrics-server` entry in `charts/apps/values.yaml`** by design; don't re-add one. ⚠️ If `MissingEndpoints` ever recurs, it's the k3s addon's Service that's missing/wrong, NOT our chart (we no longer ship it) — check `kubectl -n kube-system get svc,endpointslice metrics-server` + the k3s addon manager, not ArgoCD. Background + the now-obsolete "delete + re-own" remediation: `docs/migrations/2026-06-07-observability-datasource-audit.md` §4. (The old hetzner-k8s ADR-0015 path — `--disable metrics-server` + CP node `-replace` to run our own — is only relevant if we ever need bespoke metrics-server config; write a superseding ADR then.)
- **Grafana Tempo datasource is `:3200`, NOT `:3100`.** The grafana/tempo single-binary chart serves its HTTP query API on `server.http_listen_port: 3200` (Service port `tempo-prom-metrics`); there is no `:3100` listener (that's Loki's default). A `:3100` Tempo datasource URL fails every query with "failed to reach". In-cluster: `http://tempo.observability.svc.cluster.local:3200`.
- **Opaque per-user `user_id` UUIDs are resolved to people via a read-only Keycloak Postgres datasource (ADR-0063), NOT by re-processing logs.** The cost boards key on `user_id` = the Keycloak `sub`; a thin token (no email/name claim) leaves only the UUID. Grafana has a read-only Postgres `GrafanaDatasource` `uid: keycloak` → the Keycloak CNPG `-ro` replica, and the generated `user-directory` dashboard OUTER-joins (`joinByField` on `user_id`) the Mimir spend to `user_entity` (empty Name ⇒ non-human CI/`internal-key-*` subject). The `grafana_ro` role is **least-privilege** (SELECT on user+token tables ONLY — never `credential`/client-secret/federated-token tables; it's the auth DB) and is **NOT** granted the `realm` table, so queries filter `user_entity.realm_id` by the LITERAL `camer-digital` id (`04793949-…`, `_common.CAMER_DIGITAL_REALM_ID`), not by name. **Cross-repo:** the role + ESO secret + idempotent GRANT Job live in **home-os** `charts/home-apps/keycloak-ha` (it owns the cluster); datasource + Grafana→keycloak:5432 egress + secret-copy in **ai-helm-values**; dashboard in **ai-helm** (`tools/dashboards/.../user_directory.py`). ⚠️ out-of-band-secret-first: add `keycloak_grafana_ro_db_password` to `ssegning-aws prod/meta/test-app` before sync (else both ExternalSecrets sit in `SecretSyncedError`). See `docs/patterns/per-user-observability.md`.
- **Per-JWT (`oidc_jti`) consumption is a Loki dashboard (ADR-0067), NOT Mimir — and the extraction has a load-bearing trap.** The `jwt-tokens` dashboard (`tools/dashboards/.../jwt_tokens.py`) keys on the JWT id `oidc_jti` (an access-log BODY field, deliberately never a Mimir label — per-token cardinality, ADR-0064) × the JWT `email` LABEL (claim-only; thin tokens show `missing:*`/`unstamped:*` sentinels by design — the Keycloak DB is NOT consulted here). ⚠️ **Extract `oidc_jti` in the SAME `| json` stage the outer `sum by (oidc_jti, email)` groups on** — extracting only the unwrap field (`| json cost=…`) leaves `oidc_jti` unextracted → it collapses to `-` and every token merges into one bogus row. Dotted usage keys (`gen_ai.usage.total_tokens`) need the backtick bracket form `field=["dotted.key"]`; every `unwrap` needs `| __error__=""`; queries are RANGE not instant (the Loki plugin won't substitute `$__range` in instant). 6h default + `topk(20)`; cardinality verified low live (~handful jti/hour). See `docs/patterns/jwt-token-observability.md`.
- **⚠️ The rate-limit QUOTA read path (ADR-0070) reads counters that NO LONGER EXIST — don't debug it, retire it.** ADR-0070 built a live "who's consuming how much of their budget" view over the Lyft ratelimit service's per-account µ$ cost counters in redis-ha (keys `…rule-<N>-match-0_<x-account-id>_…_<window>`, plan = the rule index, e.g. `rule-2`=free / `rule-7`=pro, `<window>` = a 30-day epoch bucket). Those cost rules were **deleted 2026-09-05** (see *Gateway authz + rate limiting* below — the money cap is now the lightbridge-authz ledger + the 402 `budget-limiter.lua`), so nothing writes those keys. State on `main` of both repos: (1) the `prometheus-redis-exporter` observability child still runs, but `ai-helm-values .../prometheus-redis-exporter.yaml` **unset `REDIS_EXPORTER_CHECK_KEYS` and deleted every key-parsing relabel** — it now keeps only `redis_up` (exporter↔Redis liveness), so `gateway_ratelimit_spend_micro_usd` is no longer produced; (2) the `redis-datasource` GrafanaDatasource (`uid: redis-ratelimit`) + the `AI Gateway — rate-limit quota` dashboard (`tools/dashboards/.../ratelimit_quota.py`) are still deployed and **render empty**. Retiring/repointing them is open work (tracked in ai-helm-values `docs/runbooks/budget-limiter-rollout.md` §"What is NOT done"). The only per-account counters left in Redis are the per-model **RPM** counters (requests, not µ$). Budget state is read from the access log's `budget.*` fields. Still-true plugin facts if the datasource is ever reused: the `redis-datasource` plugin only dials TLS when `jsonData.tlsAuth: true` (no client cert needed — `tls-auth-clients no`), verified against `secureJsonData.tlsCACert: $__file{/etc/ssl/certs/internal-gateway-ca.pem}`. See `docs/patterns/ratelimit-quota-observability.md`.
- **Alloy pod-log labels come from K8s service discovery, never from regex-on-the-log-line.** A `loki.process` `stage.regex` with no `source` parses the **log body**; any line that mentions a `/var/log/pods/<ns>_<pod>_…/<container>/` path (Loki's own flush/query logs do constantly) overwrites that stream's `namespace`/`pod`/`container` labels → all logs ingest but are mislabeled and unfilterable ("Loki shows only useless logs"). Derive labels from `discovery.kubernetes` (role=pod, field `spec.nodeName=$NODE_NAME`) + `discovery.relabel` instead; reserve line-parsing for `level`/`trace_id` enrichment. `loki-canary` is disabled (synthetic-log noise). See the alloy `valuesObject` in `charts/observability/values.yaml`.
- **Envoy's OTel access-log sink emits format.json fields as OTLP ATTRIBUTES, not the log body (ADR-0046).** `otelcol.exporter.loki` then stores the line as `{"attributes":{...},"resources":{...}}` — a top-level `stage.json`/`| json` finds nothing (this is why the per-user dashboard was empty since rollout, and why the stream landed as `service_name="unknown_service"`). Alloy's `ai_gateway_user_attribution` stage now `stage.match`es gateway lines (`otel_envoy_accesslog` marker), flattens the line to the attributes object (`stage.output`), promotes `user_id`/`azp`/`model` labels (`-` → empty by EXACT match — UUIDs contain `-`), and pins `service_name=envoy-ai-gateway` via `stage.static_labels`. Query contract: anchor on that `service_name`; numeric body fields are STRINGS and absent ones are `"-"` → every `unwrap` needs `| __error__=""`. Validate Alloy config edits with `alloy fmt`/`alloy validate` inside a running alloy pod (helm only checks YAML, not River). See `docs/patterns/per-user-observability.md`.
- **⚠️ The Alloy `configMap.content` (the whole River config in `charts/observability/values.yaml` alloy child) is run through Helm's `tpl` by the upstream alloy chart (`templates/configmap.yaml`), so it MUST contain ZERO Go-template double-brace sequences — including inside `//` comments.** A `loki.process` `stage.template` (whose `template = "..."` is Go-template syntax) breaks **ArgoCD manifest generation** at sync time (`failed running helm … unexpected … in operand`), NOT at our gates — the alloy app goes `ComparisonError` and stops syncing while the running pods keep the OLD config. **None of our local checks catch this**: `helm template charts/observability` only renders the *Application CR* (content is a literal string there), and `alloy validate` treats `{{ }}` as harmless River string content. To map a field to empty (e.g. Envoy's `-` placeholder → no label) use `stage.replace { expression = "^-$"; replace = "" }` (regex-only, anchored = exact match), never `stage.template`. To actually exercise the `tpl` path locally, pipe the extracted content through a one-line `{{` `tpl (.Files.Get "content.txt") .` `}}` test chart, or just grep the content for `{{` (must be none). Cost a sync outage on `release-2026.06.12-v05` (ADR-0046 mechanism; fixed via `stage.replace`).
- **⚠️ ARC GitHub-runner pods that exit 0 in <1s and register `offline` = a dropped `command` (cross-repo: the runner sets live in `home-os` `charts/cd/values.yaml`, NOT here).** The `gha-runner-scale-set` (ARC) runner apps (`gis-sse-runners`/`*-runners`, deploying to Hetzner `home-remote` ns `gh-runners-*`, listener in `arc-systems`) override `template.spec.containers` for image/resources/env. Because **lists replace wholesale** in a Helm `valuesObject` merge, that discards the chart's default `command: ["/home/runner/run.sh"]` — so the stock `actions-runner` image runs its no-op default CMD, the container terminates `exit=0` with `startedAt == finishedAt` and empty logs, and the JIT-registered runner shows `status: offline`/`os: unknown`/`version: null` at GitHub and never picks up a job. Symptom seen from *this* repo's vantage: queued org jobs (e.g. `governance-check`) stall, and the ARC listener loops `decision: N` while `assigned job: 0`, churning N replacements indefinitely (a 4-day runaway on `arc-runner-set-sse`). **Fix lives in home-os** — every runner container override must restate `command` (`["/home/runner/run.sh"]`, or `["/bin/sh","-c"]` + `args: [sleep N; exec /home/runner/run.sh]` when waiting for a dind sidecar). Fast triage before blaming GitHub/the queue: `runner` container `exit=0`-in-0s + `gh api /orgs/<org>/actions/runners` showing `offline`. (Recorded here because the agent works from this repo; the runner config + its own gotcha note are in `home-os` `CLAUDE.md`.)
- **⚠️ `charts/keycloak-baseline` is NOT deployed or reconciled by anything** — no ArgoCD app, no keycloak-config-cli run, no `KeycloakRealmImport` CR. The `camer-digital` realm is **manually managed**, so editing that chart does NOT reach the cluster and the chart can be silently wrong. Change the realm in the admin console, then mirror it back into the chart. Real drift risk — verify against the live realm before trusting the chart.
- **oauth2-proxy `cookie-secret` must decode to exactly 16/24/32 bytes.** Use `openssl rand -hex 16` (32 chars). A padded base64 (44 chars) crashloops at startup: `cookie_secret must be 16, 24, or 32 bytes to create an AES cipher, but is 44 bytes`. Bites once per newly gated app (homepage).
- **Trivy scans ZERO Rust crates unless the image is built with `cargo-auditable`** — without it you only scan the base OS and a green result is meaningless. With it Trivy reports a `rustbinary` target. Applies to any Rust image behind a Trivy gate.
- **GitHub-hosted Actions runners are billing-blocked for PRIVATE adorsys-gis repos** (`The job was not started because recent account payments have failed...`). PUBLIC repos (ai-helm, lightbridge-authz) run fine on `ubuntu-latest`, which is free for public repos. Private repos (e.g. `ai-helm-values`) use **`runs-on: arc-runner-set`**: the org-scoped ARC scale set defined in `ADORSYS-GIS/wazuh` `argocd-repo/dev/arc/runner-scale-set.yaml` (Default runner group). ⚠️ **Never `runs-on: adorsys-gis-runner`**: its pools were deleted from home-os on 2026-09-15. A job on that label doesn't fail, it sits `queued` and is cancelled at GitHub's 24 h limit, so nothing ever validates and every check reads as cancelled. That silently disabled `ai-helm-values`' merge gates for three weeks (ai-helm-values#450, fixed by #461/#467). When a `queued`-forever job is on a working label, check `gh api orgs/ADORSYS-GIS/actions/runners` for an `offline` / `os: unknown` ephemeral runner wedging the scale set.
## Pod Security Standards per namespace (cluster-portability knob)
k3s on Hetzner enforces the `baseline` Pod Security Standard cluster-wide, which forbids `hostPath` / host networking. The observability collectors need them (Alloy tails `/var/log` via a hostPath; node-exporter mounts host `/proc`,`/sys`,`/`), so their namespace must be `privileged`. Mechanism: an app-renderer injects `syncPolicy.managedNamespaceMetadata.labels.pod-security.kubernetes.io/enforce` for the relevant namespace, consistently across every app in it (no metadata contention). Two carriers: `charts/apps` has a declarative map **`global.namespacePodSecurity`** (`{ ns: level }`, empty today); the **`observability` orchestrator** carries **`podSecurityEnforce: privileged`** applied to all its children (since the observability stack moved there — ADR-0020). Control-plane apps (argocd ns) are never elevated.
**This is the per-cluster-kind knob:** a cluster with no Pod Security admission can ignore it (label is harmless); a `restricted`-default cluster still lists the namespaces needing `privileged`; per-env differences override the whole map via a `$values` multi-source (ADR-0018 Source C). A namespace not listed gets no PSS label (cluster default applies).
## Sync waves
Lower waves sync first:
- `-2`: Storage backends (Mimir, Loki, Tempo, kube-state-metrics, node-exporter)
- `-1`: Operators + ESO + cert-manager + grafana-operator + Alloy (collector)
- `0`: Workloads (LibreChat, AI Gateway, per-model apps)
- `1`: Content (dashboards, opencode-wellknown)
- `2+`: Per-app post-sync work
The infrastructure → storage → collection → visualisation order is load-bearing. `MONITORING_FIX.md` is the postmortem of what happens when you violate it.
## Auth headers contract (`x-oidc-*`)
ADR-0011 defines the canonical header set Authorino injects after JWT verification. Use these names downstream:
| Header | Loki label? | Source |
|---|---|---|
| `x-oidc-user-id` | yes (`user_id`) | `auth.identity.sub` |
| `x-oidc-azp` | yes (`azp`) | `auth.identity.azp` |
| `x-oidc-user-name`, `x-oidc-iss`, `x-oidc-roles-realm`, `x-oidc-resource-access`, `x-oidc-scope`, `x-oidc-jti`, `x-oidc-email`, `x-oidc-name` | no — body only | JWT claims |
Don't reintroduce the placeholder `x-cd-*` prefix or invent new headers without an ADR. The `x-oidc-*` set survives the OPA removal; alongside it, Authorino now also stamps the **rate-limit descriptors** `x-account-id` / `x-org-id` / `x-billing-plan` (ADR-0021) — see below. The old **OPA-derived** headers (`x-project-id`, `x-account-id`-as-`LIBRECHAT`, `x-api-key-id`, `x-api-key-status`) were removed; ⚠️ `x-api-key-id` and `x-project-id` have since come back under the same names but a **different source** — the lightbridge API-key JWT's `api_key_id` claim / the `lightbridgeintrospect` metadata step (constant `""` on the internal plane) — and `x-api-key-id` now keys the per-model request-rate rule (see below).
## Gateway authz + rate limiting (ADR-0021) — OPA REMOVED, Keycloak JWT is the boundary
⚠️ The old OPA path is **gone** (2026-06-04). The `lightbridge-validation` HTTP
metadata source + the `enforce-valid-key` authorization step were **deleted** from
the AuthConfig (`security-policies.authConfigs.main` in `charts/apps/values.yaml`).
A valid **Keycloak JWT = "you're in our system, you may use the gateway."** OPA is
reserved for *future burst control* only (not wired). So: ADR-0003 (skip-OPA-for-SA)
and the OPA-derived response headers (`x-project-id`, `x-api-key-id`, the `llm_*`
dynamicMetadata) are historical — don't reintroduce them without a new ADR. The
`serviceAccountClients` allowlist + `_skipForServiceAccounts` markers are now inert
but kept for if OPA burst-control returns. (The missing `lightbridge-opa-auth`
Secret — pruned with the `secrets` app — is what caused the 404 outage that
prompted the removal; see `docs/migrations/2026-hetzner-cutover.md`.)
**Read [ADR-0021](docs/adr/0021-burst-budget-billing-and-dual-plane-authconfigs.md) before touching auth or rate limiting.** The shape now:
- **Dual-plane, AuthConfig-per-host** (one SecurityPolicy + one Authorino, indexed by Host):
- **External** — `api.ai.camer.digital` (public LB, ACME TLS): humans (opencode) + remote SAs. Full Keycloak JWT. Descriptors via CEL with defaults so `billing_plan`/`organization` claims are **optional** (`→ free` / `→ sub`).
- **Internal** — `core-gateway-internal.envoy-gateway-system.svc.cluster.local` (ClusterIP only, `api-internal` listener, `self-signed-ca` TLS): in-cluster services. Accepts **EITHER** a k8s SA token (`kubernetesTokenReview`, one-time jobs) **OR** a static `apiKey` (labeled Secret `kuadrant.io/apikey-for=internal-gateway`, long-running services like LibreChat).
- **Per-user attribution:** a long-running service (LibreChat) authenticates as itself but **forwards the end-user's Keycloak sub** (`X-LibreChat-User`); the internal AuthConfig's CEL prefers it → per-user `x-account-id` (attribution + the per-account RPM counter; the internal plane is **out of scope** for the money cap below — its AuthConfig publishes a constant `budget.enforced: false`). Trust = internal plane is first-party-only + Authorino overwrites the descriptors.
- **Money cap = the Dynamic Budget Limiter, NOT a rate limit (ADR-0137, lightbridge-authz ADR-0034).** ⚠️ The Envoy/Redis **cost buckets are GONE** — the gateway-wide monthly + weekly µ$ `shared: true` rules (and the dormant per-model `monthlyBudgetUsd` figures) were deleted **2026-09-05** (ai-helm-values#427, owner ruling, ahead of the planned 2026-10-01 date) because `effective cap = min(plan bucket, ledger)` meant a refill could not clear a spent bucket (429s nothing explains). Spend is now capped **only** by the lightbridge-authz **Postgres ledger**: Authorino's `lightbridgeintrospect` step publishes the live balance as ext_authz dynamic metadata (`budget.*`), and `charts/core-gateway` `files/budget-limiter.lua` answers **402 `budget_exhausted`** when a metered account's balance is ≤ 0 (enforcing since 2026-09-04T19:41Z). Only lightbridge **API-key** credentials are metered; the internal, GitHub-Actions (`repobinding`) and legacy-Keycloak AuthConfigs publish `budget.enforced: false`, so the limiter allows them. Observe it via the access log's `budget.*` fields, not Redis. Rollback = `budgetLimiter.shadowMode: true` in `ai-helm-values environments/prod/values/core-gateway.yaml` — **never** re-add the cost buckets. (The chart templates can still render them — `core-gateway` `backendTrafficPolicy.monthlyBudget`, default `enabled: false` — values simply leave them off.)
- **Rate limiting** = request rate only: each model's own `BackendTrafficPolicy` (`charts/ai-model`) renders **two `unit: Minute` rules** from that model's `rpmPerKey` (catalog entry in `ai-helm-values environments/prod/values/models.yaml`; fallback `requestRate.defaultRpmPerKey: 60`) — `(x-api-key-id × x-ai-eg-model)` and `(x-account-id × x-ai-eg-model)` — **no `cost:` block, no plan/tier descriptor** (owner ruling 2026-09-06). Per-model policies keep each model's rules at indices 0/1 of its own list, so adding/removing a model can't renumber another's counters (ADR-0084). `rpmPerKey: 0` = no cap for that model. The `burst:` families have been commented out fleet-wide since 2026-08-01, and `rateLimitBudgeting.plans` is now **id-only** (it survives only to hold the append-only index contract); `x-billing-plan` is still stamped but no rendered rule consumes it. Static via Helm (the AIEG CRD), no dynamic OPA.
- **⚠️ `/mcp/*` is the one carve-out from Authorino (ADR-0038).** Each MCPRoute carries `securityPolicy.oauth` (leaf `charts/mcp`, shared config in `charts/mcps` `oauth:`), and an EG route-level SecurityPolicy **displaces** the gateway-attached Authorino policy whole-policy (no merge) — so on MCP routes JWT verification is Envoy's native `jwt_authn` filter (same Keycloak issuer), the gateway itself serves the MCP-spec/RFC 9728 discovery surface unauthenticated (`/.well-known/oauth-protected-resource/mcp/<name>` path-insertion form + AS-metadata aliases + 401 `WWW-Authenticate resource_metadata` challenge; plus our non-spec path-appended alias `<path>/.well-known/oauth-protected-resource` via HTTPRoute+DirectResponse+allow-all SecurityPolicy), and the ADR-0011 `x-oidc-*` set is re-stamped via `claimToHeaders` (caveat: object/array claims arrive base64url(JSON), not plain JSON). Rate-limit descriptors are NOT stamped on `/mcp/*` (no MCP rate limiting). (The `mcps` orchestrator + its children now float from OCI on a semver range — ADR-0055 — so the old `release.sh` tag-lockstep caveat no longer applies.)
- **⚠️ External hosted MCPs go through in-cluster Caddy "normalizing proxies" (ADR-0040, supersedes ADR-0039).** Direct external-TLS MCP backends were unfixable at the Envoy layer: AIEG stamps a `dummy.transport_socket` (empty SNI) on the cluster it generates for each external MCP backend that EG's `BackendTLSPolicy`/inline `Backend.spec.tls.sni` **never reach** (verified zero effect); Envoy's **BoringSSL rejects context7's ECDSA cert** (`BAD_ECC_CERT`); and refero's `tools/list` mislabels a JSON body as `text/event-stream` so the mcpproxy's SSE parser returns **empty tools** (AIEG bug [envoyproxy/ai-gateway#2218](https://github.com/envoyproxy/ai-gateway/issues/2218)). **Fix:** `charts/mcp` `mode: proxiedExternal` renders a per-MCP **Caddy** Deployment (`caddy:2-alpine` + a generated Caddyfile ConfigMap, NO custom image) that reverse-proxies to the upstream — Caddy does the TLS (**Go TLS handles ECDSA** → context7 works), injects the credential (`header_up Authorization "Bearer {env.MCP_TOKEN}"`, token from the ExternalSecret), and (refero only) rewrites the response `Content-Type → application/json` via `header_down`. The MCPRoute points at the in-cluster Caddy Service (plain HTTP), so external MCPs become reliable in-cluster backends like brave/terraform; the ADR-0039 `EnvoyPatchPolicy` (`charts/core-gateway` `mcpBackendTls`) was **REMOVED**. Active MCPs: brave, terraform (self-hosted), context7/firecrawl/refero (proxiedExternal). ⚠️ **`MCP_TOKEN` token-bind race (2026-06-11 context7 incident):** env vars from `secretKeyRef` bind at pod start and never refresh — with `optional: true` a Caddy pod that beats ESO captures an EMPTY token and injects `Bearer ` forever, so the upstream rejects every request as **"Invalid API key"** (looks like a bad key; the key is actually fine) until a manual `kubectl delete pod`. Diagnose: `MCP_TOKEN` length 0 in the running container. Guard shipped: `charts/mcp` renders the `MCP_TOKEN` env with `optional: false` (never optional), so a keyed proxy WAITS in `ContainerCreating` for ESO instead of starting tokenless — the env + the Caddyfile `Authorization` header render only when `externalSecret.enabled: true`, so a keyless MCP disables `externalSecret` and proxies anonymously (no declared-secret-but-optional mode, by design). context7's key IS populated/valid. ⚠️ **firecrawl empty-leading-SSE-event bug (2026-06-11, ADR-0041) — RESOLVED in AIEG v1.1.0.** The workaround that pinned firecrawl's request protocolVersion down via the `openresty` proxy engine was REMOVED once AIEG's mcpproxy learned to skip non-response SSE events ([#2219](https://github.com/envoyproxy/ai-gateway/issues/2219), fixed in v1.1.0); firecrawl now proxies through Caddy like context7/refero (ADR-0041 retired). ⚠️ Drop refero's `rewriteResponseContentType` once #2218 lands. (NB the earlier **AIEG v0.6→v0.7** bump + the EnvoyPatchPolicy were interim/red-herring steps — see the doc.) Full diagnosis + repro (config_dump / stats / openssl / Caddy validation): `docs/migrations/2026-06-10-mcp-external-server-proxy-debug.md`.
## Python tooling (`tools/dashboards/`)
Per ADR-0008:
- **`uv`** is the only resolver/runner. Not `pip`, not `poetry`.
- **`ruff`** for lint + format. Not `black`, not `isort`, not `flake8`.
- **Python 3.12+**, PEP 621 `pyproject.toml`, commit `uv.lock`.
- SDK: `grafana-foundation-sdk` (Grafana Labs' official multi-language SDK, NOT `grafanalib`).
## Commit style
Conventional commits with a substantive body that explains **why** (the diff already shows what). Link the ADR when implementing one: `(ADR-NNNN)`. AI-assisted commits get the trailer:
```
Co-Authored-By: Claude Opus <version> (1M context) <noreply@anthropic.com>
```
(use the running model version, e.g. `Opus 4.8`)
See recent `git log` for the established length and style — bodies of 20–60 lines are common for non-trivial changes.
**Conventional Commits are ENFORCED** (full spec: `docs/commit-conventions.md`). The `type` drives release-please/ADR-0082 version bumps (`feat`→minor, `fix`→patch, `!`/`BREAKING CHANGE:`→major; pre-1.0 charts bump minor), and the scope is the chart dir name (attribution is by file path). Because merges are squash (`COMMIT_OR_PR_TITLE`) — and merge/rebase are also enabled — **both the PR title and every commit must be valid**. Enforced by the local `commit-msg` hook (`git config core.hooksPath .githooks`) + the `Commit Lint` CI gate, both delegating to the single validator `tools/commit-lint.sh` (dependency-free POSIX sh — no npm). Change the rule in that one script; keep the doc's type table + `release-please-config.json` `changelog-sections` in sync.
## Local shell convention
Default shell on the maintainer's laptop is **zsh** (not bash). For shell commands that depend on shell features, prefer POSIX-portable; otherwise note zsh assumptions explicitly. CI workflows use GitHub Actions' implicit `bash` — that's intentional, don't change it (the user's "use zsh locally" preference is local-only).
## Tool quirks to remember
- **Edit tool** requires Read first when the target file already exists (Write also requires Read first for overwrites). Failure mode: `File has not been read yet`. Mitigation: Read with a small `limit:` if the file is large.
- **`gh` CLI** sometimes needs `zsh -i -c '...'` to pick up `GITHUB_TOKEN` from interactive profile. The harness's Bash tool runs zsh but non-interactively.
- **`helm template` warnings** about `~/.kube/config` group/world readability are noise — ignore.
- **`helm lint`** WARNING-level output ≠ failure. Look for `1 chart(s) failed`.
- **bjw-template-only charts fail `helm lint --strict`** with `[WARNING] templates/: directory not found` → `1 chart(s) failed`. This is expected: charts like `mcpo`/`lmcache` carry no own `templates/` dir — every manifest comes from the `bjw-template` subchart. **It's fine for us** — non-strict `helm lint` passes (`0 chart(s) failed`) and the real gate, `helm template … --dry-run`, renders cleanly. Don't add an empty `templates/` dir to silence it; judge these charts by render + non-strict lint. **The `helm-lint` CI now does this automatically:** it applies `--strict` only when a chart has its own `templates/` dir, and lints subchart-only charts non-strict (so the workflow is green without weakening the render gate). Before this, `--strict` ran on every chart and CI was red repo-wide on these charts + `ai-model`.
- **Render-only LEAF charts (e.g. `ai-model`) can't be lint/render-tested standalone** — their templates `fail`-guard on per-model values the orchestrator injects (ADR-0012), so `helm lint`/`helm template` with the default (empty) `values.yaml` aborts by design. **Convention:** mark such a chart with the Chart.yaml annotation `ai-helm.adorsys-gis.github.io/lint-mode: ci-values` and add a representative fixture at `charts/<chart>/ci/lint-values.yaml`. The `helm-lint` CI then skips the default lint+template for that chart and instead **lints AND renders each `ci/*-values.yaml` fixture** (so the leaf is still fully covered). Add a fixture when you add a new render-only leaf — don't exclude the chart from CI, and don't loosen its `fail` guards to make defaults render.
- **The `.opencode/README.md`** is stale (refers to an unrelated "Azamra monorepo"). Ignore it. The canonical agent/CI rules live in `.github/workflows/opencode.yml`.
## `.well-known/opencode` is NOT OIDC discovery
If you're touching `charts/librechat-opencode-wellknown/`, read ADR-0014 first. The endpoint is opencode-proprietary; the JSON has an `auth.command` + `config` shape. ⚠️ **Since ADR-0141 the `config` is native OpenCode v2** (`plugins` as `{package, options}`, `providers.*.settings`, `mcp.servers`, ordered `permissions` rules, `agents` with `system`/`request.body`) and every developer must run OpenCode v2. ⚠️ **Temporary exception (ADR-0142, sunset 2026-12-09):** a FROZEN v1 descriptor (`files/wellknown-v1.yaml`, `@vymalo/*@0.17.0`) is also served at `/opencode-v1/.well-known/opencode`. Never edit it or sync it with v2; every bridge line is marked `TEMPORARY … (ADR-0142)` so it can be removed in one PR. The `config.plugins` array triggers opencode's auto-install (bun-install at startup, cached under `~/.cache/opencode/node_modules/`). Two traps: the `@vymalo` opt-ins (`modelsInfo*`, `rateLimit`) must be FLAT on `settings` (0.18 ignores a `settings.meta` block, with no error), and permission rules are ordered (`findLast`), so a rule's position is behaviour. Check any edit by running the render through OpenCode's own `ConfigNormalize.normalize` (v2 `packages/core/src/config/normalize.ts`): it must report zero diagnostics and return the input unchanged (a fixed point). End users only run `opencode auth login <url>` — no manual install. Sources: `packages/opencode/src/cli/cmd/providers.ts` + `config/config.ts` in the opencode repo.
## Where the cluster's actual state lives that this repo doesn't track
- **Secrets + ESO** — the External Secrets Operator (controller + CRDs) is **installed externally**, not by this repo (it runs in the `external-secrets` namespace, Helm-managed). The `ClusterSecretStore` is `ssegning-aws` (cluster-scoped, external). This repo now **owns the ExternalSecret CRs in-chart** (ai-models-backends, librechat-app, observability-secrets, + `environments/<env>/deps/*` overlays), all referencing `ssegning-aws` — the old wholesale `secrets` Application from `ai-ops-secrets.git` was **removed (2026-06-04)**. App secrets pull from key `ai/camer/digital/prod/env`, platform secrets from `prod/meta/test-app`. Don't re-add an ESO operator or a ClusterSecretStore chart here (the old Vault `bootstrap-secrets` store config was removed — it was never the store actually used).
- **Keycloak realm clients/scopes/groups** — `charts/keycloak-baseline/values.yaml` defines them; keycloak-config-cli applies. Client secrets come from ESO at sync time.
- **Backups** — `charts/*-backup/` define the CronJobs; the S3 buckets + retention policies are out-of-band.
- **ArgoCD root Application** (`ai-apps-v2`) — applied **manually** on the ArgoCD cluster (no `ai-gitops` repo). The chart `charts/apps/` in this repo is what it points at.
- **Redis** — deployed by the `home-os` repo (`charts/home-apps/redis-ha`) as `redis-ha-redis` in the `redis-system` namespace. This repo only *consumes* it (LibreChat, LiteLLM proxy, Envoy rate-limit point at `redis-ha-redis.redis-system.svc.cluster.local:6379`). Don't re-add a redis chart here. **Auth:** redis-ha requires a password (remote ref `prod/meta/test-app` / `redis_password` in the `ssegning-aws` ClusterSecretStore). Consumers read it from a local Secret `redis-ha-redis-auth` (key `redis-password`) — LibreChat via `REDIS_PASSWORD`, LiteLLM via `REDIS_PASSWORD` (split `REDIS_HOST`/`REDIS_PORT`), Envoy ratelimit via the `REDIS_AUTH` env injected through the `rateLimitDeployment.patch`. That Secret must exist in **each consumer namespace** (`converse`, `envoy-gateway-system`) via its own ExternalSecret — the repo references it by name only (secrets are provisioned externally; same as every other secret here). **TLS:** redis-ha is **TLS-ONLY** (`redis.conf`: `port 0` / `tls-port 6379`), server cert from the internal `self-signed-ca` ClusterIssuer ("Home SSegning Root CA"), `tls-auth-clients no` (clients need no client cert, only to trust that CA). So every consumer must connect over TLS **and** trust the internal CA — a plaintext client gets `connection reset by peer`. The Envoy ratelimit does this via `REDIS_TLS=true` + `REDIS_TLS_CACERT` pointing at a cert-manager `Certificate` (issued from `self-signed-ca`, `ca.crt` only) mounted from the `eg` umbrella overlay. New consumers need the same. ⚠️ **If redis-ha-haproxy logs look noisy or LibreChat logs Redis connection errors, it's very likely NOT this repo's problem** — see `docs/architecture/06-networking-tls.md` (HAProxy-for-TLS-Redis gotchas table) and `docs/migrations/2026-hetzner-cutover.md` §2026-07-25 for two root causes already found and fixed durably: (1) haproxy's own liveness/readiness probes spamming `Connection closed during SSL handshake` (fixed in `home-os` PR #116/issue #117), and (2) the Hetzner cluster's CoreDNS running as a single replica causing intermittent cluster-wide DNS blips (now 3 replicas, one per CP node — durable across k3s restarts/upgrades since v1.25.5+k3s1 stopped hardcoding replica count on reconcile; the from-scratch-bootstrap gap is closed via `hetzner-k8s` PR #30).
- **Traefik** — the ingress controller + the `traefik` IngressClass are deployed externally (runs in the `traefik` namespace). This repo only sets `ingressClassName: traefik` / `className: traefik` on its Ingresses. Don't re-add a traefik chart here.
- **CloudNativePG** — the `cnpg` Postgres operator **and** the Barman Cloud backup plugin (`cnpg-barman-cloud`) are deployed externally (`cnpg-system`; CRDs `postgresql.cnpg.io` + `barmancloud.cnpg.io`). This repo only defines CNPG `Cluster` CRs (e.g. `charts/lightbridge-db`) that the external operator reconciles. Don't re-add a cnpg or plugin-barman-cloud chart here.
- **OpenTelemetry Operator** — installed externally (`opentelemetry-system`; CRDs `opentelemetry.io`). This repo only defines `OpenTelemetryCollector` CRs (`charts/core-gateway`: the `-traces` collector → Alloy → Tempo) that the external operator reconciles. Don't re-add an otel-operator chart here. There are **two** collectors: the `-traces` one above and a **`<fullname>-usage`** collector (`charts/core-gateway/templates/otel.yaml`). ⚠️ `-usage` was removed once (948814de) and has since been **restored** as a fan-out point: Envoy's access-log OTLP sink (`templates/envoy-proxy.yaml`) now targets `<fullname>-usage-collector`, NOT Alloy directly, and that collector re-exports the same stream to **both** `otlp/alloy` (→ Loki, ADR-0005 per-user observability — unchanged) **and** `otlphttp/lightbridge_usage` → lightbridge-authz-usage's OTLP/HTTP ingest (`https://lightbridge-usage.converse.svc:3000/v1/otel/logs`, `X-Source: eaig` header required, TLS verified via the `<fullname>-usage-ca` Certificate) → `usage_events`, i.e. billing. Don't "simplify" the sink back to Alloy — that silently drops the billing leg.
- **cert-manager** — controller (in `kube-system`, `--cluster-resource-namespace=kube-system`), CRDs, **and** the shared cluster-scoped ClusterIssuers (`cert-home-cert-http`, `self-signed-ca`, `cert-cloudflare`, `cert-route53`) are deployed by `home-os` (`charts/cert`, apps `cert` + `cert-remote`). home-os has **`config.enableGatewayAPI: true`** so cert-manager solves ACME HTTP-01 via `gatewayHTTPRoute` + honours the `cert-manager.io/cluster-issuer` gateway-shim. This repo only references issuers via annotations. ⚠️ The `cert-home-cert-envoy` issuer is **RETIRED** — `api.ai.camer.digital` TLS now comes from an **in-chart ns ACME `Issuer` + HTTP-01 `gatewayHTTPRoute`** solver (`charts/core-gateway`, `gateway.acmeHttp01.enabled`), no DNS token. `cert-cloudflare` (DNS-01) is **live** — its `cloudflare-secret` in kube-system has been ESO-provisioned since 2026-07-22 — but it can only issue for names in zones Cloudflare actually serves. `camer.digital` is on Cloudflare; `ai.camer.digital`, `coder-ai.camer.digital` (AWS Route53) and `observe.camer.digital` (Google) are **NS-delegated away**. Point a DNS-01 `Certificate` at a delegated name via `cert-cloudflare` and it fails *silently-ish*: cert-manager writes the `_acme-challenge` TXT into the Cloudflare zone, the `Challenge` reports `Presented: true`, and then stalls forever on `not yet propagated` because Route53 answers the lookup instead — this is exactly what happened to `traefik/ai-certificate`, stuck 21-32 days (#993). **Always `dig +short NS <parent-of-your-wildcard>` first** — non-Cloudflare NS ⇒ `cert-cloudflare` cannot issue for it.
✅ **RESOLVED (2026-08-23, home-os#138/#139): a dedicated `cert-route53` ClusterIssuer**, not a second solver bolted onto `cert-cloudflare` — an incident showed cert-manager validates every solver's `secretRef` when it reconciles the *issuer*, so a momentarily-missing Route53 credential would otherwise have taken down all ~20 unrelated certs `cert-cloudflare` already serves. `cert-route53` is enabled + ESO-credentialed (`route53-secret`, from the same `prod/meta/test-app` entry as the Cloudflare token) only for `cert-remote`/`home-remote` — the one cluster serving `ai.camer.digital` names — covers `dnsZones: [ai.camer.digital, ai.kivoyo.com]`, and is verified `Ready=True` live. ⚠️ Residual sharp edge: cert-manager does **not** re-reconcile an Issuer when a previously-missing solver Secret later appears — if `route53-secret` isn't present on first apply, `cert-route53` sticks in `InvalidSolver` until the controller restarts (the `cert-remote` Application does go Degraded, so it's visible, just not self-healing).
**`traefik/ai-certificate` itself was deleted, not reissued** (home-os#140/#141): none of its 5 SANs were ever actually requested by anything hitting `traefik/traefik-gateway` — its only real consumer, knative-serving, serves hostnames under `mlops.camer.digital` (no `.ai`, not delegated) — so it was dead config, replaced by a correctly-scoped `serverless-gateway-certificate` (`*.mlops.camer.digital`, `cert-cloudflare`, no Route53 needed). A wildcard under a delegated zone needs `cert-route53`, not `cert-cloudflare`. ⚠️ **Still open:** the nine orphaned `_acme-challenge` TXT records (ai, coder-ai, coder-ai.ai, mlops, serverless.mlops) from the repeated failed attempts were still sitting in the Cloudflare zone as of 2026-08-27 — cleanup tracked in #1050, not yet confirmed done. Don't re-add a cert chart here. **Vanity-domain redirects (ADR-0053):** we own `kivoyo.com` (DNS already pointing at our LBs) and redirect it to the canonical `camer.digital` — `api.ai.kivoyo.com → api.ai.camer.digital` via `charts/core-gateway` `gateway.redirectHosts[]` (each entry → its own HTTPS listener + ACME cert reusing the in-chart issuer + a `RequestRedirect` HTTPRoute, `307`), and `ai.kivoyo.com → ai.camer.digital` via a Traefik `redirectRegex` Middleware + Ingress in `charts/librechat-app` `rawResources` (`302`). Temporary + path-preserving; add the next vanity host the same way (one `redirectHosts` entry).
## Documentation conventions — "update the docs" means ALL of them
> **Maintainer's rule:** when the maintainer says **"update documentation"** or
> **"upgrade the doc"**, they mean **every** affected surface, not just the one
> nearest doc. For any substantive change, sweep this whole set and update what's
> relevant:
> 1. **The topical doc** under `docs/`, filed by **intent** into one of the four
> dirs — **`docs/playbooks/`** (runbooks, setup, how-to), **`docs/integrations/`**
> (consuming a specific product/surface: LibreChat, opencode,
> Keycloak-as-IdP), **`docs/patterns/`** (reusable patterns, concept explainers,
> reference/research), **`docs/migrations/`** (cutovers, upgrades, point-in-time
> audits) — or an existing subsystem subdir (`adr/`, `architecture/`, `models/`,
> `models-chart-docs/`, `cnpg-native-backup/`, `secret-management/`, …).
> Front-matter docs (`architecture.md`, `arc42.md`, `continuous-delivery.md`,
> `commit-conventions.md`) stay at `docs/` root. Each topical dir has a local
> `README.md` index — add your file there **and** to `docs/README.md`. Long-lived
> subsystem guides drop the date prefix (`self-hosted-model-serving.md`, not
> `2026-…`); dated files are point-in-time change-logs/audits (→ `migrations/`).
> 2. **ADRs** — write a new ADR for any decision with non-obvious consequences or
> a locked-in contract; amend a prior decision with a *new* ADR (the old body
> is immutable). Update `docs/adr/README.md`.
> 3. **`docs/arc42.md`** — the formal architecture description. New chart → §5
> building-block table; new ADR → §9 decisions; new risk → §11.
> 4. **`docs/architecture.md`** — the system map (topology, sync waves, "where
> every choice is documented").
> 5. **`docs/README.md`** — the docs index.
> 6. **`CLAUDE.md`** (this file) — if a convention/contract/gotcha changed.
> 7. **User memory** — if it's a durable preference or hard-won lesson.
**Self-hosted models — the GPU fleet (ADR-0094/0095).** ⚠️ **Two generations exist;
do not mix them up.**
*Current.* ⚠️ **ONE** Hetzner Robot GPU node (`hetzner-k8s-gpu-1`, RTX 4000 SFF Ada,
**20475 MiB**, driver 550/**CUDA 12.4**, Ada ⇒ **FP8 IS supported here**) is on
**`home-remote`** — the same cluster as the gateway. `hetzner-k8s-gpu-2` was
**removed 2026-09-15** (ADR-0138), so **one card ⇒ ONE served model**: the fleet
runs `qwen3-5-2b` and `z-image-turbo` is disabled. Enabling a second model now
leaves it `Pending` on `Insufficient nvidia.com/gpu` — say which one you are
disabling first. Much of the prose below (and ADR-0092/0094/0114) was written
for the two-card fleet; where it says "two cards", read one. So a model is an ORDINARY
workload: **NOT `homeCluster`**, no Ingress, no cert, no DNS, no static API key, no
Caddy sidecar. `charts/inference` (orchestrator) + `charts/inference-server`
(generic leaf) serve them in ns `inference`; the gateway `Backend` points at
`<model>.inference.svc.cluster.local:8080` and a `CiliumNetworkPolicy` is the
control. **Adding/replacing a model = ONE ~15-line entry in
`ai-helm-values environments/prod/values/inference.yaml`** (ADR-0129 — NOT the
chart any more; its defaults are an empty skeleton that refuses to render) (+ its
gateway backend + model
entry to make it user-reachable). Do NOT create a new chart per model.
- Engine profiles live in the ORCHESTRATOR's `_helpers.tpl` — a Helm parent can't
compute SUBCHART values at render time, which is why the old charts hardcoded
their seed repo/glob. **THREE profiles: `llamacpp`, `vllm` (opt-in LMCache) and
`localai` (IMAGE GENERATION, ADR-0100/0102)**, each rendering **ONE container**; the
optional `apiKey` is enforced natively by every engine, and *how* it is taken is
profile DATA (`engines.<name>.apiKey.mode: file|env`), not a template branch —
same for `metrics` and `devShm`. **Caddy was only ever needed for
`kserve/huggingfaceserver`**, which ignores `VLLM_API_KEY` (ADR-0022) and is
deliberately not an engine profile — don't reintroduce a proxy sidecar.
- **⚠️ The image engine is `localai` — OFF THE SHELF (ADR-0102).** We briefly
maintained a first-party Rust/Candle server for this and should not have: the
ADR-0100 survey missed OpenAI-compatible multi-backend servers entirely.
`quay.io/go-skynet/local-ai` — **pin the `-gpu-nvidia-cuda-12` tag**, because
`master`/`latest` are CUDA 13 against our driver 550 (same trap as
`server-cuda13` / `-cu129`). It is shaped differently from the text engines and
the profile says so as DATA: `seedJob: false` (LocalAI downloads its own model
AND its inference backend, so no seed Job, no HF token, no pinned SHA) and
`writableModelStore: true` (so `/models` is mounted RW and must persist
`backends/`). Configured by ENV (`MODELS`, `MODELS_PATH`, `BACKENDS_PATH`,
`API_KEY`), not flags. ⚠️ With no seed Job there is one controller, so bjw names
the Deployment `<model>` and **`deploy/<model>-main` does not exist** for image
models. ⚠️ A gallery reference is NOT a pin — the bytes can move under us.
- **⚠️ Two traps an image model taught us the hard way.** (1) A model's advertised
precision is marketing — Z-Image-Turbo is documented everywhere as "FP8, ~8 GB";
the diffusers repo is **FP32, ~33 GB**. (2) A server that loads its model
**lazily** deadlocks against a readiness-gated `/health`: not-Ready ⇒ no
endpoint ⇒ no request ⇒ never loads. Every fleet engine must pre-load at
start-up. Both were discovered the expensive way; both are why ADR-0101 says a
model is federated only once MEASURED on the hardware it runs on — "it worked
on the old hardware" is not evidence.
- GPU placement is a `nvidia.com/gpu: 1` REQUEST (+ toleration/nodeSelector/
runtimeClass). ⚠️ **1 card ⇒ 1 model** since gpu-2 was removed (ADR-0138); a
2nd enabled model queues `Pending`. ⚠️ The **seed Job**
needs the same nodeSelector/toleration despite needing no GPU. ⚠️ **Longhorn was removed
from hetzner-prod on 2026-10-30 (ADR-0140)**, so there is currently NO ReadWriteMany
StorageClass for the weights volume: `charts/inference-server` still defaults
`pvc.storageClassName` to `longhorn`, which no longer exists, and `hcloud-volumes` (the
cluster default) is ReadWriteOnce. Choose and set an RWX class in the fleet catalog before
re-enabling any model. ⚠️ The CNP MUST allow `fromEntities: [host, remote-node, health]` or
KUBELET PROBES fail and the pod never goes Ready (looks like a crash-loop).
- `check-model-catalogs.sh` fails if a cluster-local gateway backend has no server
behind it. Serving *without* federating is allowed on purpose — that's how a
model is load-gated before users can reach it. ⚠️ It lives in **`ai-helm-values`**
now (ADR-0126), not here: the gateway half of the comparison moved to that repo,
and only that repo can see both (it pulls these charts anonymously from public
OCI; this repo cannot read a private one). Don't re-add a copy here that renders
`charts/ai-models` defaults — they're an empty skeleton, so it would pass
vacuously and guard nothing.
*Legacy.* The `charts/model-serving-*` charts (seven left) target the OTHER
cluster (`admin@homeos`) over a public edge with `homeCluster: true`. ⚠️ **Since
2026-07-27 (ADR-0100) ALL are `enabled: false`** — `zimage-turbo` was the last
live one and moved to the fleet, so the generation is now a rollback surface only
and ADR-0094's reason for retaining it is spent. **Retained, not deleted**
(deleting them is a decommissioning exercise on that cluster) — don't copy their
shape, and don't "fix" them to match the new pattern. ⚠️ The one EXCEPTION is
`model-serving-zimage-turbo`, **deleted** by ADR-0106: its image never existed, so
it was not a rollback surface, and leaving it disabled is what let a stale branch
turn it back on. Papers: [`docs/models/`](docs/models/).
*Where the knowledge lives.* The GitOps *how* is
[`docs/patterns/self-hosted-model-serving.md`](docs/patterns/self-hosted-model-serving.md).
The **inference** knowledge — VRAM budgeting, quantization, engine selection,
benchmarks, runbooks, the add/replace/roll-back/measure recipes — lives in the
team's **`inference-ops`** repo (checked out alongside this one; Diátaxis + ADRs +
immutable benchmark reports). Put inference knowledge THERE, not here;
ai-helm ADRs own the GitOps shape, inference-ops ADRs own the inference decisions.
⚠️ Pricing is **nominal, not cost-recovery** (ADR-0128, amends ADR-0028): the fleet
costs the same €/hour idle or saturated, so cost-recovery rates only pushed traffic
onto SaaS invoices we then pay again. Self-hosted models carry a token charge
(`0.001/0.0002/0.005` per 1M; `0.00001`/image) ~2 orders of magnitude under the
cheapest SaaS entry — non-zero on purpose, so cost series stay alive and are not
confused with missing data. ⚠️ Coefficients render at `%.4f`, so anything below
~0.00005 silently becomes free. The €/hour-TCO derivation (ADR-0028/0096/0104) is
still the right answer to "what does this cost US to run" — it is just no longer
what users are charged; that question lives in inference-ops.
## When you finish substantive work
- Update `docs/README.md` index if you added a doc.
- Update `docs/adr/README.md` index if you added/superseded an ADR.
- If you renamed/restructured charts, check `charts/apps/values.yaml` doesn't reference a deleted path.
- `helm template <touched chart>` is the fastest pre-commit smoke test.
- For PR work: tasks are tracked in the harness; mark them as you go.
## Backlog: at most 5 open `Story` issues per repo
A standing WIP limit (maintainer's call, 2026-09-01), org-wide via
[ai-governance](https://adorsys-gis.github.io/ai-governance/) — see
`docs/02-user-story-template.md` §12a there. **Only `Story` is capped**; `Task`, `Bug` and
`Feature` are not, and the dev-ticket form (`type: Task`) is the default shape for
implementation work, so most filing is unaffected.
At the limit, either don't file the Story or say which open one it displaces. **Parking a Story
means folding it into its Epic as a checklist item, never deleting it** — a cap that loses scope
is worse than no cap. A Story under an active Epic still counts; epics do not exempt their children.
> ⚠️ `gh issue create` and raw API calls **bypass the issue forms**, so they stamp no issue type
> — a `Story` filed that way arrives untyped and is invisible to the count. Stamp it afterwards
> with `updateIssueIssueType` (org-level type ids; `Task` is `IT_kwDOCFHHUc4BMIZb`). This is the
> same mechanism by which the type taxonomy decayed before the forms carried `type:`, and it bit
> twice on 2026-09-01.
<!-- ai-governance:stanza -->
<!-- BEGIN: AI Governance stanza (managed by ADORSYS-GIS/ai-governance) -->
## AI Governance
AI may accelerate the work, but humans own intent, verification, and consequences.
AI output is not truth: review AI-generated code as untrusted, and never submit work you cannot explain.
When opening issues or pull requests in this repo:
- Use the provided **issue forms** (Epic, User Story, Dev Ticket) and the **pull request template** — do not open blank issues/PRs.
- Fill in the **AI Usage Declaration** honestly (what AI was used for, what you verified).
- Include a **source-of-truth link** (a URL or `#123` reference). No source of truth means the work is not ready.
- Provide **verification evidence** (commands, logs, links, or checked verification boxes). No evidence means it is not done.
Source of truth and full doctrine: https://adorsys-gis.github.io/ai-governance/
This stanza is intentionally thin — read the site; do not duplicate the doctrine here.
<!-- END: AI Governance stanza -->