justfile
Rama main · réplica de lectura del repositorio original.
set dotenv-load := true
# Bothy - five containers, ONE compose project on purpose (2026-08-16): the web
# app on :80, bothy-files (editor + config forms), bothy-ops (container + cluster
# actions) and the two socket proxies. Eight containers in four projects until
# 2026-09; see apps/bothy/compose.yml for what merged and what did not.
#
# It used to be three: `portal`, `portal-next` and `portal-files`. Bothy groups
# systems by com.docker.compose.project, so it rendered ITSELF as three separate
# cards in its own Overview - the "why are there 3 portals?" bug. One project,
# one card.
#
# The three files stay separate and are pulled together by `include:` in
# apps/bothy/compose.yml - read that file for why it is `include:` and NOT
# repeated `-f` (the short version: with `-f`, every relative path resolves
# against the FIRST file's directory, which built the editor tier from the web
# tier's Dockerfile and moved its audit log).
#
# apps/bothy-files/compose.yml was in NO lifecycle recipe before this: it ran,
# but `just down` never stopped it and `just up` never started it. It is reached
# through exactly one path now, so it cannot fall out again.
BOTHY := "-f apps/bothy/compose.yml"
# List recipes
default:
@just --list
# Create the shared docker networks (idempotent)
network:
-docker network create devnet 2>/dev/null || true
# opsnet / controlsocknet: the action tier, and the reason it is TWO networks
# rather than one. opsnet is traefik + bothy-ops and nothing else - bothy-ops
# can stop containers and scale cluster workloads and authenticates nobody, so
# the network IS the boundary. controlsocknet is bothy-ops + the two socket
# proxies: traefik must not be ABLE to reach a proxy that can mutate
# containers, so the edge meets the service on one network and the service
# meets its proxies on another. bothy-ops' way out to the cluster is
# minikube's own network, added by apps/bothy-ops/compose.cluster.yml.
#
# controlnet, kubenet and confignet were retired 2026-09 with bothy-control,
# bothy-kube and bothy-config. They are no longer created; remove the old ones
# by hand once nothing is attached (`docker network rm ...`).
-docker network create opsnet 2>/dev/null || true
-docker network create controlsocknet 2>/dev/null || true
# socketnet: traefik + bothy-socket-read. docker-socket-proxy has NO auth, so
# network reachability IS authorisation - on devnet, any of ~20 containers
# could read the docker socket through it. That proxy is POST=0, so nothing on
# socketnet can mutate a container through it.
-docker network create socketnet 2>/dev/null || true
# filesnet: traefik + bothy-files, and nothing else. bothy-files holds
# read-write bind mounts on two git repos and has no auth of its own, so the
# network IS the boundary - authorisation happens at the edge in front of it.
# Putting it on devnet would hand ~20 containers write access to the docs.
-docker network create filesnet 2>/dev/null || true
# Bring up EVERYTHING
# `bootstrap` first, and that is not decoration: on a fresh clone Prometheus is
# started with --web.config.file pointing at a gitignored file nothing creates,
# and three directories the services write to do not exist. README.md's own rule
# applies - if a rule can be forgotten, it will be, so put it somewhere it
# cannot be skipped.
#
# Portainer and Dozzle were retired on 2026-08-17 and DELETED on 2026-08-18,
# along with `up-mgmt` and mgmt/. Bothy grew its own answer to both - the Control
# tier manages containers, and Loki plus Grafana hold the logs - so keeping a
# second, unauthenticated copy of each was not a rollback path, it was two more
# things to secure. Dozzle could not even start without a gitignored users file
# that nothing generated, so `just up` used to fail partway through on every
# fresh clone at a service the box does not use.
#
# `git revert` brings them back if that turns out to be wrong; a compose file
# kept on disk "just in case" is how this repo ended up with four of them.
# THE RE-INVOCATION IS THE POINT, and it fixes a bug that made the documented
# one-command install impossible.
#
# `set dotenv-load` reads .env when just PARSES this file - before any recipe
# runs, and therefore before `bootstrap` generates the five credentials a fresh
# clone does not have. So on a first `just up`, compose was handed the values
# that were on disk a moment BEFORE bootstrap wrote the real ones, and died with
#
# required variable OAUTH2_COOKIE_SECRET is missing a value
#
# on a .env that visibly contained one. Found by `just ci-install`, on its first
# complete run, which is the entire reason that recipe exists.
#
# A nested just is a new process, so it re-reads .env and sees what bootstrap
# wrote. That alone is NOT enough, and the gap is the subtle half: just's
# dotenv-load does not override a variable already in the environment (verified),
# and an EMPTY value in .env is never exported at all.
#
# So the blank OAUTH2_COOKIE_SECRET was simply absent, and the child picked up
# the generated one - which made this look fixed. Every key shipping a non-empty
# PLACEHOLDER was exported by the parent and kept by the child:
#
# POSTGRES_PASSWORD Postgres initialises with `changeme-generate-one` while
# .env holds a real password, and the next `just up`
# cannot connect to the database it just created.
# BOX_IP oauth2-proxy dials the placeholder address for OIDC discovery,
# times out, crash-loops, and every gated route answers
# 502 - on a stack whose .env plainly says otherwise.
#
# Hence `env -u` over everything bootstrap may write, not just the secrets. The
# list is scripts/lib/bootstrap-keys.sh, shared with bootstrap so the two cannot
# drift; both failures above were found by `just ci-install`, one after the
# other, because fixing the first revealed the second.
# Bring the whole box up, in dependency order. Runs bootstrap first.
up: bootstrap
#!/usr/bin/env bash
set -euo pipefail
unset_flags=()
for k in $(bash -c '. scripts/lib/bootstrap-keys.sh && printf "%s " $BOOTSTRAP_KEYS'); do
unset_flags+=(-u "$k")
done
env "${unset_flags[@]}" just _up-all
# Not called directly - `up` runs it in a re-read environment. Split out rather
# than inlined so the dependency ORDER, which is load-bearing, stays declarative.
#
# DATA BEFORE AUTH, and that ordering is a bug fix rather than a preference.
# `keycloak-db-init` creates Keycloak's role and database inside the SHARED
# postgres, which lives in a different compose project - so `depends_on` cannot
# reach it and the script waits on `pg_isready` for sixty seconds instead. A wait
# cannot help when the thing waited for has not been started yet: with auth
# ahead of data, a genuinely fresh box spent a minute failing to resolve the name
# `postgres`, exited 2, and took `just up` down with it.
#
# It never showed up here because postgres was ALREADY RUNNING from the previous
# `just up`, every time. That is the shape of every bug this install job exists
# to find: correct on the machine it was written on, and only there. Found by
# `just ci-install` against an empty daemon.
#
# monitoring moves after data for the same reason, one step milder:
# postgres-exporter dials postgres, and starting it first just means a few
# restarts before it settles.
_up-all: network up-edge up-data up-auth up-monitoring up-apps
@echo "All stacks up. Run 'just urls' for access."
# Edge: Traefik - the single front door on :80. It no longer routes anything by
# Host header (the name layer was deleted 2026-08-12); what it still owns is the
# portal's host-less /-/api/* data plane and the catch-all that serves the portal
# on the bare IP. Must come up before anything that expects to be routed.
# Edge: Traefik on :80. Must be up before anything that expects routing.
up-edge: network
docker compose -f edge/compose.yml up -d --wait --wait-timeout 180
# Identity: Keycloak (the IdP) + oauth2-proxy (what Traefik's forwardAuth asks).
#
# One `up -d` brings the whole chain up in order, because the compose file
# expresses the ordering itself: the database bootstrap must EXIT SUCCESSFULLY
# before Keycloak starts, and Keycloak must be HEALTHY before oauth2-proxy and
# the post-import fixes run. So this blocks for as long as that takes - a minute
# or two on first boot, while Keycloak builds and creates its schema.
#
# Nothing here enforces anything yet: it defines the `sso@file` middleware,
# it does not attach it to any router. See edge/dynamic/auth.yml.
# Identity: Keycloak + oauth2-proxy. Blocks until Keycloak is healthy.
up-auth: network
#!/usr/bin/env bash
set -euo pipefail
# --wait, like every up-* recipe - with one service held out of it.
#
# keycloak-init is a one-shot that NOTHING depends on. Compose's --wait
# accepts an exited container only when some service depends on it with
# `service_completed_successfully` (keycloak-db-init has keycloak); any other
# exit - even exit 0 - fails the wait. So a --wait over the whole file raced:
# it passed if keycloak-init was still running when checked and failed with
# "container keycloak-init exited (0)" if it had already finished. Measured
# on a throwaway project with compose v5.3.1, 2026-09-18.
#
# 600s, not 180: the timeout also covers the dependency waits inside `up`,
# and keycloak is only declared unhealthy after start_period 180s + 20 x 15s
# retries = 480s, plus keycloak-db-init's own 60s wait for postgres. A
# shorter timeout would fail a slow first boot that compose itself would
# still have let through.
svcs=$(docker compose -f auth/compose.yml config --services | grep -vx keycloak-init)
# shellcheck disable=SC2086
docker compose -f auth/compose.yml up -d --wait --wait-timeout 600 $svcs
# ...then keycloak-init on its own, and its EXIT CODE is checked. Before
# this, a failed post-import fix (no seed user, master still sslRequired)
# left `just up-auth` returning 0. --no-deps: keycloak is healthy already.
docker compose -f auth/compose.yml up -d --no-deps keycloak-init
rc=$(docker wait keycloak-init)
if [ "$rc" != 0 ]; then
echo "keycloak-init exited $rc - see: docker logs keycloak-init" >&2
exit 1
fi
# One resolver, not a second copy of the dance - see scripts/lib/box-addr.sh
# for the order. This one matters more than most: the address printed here
# has to be the address Keycloak's issuer was built from, or the admin
# console link works and the login it leads to does not.
IP=$(bash scripts/lib/box-addr.sh)
echo ""
echo " Keycloak admin http://$IP:8090/admin (admin / DEV_LOGIN_PASSWORD in .env - the shared dev login)"
echo " OIDC discovery http://$IP:8090/realms/devbox/.well-known/openid-configuration"
echo ""
echo " The 'issuer' in that document must equal oauth2-proxy's"
echo " --oidc-issuer-url exactly. If they ever differ, the symptom is a"
echo " redirect loop, not an error. Check it after changing BOX_IP:"
echo " curl -s http://$IP:8090/realms/devbox/.well-known/openid-configuration | jq -r .issuer"
# compose.cluster.yml is added ONLY when the docker network `thales-scc` exists
# (the minikube cluster). It joins victoriametrics to that network so it can
# scrape the node; as an `external` network in compose.yml itself it would make
# this recipe fail on every box without the cluster. See that file's header.
# The legacy prometheus/promtail services sit behind compose profiles and are
# NOT started here.
# Observability: grafana, victoriametrics, loki, alloy, cadvisor, node-exporter (+ the cluster, if present)
up-monitoring: network
#!/usr/bin/env bash
set -euo pipefail
files=(-f monitoring/compose.yml)
# node-exporter's textfile collector reads this (read-only, never
# auto-created; 755 because node-exporter runs as nobody). Update discovery
# writes bothy_updates.prom into it.
mkdir -p -m 755 "${STATE_ROOT:-$HOME/.local/state}/bothy/textfile"
if docker network inspect thales-scc >/dev/null 2>&1; then
files+=(-f monitoring/compose.cluster.yml)
# The mount source must be a directory before compose creates it as root.
mkdir -p monitoring/kube-auth
[ -f monitoring/kube-auth/token ] || echo "note: no monitoring/kube-auth/token - run 'just k8s-monitoring' (kubelet jobs stay DOWN until then)"
fi
# --wait: healthy where a healthcheck exists (cadvisor's comes from its
# image), merely RUNNING for loki, which has none - so this proves loki
# started, not that it is ready. See `just doctor` for that.
docker compose "${files[@]}" up -d --wait --wait-timeout 180
# Data services: postgres (+ its exporter).
#
# redis and kafka were RETIRED 2026-08-12 and DELETED on 2026-08-18. Both were
# measured completely idle - redis `DBSIZE` = 0, kafka `--list` = zero topics -
# while together holding ~1,140 MB of the box's 4,678 MB of container memory.
# Their volumes and images went at retirement; the compose files went with the
# rest of the dead weight.
#
# They outlived wiki and mgmt by an afternoon on the argument that they were
# idle rather than REPLACED, so if either were wanted again it would be wanted
# as itself. That argument is what `git revert` is for. A compose file kept on
# disk for a service nobody starts is not a rollback plan, it is a file every
# reader has to ask about - and this repo had four of them.
#
# THEIR DATA IS GONE. An earlier version of this comment said the volumes were
# "PRESERVED, not deleted" - that was true for about twenty minutes. The
# `kafka_data` and `redis_data` volumes were removed afterwards, along with the
# images (apache/kafka, kafbat/kafka-ui, danielqsj/kafka-exporter, redis:8-alpine,
# oliver006/redis_exporter). Verified 2026-08-12: neither name appears in
# `docker volume ls`. So the commands below do NOT restore anything - they pull
# fresh images and start an EMPTY broker / an EMPTY keyspace:
# docker compose -f data/redis/compose.yml up -d # new, empty
# docker compose -f data/kafka/compose.yml up -d # new, empty
# Nothing that was in them comes back. (Nothing was in them, which is why they
# went; the point is that the instruction must not promise otherwise.)
#
# postgres stays: it is in active use - Keycloak's database lives there.
# Data: Postgres, bound to loopback only.
up-data: network
@bash scripts/pg-volume-guard.sh data/postgres/compose.yml postgres
docker compose -f data/postgres/compose.yml up -d --wait --wait-timeout 180
# Apps: Bothy - one project, five containers (apps/bothy/compose.yml).
#
# compose.cluster.yml is added ONLY when the docker network `thales-scc` exists
# (the minikube cluster). It joins bothy-ops to that network and mounts its
# token; as an `external` network in compose.yml itself it would fail this
# recipe - and with it the file editor and container actions - on every box
# without the cluster. Without it the kube verbs answer 503 "cluster
# unavailable" and nothing else changes. This replaces `just up-kube` (2026-09).
#
# Both undo nets must exist before bothy-files starts: policy.toml declares them
# and the service refuses to boot without them - a safety net nobody notices is
# missing is worse than none. Docker would create them root-owned otherwise.
#
# `services` (optional) limits the build and the `up` to those services - the
# own-code updater brings bothy-files, bothy-ops and bothy-web up one at a time,
# web last. Two environment switches, both the updater's (docs/plans/updates.md
# step 6) and harmless by hand:
# BOTHY_IMAGE_TAG the tag of bothy-{web,files,ops}; default HEAD's sha
# BOTHY_UP_NO_BUILD=1 do not build: run the images already tagged so (the
# updater built them from a temporary worktree first; its
# rollback re-runs the previous commit's)
#
# Apps: Bothy's five containers, plus the cluster overlay when thales-scc exists.
up-apps *services: network
#!/usr/bin/env bash
set -euo pipefail
state="${STATE_ROOT:-$HOME/.local/state}"
mkdir -p "$state/bothy/trash" "$state/bothy/config-trash"
mkdir -p -m 700 "$state/bothy/inventory"
# The Settings credentials/backups pages read this; refresh it now so they
# are not empty until the timer's first run. Metadata only - see its header.
python3 apps/bothy-ops/inventory.py || echo "note: the admin inventory could not be written - Settings > Credentials will say so"
# Settings > Updates reads available.json from here (read-only mount, never
# auto-created). NOT refreshed here: discovery asks registries on the
# internet, which `up` should not wait on - the timer, or `just updates-discover`.
mkdir -p -m 700 "$state/bothy/updates"
# ...and its spool, bothy-ops' only read-WRITE mount besides its audit dir:
# Settings > Updates writes one request file here and the host's
# bothy-updater.path picks it up (build step 4). 700, never auto-created.
mkdir -p -m 700 "$state/bothy/updates/spool"
files=({{BOTHY}})
# The Users & roles page: only once `just admin-client` has written a secret.
if [ -f apps/bothy-ops/secrets/keycloak-admin-client-secret ]; then
files+=(-f apps/bothy-ops/compose.admin.yml)
fi
if docker network inspect thales-scc >/dev/null 2>&1; then
if [ -d apps/bothy-ops/secrets ]; then
files+=(-f apps/bothy-ops/compose.cluster.yml)
else
echo "note: thales-scc exists but apps/bothy-ops/secrets does not - run 'just kube-token', then 'just up-apps' again (kube verbs answer 503 until then)"
fi
fi
# BUILD FIRST, every time. bothy-web, bothy-files and bothy-ops are `build:`
# images, and a plain `up` never rebuilds an image that already exists - so
# until 2026-09-18 `bothy upgrade` (pull, then `just up`) pulled new app code
# and kept serving the old image, green and healthy, forever. `upgrade.yml`
# compared data, not code, so nothing noticed.
#
# A separate `build` rather than `up --build`: it fails BEFORE anything
# running is touched (a broken build leaves the old containers serving), and
# it reads as its own step in the log. It costs nothing when nothing changed:
# BuildKit's layer cache answers every step and returns the same image ID,
# so `up` recreates nothing - the second-`just up` assertion in upgrade.yml
# holds that.
#
# BOTHY_REVISION is baked in as the OCI revision label (and bothy-web's
# /version.json). It is the last layer of each Dockerfile, so a new commit
# that changes no app source re-runs one metadata step and recreates the
# three containers onto an image that says which commit it is. That churn is
# the price of "docker inspect answers which code is running" - and it is
# what upgrade.yml compares against HEAD. HEAD, not "HEAD plus local edits":
# a dirty tree builds what is on disk and is labelled with the commit under it.
#
# BUILDX_NO_DEFAULT_ATTESTATIONS is LOAD-BEARING for "costs nothing". buildx
# attaches a provenance attestation by default and it carries the build's
# timestamp, so on the containerd image store (Docker 29's default, and this
# box's) every build of an UNCHANGED tree got a new image ID - and `up`
# recreated all three containers on every `just up`. Measured 2026-09-18:
# three cached builds, three IDs; with this set, three builds, one ID.
# compose's own `build: provenance: false` did NOT help on compose v5.3.1
# (same test), which is why it is an env var here and not a compose key.
# Nothing is pushed anywhere, so nothing reads the attestation.
BOTHY_REVISION="$(git rev-parse HEAD 2>/dev/null || echo unknown)"
# The image TAG is the commit too (compose.yml: bothy-web:${BOTHY_IMAGE_TAG}),
# so the previous commit's images survive this build under their own name.
BOTHY_IMAGE_TAG="${BOTHY_IMAGE_TAG:-$BOTHY_REVISION}"
export BOTHY_REVISION BOTHY_IMAGE_TAG BUILDX_NO_DEFAULT_ATTESTATIONS=1
# `${a[@]+"${a[@]}"}`: an EMPTY array under `set -u` is an unbound-variable
# error on bash 3.2 (macOS), and both of these are usually empty.
svcs=({{services}})
up_flags=()
if [ "${BOTHY_UP_NO_BUILD:-}" = 1 ]; then
echo "BOTHY_UP_NO_BUILD=1: running the images tagged $BOTHY_IMAGE_TAG, not building"
up_flags=(--no-build)
else
docker compose "${files[@]}" build ${svcs[@]+"${svcs[@]}"}
fi
# --remove-orphans: a service dropped from the project (socket-proxy became
# socket-read in 2026-09) must not keep running under its old definition.
# --wait: return only once every Bothy container is HEALTHY. Traefik routes
# nothing to a container still `starting`, and bothy-web carries the `/`
# catch-all - so "up" returning early meant a few seconds in which the box
# answered Traefik's 404 on its front page (CI's install job, 2026-09-18).
docker compose "${files[@]}" up -d --remove-orphans --wait --wait-timeout 180 \
${up_flags[@]+"${up_flags[@]}"} ${svcs[@]+"${svcs[@]}"}
# Images named by commit accumulate: keep each one's three newest. `image rm`
# of a tag a container still runs is refused by Docker, so this never removes
# what is running - and the updater's rollback re-tags what it needs by id.
for repo in bothy-web bothy-files bothy-ops; do
old=$(docker image ls "$repo" --format '{{{{.Tag}}' 2>/dev/null | grep -E '^[0-9a-f]{40}$' | tail -n +4 || true)
for t in $old; do docker image rm "$repo:$t" >/dev/null 2>&1 || true; done
done
# ...and healthy is not yet ROUTED: Traefik batches provider changes (2s
# throttle), measured 1.2-1.9s after --wait returns. Wait for the catch-all
# itself to be in the router table, so the next command sees the front page.
for _ in $(seq 1 30); do
curl -fsS -m 2 http://127.0.0.1/-/api/traefik/http/routers 2>/dev/null \
| grep -q '"bothy-web-fallback@docker"' && break
sleep 1
done
# Stop everything (keeps volumes/data)
down:
-docker compose -f auth/compose.yml down
-docker compose -f edge/compose.yml down
-docker compose {{BOTHY}} down
# apps/wiki and mgmt/ were DELETED on 2026-08-18, so there is nothing here to
# bring down any more. The `wiki` database they were kept for went with them:
# its content was superseded by the Files tier, and it is in the nightly dump
# if that turns out to be wrong.
-docker compose -f data/postgres/compose.yml down
-docker compose -f monitoring/compose.yml down
# Stop everything AND delete all data volumes (DESTRUCTIVE)
# Sits one keystroke from `down` in the recipe list and deletes every data
# volume that still exists: postgres, prometheus, grafana, loki, portainer.
# (redis_data and kafka_data were already deleted on 2026-08-12 - those two
# lines below are now no-ops on this box.)
# DESTRUCTIVE: delete every data volume. Asks for confirmation; irreversible.
nuke:
@printf "This deletes ALL data volumes. Type yes to continue: " && read ans && [ "$ans" = yes ] || (echo aborted; exit 1)
-docker compose -f auth/compose.yml down -v
-docker compose -f edge/compose.yml down -v
-docker compose {{BOTHY}} down -v
-docker compose -f data/postgres/compose.yml down -v
-docker compose -f monitoring/compose.yml down -v
# Show running containers
ps:
docker ps
# One-glance health check of the whole environment
# Plain `just doctor` never exits non-zero, and should not: the everyday use is a
# human reading the report, and for them an exit code adds nothing while making
# the recipe unusable under `set -e`. The flag is for a caller that wants a
# verdict rather than a report - which until now meant nothing could gate on the
# most complete picture of the stack in this repo.
#
# One-glance health check of the whole environment. `just doctor strict` also exits non-zero on any fault.
doctor mode="":
@bash scripts/doctor.sh {{ if mode == "strict" { "--strict" } else { "" } }}
# Reproduce the Install CI job on this box, inside docker-in-docker.
#
# NOT a clone in /tmp, which is what everyone reaches for first and which does
# not work: every compose file here pins a top-level `name:`, compose project
# names are GLOBAL to the daemon, and a second `just up` therefore ADOPTS the
# running stack and recreates its containers from the other directory. That has
# already happened here once. A separate daemon is the only real isolation.
#
# Pulls ~6 GB inside the container and takes about as long as the CI job. The
# container is left running afterwards so you can run the suites against it.
#
# Run the whole download-and-run install in an isolated daemon, like CI does.
ci-install:
@bash scripts/ci-install.sh
# Verify the access model still holds - run after ANY edge/routing change.
# `just verify selftest` additionally proves the template-error probe can fail,
# by voiding a dynamic file on purpose and cleaning it up again.
# Prove the edge actually routes - 23 checks against the running stack.
verify mode="":
@VERIFY_SELFTEST={{ if mode == "selftest" { "1" } else { "" } }} bash scripts/verify-access.sh
# Could anyone but its author run this? Counts the absolute paths and tailnet
# addresses that are baked in, against a baseline that should only ever shrink.
# Runs against the TREE, so it needs nothing up - which is why it is its own
# recipe rather than a section of `verify`.
# Count the paths and addresses baked in that exist on only one machine.
portability:
@bash scripts/checks/portability.sh
# Re-apply the mechanical half of docs/brand/patterns/dataviz.md to
# monitoring/dashboards/*.json - one datasource syntax, one refresh interval,
# series coloured by entity rather than by rank, gaps left as gaps.
#
# THE WORKFLOW THIS EXISTS FOR. Three of the five dashboards are upstream imports
# (grafana.com 193, 455 and 1860) and the supported way to update one is to
# re-import it whole, which only stays cheap while nobody has hand-edited it. So:
# drop the new export in, run this, read the diff, commit. It is idempotent, so
# running it on an already-normalised tree prints "already normalised" and writes
# nothing.
#
# It refuses one rule rather than fixing it - a series drawn against a second
# y-axis - because the fix is to SPLIT the panel, which is a judgement about what
# the two halves are for. scripts/checks/dashboards.sh runs the same rules with
# --check and holds them in CI.
# Re-apply the chart conventions to the Grafana dashboard files.
dash-normalise:
@python3 scripts/normalise-dashboards.py
# Re-render docs/diagrams/*.mmd to docs/assets/diagrams/*.svg. Only the stale
# ones; pass `all` to force every one.
#
# THE SVGs ARE OUTPUT, NEVER SOURCE. apps/bothy-files/policy.toml flags
# `**/*.svg` as `caution` on write, which is right for an SVG a person might edit
# and wrong for these seven - they are machine-generated from the .mmd beside
# them, and scripts/checks/diagrams.sh (CI tier 0) fails the moment the two
# disagree. Edit the .mmd and run this; never touch the .svg.
#
# NOT IN CI, and it cannot be: it drives the headless Chromium under
# ~/.cache/ms-playwright, which a GitHub runner does not have. That is why the
# check hashes rather than re-renders.
# Re-render the architecture diagrams from their mermaid sources.
diagrams *args:
@python3 scripts/gen-diagrams.py {{ if args == "all" { "--all" } else { args } }}
# Make a fresh clone able to start: check what is missing, create the
# directories the services write to, and generate the gitignored files that
# nothing else creates. Idempotent - a second run says so rather than acting.
# `up` depends on it, because a rule that can be forgotten will be.
# Make a fresh clone runnable: check prerequisites, create dirs, generate secrets.
bootstrap *args:
@bash scripts/bootstrap.sh {{ args }}
# Checks for the editor tier: path-safety unit tests, per-route role enforcement,
# a full anonymous-refused / login / write round trip, the undo net, and a scan
# of everything the portal SERVES for anything that looks like a credential.
# Checks for the editor tier. `offline` skips what needs the stack up; `ci` skips the box-specific credential survey.
files-check mode="":
@bash apps/bothy-files/checks/run.sh {{ if mode == "offline" { "--offline" } else { if mode == "ci" { "--skip-survey" } else { "" } } }}
# Checks for bothy-ops: the guard as a pure unit, the socket-proxy grants, both
# HTTP surfaces against stand-ins (a counting fake daemon, a TLS fake apiserver),
# the edge/RBAC/compose wiring, and - unless `offline` - a real container
# transition through real throwaway proxies.
# Checks for bothy-ops (container + cluster actions). `offline` skips the one that needs docker.
ops-check mode="":
@bash apps/bothy-ops/checks/run.sh {{ if mode == "offline" { "--offline" } else { "" } }}
# Two declarations are the only hand-written cluster wiring, one per account that
# reaches the cluster from this box. This regenerates everything derived from them:
# apps/bothy-ops/catalog.toml -> the edge routers, bothy-kube's Role, the
# can-i rows of `just kube-token`, the UI's dev catalog
# apps/bothy-collector/reads.toml -> bothy-collector's ClusterRole and the
# can-i rows of `just collector-token`
# `check` fails on drift instead of writing (CI runs that). Apply the RBAC with
# `just kube-token` / `just collector-token`.
# Regenerate the kube edge routers, both Roles and both can-i row sets from their declarations. `check` only diffs.
ops-wiring mode="":
@python3 scripts/gen-ops-wiring.py {{ if mode == "check" { "--check" } else { "" } }}
# The Settings area's Keycloak client `bothy-admin` (view-users only), and its
# secret in apps/bothy-ops/secrets. `--rotate` issues a new secret; `--revoke`
# disables the client and deletes the file.
# Create (or rotate/revoke) the read-only Keycloak client behind Settings > Users.
admin-client *args:
./scripts/keycloak-admin-client.sh {{args}}
# Settings > Credentials and Backups are served from a metadata-only file written
# on the host - names, modes, ages and sizes, never a value. The timer in
# host/systemd/bothy-inventory.timer runs this every five minutes.
# Rewrite the host-side credential and backup inventory that Settings serves.
admin-inventory:
python3 apps/bothy-ops/inventory.py
# Settings > Updates is served from a file written on the host: every pin in
# apps/bothy-ops/updates.toml against the registries, GitHub and the helm index,
# and what is running. READ-ONLY - it pulls nothing and changes no pin. The timer
# host/systemd/bothy-updates-discover.timer runs it every six hours; answers are
# cached (~/.cache/bothy/updates.json), so a rerun is cheap. `args` go to the
# program: --no-cache, --only grafana,loki, --json, --state-dir DIR.
# Discover newer versions of every component (read-only) and print the table.
updates-discover *args:
python3 apps/bothy-ops/discover_updates.py {{args}}
# The host updater (docs/plans/updates.md step 4). It deploys ONLY what the
# checked-out main already pins - a merged Dependabot PR, say - for the stateless
# and time-series components, with a pre-flight, a snapshot, canaries on response
# bodies and an automatic rollback. Settings > Updates queues a request and
# host/systemd/bothy-updater.path runs it; these are the same program by hand,
# over Tailscale SSH, when the browser is not an option.
# just update-plan loki the plan for one component (read-only)
# just update-groups what each recipe would apply all at once (read-only)
# just update-status the current job, the last ten and the queue
# Show the plan the host updater would run for a component (read-only).
update-plan component:
cd apps/bothy-ops && python3 -m updater plan --component {{quote(component)}}
# When two services of one compose project are behind, no single-component apply
# can succeed - the scope check, rightly, refuses both. The group is the action
# that closes that, and it is a click in Settings > Updates rather than a recipe
# here, because the union of its members' classes may need the recipe's name
# typed. From a shell the answer stays `just up-<recipe>`.
#
# `just` takes the LAST comment line as the description, so the summary goes
# here and the reasoning stays above it.
# Show what each `apply` recipe would apply all at once, and why one is refused (read-only).
update-groups *group:
cd apps/bothy-ops && python3 -m updater groups {{ if group == "" { "" } else { "--group " + quote(group) } }}
# Show the updater's current job, recent history and queue.
update-status:
cd apps/bothy-ops && python3 -m updater status
# The automatic channel (docs/plans/updates.md step 7): bothy-updater-auto.timer
# runs `update-auto` at 03:30. A rollback or a failure pauses auto for that
# component until an operator clears it - here, or Settings > Updates.
# just update-auto --dry-run what tonight would do, and why (writes nothing)
# just update-pauses what is paused, and the last night's decision
# just update-unpause loki clear loki's pause (audited, as you)
# Run the automatic-update night job by hand (it still keeps to the window).
update-auto *args:
cd apps/bothy-ops && python3 -m updater auto {{args}}
# List the components whose automatic updates are paused, and why.
update-pauses:
cd apps/bothy-ops && python3 -m updater pauses
# Clear the automatic-update pause of one component (an operator's decision).
update-unpause component:
cd apps/bothy-ops && python3 -m updater unpause {{quote(component)}}
# The updater never replaces itself mid-run (docs/plans/updates.md step 6): the
# systemd units run ~/.local/lib/bothy-updater/current, a copy of the updater's
# files at one commit. This copies HEAD's (from git, not the working tree) and
# switches `current` atomically. An update of Bothy that changes the updater only
# STAGES the new copy; Settings > Updates then says this switch is pending.
# Install (or switch to) HEAD's updater in ~/.local/lib/bothy-updater - its systemd units run from there.
install-updater:
cd apps/bothy-ops && python3 -m updater install
# What it saves, and how, is listed at the top of scripts/backup.sh.
# Back up postgres, grafana, .env, VictoriaMetrics, Loki, alloy, audit/trash and notes now (the nightly timer also runs this).
backup:
@bash scripts/backup.sh
# RESTORE. Each takes an explicit file - there is no "latest" default, on
# purpose - shows what it will overwrite, asks unless `--yes`, stops only the
# service it replaces, and verifies afterwards. See docs/guide/backups.md.
# just restore-postgres ~/backups/postgres/pg-<ts>.sql.gz
# Restore postgres from a pg_dumpall file (drops + recreates the databases in it; restarts keycloak).
restore-postgres file *flags:
@bash scripts/restore.sh postgres {{quote(file)}} {{flags}}
# Restore grafana.db from a backup (stops grafana, replaces the file, starts it, checks /api/health).
restore-grafana file *flags:
@bash scripts/restore.sh grafana {{quote(file)}} {{flags}}
# Restore .env from a backup (the current one is saved first; takes effect at the next `just up`).
restore-env file *flags:
@bash scripts/restore.sh env {{quote(file)}} {{flags}}
# Restore VictoriaMetrics from a vm-<ts>.tar or an unpacked snapshot directory (replaces ALL its data).
restore-victoriametrics path *flags:
@bash scripts/restore.sh victoriametrics {{quote(path)}} {{flags}}
# Restore Loki from a loki-<ts>.tar.gz (replaces ALL its data).
restore-loki file *flags:
@bash scripts/restore.sh loki {{quote(file)}} {{flags}}
# Follow a container's logs, e.g. `just logs grafana`
logs service:
docker logs -f --tail 100 {{service}}
# Open a psql shell on the dev database
psql:
docker exec -it postgres psql -U ${POSTGRES_USER:-dev} -d ${POSTGRES_DB:-dev}
# `just redis` is GONE (2026-08-12) - redis was retired as idle (0 keys). A
# recipe that always fails is worse than a missing one: it looks like breakage.
# `docker compose -f data/redis/compose.yml up -d` starts a NEW, EMPTY redis:
# the redis_data volume and the images were deleted after the retirement, so
# there is nothing left to restore.
# Regenerate the portal's read-only Prometheus route (edge/dynamic/bothy-prom.yml).
#
# That file carries the basic-auth header the edge injects on the portal's
# behalf, so it is GITIGNORED and generated from .env rather than committed.
# Run this on a fresh clone, and again after changing DEV_LOGIN_*. Traefik
# watches ./dynamic, so no restart is needed.
# Regenerate the portal's Prometheus data-plane route.
bothy-prom-route:
./scripts/gen-bothy-prom-route.sh
# (Re)apply the cluster side of monitoring into minikube thales-scc: namespace
# `monitoring`, kube-state-metrics (pinned helm chart, NodePort 30808), the
# kubelet scrape identity, the Alloy DaemonSet shipping pod logs to Loki, and
# the metrics-server addon - then refresh the kubelet token VictoriaMetrics
# scrapes with. Idempotent. See k8s/monitoring/README.md. The compose side is
# `just up-monitoring`. `just k8s-monitoring ksm` / `alloy` apply ONE add-on
# only - what the host updater's cluster class runs (docs/plans/updates.md step 8).
# Apply cluster monitoring (KSM, alloy, RBAC) into thales-scc and refresh the token.
k8s-monitoring *part:
./scripts/k8s-monitoring.sh {{part}}
# Rewrite monitoring/kube-auth/token from the cluster's prometheus-scraper-token
# Secret. GITIGNORED (a credential), mode 600 owned by uid 65534 so the
# container's `nobody` can read it (the legacy Prometheus; VictoriaMetrics runs
# as root and reads it either way). No restart needed: the token is re-read on
# every scrape.
# Regenerate the kubelet bearer token VictoriaMetrics scrapes thales-scc with.
kube-prom-token:
./scripts/gen-kube-prom-token.sh
# The OCI cluster's datasources in Grafana (OCI Prometheus / Loki / Tempo):
# writes monitoring/oci-auth.env (GITIGNORED, mode 600) from the node's
# bothy-grafana-token Secret over Tailscale SSH, then recreates grafana so it
# reads it. The account may only GET through those three Services' proxy.
# Write Grafana's read-only credential for the OCI cluster, then restart grafana.
oci-grafana-token:
./scripts/gen-oci-grafana-token.sh
docker compose -f monitoring/compose.yml up -d grafana
# bothy-ops' cluster credential: applies k8s/rbac/bothy-kube.yaml, then writes
# apps/bothy-ops/secrets/{token,ca.crt} (mode 600, gitignored) and prints what
# the account can and cannot do. `--rotate` revokes and reissues; `--revoke` stops.
#
# `just up-kube` is GONE (2026-09): bothy-kube merged into bothy-ops, and
# `just up-apps` adds the cluster overlay by itself when thales-scc exists.
# Issue (or rotate/revoke) bothy-ops' ServiceAccount token for the cluster.
kube-token *args:
./scripts/gen-kube-token.sh {{args}}
# The portal collector's cluster credential: applies k8s/rbac/bothy-collector.yaml
# (a ClusterRole holding `list` on five kinds and nothing else, generated from
# apps/bothy-collector/reads.toml), then writes
# apps/bothy-collector/secrets/kubeconfig (mode 600, gitignored) and prints the
# can-i table. `--rotate` revokes and reissues; `--revoke` stops.
#
# Run it once per cluster REBUILD: a new cluster has a new CA and no bothy
# namespace, and until this runs the collector falls back to ~/.kube/config -
# which on minikube is the admin certificate, and says so on stderr.
# Issue (or rotate/revoke) the portal collector's read-only ServiceAccount kubeconfig.
collector-token *args:
./scripts/gen-collector-token.sh {{args}}
# Headlamp's cluster credential: applies k8s/rbac/bothy-browse.yaml (the built-in
# `view` ClusterRole - no secrets, exec or port-forward), then writes
# apps/headlamp/secrets/kubeconfig (mode 600, gitignored) and prints the can-i
# table. `--rotate` revokes and reissues; `--revoke` stops.
# Issue (or rotate/revoke) Headlamp's read-only ServiceAccount kubeconfig.
headlamp-token *args:
./scripts/gen-headlamp-token.sh {{args}}
# Headlamp, the read-only cluster console, behind its own oauth2-proxy on :8110
# (Keycloak client `headlamp`, role `viewer`). NOT part of `up`, for the same
# reason as up-kube: it joins minikube's `thales-scc` network, which exists only
# while the cluster does. Idempotent: the token is issued only if absent, and
# the Keycloak client (plus its two .env secrets) is re-asserted every run.
# The script writes .env, so compose runs in a nested just that re-reads it.
# Start Headlamp (read-only cluster console, SSO on :8110). Needs the cluster and Keycloak.
up-headlamp:
#!/usr/bin/env bash
set -euo pipefail
docker network inspect thales-scc >/dev/null 2>&1 || {
echo "no docker network thales-scc - start the cluster first: minikube start -p thales-scc" >&2; exit 1; }
[ -f apps/headlamp/secrets/kubeconfig ] || ./scripts/gen-headlamp-token.sh
./scripts/keycloak-headlamp-client.sh
env -u HEADLAMP_OAUTH2_CLIENT_SECRET -u HEADLAMP_OAUTH2_COOKIE_SECRET \
just _up-headlamp
echo ""
echo " Headlamp http://$(bash scripts/lib/box-addr.sh):8110 (Keycloak login, role viewer)"
_up-headlamp:
docker compose -f apps/headlamp/compose.yml up -d --wait --wait-timeout 180
# Stop Headlamp and its proxy. Run before `minikube delete`, or the network cannot go.
down-headlamp:
docker compose -f apps/headlamp/compose.yml down
# Print access URLs. Pure-IP-over-tailscale model: every service has a published
# host port on this node's tailnet IP.
#
# Traefik Host-name routing is DELETED - not
# dormant, not waiting on a DNS layer. Zero Host() rules remain in the router
# table. A new service publishes a port; it does not declare a name.
# The order is the point. VERSION is edited and COMMITTED first, by hand, in its
# own change - then this tags what is already on main. Tagging first and editing
# after leaves a tag pointing at a commit that claims the wrong version, which is
# the exact disagreement scripts/checks/version.sh exists to catch.
#
# Tag the current main as the release named in VERSION. Does not push.
release:
#!/usr/bin/env bash
set -euo pipefail
v=$(tr -d '[:space:]' < VERSION)
bash scripts/checks/version.sh || {
echo "Fix VERSION before tagging - see above."; exit 1; }
git diff --quiet || { echo "working tree is dirty - commit first"; exit 1; }
[ "$(git rev-parse --abbrev-ref HEAD)" = main ] || {
echo "releases are cut from main"; exit 1; }
git tag -a "v$v" -m "Bothy v$v"
echo "tagged v$v - push it with: git push origin v$v"
# Print every address on this box, and what is deliberately not running.
urls:
#!/usr/bin/env bash
# Identity is read from tailscale at run time and never stored in this repo:
# the node name/IP identify the owner's tailnet, and this repo is public.
IP=$(bash scripts/lib/box-addr.sh)
HOST=$(tailscale status --json 2>/dev/null | jq -r '.Self.DNSName // ""' | sed 's/\.$//'); HOST=${HOST:-<this-node>}
echo "This box is a tailnet node: $HOST / $IP - reachable directly."
echo "Access is by IP:port ($HOST:<port> also works - MagicDNS is free)."
echo ""
echo " ⭐ START HERE http://$IP/ (the portal - index of everything)"
echo ""
echo " Stack services:"
echo " Grafana http://$IP:3000 (unified dev login)"
echo " Metrics (VM) http://$IP:8428/vmui (VictoriaMetrics; Prometheus API - unified dev login)"
echo " cAdvisor http://$IP:8082"
echo " node-exporter http://$IP:9100"
echo " Loki http://$IP:3100 (API only; 404 at / is normal)"
echo " Keycloak http://$IP:8090/admin (identity - admin / shared dev login)"
echo " Headlamp http://$IP:8110 (read-only cluster console - Keycloak"
echo " login, role viewer; only while the"
echo " thales-scc cluster runs: just up-headlamp)"
echo ""
echo " Files (raw) http://$IP:8100 (SANDBOX ORIGIN - raw file bytes"
echo " only. A different port is a"
echo " different origin, so a hostile"
echo " SVG/PDF served here cannot touch"
echo " the portal session. Nothing else"
echo " may be routed to :8100.)"
echo ""
echo " Deleted 2026-08-18 - Bothy replaced each of them:"
echo " Portainer (was :9000) start/stop/restart is the Control tier"
echo " Dozzle (was :8080) logs are on each service page, from Loki,"
echo " which also keeps them after a container is gone"
echo " Wiki.js (was :3001) docs and notes are the Files tier"
echo " WHAT YOU LOSE, said plainly rather than discovered later:"
echo " · a shell in the browser. Bothy Control does exec deliberately"
echo " NEVER - that is root on this box. Use Tailscale SSH:"
echo " ssh devssh@$HOST then docker exec -it <name> sh"
echo " · image, volume and network management. Those are docker CLI."
echo " · a true streaming tail. Bothy polls Loki on a range instead."
echo " Retired 2026-08-17, deleted with mgmt/ and apps/wiki/ the next day,"
echo " along with the 'wiki' database - superseded, and in the nightly dump."
echo " \`git revert\` brings the compose files back if that was wrong."
echo ""
echo " Not running right now - normal, not a fault:"
echo " Tilt http://$IP:10350"
echo " NOT a stack service - a per-project dev server under ~/projects"
echo " (CVOps, Tals) that you start by hand. Nothing is wrong when this"
echo " port is dead; that is its normal state between sessions:"
echo " tilt up --host=0.0.0.0"
echo ""
echo " Reserved by projects (listed so the stack never picks them):"
echo " CVOps nginx http://$IP:8086 (project edge, compose mode)"
echo " CVOps garage http://$IP:3900 (S3 blobs - the BROWSER fetches"
echo " these directly, so it must stay"
echo " 0.0.0.0-bound, not loopback)"
echo " Tals frontend http://$IP:5173 (ports in Tals' .ports.lock)"
echo " Tals pre-prod http://$IP:31080 (minikube profile thales-scc, namespace"
echo " thales-pre-prod; break-glass login)"
echo " Tals dev http://$IP:31081 (namespace thales-dev; seeded, demo logins)"
echo " Both held open by the user units"
echo " thales-{preprod,dev}-forward; the"
echo " cluster starts via minikube.service."
echo ""
echo " Retired 2026-08-12 - measured idle, then DELETED. Data NOT recoverable:"
echo " Kafka + Kafka-UI (was :8081) zero topics ~1,110 MB"
echo " Redis (was :6379) zero keys ~30 MB"
echo " minikube (local k8s) zero user pods ~1,046 MB"
echo " ^ REBUILT 2026-09-01 as profile \"thales-scc\" - an OpenShift"
echo " restricted-v2 SCC emulation (PSA restricted + Kyverno) running the"
echo " thales-dev and thales-pre-prod namespaces, started at boot by"
echo " minikube.service. The line above records the 2026-08-12 deletion."
echo " All three were first stopped and kept, then removed for real:"
echo " the kafka_data and redis_data volumes and every image are gone,"
echo " and 'minikube delete' ran at 13:56 (no profile remains)."
echo " The compose files survive, so these commands still WORK - but"
echo " each one builds something BRAND NEW AND EMPTY, it does not"
echo " resume anything:"
echo " docker compose -f data/kafka/compose.yml up -d # empty broker"
echo " docker compose -f data/redis/compose.yml up -d # empty keyspace"
echo " minikube start # empty cluster"
echo ""
echo " Windows-host mirrors via portproxy: http://100.93.197.10:<port> from the"
echo " tailnet, or localhost:<port> on the host. Same ports; LAN is NOT served."
echo ""
echo " Logins: one unified dev login everywhere a login exists - username is"
echo " the owner gmail, password = DEV_LOGIN_PASSWORD in the gitignored .env."
echo ""
echo " Name-based routing and the Traefik dashboard were"
echo " DELETED on 2026-08-12 - publish a port, never a Host() rule."
echo " Route data is still live at http://$IP/-/api/traefik/http/routers"
echo ""
echo "Databases stay off the network (loopback-bound). Reach via ssh tunnel:"
echo " ssh -L 5432:localhost:5432 $USER@$IP"
echo " (6379/redis and 9092/kafka were retired 2026-08-12 - nothing listens"
echo " on them now; add those -L flags back if you restart either stack.)"
echo ""
echo "If the tailnet IP serves headers but bodies stall, or SSH hangs at KEX:"
echo " that is the large-packet blackhole - restart tailscaled ON THIS BOX."
echo " See docs/kb/incidents/2026-08-08-wsl-node-large-packet-blackhole.md."