docs/ARCHITECTURE.md
Rama main · réplica de lectura del repositorio original.
# Architecture
One page. The decisions behind it live in the ADRs linked at the end; this is
the map.
## Two processes, one boundary
```
┌────────────────── always on ──────────────────┐
Alertmanager ──► ingest-api ──► Redis Stream ──► analyzer-worker REFLEX
webhook (FastAPI) (consumer group) collect → redact → one LLM call per alert
budget → 1 call → hard timeout, daily cap
persist → notify dedup, dead-letter
│
├─► Postgres (redacted incidents) ◄─────┐
├─► Slack / Telegram (hypothesis) │
└─► web-ui (read-only history) │
│
Engineer in Telegram ──► minus20-agent DELIBERATE │
"why did billing-api crash?" bounded tool loop, on demand ──────────────────────────────┘
"what changed in the last hour?" tools: closed, read-only (events, logs, metrics,
"/report 7" rollouts, incident history, memory)
"/wrong <id> it was X" memory: pgvector, local embeddings
feedback written back, becomes memory
```
**Reflex** reacts to every alert with exactly one bounded model call. It never
depends on the agent. If the model, the budget or the agent is gone, the raw
alert is still delivered (ADR-0001, ADR-0003).
**Deliberate** runs only when a human asks. It may call tools in a loop, but
every tool observes and none can change anything; the loop is capped by tool
calls, wall clock and a daily budget of its own (ADR-0005).
Both run from the same container image with different entrypoints
(`python -m app.worker`, `python -m app.agent`) and share one read-only
ServiceAccount. A third entrypoint, `python -m app.mcp_server`, serves the
agent's tool registry over the Model Context Protocol so an external agent
(Gemini CLI, Claude Code) can use Minus20 as its memory of the cluster;
it adds no tool and no permission.
## Ports and adapters
The worker depends on interfaces (`services/analyzer-worker/app/ports.py`),
never on vendors. Choosing an adapter is one environment variable; swapping a
vendor is never a code change (ADR-0002).
| Port | Adapters | Selected by |
|---|---|---|
| `Collector` | `K8sEventsCollector`, `K8sPodLogsCollector`, `PrometheusCollector`, `LokiCollector`, `CloudWatchLogsCollector`, `CloudWatchMetricsCollector`, `StubCollector`, fanned out by `AggregateCollector` | `MINUS20_COLLECTORS` |
| `LLMBackend` | `OllamaBackend` (local, $0), `OpenAICompatibleBackend` (DeepSeek, Groq, Gemini, vLLM…), `AnthropicBackend`, `StubBackend`; `FallbackBackend` tiers any two | `MINUS20_LLM_PROVIDER`, `_LLM_FALLBACK_PROVIDER` |
| `Notifier` | `SlackNotifier` (Incoming Webhook), `TelegramNotifier`, `StubNotifier` | `MINUS20_NOTIFIER` |
| `IncidentStore` | `PostgresStore`, `InMemoryStore` | `MINUS20_STORE` |
| `Budget` | `RedisBudget` (shared across replicas), `InMemoryBudget` | Redis present or not |
| `Deduplicator` | `RedisDeduplicator` (`SET NX EX`), `InMemoryDeduplicator` | Redis present or not |
| `ChatBackend` (agent) | `OllamaChat`, `OpenAIChat`, `FallbackChat`; vision routed separately | `MINUS20_LLM_PROVIDER`, `_VISION_PROVIDER` |
| `Embedder` (agent) | `OllamaEmbedder` | `MINUS20_EMBED_MODEL` |
Tests inject fakes at the same ports (`httpx.MockTransport`, fake pools, fake
Kubernetes clients), which is why the whole suite runs offline in about a
second.
## Why Python
The services are network-bound: almost all of their time is spent waiting on
Loki, Prometheus, the Kubernetes API and the model. The runtime is not in the
latency. Python gave a solo project faster iteration and the ecosystem the
product actually depends on (the official MCP SDK, pydantic, httpx,
kubernetes_asyncio, asyncpg with pgvector, every LLM client). What Go or
Rust would have bought: images of ~20 MB instead of ~200 MB, faster starts,
and fewer base-image CVEs to chase (Trivy found 12 in `python:3.12-slim`
on its first run; the Dockerfiles now apply security updates at build). The
hot path, `ingest-api`, is a hundred lines and can be rewritten on its own if
it ever becomes the bottleneck. A Kubernetes operator with CRDs would be
written in Go; this is not one.
## What flows where, and what never leaves
1. Alertmanager POSTs the alert. `ingest-api` validates it with Pydantic and
publishes it to a length-capped Redis Stream. Nothing else is parsed here.
2. The worker reads it through a consumer group, checks the dedup window
(the same alert again) and the storm window (a different pod, same
alertname and namespace, moments later: recorded and counted as a member
of the first incident, not analysed), and runs the collectors: Kubernetes events (`kubectl describe`-level),
pod logs, PromQL instant queries, LogQL over a bounded window, CloudWatch. Fixed set, small caps.
A *different* alert in a namespace with a fresh incident is attached to it
(status `correlated`); after a short settle period one revision call over
all attached contexts replaces the leader's hypothesis, the previous cause
kept as `revised_from` (ADR-0006). The first hypothesis is never delayed.
3. **Redaction** masks e-mails, IPs, tokens, cloud keys, secret-shaped
values, payment card numbers (Luhn), IBANs (mod-97) and international
phone numbers in everything collected. It runs before any model call and has no
off switch (ADR-0002).
4. One structured model call produces a hypothesis with evidence, the cheapest
disproof, blast radius and next steps (ADR-0004). The context is framed as
untrusted data; the model holds no tools on this path.
5. The redacted incident is stored **together with the exact redacted context
the model was shown** (the audit trail: what did it see when it said that),
the message is delivered, the cost is charged to the daily budget. Failure
at any step still delivers the alert.
6. A message read by a worker that dies mid-analysis is reclaimed by the next
worker (`XAUTOCLAIM` after an idle threshold) and analysed, bypassing the
dedup key the dead worker already set. Each pod is its own consumer, a
worker stops reading on SIGTERM and finishes the message in hand, and dead
consumer names are pruned at startup. Every log line passes through the
redactor before it is written, exception text included.
7. Two clocks are recorded per incident: the model call (`latency_ms`) and
the wall clock from the alert firing to the hypothesis
(`time_to_hypothesis_ms`), which is what an engineering manager asks for.
With the Ollama backend and local embeddings, no byte of log, metric or
event leaves the cluster. That is the difference between "we redact" and
"there is nothing to redact from".
## Security posture in one paragraph
Model output is never executed. The worker's model has no tools. The agent's
tools are a closed registry of read-only observations (`app/agent/tools.py`);
adding one is a code review of its verbs. RBAC grants get/list on events,
pods, pods/log, nodes, replicasets and deployments, and nothing with a write verb.
Credentials live in Secrets, never in values or the ConfigMap; the Telegram
token is kept out of logs by pinning httpx to WARNING. A prompt injection in a
log line can produce a wrong sentence in chat; it cannot produce an action.
## Deployment shape
Helm chart `deploy/minus20` (published as OCI at
`ghcr.io/maxuver/charts/minus20`): ingest-api, analyzer-worker, Redis,
optional Postgres (pgvector image), optional web-ui, optional agent. Images
come from GHCR on every push to `main`, gated on tests, SAST, SCA and secret
scanning. `infra/terraform` stands up an ephemeral EKS for the same chart.
## Decisions
- [ADR-0001](adr/0001-event-driven-pipeline-over-agent-loop.md) — one bounded call per alert, not an agent loop
- [ADR-0002](adr/0002-llm-privacy-and-pluggable-backends.md) — mandatory redaction, backend is configuration
- [ADR-0003](adr/0003-graceful-degradation.md) — the AI is an overlay; the alert always arrives
- [ADR-0004](adr/0004-hypothesis-evidence-and-blast-radius.md) — a hypothesis carries evidence, its disproof and blast radius
- [ADR-0005](adr/0005-reflex-and-deliberate-agent.md) — reflex and deliberate agent as separate processes
- [ADR-0006](adr/0006-cross-alert-correlation.md) — different alerts of one namespace: one incident, one bounded revision