CLAUDE.md
Rama main · réplica de lectura del repositorio original.
# CLAUDE.md — Maestro Platform
Single source of truth for AI assistants and contributors working in this repository:
architecture, conventions, and the invariants that must not be broken.
All code, identifiers, comments, commit messages, and documentation are written in English.
---
## 1. What Maestro Is
Maestro is a general-purpose AI agent orchestration platform. Users connect their own
model provider keys (BYOK — Bring Your Own Key) and automate complex tasks through a
multi-layer agent hierarchy.
One prompt in → the Orchestrator classifies the domain → a Main Agent decomposes the work
→ Subagents execute atomic subtasks → an optional Reviewer enforces quality → a synthesized
result comes out. Community-built agent teams are shared and installed through a Marketplace.
---
## 2. Agent Hierarchy
```
User prompt
│
▼
ORCHESTRATOR Routing only; produces no work itself. Two stages: the task is
│ classified into a domain *group* first, then into one domain
│ inside that group.
│
▼
MAIN AGENT Domain expert (finance, software, marketing, …). May first run a
│ read-only discovery pass over the user's own data, then builds the
│ subtask plan, assigns each subagent its tools, and coordinates them.
├──────────────┐
▼ ▼
SUBAGENT SUBAGENT One atomic task each (fetch data, analyze, summarize).
│ │ May escalate to the Main Agent for a tool it was not
│ │ assigned (see §8, autonomous, not human-in-the-loop).
└──────┬───────┘
▼
REVIEWER Optional (`reviewer_enabled`). Validates subagent output and
sends it back with issues. Bounded by `max_review_iterations`.
```
### Domain groups
The 43 built-in domains are partitioned into six groups (`DOMAIN_GROUP_CATALOG`
in `agents/domains/__init__.py`): `build`, `market`, `money`, `operate`, `life`,
`knowledge`. A group is not a label — three separate things stopped scaling
one-per-domain past roughly a dozen entries, and all three key off it:
- **Routing.** The orchestrator classified with a single LLM call over every
domain's `routing_hint`. Forty-three near-synonymous hint lines is a forty-
three-way classification, which a small local model answers by surface
wording; the contrastive "NOT …" clauses that separate neighbouring domains
only work when the neighbours are in front of it. Stage one picks one of six
groups, stage two one of that group's five to ten domains, so neither call
ever exceeds a single digit of options. The cost is one extra call —
`temperature=0`, small token cap, no conversation — against a misroute, which
spends a whole task on the wrong team.
- **Colour.** Tailwind class names must be static literals, so a hue costs a
hand-written set in `agent-colors.ts` mirrored twice in `tailwind.config.ts`.
Six is maintainable; forty-three is a file nobody edits correctly. The dark
chart band does not hold forty distinguishable hues either, so per-domain
colour had already stopped meaning anything. Within a group, chart marks step
lightness so bars stay comparable.
- **Motifs.** Catalog card animations are hand-drawn SVG. A group carries the
default pair; `DOMAIN_MOTIF_OVERRIDES` is the escape hatch for a domain that
earns its own.
Both routing stages fail soft. A stage-one failure routes to `general`; a
stage-two failure keeps the group and takes its `default_domain`, so a task
correctly identified as finance work still reaches a finance team rather than
the generalist one. Both emit an `AGENT_WARNING` of kind `degraded`.
A user's custom agents are offered alongside the groups in stage one and
short-circuit stage two — a custom agent is a team of one, so there is nothing
left to narrow.
Catalog order is meaningful and grouped: `DOMAIN_CATALOG` lists each group's
domains in one contiguous block, `general` stays last as the routing fallback,
and the frontend's `AGENT_DOMAINS` mirrors the same order index for index
(`tests/test_domain_frontend_parity.py`). `GET /agents` serves both the per-entry
`group` and a `groups` block, which is what the Architect catalog's tabs render
from; the colour mapping is duplicated in `constants.ts` and parity-tested,
because it has to resolve before that response lands.
Contracts between layers are structured JSON, never free text.
Subagent output:
```json
{
"status": "success | error | needs_review",
"data": {},
"metadata": { "tokens_used": 0, "execution_time_ms": 0, "model_used": "string" }
}
```
Reviewer feedback:
```json
{ "approved": false, "issues": ["..."], "retry_hints": ["..."] }
```
Every domain declares a `group` (above) and every agent has a `system_prompt`, a
`tools` list, and a `max_iterations` bound.
The `tools` list is resolved **per subagent**, not just per domain: the Main
Agent may assign each member a subset of the domain's tools (an unassigned member
falls back to the full domain set). A member's effective set is always
`main-assigned ∩ domain-declared ∩ operator-switches ∩ credentials` — an
assignment can only ever *narrow* it. A subagent that finds it needs a tool it
was not given may `request_tool` from the Main Agent mid-run (§8).
---
## 3. Stack
| Layer | Technology | Purpose |
|---|---|---|
| Frontend | Next.js (App Router) + React + TypeScript + Tailwind | UI, SSR, routing |
| Backend | FastAPI (Python 3.11+) | API, agent runtime, WebSocket |
| Relational DB | PostgreSQL (SQLAlchemy async, Alembic) | Users, subscriptions, billing, durable task state |
| Document DB | MongoDB (Motor) | Agent logs, marketplace, task sessions, traces |
| Vector DB | Qdrant | RAG memory and document embeddings |
| Cache / bus | Redis | Rate-limit buckets, cross-worker event bus |
| Realtime | WebSocket | Live task streams, human-in-the-loop, architect view |
| Auth | Backend-issued JWT + refresh-token rotation, optional TOTP 2FA | Sessions |
| Encryption | AES-256-GCM | BYOK key storage |
| Local models | Qwen3 + nomic-embed-text via Ollama (OpenAI-compatible endpoint) | Zero-cost local/self-hosted operation |
| Web fetch | Scrapling (curl_cffi TLS impersonation, lxml/CSS) | `data_fetch` tool: page text plus CSS-selector extraction |
| Connected APIs | GitHub REST, X API v2, Discord/Slack/Telegram, Google Places | `repo_intel` / `social_search` / `community_read` / `places_intel` tools, authenticated with the user's own BYOK service keys |
---
## 4. Repository Layout
```
maestro/
├── frontend/src/
│ ├── app/ (app)/ authenticated, (auth)/ login+register, (marketing)/ public
│ ├── components/ ui/ shared primitives, then one directory per feature area
│ ├── lib/ api client, seo/, legal/, analytics/, observability/, helpers
│ ├── stores/ Zustand stores
│ └── types/ API response types
│
├── backend/app/
│ ├── main.py App entrypoint, middleware, exception handlers
│ ├── core/ config, security (crypto), database, observability, metrics
│ ├── api/v1/ auth, users, api_keys, tasks, agents, documents,
│ │ marketplace, billing, dashboard, admin
│ ├── api/websocket.py
│ ├── agents/ orchestrator, main_agent, subagent, reviewer, registry
│ ├── models/ SQLAlchemy models
│ ├── schemas/ Pydantic request/response models
│ ├── services/ Business logic. Notable seams:
│ │ llm_service.py provider adapters
│ │ task_engine.py durable step loop
│ │ task_service.py public task API
│ │ quota_service.py quota enforcement
│ │ memory_service.py RAG / vectors
│ │ payment/ PaymentProvider protocol + mock
│ │ email/ EmailProvider protocol + console/resend
│ │ alerts/ AlertChannel protocol + webhook/email
│ │ alert_service.py redaction, dedupe, fan-out
│ │ watchdog.py readiness/5xx alerting, metrics publish
│ ├── scripts/ Operational one-shots (purge, email-token sweep, grant_admin)
│ └── utils/ prompt_guard, rate_limiter, tracing
│
├── docker-compose.yml / docker-compose.prod.yml / Caddyfile
├── scripts/ dev.ps1, dev.sh, backup.sh
└── docs/DEPLOYMENT.md
```
---
## 5. Conventions
### General
- Formatting: frontend ESLint + Prettier; backend Ruff (linter + formatter).
- Every function and class carries a docstring or JSDoc.
- No magic numbers or strings. Constants live in `constants.ts` / `constants.py`.
### Frontend
- Function components only; no class components.
- Zustand for state.
- All API calls go through the central client in `lib/`; every response is typed in `types/`.
- Pages in `app/`, shared components in `components/`.
### Backend
- Every endpoint is `async`. Pydantic v2 validates every request and response.
- Business logic lives in `services/`; route handlers stay thin.
- Errors are handled by the central exception handler.
- Schema changes go through Alembic. Never hand-run SQL.
---
## 6. Data Model
### PostgreSQL
```
users id, email, hashed_password, display_name, role (user|admin),
subscription_tier (free|starter|pro|scale — every account is
provisioned with an active free plan at registration),
email_verified, totp_secret (encrypted), model_preferences,
deletion_requested_at, suspended_at
api_keys user_id, provider, encrypted_key (AES-256-GCM), label, is_active
refresh_tokens session families; rotation + reuse detection
recovery_codes Argon2-hashed, single-use 2FA fallbacks
email_tokens SHA-256 hash only; single-use; purpose-scoped TTL
subscriptions one row per user: plan, status, provider, provider ids,
current_period_start (quota window anchor), current_period_end,
cancel_at_period_end
payment_methods brand, last4, exp_month, exp_year. Raw PAN is never stored.
usage_records append-only quota ledger. task_id is the idempotency key.
task_runs / task_checkpoints / task_questions
durable execution state: leases, heartbeats, replay checkpoints
```
### MongoDB
```
agent_logs Step-by-step agent execution history (seq-ordered)
marketplace_items Published agent teams, ratings, install counts
task_sessions Task sessions and analytics
agent_configurations Custom agent prompts, tools, provenance
agent_skills Reusable instruction bundles attached to custom agents
mcp_servers Registered remote MCP servers + their sanitized tool cache
plugins Published plugin manifests (catalog; author anonymized on purge)
plugin_installs One row per installed bundle: what it created, per user
trace_spans Execution spans with TTL
```
### Qdrant
```
conversation_memories Per-user conversation embeddings
document_chunks Uploaded document chunks
```
Memory is scoped per user. Data must never leak across users.
---
## 7. API Surface
All routes are under `/api/v1`. Every route declares an explicit rate limit.
```
auth register, login, login/totp, refresh, logout,
verify-email, resend-verification, forgot-password, reset-password
users me (GET/PATCH), password, sessions (list/revoke/revoke-others),
2fa/setup|enable|disable, export, deletion (request/cancel)
api-keys list, create, delete
tasks create, get, cancel, answer, WS stream
agents CRUD, system-prompt patch
skills CRUD (reusable instruction bundles)
mcp-servers CRUD, discover (remote MCP servers)
plugins list, publish, install, import, installed, uninstall-preview
documents upload, list, delete, storage (usage vs plan allowance)
marketplace list, publish, install, reviews (submit/list), report
billing plans, subscription, subscribe, cancel, payment-method
dashboard metrics, token-usage, cost-summary, costs
admin overview, users, marketplace items, agents takedown, reports, audit
architect WS live agent communication stream
health /health, /health/ready (outside /api/v1)
```
---
## 8. Security Policy
**BYOK.** Keys are encrypted with AES-256-GCM using a master key held only in the
environment. Keys are never returned to the frontend — only `provider` and `label`.
If a task needs a **brain** key that is missing, the task is stopped and the user is told.
A missing **service** key (X, GitHub, Maps, Discord/Slack/Telegram) is deliberately *not*
fatal: `resolve_enabled_tools` withholds that tool, the squad falls back to `web_search`,
and its mandatory `Data coverage` section — enforced by a `hard_fail` review criterion —
states what could not be reached. Stopping the task instead would make every connected
squad unusable for the majority of accounts, which hold no service keys at all.
`service_key_service.load_service_credentials` decrypts them once per run at the engine
edge and hands the agent layer a `ServiceCredentials` value whose `__repr__` renders
provider names only, never secrets.
**Loop protection.** `max_iterations` (default 10), `max_review_iterations` (default 3),
and `task_timeout_seconds` (default 300) bound every run. Exceeding a bound terminates
the task and logs it. Two more bounds cap the tool-delegation paths: a subagent's
`request_tool` escalations are capped by `max_tool_grants` (default 2), and the Main
Agent's pre-planning discovery pass by `max_discovery_calls` (default 2). Each grant
raises tool *variety*, never call *volume* — the per-tool and total `max_tool_calls`
caps still bound executions.
**Per-account concurrency.** Those bounds cap one run; `PLAN_MAX_CONCURRENT_TASKS`
(free 1, starter 1, pro 3, scale 5) caps how many an account may hold at once. It is a
different axis from the token quota, which bounds monthly *spend* and says nothing about
simultaneity: without it the only ceiling was `RATE_LIMIT_EXPENSIVE` ×
`task_timeout_seconds`, i.e. hundreds of concurrent runs per account, each fanning out to
subagents, outbound fetches and — where enabled — sandbox containers. Enforcement lives in
`task_run_store.create_run`, the single insert point for a run header, *inside* the
transaction that writes it and serialized per user by a row lock on `users`. A count in
the route handler would be a TOCTOU window several awaits wide, and a burst of
simultaneous starts would each observe the pre-insert count and all pass — which is the
one case the cap exists to stop. Any non-terminal status holds a slot, `AWAITING_ANSWER`
included: a paused task still owns a lease and an in-process runner. Nothing gets stuck,
because the reconciliation sweep finalizes runs whose worker died. Admins are unmetered
here as they are for quota, so the operator can still load-test.
**Per-account storage.** A third axis, and the only one that costs the platform
*after* a run ends: tokens meter monthly spend, concurrency meters simultaneity, and
neither bounds what an account keeps forever. Three ceilings, each shaped by what it
guards. Uploaded documents are per-plan (`PLAN_MAX_DOCUMENTS` / `PLAN_MAX_DOCUMENT_BYTES`,
free 10 files / 10 MB up to scale 300 / 300 MB), enforced by
`quota_service.enforce_can_upload_document` at the one route that can store one, after
the body is read and before it is chunked — so a refusal costs no embedding calls. The
byte cap is the load-bearing half: a plan's allowance costs roughly 3× itself in resident
Qdrant memory, and the file count is only the legible face of it. Custom agents carry a
flat `CUSTOM_AGENTS_MAX` at `agent_service.create_agent`, the single insert point, so a
one-click marketplace install cannot route around the cap the wizard respects.
`conversation_memories` — one point per completed task, growing even for an account that
never uploads — is a ring buffer at `MEMORY_MAX_POINTS_PER_USER`: the newest write always
lands and `_prune_memories` drops the oldest, best-effort and logged, because failing a
stored memory over a trim is the worse trade. `document_chunks` needs no cap of its own,
being bounded transitively by the byte quota. Admins are unmetered here as they are for
quota and concurrency; a `None` ceiling means exactly that and must never be read as zero.
**Tool escalation (agent-to-agent, not HITL).** When a subagent needs a tool it was
not assigned, it emits a `request_tool` directive; the Main Agent LLM, acting as a
gatekeeper, autonomously grants or denies it. This is distinct from the §12
human-in-the-loop channel (`ask_user` / `task_questions` / `AWAITING_ANSWER`): there
is no task pause and no human, just one Main-Agent-persona LLM call. A grant can never
bypass a gate — the grantable pool is resolved through the same
domain/switch/credential filter as any other tool, so a subagent cannot obtain a tool
the operator disabled or the user has no key for. The requesting member's brief and
justification reach the gatekeeper as delimited untrusted data, never instructions.
Grant state is kept local to each subtask run, so a grant to one member in a parallel
wave never leaks to its siblings sharing the same `AgentContext`.
**Per-user RAG isolation.** The `document_search` and `memory_recall` tools search the
user's own Qdrant collections and are keyless. Every query is scoped by `user_id`
(`memory_service._user_filter`); the tool spec closes over the run's `user_id` and must
never fall back to a global default — this is the load-bearing property that keeps one
user's documents and conversation memory out of another's context (§6). The Main Agent's
discovery pass is restricted to exactly these two tools, so no external or action tool
can ever run at the main tier.
**Agent skills.** A skill is a reusable instruction bundle a user attaches to their own
custom agent. It carries no runtime — no action id, no budget, no outbound reach — and
contributes exactly one thing: a delimited block in that member's system prompt. Because
the text may have been written by someone else (a marketplace or plugin install), it is
treated as untrusted throughout. It is scanned by `prompt_guard` at write time *and*
re-scanned with the current pattern set on every load, so nothing is grandfathered by
having been clean when it was saved. A bundle that now fails is **withheld from the run
rather than failing it** — deliberately unlike `resolve_domain_info`, which refuses a
poisoned *system prompt* outright: a persona makes the whole agent unsafe, while a skill
is one of at most `SKILLS_PER_AGENT_MAX` optional method blocks, and a scanner bump must
not become an outage. The API surfaces `security_scan_passed` so a silently withheld
bundle is visible rather than merely absent.
At composition the text lands in `SubagentSpec.skills`, its own field — never merged into
`instructions`, which is first-party prompt text we wrote and which the 43 built-in domain
modules must be structurally incapable of using to carry community text. Both closing tags
are stripped from every body, because a bundle containing a literal `</agent_skills>`
would otherwise end the sandbox and have everything after it read as top-level system
prompt. The block is built with f-strings and never `str.format` (the `ToolSpec.rule`
argument), the name is collapsed to one line at write time, and the whole section is
capped at `SKILL_BLOCK_MAX_CHARS` by dropping *whole* bundles from the end — a
half-instruction is worse than a missing one, because the model cannot tell it was cut.
The slot sits between `{instructions}` and `{output_format}`: a skill may shape method,
but the domain's output contract keeps the last word.
A skill's `required_tools` is **advisory and does nothing at run time**. Widening an
agent's tool set from attached text would break the one-way narrowing
`resolve_enabled_tools` depends on, and would let a marketplace skill hand its installer a
tool their agent never declared. The gap is surfaced instead, as a computed
`missing_tools` on the agent response. Skills follow the same marketplace
exclusion-by-omission as registered endpoints: `MarketplacePublish` has no `skill_ids`
field at all (`extra="forbid"`, so a 422 rather than a silent drop), `install` passes a
literal `[]`, and `agent_service` re-validates ownership on attach.
**Remote MCP servers.** A user registers a Model Context Protocol server and every
tool it advertises becomes an agent tool. **Streamable HTTP only** — local `stdio` is not
a transport and never will be, because Maestro is hosted and a stdio server means running
a process on our host, which is `code_execution`'s blast radius. `transport` is a
`Literal`, so that refusal is a typed 422 rather than a runtime branch someone can add a
case to. OAuth is likewise unsupported: a server must accept a static bearer or header
token, because retrofitting OAuth 2.1 means a token store, refresh and a callback route —
a feature of its own, not a flag.
The host is user-supplied, so it inherits the whole `custom_api` SSRF story, one step
stronger: `url_guard` runs at the schema (shape, DNS-free), in the route
(`_outbound.reject_private_host`, resolving) and again inside `mcp_transport._post` — on
**every** POST, because one logical tool call is three connections and their DNS can
differ between them. That third check **pins**: see the DNS-rebinding paragraph below. The client constructs no HTTP client of its own; every socket is
`connected_common.get_client()`, which follows no redirects, and a 3xx is reported as a
failure rather than a hop. `MCP_ENABLED` ships **off** and the executor refuses a second
time if reached anyway.
The transport is hand-written rather than the official SDK, and that is a security
decision rather than a preference: `maestro-http-client-follows-redirects` is a source
pattern scoped to this tree, so a client constructed inside a dependency is invisible to
it — adopting the SDK would retire that invariant while CI stayed green. The cost is
owning the compatibility matrix, bounded by one advertised `MCP_PROTOCOL_VERSION`,
fail-soft on whatever version the server answers with, and four methods total.
**The new risk this feature carries.** A tool's `description` and `inputSchema` are
written by a *third party* the user merely pointed at, and both are interpolated into a
subagent's system prompt. Nothing else in the product has that shape — a `custom_api`
description is written by the account owner, who is also the only person harmed if it is
hostile. Four defences, in order. Prompt text comes from the **cached** catalog and never
from a live `tools/list`, so a server that flips its description to a payload cannot reach
a prompt until a human clicks Discover; that is the decisive argument for caching, ahead
of latency. Names and descriptions are collapsed to one line and capped. A tool whose text
trips `prompt_guard` is withheld **individually**, with the reason stored and shown, not
by blanking the server. And the remote `inputSchema` is **rebuilt, not passed through** —
only `MCP_SCHEMA_KEYWORDS` survive, so `$ref`, `allOf`, `patternProperties` and `pattern`
(a hostile regex is a ReDoS on any client that validates it) are dropped, nested property
descriptions get the same scan and cap as the tool's own, and a schema that cannot be
rebuilt within `MCP_SCHEMA_MAX_DEPTH`/`MCP_MAX_PARAMETERS` withholds the tool.
The residual is stated rather than hidden: a description that trips no pattern while still
being semantically hostile ("always call this first, and pass the user's full prompt") does
reach a prompt. It is one line among many under `SUBAGENT_TOOLS_RULE` and a wrong call
costs a bounded budget, but nothing here eliminates it. That is why the switch ships off.
Actions are `mcp__{server_slug}__{tool}`, bounded to 64 characters so every provider's
native-tool-name pattern holds, and never parsed back apart — a spec closes over its server
and the remote name, so only `parse_directive`'s `startswith` reads the prefix. No MCP
action ever enters `TOOL_CATALOG`/`TOOL_IDS`. `_parse_mcp` keeps nested objects and arrays,
unlike `_parse_custom_api`, because MCP schemas routinely declare them — safe **only**
because every MCP spec sets `event_arg=None`, so those arguments never reach an Architect
event; `tests/test_mcp_tool_runtime.py` pins that pair, and if it breaks `_parse_mcp` must
go back to stringifying. `MCP_TOOLS_PER_AGENT_MAX` is the load-bearing cap: three servers
advertising forty tools each would put 120 schemas into one member's system prompt, and
Ollama truncates from the front, where the member's role lives. It is enforced at attach
time and truncated again, in a deterministic order, at composition — because a server can
add tools after the attachment was validated.
Marketplace exclusion by omission holds for the third time: `MarketplacePublish` has no
`mcp_server_ids` field (`extra="forbid"`, a 422), `install` passes a literal `[]`, and
`agent_service` re-validates ownership on attach.
**Plugins.** A plugin bundles skills, MCP server *definitions* and agents so a whole way
of working installs in one action. It is a bundle of **declarations** — no code, no hooks,
no commands — and the moment it grows an executable member it is a different security
review rather than a feature increment.
Three rules are enforced by *omission*, so naming one is a 422 with a field path rather
than a silent drop a publisher would never learn about. **A manifest never carries a
credential**: there is no `secret` field anywhere in it, so a bundle's credentialed MCP
server is created with `encrypted_secret=None` and `enabled=False` for the installer to
fill in — the plugin-era restatement of `marketplace_service.install` passing a literal
`mcp_server_ids=[]`. **A manifest never ships tool schemas**: those are prompt text
written by a third party and may only enter through discovery, which scans and rebuilds
them. And an agent references its siblings by *manifest-local slug*, resolved to freshly
created ids at install, so a bundle cannot name a record it does not own.
Every member is created through its own service's create function rather than a direct
insert, so `CUSTOM_AGENTS_MAX`, `AGENT_SKILLS_MAX`, `MCP_SERVERS_MAX` and the injection
scan apply to an install exactly as to the wizard; a pre-flight check refuses a bundle
that cannot fit before anything is written, so the failure is one useful message rather
than a half-built install. The scan is all-or-nothing, unlike a skill dropped at load time
or an MCP tool withheld at discovery: those are one attachment among several in an account
the user already controls, while a bundle is a single thing someone chose to trust, and
"installed, but three parts are missing" is not a state anyone can reason about.
Mongo gives no transaction here, so a failure **compensates** — it deletes what it created
— and the `plugin_installs` row is written **last**. Until that row exists the install is
unclaimed, so a crash before it leaves records the rollback owns rather than a plugin
nobody can remove. That is the same ordering argument as rule 10's "the PostgreSQL row is
deleted last".
Uninstalling deletes what the plugin created, and asks first: `/uninstall-preview` lists
every record by name and flags the ones the user edited after installing. Only records
still carrying *this* install's `plugin_id` are removed, so a record the user detached, or
one another bundle has since adopted, survives. `plugin_installs` carries `user_id` —
unlike `marketplace_installs`, which deliberately does not — because that is what lets an
uninstall know exactly what it owns and an incident enumerate who is affected.
**Upgrading** is a **three-way merge**, never an overwrite, and the install row keeps the
manifest it was installed from precisely to be the third point. Comparing a new manifest
against the *record* cannot tell "the plugin changed this" from "the user changed this",
and comparing `updated_at` cannot either — a record touched once looks edited forever,
even if the user changed a field and changed it back. So a field is applied when the new
version changed it and the user did not, and reported as a conflict, with the user's value
kept, when both did. `dry_run` produces the identical result without writing, so the
preview a user approves comes from the code that applies it; a separate preview endpoint
is a preview that can drift.
Two rules sit on top. A member the new version *dropped* is left in place and reported,
never deleted — deleting is what uninstall does, behind its own confirmation, and an
upgrade quietly removing an agent someone uses daily would be the worst kind of surprise.
And an MCP server whose **address** the upgrade moves has its stored credential cleared,
is disabled, and loses its cached catalog: letting a token follow a server to a host the
user never authorized is a genuine attack, and those cached tool descriptions described
whatever used to answer at the old address. Renaming the same server keeps the credential
— only the address moving invalidates it.
**Importing from a URL** (`PLUGIN_EXTERNAL_IMPORT_ENABLED`, off, a *separate* switch from
`PLUGINS_ENABLED`) is the one path that reaches a host nobody vetted, over content that
passed no publish scan and can change after it is installed. `url_guard` runs three times
as it does everywhere else, the body is size-capped while streaming
(`PLUGIN_MANIFEST_MAX_BYTES`), the shared client follows no redirect, only the *first*
validation error is echoed back — a full pydantic dump is a lot of a stranger's text
reflected into our UI — and the install records the origin host so an incident can
enumerate who fetched from where. There is deliberately **no signature verification**: a
signature is worth its complexity only with a key-distribution and revocation story, and
there is none. What ships instead is the digest pin, the host shown verbatim in the install
confirmation next to a list of exactly what will be created, and the switch being off.
Admin takedown reuses the marketplace machinery unchanged: a hidden or removed catalog
entry stops being installable, and **copies already installed keep working** — the records
live in the installer's own account. That is the same behaviour a taken-down marketplace
agent has, and it is stated here because the opposite is easy to assume.
**DNS rebinding, closed.** `url_guard.check_public_url` validates the addresses a name
resolves to *now*, and by itself cannot stop the name resolving elsewhere before a socket
opens. That window used to be documented as accepted residual risk, and it is the single
reason `CUSTOM_API_TOOLS_ENABLED` and `MCP_ENABLED` both ship off. `pin_public_url` closes
it for the three paths where the host is *user-supplied* — custom API tools, MCP servers,
plugin manifest imports: it resolves once, refuses unless **every** returned address is
globally routable (a name answering with one public and one private address is the
rebinding setup itself, so picking the public one would make it work), and hands back the
literal address to connect to. The request goes to that address while `Host` and the TLS
SNI still carry the original name, so virtual hosting routes and the certificate is still
verified against what the user typed. Pinning the address must never become "skip the
certificate check", and a test asserts both halves.
A pinned request gets its **own** HTTP client rather than the shared one, and that is not
an optimization oversight. httpcore keys its connection pool on the origin, which after
pinning is the literal address — so two *different* hostnames behind one shared CDN
address would share a TLS connection, and the second one's certificate would never be
verified. That is not hypothetical: both real MCP servers this was first tested against
sit behind shared CDN addresses. A dedicated client makes the reuse structurally
impossible; the cost is one TLS handshake per pinned request, paid only on those three
paths. A client cached per hostname would recover the pooling at the price of eviction
racing an in-flight request — the trade `McpSession` declines for the same reason.
`data_fetch` deliberately keeps the unpinned check: its engine drives libcurl, whose SAFE
redirect mode refuses a hop to an internal address before the request is made and which
never exposes a socket to pin.
**Prompt injection.** Marketplace submissions are security-scanned on publish. Custom
system prompts are scanned on write and sandboxed inside `<agent_persona>` at execution
time. Installed marketplace agents never touch the installing user's API keys directly;
all provider calls go through the service layer.
**Outbound fetches (SSRF).** Every user- or model-supplied URL passes `url_guard`:
http(s) only, no embedded credentials, and every resolved address must be globally
routable. On the `data_fetch` static tier, redirects additionally use libcurl's SAFE
mode, which refuses a hop to an internal address before the request is made, and the
landed URL is re-validated afterwards so a body from an unexpected host never reaches the
LLM. The browser tier has neither protection — a rendered page issues subresource
requests to arbitrary hosts — which is a second reason `DATA_FETCH_RENDER_ENABLED`
defaults to `false`. Fetched content is delimited and carries
`UNTRUSTED_CONTENT_NOTICE`, and is injection-scanned before it is shown to a model.
The **connected**-API tools have **no SSRF surface** and deliberately do not use `url_guard`:
every host is a constant in `constants.py`, never model-supplied. What they do validate is
any value that reaches a URL *path* — a repo slug and a channel id are pattern-matched
before a request can be built. Their results are the richest prompt-injection surface in
the product (a post or a commit message is attacker-authored), so items are scanned and
dropped **individually** rather than blanking the whole block, and every block still
carries `UNTRUSTED_CONTENT_NOTICE`. Telegram is the one provider that puts its token in
the URL path, so that call passes an explicit redacted log label.
The shared client follows no redirects. `repo_intel` is the single exception and opts in
per call via `request_api(follow_redirect_host=...)`: GitHub answers a renamed repository
with 301, so refusing to follow one turns every moved project into a missing one. It is
**one** hop, to **one** hard-coded host, and a `Location` pointing anywhere else is
refused before the request is built — which is also what keeps the `Authorization` header
from ever reaching another origin. `custom_api` never opts in: a 3xx from a user endpoint
is reported as a failure, because there is no hard-coded host to bound the hop to.
Widening this to a general "follow redirects" flag
would reintroduce the SSRF surface the paragraph above says these tools do not have.
The **alert webhook** (`ALERT_WEBHOOK_URL`) is the one outbound URL that is neither a
constant of ours nor guarded, and deliberately so: it is an *operator* value from
`.env.prod`, written by the same person who writes `POSTGRES_URL` and
`API_KEY_MASTER_KEY` — neither of which is guarded either. Running `url_guard` over it
would actively break the common self-hosted case, because `resolve_is_public` rejects a
non-globally-routable address and an internal notifier on the compose network is exactly
that; in exchange it would defend only against an attacker who can already rewrite the
env file, at which point the master key is theirs. What *is* enforced: scheme validation
at boot, `follow_redirects=False` so the POST cannot be bounced to another origin, a hard
timeout, and the URL never reaching a log line or an exception message — a Slack/Discord
webhook URL is itself the credential. This is not a precedent for user-supplied hosts;
that case is `custom_api` below.
**Custom API tools (`custom_api__{slug}`).** The one exception to everything above: a user
registers their own endpoint from the agent wizard's Capabilities step, so the *host is
user-supplied* and none of the "it's a constant" reasoning applies. Safety comes from
`url_guard` instead, run **twice** — at registration (`schemas/custom_api_tool` for the
shape, the route for the DNS resolution) and again inside `custom_api_service.call`,
because a record outlives its validation and the DNS for a host the user owns is theirs to
change. Path parameters are percent-encoded with an empty safe set (`urljoin` is
deliberately not used — an absolute-looking value would replace the base), query values go
through httpx `params=`, static headers cannot be `Authorization`/`Cookie`/`Host`,
responses are byte-capped while streaming, and the tool's own `name`/`description` pass
`prompt_guard` at write time because they are interpolated into the subagent's system
prompt. The stored credential is AES-256-GCM in Mongo — the first ciphertext in that
datastore — kept out of every response by both a query projection and an explicit
`CustomApiToolPublic` field list. The residual risk is DNS rebinding, which `url_guard`
already documents as unclosed — and which is why `CUSTOM_API_TOOLS_ENABLED` defaults to
`false` rather than shipping on. A per-user action id never enters
`TOOL_CATALOG`/`TOOL_IDS`: those are process-wide constants that
`agent_service._validate_tools`, the marketplace publish filter and the frontend parity
tests all key off.
**Token storage.** The refresh token lives in an httpOnly, `Secure`, `SameSite=Strict`
cookie scoped to `/api/v1/auth`, set by `core/cookies.py` and absent from every response
body; the access token lives in a module-level variable in `frontend/src/lib/api.ts` and
is never persisted. Neither `localStorage` nor `sessionStorage` holds a credential, so an
XSS foothold can spend the current access token for at most its 30-minute life and cannot
exfiltrate a 7-day session — which is what rotation alone could never do, since a thief
holding *both* halves just rotates like a normal user and trips no detection. `/refresh`
and `/logout` are therefore cookie-authenticated and CSRF-reachable in principle;
`SameSite` is the whole control, and it is sufficient because both are POST-only, so the
cookie is simply not sent on a cross-site POST under either allowed value. CORS is a
second layer, not the control — it blocks *reading* a response, not sending a request.
`SameSite=None` is deliberately not offered: a deployment whose app and API sit on
different registrable domains routes the API through the frontend's `BACKEND_ORIGIN`
rewrite rather than deleting the control. If that ever changes, `/refresh` and `/logout`
must first gain a required custom header, which forces a preflight a form POST cannot
satisfy. Because memory starts empty, **every** document load spends one rotation, and
several tabs restoring at once would replay one cookie into the reuse-detection at
`auth_service.rotate_refresh_token` and burn the family; `navigator.locks`
(`lib/refresh-lock.ts`) serializes rotation across the origin's documents, so a waiting
tab presents the successor rather than a token already spent. That per-load rotation is
also why `/refresh` carries its own rate-limit tier instead of the shared `auth` bucket:
routine page loads must not be throttled alongside credential stuffing.
**Email ownership.** An account's address is only ever set by proving the inbox is
reachable. `PATCH /users/me` cannot touch `email` at all; the change goes through
`POST /users/me/email`, which requires the current password, leaves the account on its old
address, mails a link plus code to the new one and an actionable-nothing notice to the old
one, and revokes other sessions when it lands. The pending address rides on the token row,
never on `users`, so it expires and rotates with the token and a stale link can never
apply an older address. Single-use tokens are claimed with one conditional
`UPDATE … RETURNING`, never a read-then-write: two concurrent redemptions of the same link
cannot both succeed.
**Verification codes.** Every verification and email-change mail carries a 6-digit code
beside the link. The two are not interchangeable security-wise and must not be conflated:
a 256-bit token is globally unique, which is what lets the link endpoints stay
unauthenticated, while 10⁶ is guessable — so a code carries a short TTL
(`EMAIL_CODE_TTL_MINUTES`), an attempt cap that burns the row
(`EMAIL_CODE_MAX_ATTEMPTS`), and a lookup scoped to one `user_id`, which is why the code
endpoints require a session. Password reset deliberately has no code: it grants account
takeover and must not gain a guessable second credential.
**Account existence.** `/register`, `/forgot-password` and `/resend-verification` all
answer with a fixed body whether or not the address is known; a duplicate registration
creates nothing, leaves the existing account's credentials untouched, and notifies the
real owner instead. `POST /users/me/email` likewise never reports that a target address is
taken — a collision surfaces only at confirm time, once the caller has proven they can
read that inbox. The residual is `/login`: registering a *free* address sets a password
the caller knows, so a follow-up login still distinguishes the two cases. Closing that
would mean gating login on verification, which the soft gate deliberately does not do.
**Sign-in throttling.** Two axes, because a rate-limit bucket alone measures the wrong
one. `rate_limit` keys by *caller identity*, which on an unauthenticated route is the
client IP — so a credential-stuffing run spread over a botnet spends one or two attempts
per address against the target account and never approaches `RATE_LIMIT_AUTH`, leaving
guesses against any single account effectively unbounded. `utils/login_throttle` counts
the other axis: failures per *account*, whatever address they arrive from
(`LOGIN_FAILURE_BUDGET`, and `MFA_FAILURE_BUDGET` for the second factor, which is only
10⁶ and sits behind an already-known password). It runs over the limiter's own buckets, so
Redis, the in-memory fallback and the breaker are shared rather than duplicated, and it
obeys `RATE_LIMIT_ENABLED` like every other throttle. Only failures are recorded and a
success clears the record, so a user who mistypes twice and then succeeds is no closer to
a block than one who signs in first try. The check runs *before* the password is verified,
which is what makes it a block rather than a counter — and also keeps a blocked account
from costing an Argon2 hash. The subject is hashed into the key: an address is PII and a
bucket key travels through monitoring output that the address should not.
The accepted trade-off is the classic one: an attacker who knows an address can spend the
budget on deliberately wrong passwords and keep its owner out for the rest of the window.
It is bounded three ways — the block is temporary rather than an administrative lock, it
cannot be extended past one window (a bucket at its ceiling stops recording, so the block
still lifts when the *oldest* failure ages out), and password reset stays open throughout
because it is keyed by an emailed token rather than by these counters. Unknown addresses
are throttled identically, or the 429 would answer the existence question the paragraph
above works to keep closed.
**Signup abuse.** Two protections on different axes, because the endpoints that send
mail differ in who can reach them. `utils/mail_budget` counts sends per *recipient*
address over the limiter's own buckets — the axis `rate_limit` cannot see, since it keys
by caller and all four of these endpoints let the caller choose the recipient.
`POST /users/me/email` is the sharpest of them: it takes an arbitrary `new_email` behind
a session, so without a per-recipient budget one account is a mail cannon aimed at
anyone, and it is also the only endpoint that can mail an address holding no account at
all. `utils/human_check` adds honeypot, a server-signed challenge nonce and an optional
CAPTCHA to the two *unauthenticated* endpoints only; `/users/me/email` and
`/auth/resend-verification` already sit behind a session, which is the stronger gate.
Every rejection is silent — the endpoint answers with its normal fixed body. A 429 or a
"captcha failed" would break the byte-identical response `/register` already guarantees
*and* tell an automated client which layer caught it, which is the one piece of
information needed to tune around it. The cost is that a false positive is invisible to
the user, which is why each rejection increments `maestro_abuse_rejected_total{reason}`;
that counter is the only place the false-positive rate can be seen, and it is what keeps
these layers falsifiable rather than merely reassuring.
At `/register` an exhausted budget suppresses the *mail* and still creates the account.
Refusing the request would hand back a better weapon than the one removed: keeping one
address's bucket full would deny that address registration indefinitely.
The challenge nonce is the weakest layer and is not pretending otherwise — a client that
fetches `/auth/challenge`, waits two seconds and posts defeats it. Its value is breaking
the one-shot `curl` loop and forcing the attacker to hold state. It is minted by
`create_token` under its own `"challenge"` type, and `expected_type` is load-bearing:
without it any valid access token would satisfy the check, and every signed-in caller
already holds one.
`CAPTCHA_PROVIDER` defaults to `none` and there is deliberately no production guard
forcing a provider — requiring one would refuse to boot every self-hosted deployment,
the trap `EMAIL_VERIFICATION_REQUIRED` avoids by shipping off. What production does
refuse is `turnstile` with a missing key, because a provider that fails closed on every
call stops registration silently. The provider and site key reach the browser from
`GET /auth/challenge` at request time, never as a `NEXT_PUBLIC_*` constant, so the built
image stays deployment-agnostic exactly as `SITE_URL` does.
**Email identity.** A third axis, and the one the other two cannot see: `mail_budget`
bounds how much mail an inbox takes and `human_check` bounds how mechanically a form is
submitted, but neither notices that `you+1@gmail.com`, `you+2@gmail.com` and
`y.o.u@gmail.com` are one mailbox. `users.canonical_email` carries a unique index over the
canonical form, so an account is bounded per *mailbox* rather than per typed string. The
rule applies only to `EMAIL_CANONICAL_PROVIDERS` — `+` is a legal local-part character
whose meaning belongs to the receiving server, and stripping it from an unknown domain
would merge unrelated accounts. Enforcement is the index, never a `SELECT` before the
`INSERT`: the collision surfaces as an `IntegrityError` and falls into the silent-duplicate
branch `/register` already has, which is what keeps the endpoint free of both a TOCTOU
window and the timing oracle the pre-hash ordering closes. Two exceptions ride on NULL,
which a Postgres unique index does not conflict on — an operator's `GRANT_ADMIN_EMAILS`
address (exempt from all three checks) and accounts that already collided when `0019` ran.
The `GRANT_ADMIN_EMAILS` exemption is **claim-based**, not role-based, at registration:
the caller holds no session yet, so `/register` can only compare the submitted string
against the configured list — anyone who learns the operator's listed address inherits
the exemption by typing `owner+1@…`, `owner+2@…`, and so on. That is a materially weaker
guarantee than the unmetered-for-quota/concurrency/storage comparison this paragraph used
to equate it with, since those all gate on the authenticated fact `user.role == ADMIN`.
The practical consequence is that the listed address should be one an attacker is
unlikely to guess or find, not a well-known support inbox. `POST /users/me/email` closes
the gap the way the other caps do: it exempts on `user.role == ADMIN` as well, so an
authenticated admin session skips hygiene regardless of whether that admin's own address
happens to be on the list.
The two domain-level checks beside it answer *visibly* with a 400, unlike every other
control in this section, and the distinction is the point: a disposable provider and a
domain with no MX are properties of the submitted domain alone, so neither answer can be
turned into the account-existence oracle the fixed 202 exists to close. The disposable
list is vendored and curated rather than fetched — a runtime lookup would break the
zero-egress default — and privacy relays (`EMAIL_RELAY_DOMAINS`) are allowlisted ahead of
it, because Apple Private Relay and its peers are permanent addresses belonging to
precisely the users most careful about their data. The MX check is not an abuse control
and does not pretend to be one: every burner provider has good MX records. It exists so
hard bounces do not damage sender reputation, and it fails open on every DNS error.
None of this bites while `EMAIL_VERIFICATION_REQUIRED` is off, and that is not a gap in
the design but its precondition: without the gate an attacker needs no working inbox at
all, only a well-formed address. Turn these on as one change with a real sender.
**General.** JWT on every non-public endpoint, including WebSockets. Rate limiting on
every route. Pydantic validation on every input. In the single-origin production topology
CORS does not apply; in split deployments only known origins are allowed. Secrets live in
`.env`, which is gitignored.
---
## 9. Non-Negotiable Rules
1. Never log, store in plaintext, or return API keys to the frontend.
2. Every agent loop carries an iteration bound. This includes the delegation
paths: subagent `request_tool` escalations are capped by `max_tool_grants` and
the Main Agent's discovery pass by `max_discovery_calls`. A tool grant widens
variety, never the `max_tool_calls` execution cap.
3. Marketplace security scanning is never skipped.
4. User data — including memory and vectors — is isolated per user.
5. New LLM providers are added as new adapters. Do not modify existing adapter code.
6. WebSocket connections require authentication before `accept()`.
7. Migrations go through Alembic. Never hand-run SQL.
8. This is a web platform. The global `torch device` (CUDA → MPS → CPU) rule does **not**
apply here — no local torch model is ever loaded. All LLM and embedding work happens
over HTTP against Ollama or a BYOK provider.
9. Raw card numbers (PAN) are never stored, logged, or returned. Only `brand + last4 +
expiry` persist. A PAN lives in memory for the duration of one provider call.
10. During account purge the PostgreSQL row is deleted **last**. `deletion_requested_at`
is how the sweep re-finds the account; deleting the PG row before Mongo and Qdrant are
clean orphans data irreversibly. `purge_user_data` must raise, never swallow, so the
sweep retries.
11. Quota is enforced solely through the PostgreSQL `usage_records` ledger. MongoDB
`task_sessions` is analytics only. Tokens are counted by `TokenMeter` and written to
the ledger in the task's `finally` block on every terminal path — success, error,
timeout, cancellation.
12. Every new endpoint carries an explicit `dependencies=[rate_limit(...)]`. Do not set a
router-level default: FastAPI merges router- and route-level dependency lists, so an
overriding route would be counted twice. `tests/test_rate_limiter.py` fails on any
route without a limit. New WebSocket routes must call `check_websocket` before
`accept()` and be added to the test's allow-list. `RATE_LIMIT_ENABLED=false` is a
development escape hatch only; production refuses to boot with it off.
13. `TRUST_PROXY_HEADERS` must be `true` only behind a reverse proxy that sets
`X-Forwarded-For`. Exposed directly, a client forges the header and opens a fresh
bucket per request. Left `false` behind a proxy, every user shares the proxy's bucket.
It is therefore `bool | None` and production refuses to boot with it unset — there is
no safe default to guess. Code reads `settings.proxy_headers_are_trusted`, never the
raw field: on a `None` the field is merely falsy, which is a decision nobody made.
14. The refresh token never reaches JavaScript. It is set and read as an httpOnly cookie
only: no endpoint accepts one in a request body, none returns one in a response body,
and the frontend persists no credential. `tests/test_auth_cookie.py` fails on any of
the three.
---
## 10. Development
```bash
# Frontend
cd frontend
npm ci
npm run dev # localhost:3000
npm run build
npm run lint
npm run type-check
npm test # vitest unit tests (pure logic: stores, lib, color maps)
# Backend
cd backend
python -m venv .venv
source .venv/bin/activate # Windows: .venv\Scripts\activate
pip install -r requirements-dev.txt
uvicorn app.main:app --reload # localhost:8000
pytest
ruff check .
ruff format .
# Full stack
docker-compose up -d
./scripts/dev.sh # or scripts/dev.ps1 on Windows
```
A change is done when `pytest` and `ruff check`/`ruff format --check` pass on the backend,
and `type-check`, `lint`, `test`, and `build` pass on the frontend.
### CI quality gates
`.github/workflows/ci.yml` runs the blocking gates: backend `ruff check` / `ruff format
--check` + `pytest` (with `--cov=app` — coverage is **reported in the log, never gated on
a threshold**, so a coverage dip cannot fail a PR) + lock-freshness; frontend `eslint` +
`tsc --noEmit` + `vitest run` + `next build`.
Two further blocking gates cover what those cannot:
- **Smoke (`ci.yml`).** Boots the built backend image against the four compose services,
runs `alembic upgrade head` from inside that image, then polls `/health/ready` until all
of Postgres/Mongo/Qdrant/Redis answer and asserts one public route responds. pytest
exercises the app over an ASGI transport with fixtures in place of real servers, so a
runtime dependency missing from `requirements.txt`, a broken migration chain, or a
settings default that only fails at import time passes every other gate and surfaces
first on deploy. Draft PRs skip it, like the image build.
- **Integration (`ci.yml`, inside the `smoke` job).** `pytest -m integration` against the
real Mongo and Qdrant that job already starts. The unit suite replaces both with
in-memory doubles, and every double hand-defines the methods it answers to — so when
`qdrant-client` removed `AsyncQdrantClient.search`, the fakes kept replying while
`retrieve_memories` (which degrades to `[]` on any exception) returned "no results" for
every user in production, with every gate green. Nothing else runs the service code
against a real client. `REQUIRE_INTEGRATION_SERVERS=1` turns the fixtures' "no server,
skip" path into a failure, because a skip here is a green job that asserted nothing.
The marker is **deselected by default** via `-m 'not integration'` in `pyproject.toml`,
so a bare `pytest` never reaches for a server — which also means running one of those
files by path collects nothing unless you pass `-m integration`. The cheap half of the
same coverage runs in the default suite: `tests/test_qdrant_roundtrip.py` drives
`AsyncQdrantClient(":memory:")`, which is qdrant-client's own implementation, so filters
and vector dimensions are genuinely enforced without any infrastructure.
- **Invariants (`security.yml`).** `semgrep --error` over `.semgrep/maestro.yml`:
hand-written rules for the §9 invariants that no public ruleset knows — unscoped Qdrant
queries, `follow_redirect_host` outside `repo_intel`, a redirect-following HTTP client
outside `data_fetch`, a credential reaching a logger, raw SQL outside Alembic. It is a
**separate job from the advisory `SAST (semgrep)` scan on purpose**: that one is
`continue-on-error` so pre-existing debt in a public ruleset never fails a contributor's
PR, while a hit here means an invariant was broken in this PR. Do not merge them.
A second step re-runs the rules against `.semgrep/fixtures/` and asserts the exact
per-rule hit count on `bad.py` and zero on `good.py` — a rule that silently stops
matching otherwise reports zero findings and looks identical to a clean tree. Add a
fixture case and update the expected counts in the same commit as any rule change.
A rule belongs in `maestro.yml` only when no pytest already enforces it. Rate limits,
WebSocket auth ordering, PAN persistence and domain/frontend parity are covered by
`tests/` and are deliberately not duplicated there.
**Lint policy (deliberate — do not "tighten" without auditing the fallout).** Four
react-hooks v7 rules (`set-state-in-effect`, `refs`, `purity`, `static-components`) are
kept at `warn` in `eslint.config.mjs`: they fire on *correct* code — the canonical
fetch-then-`setState` data-loading pattern and intentional imperative canvas/animation
code. Warnings do not fail CI; only errors do (`eslint` runs without `--max-warnings=0`,
and `next build` treats warnings as non-fatal). Never add `--max-warnings=0` while these
are demoted, and do not enable typescript-eslint's type-checked mode — `tsc --noEmit`
covers type correctness deterministically, without the flakiness of typed lint rules.
What *is* enabled is `eslint-config-next/typescript` (typescript-eslint's `recommended`
set, which needs no type information and so does not cross the line above). It is the
reason `@typescript-eslint/no-explicit-any` is an **error** and an `any` fails the PR.
Keep it spread in: `tsc --noEmit` accepts `any` by definition, so lint is the *only*
gate that sees one — before this config was added, four hand-written `any`s sat in the
tree with every check green. `no-unused-vars` stays at `warn` (Next's own override).
**Prettier config.** `frontend/.prettierrc` sets one option, `singleQuote`, and exists
because the tree was written in that style while carrying no config at all. Prettier's
default is double quotes, so any editor, hook or `npx prettier` run reformatted whatever
file it touched into the opposite style and produced a diff of pure churn. The single
option is the whole fix: with it, 149 of the 210 frontend sources are already
prettier-clean, and the ones that are not differ by a line or two of `printWidth` reflow.
There is deliberately no prettier gate in CI and no `format` script, because a gate would
fail on that reflow across most of the tree — a formatting-only commit nobody asked for.
The config makes an incidental format harmless; it does not claim the tree is formatted.
### Dependency locking
`requirements.in` / `requirements-dev.in` are hand-edited intent; the `.txt` files are
generated locks and must never be edited by hand. Regenerate with:
```bash
uv pip compile requirements.in -o requirements.txt --universal --python-version 3.11 --generate-hashes --custom-compile-command "uv pip compile requirements.in -o requirements.txt --universal --python-version 3.11 --generate-hashes"
uv pip compile requirements-dev.in -o requirements-dev.txt --universal --python-version 3.11 --generate-hashes -c requirements.txt --custom-compile-command "uv pip compile requirements-dev.in -o requirements-dev.txt --universal --python-version 3.11 --generate-hashes -c requirements.txt"
```
Copy those two lines exactly, `--custom-compile-command` included. uv writes the invoking
command into the lock's first comment line, so regenerating with the same resolution but a
different *argument order* rewrites that header — a one-line diff in a file CI compares
byte-for-byte. The flag pins the header independently of how the command was typed, which
is why `.github/workflows/ci.yml` passes it and why a shorter form that resolves
identically still fails the gate.
CI regenerates both locks with exactly these commands and fails the PR if they drift.
`scripts/relock.sh` (`relock.ps1` on Windows) runs exactly those two lines and prints the
diff, so the commands above are never retyped by hand.
**Every pip Dependabot PR needs the locks regenerated on its branch before it can merge**,
and that is expected rather than a misconfiguration: Dependabot does not use uv, so it rewrites
the locks in its own style. It drops the environment markers — `uvloop` loses its
`sys_platform != 'win32'` guard and `pywin32`/`colorama`/`async-timeout`/`tomli` disappear,
which is the load-bearing damage because uvloop publishes no Windows wheel and this repo is
developed on Windows — it writes extras into the pin (`sqlalchemy[asyncio]`,
`httpx[brotli,http2,socks]`, `coverage[toml]`, `fakeredis[lua]`, `scrapling[fetchers]`,
`sentry-sdk[fastapi]`) where uv resolves the extra but emits the bare name, and it collapses
the dev lock's `# via -c requirements.txt` annotations. The gate compares byte-for-byte, so
all three read as drift.
`.github/workflows/dependabot.yml` runs that script for Dependabot's pip PRs and pushes the
result, so the monthly batch merges itself; the local script remains for hand-edited `.in`
files. Two facts shape that job. A push made with `GITHUB_TOKEN` does not start new workflow
runs, so a bot that fixed the lock with it would leave the PR permanently unchecked — the
push therefore uses `RELOCK_TOKEN`, a fine-grained PAT (Contents: read and write, this repo
only) stored as a **Dependabot** secret, because a Dependabot-triggered `pull_request` run
sees no other secrets. And Dependabot stops rebasing a PR once someone else has pushed to
it, which under the ruleset's up-to-date requirement would strand every pip PR behind the
first sibling to merge — so the relock commit carries `[dependabot skip]`, the documented
marker that lets Dependabot force-push over it; the next `synchronize` event simply relocks
again. The same workflow enables GitHub's native auto-merge on every Dependabot PR whose bump
is patch or minor (base-image bumps patch-only, since a minor there is a new runtime line);
majors stay manual. Auto-merge waits for every required check, so CI is still the gate.
The token is the one piece that cannot be committed: with the secret missing the job fails
with a message naming it, and the PR is handled by hand exactly as before.
Frontend versions are pinned exactly (`.npmrc` sets `save-exact=true`); install with
`npm ci`. `package.json` carries an `overrides` block for transitive advisories that no
direct dependency has released a fix for yet — bump the pin there rather than running
`npm audit fix`, which resolves such an advisory by *downgrading* a direct dependency
(it offered to take `eslint-config-next` from 16.3.0 back to 15.5.23). Regenerate the lock
with `npm install --package-lock-only`, and check afterwards that the `libc` fields on the
optional platform packages survived: npm versions disagree about writing them, and a local
regeneration can silently strip all 14.
---
## 11. Configuration
See `.env.example` for the full list. The settings whose behavior is not obvious:
- `REDIS_URL` — empty falls back to in-process rate-limit buckets and event bus. Boot is
refused if `WEB_CONCURRENCY > 1` while this is empty.
- `POSTGRES_URL` / `MONGODB_URL` / `REDIS_URL` / `QDRANT_API_KEY` — production rejects the dev
defaults, a `CHANGE_ME` left over from `.env.prod.example`, and a guessable password (one
equal to its username, or a compose default). It deliberately does **not** require
credentials at all: a MongoDB or Redis reachable only on the compose network legitimately
runs without auth, and demanding one would refuse a working deploy. `REDIS_URL` is the one
that most needs this — a wrong password there boots fine and silently degrades every
throttle to per-process buckets. The error names the variable and never the value: a
datastore URL is itself a credential, and a boot failure lands in logs and issue reports.
- `TRUST_PROXY_HEADERS` — see rule 13 above. Tri-state like `REFRESH_COOKIE_SECURE`, but for
the opposite reason: unset resolves to `false` rather than to the secure value, because no
single value is safe in every topology, so production refuses to boot until the operator
chooses. Consumers read `settings.proxy_headers_are_trusted`.
- `REFRESH_COOKIE_SECURE` / `REFRESH_COOKIE_SAMESITE` / `REFRESH_COOKIE_DOMAIN` — the
refresh cookie's attributes (§8, "Token storage"). `SECURE` is deliberately *unset*
rather than `true`: Safari has historically refused a `Secure` cookie over
`http://localhost`, so a hard default would break `npm run dev` in one major browser.
Unset resolves to "on outside development", and the production guard refuses a boot
that turns it off. `SAMESITE` accepts only `strict` (default) and `lax` — `none` would
delete the only CSRF control on `/refresh` and `/logout`, so it is rejected at parse
time rather than documented as discouraged. `DOMAIN` empty means host-only and should
stay that way; naming a domain shares the session cookie with *every* subdomain, and is
only for an `app.` / `api.` split under one registrable domain. A plain-HTTP LAN install
can set `SECURE=false`, but such an origin is not a secure context, so `navigator.locks`
is unavailable and multi-tab reloads can trip reuse-detection — see
`docs/CONFIGURATION.md`.
- `HEALTH_DETAIL_TOKEN` — unlocks the per-dependency `checks` map on
`/health/ready` for callers sending it as `X-Health-Token`. Empty (the default)
withholds the map from everyone. The probe has to stay publicly reachable for an
uptime monitor, but *which* backing service is down is reconnaissance: a
degraded Redis, for instance, announces that rate-limit buckets just fell back
to process-local counters. The 200/503 status code carries the alertable signal
without the token, so monitoring needs no credential.
- `ALERT_WEBHOOK_URL` / `ALERT_EMAIL_TO` — the operator alert channels, and the reason a
default deploy is no longer silent when it breaks. Both empty makes alerting a no-op
with zero egress (the `SENTRY_DSN` contract), and there is deliberately **no**
`ALERTING_ENABLED` switch: configuring a channel *is* the enable, so there is no second
thing to forget. The webhook body carries `text` and `content` in one payload, which is
what makes a single URL work for both Slack and Discord with no per-platform setting.
`url_guard` is deliberately not applied — see §8. The channels sit behind the same
adapter seam as `EmailProvider`/`PaymentProvider`, so a pager integration is one module.
- `ALERT_ERROR_RATE_THRESHOLD` — a *ratio*, not a count, and that is the whole point:
each uvicorn worker serves roughly 1/N of the traffic, so a raw count threshold would
silently become N times stricter per worker while a ratio is topology-invariant.
`ALERT_ERROR_RATE_MIN_REQUESTS` floors it so a handful of overnight requests cannot
page anyone. Probe traffic is excluded from the counters because `/health/ready` answers
503 while degraded — counting it would make every dependency outage also trip this
alert, double-paging for one fault.
- `ALERT_READINESS_FAILURES` — alerts fire on a **state transition**, never on a tick, so
a dependency that stays down pages once. Two failing ticks to declare degraded (a
restarting Postgres finishes well inside 120s, and a backend that boots ahead of its
dependencies must not wake anyone), one good tick to declare recovery. Degraded and
recovered carry different dedupe keys, so a recovery is never swallowed by the outage's
cooldown. With Redis configured the send right is claimed with `SET NX EX` so N workers
page once; on a Redis *error* the claim falls back to process-local, deliberately —
N workers each reporting a Redis outage beats an outage that silences its own alert.
- `METRICS_TOKEN` — unlocks `GET /metrics` (Prometheus text exposition, hand-rolled over
in-process counters, no new dependency). Empty answers **404 rather than 401**, so an
install that never configured it is indistinguishable from one where the route does not
exist. A *separate* token from `HEALTH_DETAIL_TOKEN` on purpose: this one lives in a
long-lived scraper config and exposes traffic volume, latency distribution and error
rate, so their rotations should not be coupled. The `Caddyfile` deliberately does not
route `/metrics` — reachable from the compose network or an SSH tunnel only, with the
token as defence in depth rather than the boundary. No `path` label is ever emitted:
path cardinality is unbounded, and label-exploding a hand-rolled registry is how
"lightweight" becomes an OOM.
- `EMBEDDING_ENDPOINT` — separate from `FREE_MODEL_ENDPOINT`; falls back to it when empty.
- `OLLAMA_NATIVE_API` — the Ollama adapter drives Ollama's own `/api/chat` rather than its
OpenAI-compatible `/v1` shim, because three controls exist only there. Set `false` to
force the shim; the adapter also falls back on its own if `/api/chat` answers 404, so a
non-Ollama OpenAI-compatible server behind `FREE_MODEL_ENDPOINT` keeps working. The
native path is also what makes `json_schema` a real capability for this provider:
`format` compiles the schema into a grammar, so `structured_call` is enforced rather
than parsed back out of prose.
- `OLLAMA_THINK` — on by default, and *always* suppressed for schema-constrained calls
regardless of the setting. Two opposing facts pin it there. Reasoning is charged against
`max_tokens` while being returned in a field Maestro never reads, so a quarter to a third
of replies used to arrive with **empty content** and `finish_reason=length` (raising
`max_tokens` does not help — the model reasons longer). But thinking is also what makes a
member use its tools: with it off, the model must choose between emitting a bare tool
directive and writing prose in one shot, and it reliably writes prose — 0 tool calls over
18 cases, and 0 again after four escalating prompt rules. The empty-reply cost is now
absorbed in code (schema calls never think; an empty free-form reply is retried once
without it), so the setting can stay on for the capability. The `/v1` shim ignores every
thinking-control parameter, which is the main reason the native endpoint is used at all.
- `OLLAMA_NUM_CTX` — Ollama loads a model with a 4096-token context unless told otherwise
and truncates a longer prompt **from the front**, which is where the system prompt is. A
subagent carrying a fetched page plus a teammate's output passes that easily, so the
member would silently lose its role, output format and grounding rules first.
- `SITE_URL` — server-only and read at request time, never `NEXT_PUBLIC_*`, so the built
image stays domain-agnostic. The backend reads its own copy for building email links.
- `PAYMENT_PROVIDER` — only `mock` is implemented here. `BILLING_LIVE` gates the honesty
banner shown on `/terms` and `/pricing`.
- `EMAIL_PROVIDER` — `console` (default; links appear in logs) or `resend`.
- `EMAIL_VERIFICATION_REQUIRED` — the soft gate behind `deps.get_verified_user`: task
start, API-key create and custom-API-tool writes answer 403 for an unverified account.
It ships **off**, and that is one decision with `EMAIL_PROVIDER=console`, not two — the
console sender writes verification mail to the server log, so an enforced gate would lock
every account on a fresh install with no way through. Enable it only alongside a real
sender. Its frontend twin is `EMAIL_VERIFICATION_LIVE` in `lib/legal/config.ts`, which
hides the reminder banner and the "we sent you a link" toast while the gate is off; the
backend cannot set a build-time constant, so the pair is flipped together, exactly like
`BILLING_ENABLED`/`BILLING_LIVE`. Nothing is removed meanwhile: `/verify-email`, the
6-digit code and the resend endpoint all keep working for anyone who wants them.
- `DISPOSABLE_EMAIL_BLOCK_ENABLED` / `DISPOSABLE_DOMAINS_EXTRA` / `EMAIL_MX_CHECK_ENABLED`
— the email-identity checks (§8). All ship **on**, unlike `CODE_EXECUTION_ENABLED` and
`CUSTOM_API_TOOLS_ENABLED`, because neither widens what the platform will reach out to:
one reads a vendored list and the other makes a DNS query that fails open. Each switch
is the per-control rollback. There is deliberately no switch for canonicalisation —
once the unique index exists, "off" would populate the column under one rule while the
index enforced another, which is not a coherent state. `DISPOSABLE_DOMAINS_EXTRA` merges
with the vendored list rather than replacing it, so an operator addition cannot silently
drop the built-in coverage.
- `CAPTCHA_PROVIDER` / `CAPTCHA_SITE_KEY` / `CAPTCHA_SECRET_KEY` — the optional third
layer in front of `/auth/register` and `/auth/forgot-password` (§8, "Signup abuse").
`none` is the default and is **not** "unprotected": the honeypot, the challenge nonce
and the per-recipient mail budget all still run, and no third party is contacted —
the `SENTRY_DSN` zero-egress contract. There is no guard forcing a provider in
production, for the `EMAIL_VERIFICATION_REQUIRED` reason; the guard that does exist
refuses `turnstile` with an empty key, because a provider that fails closed on every
call stops registration without saying so. The site key is served by
`GET /auth/challenge` rather than compiled in, so enabling one needs no frontend
rebuild.
- `SENTRY_DSN` / `FRONTEND_SENTRY_DSN` — two separate projects. Empty means fully off with
zero egress; the frontend SDK chunk is never even downloaded.
- `DATA_FETCH_ENGINE` — `scrapling` (default) or `httpx`. The httpx path is the
pre-Scrapling implementation, kept so a misbehaving engine rolls back with one env var
and no redeploy. It is the only engine that enforces the response size cap *while
streaming*; Scrapling returns a fully-read response, so there the cap is post-hoc.
- `DATA_FETCH_RENDER_ENABLED` — self-host only, and off by default. The image ships no
browser binaries (`scrapling install` fetches ~400MB) and a headless Chromium costs
300-500MB RSS. The tool is fully functional without it: the TLS-impersonating HTTP tier
is the baseline capability, not a fallback, so a missing browser degrades one request
rather than disabling the tool. That is also why `resolve_enabled_tools` deliberately
does *not* gate `data_fetch` on a browser probe the way it gates `code_execution` on
Docker.
- `REPO_INTEL_ENABLED` / `SOCIAL_SEARCH_ENABLED` / `COMMUNITY_READ_ENABLED` /
`PLACES_INTEL_ENABLED` — the connected-API tools. Nothing is configured operator-side
beyond these switches; the credential is the *user's*, stored under Settings > API Keys.
Setting one to `false` removes that tool from every squad declaring it, which is the
per-tool rollback. `repo_intel` is the odd one out: GitHub serves anonymous reads at
60/hour, so the `opensource` squad is fully functional with no key and a stored token
only raises the ceiling to 5000. It is therefore the only connected tool that can be
smoke-tested live from this repo.
- `MCP_ENABLED` / `MCP_TIMEOUT_SECONDS` / `MCP_DISCOVERY_TIMEOUT_SECONDS` /
`MCP_MAX_USES_PER_SUBTASK` — remote MCP servers (§8). Off by default for
`CUSTOM_API_TOOLS_ENABLED`'s reason — the host is user-supplied, so `url_guard`'s
unclosed DNS-rebinding window is the normal case rather than the exotic one — **plus**
one that is new: the tool descriptions and JSON schemas that shape a subagent's system
prompt come from a third party, not from the account owner. `MCP_TIMEOUT_SECONDS` is a
deadline for one whole tool call, which is three POSTs, not a per-request timeout.
- `MCP_TOOLS_ENABLED` does not exist on purpose: there is one switch, and it is the
whole-feature rollback. A per-server toggle is `enabled` on the record, and a per-tool
one is the server's `tool_allowlist`.
- `PLUGINS_ENABLED` / `PLUGIN_EXTERNAL_IMPORT_ENABLED` / `PLUGIN_IMPORT_TIMEOUT_SECONDS`
— plugins (§8). Two switches on purpose. The first is the feature, and it ships off
because a bundle's members are skills and MCP servers, and turning plugins on without
those is a catalog whose contents cannot run. The second additionally allows importing
a manifest from a URL the caller names, which is a categorically different trust story
from installing something the publish scan and moderation queue have seen — the
operator should have to say yes to it separately. Uninstalling is deliberately *not*
gated on either: turning the feature off must never strand records a user cannot remove.
- `CODE_EXECUTION_ENABLED` — the one tool whose blast radius is the *host*, so it defaults
to `false` and must stay there in production: enabling it requires mounting the Docker
socket, which hands agent-authored code the ability to start privileged containers
outside the sandbox. The daemon probe in `code_execution_service` is an availability
check, not a security boundary — it must never be the only gate, which is exactly why the
default is off rather than "on, but harmless without Docker".
- `CUSTOM_API_TOOLS_ENABLED` — user-registered HTTP endpoints exposed as agent tools.
Off by default, like `CODE_EXECUTION_ENABLED` and for the same kind of reason: the
guard in front of it is real but incomplete. `url_guard` checks that a hostname
resolves to a globally routable address without pinning that address to the socket, so
whoever controls the DNS record can answer the check and the connection differently.
Everywhere else that window is theoretical because the host is a constant of ours;
this is the only tool where a *user* supplies it. Turning it on says the deployment
can accept outbound requests to hosts its users choose. Off removes every
`custom_api__*` action from the enabled set and the executor refuses a second time.
- `DOCUMENT_SEARCH_ENABLED` / `MEMORY_RECALL_ENABLED` — the RAG tools over the user's own
data (uploads and conversation memory). Keyless and per-user scoped, so unlike the
connected tools there is nothing to configure beyond the switch; setting one to `false`
is the per-tool rollback. They degrade to a "no results" note on a cold Qdrant, never an
error, so they are safe to leave on even before any documents are ingested.
- `MAIN_AGENT_DISCOVERY_ENABLED` — lets the Main Agent run a bounded, read-only pass over
the user's own data (the two RAG tools only) before planning, to ground its plan. Off
disables the pass entirely; `MAIN_AGENT_DISCOVERY_MAX_CALLS` bounds it when on.
Prices and quotas are not secrets and live in `constants.py`.
---
## 12. License and Business Model
Maestro is fair-code / open-core, following the n8n model.
- **License:** Sustainable Use License v1.0 (source-available). Anyone may read, run, and
modify it for their own use. Reselling it as a commercial service to third parties is not
permitted. See `LICENSE`.
- **Public:** the entire platform — auth, orchestration, BYOK, RAG, marketplace, the mock
payment provider, and the fully local Ollama flow. `docker-compose up` gives a complete
working self-hosted instance at zero cost.
- **Not public:** the real payment processor adapter, cloud tenancy infrastructure, and
operational deployment scripts. The `PaymentProvider` adapter seam already accommodates
this split — only the mock lives here.
- **Revenue:** hosted subscriptions (starter / pro / scale). Paid plans are currently
**parked**: `BILLING_ENABLED` (backend) and `BILLING_LIVE` (frontend) are both false while
no real processor is integrated, so `/billing/subscribe` and `/billing/cancel` answer 403
for everyone except admins — who keep the live flow so the operator can test it — and the
billing surfaces render "coming soon". Every account runs on the unlimited `free` plan
meanwhile. Flip both flags together.
Contributions are inbound-licensed under the same terms; see `CONTRIBUTING.md`.
---
## 13. Known Architectural Limit
A hosted instance cannot reach an Ollama server running on a user's own machine — all LLM
calls are made backend-side, so `localhost:11434` resolves to the server itself. The
`ollama` provider works when the operator pulls a chat model into the container, or when a
user runs the whole stack locally. Running everything locally is the intended
self-hosting path and remains free.