api/AGENTS.md
Rama main · réplica de lectura del repositorio original.
# API conventions (OSS + EE)
Scope: everything under `api/`. This file loads when you work in the API. The repo-wide
root conventions live in `/AGENTS.md`.
## Environment config conventions
- For API configuration, add new environment variables to `api/oss/src/utils/env.py` and
consume them via the shared `env` object.
- Avoid calling `os.getenv(...)` directly in feature code when the value is part of
application config.
- Avoid local imports inside helper functions for configuration lookup; prefer
module-level imports unless there is a proven circular dependency.
## Formatting
Run `ruff format` then `ruff check --fix` within the API folder (from the repo root:
`ruff format` then `ruff check`). Fix all errors before committing.
## Code comments
Keep AI-generated in-code comments minimal. Comment only the non-obvious *why* (a subtle invariant, a workaround, a pool/concurrency hazard), never the *what* the code already states. Prefer one terse line over a multi-line block; do not narrate steps, restate the code, or leave TODO-style prose. If a comment is needed to explain *what* a block does, rename or extract instead.
## API Architecture Patterns
Use this section for all new work.
### API repo map
- `api/oss/src/*` is the OSS baseline API (new + legacy coexist here).
- `api/ee/src/*` is the EE extension API (billing, organizations, workspace, meters,
subscriptions, throttling).
- `api/entrypoints/*` is the composition root (dependency wiring and route mounting).
OSS and EE relationship:
- OSS app is assembled first in `api/entrypoints/routers.py`.
- When `is_ee()` is true, EE extends that app via:
- `ee.extend_main(app)` for extra routers/features.
- `ee.extend_app_schema(app)` for OpenAPI/security metadata.
- EE is additive over OSS, not a separate API architecture.
RBAC / access enforcement:
- RBAC enforcement is an OSS feature and is always-on in both editions. The
permission vocabulary, default role catalog, and enforcement service live in
`api/oss/src/core/access/`. Handlers call `check_action_access(...)`
unconditionally; it decides for the request's principal (the `RequestAccess` the auth
middleware sets), and there is no edition gate around the check itself.
- Roles apply on every edition and plan; no plan lets a member pass a check their role
lacks. The plan decides only which roles an invite or a role change may set:
`AccessService.offered_roles` is the one reader, through `RbacEntitlementInterface`
(`oss/src/core/access/interfaces.py`); `entrypoints/access.py` wires
`PlanRbacEntitlement` (EE, `Flag.RBAC`) or `NoRbacEntitlement` (OSS). With the
entitlement (Business, Enterprise) every catalog role but `owner` is offered and the
floor role is `viewer`; without it (OSS, Hobby, Pro) `admin` and `member` are offered
and the floor role is `member`. The same entitlement limits organization credentials
to selected projects (`ee/src/core/secrets/limits.py`). OSS never imports `ee.*` at
module top.
- Custom roles are an EE feature: `AGENTA_ACCESS_ROLES` / `AGENTA_ACCESS_ROLES_OVERLAY`
are parsed/applied in `api/ee/src/core/access/permissions/role_overrides.py` and
ignored in OSS (which enforces the code-default catalog).
Project membership and access:
- Membership is decided only by `MembershipsService.resolve` (`oss/src/core/memberships/`) and its uncached form `read_membership` (same policy and DAO read; use it only where another cache keeps the outcome, today the Bearer workspace-only and no-parameter branches). No other membership reader or policy.
OSS: every organization member is a member of every project, with the organization role.
EE: a member needs an organization row and a `project_members` row; the project row's
role applies. The edition policy is picked only in `entrypoints/access.py`.
- Membership writes go only through the four `MembershipsService` writers:
`set_member_role`, `remove_member`, `create_project` (seeds every organization member in
the same transaction) and `transfer_ownership`. Each writer clears the access caches with
`invalidate_access_caches` after the commit. An admin tool that writes membership rows
directly must call `invalidate_access_caches` too.
- `check_action_access` delegates to `AccessService.check_request`. Every process that
checks access calls `bootstrap_access()` at startup (the API and both workers do).
- No new code reads `workspace_members` or `project_members` to decide access.
- No deployed code reads or writes `workspaces`, `workspace_members`,
`projects.workspace_id` or `meters.workspace_id`, and the runtime ORM maps none of them
(workspace removal step 2b; step 3 drops them). The API answers the organization id where
a workspace id was, and a legacy workspace id is an unknown id. Migrations that need
these objects use frozen models, never `oss.src.models.db_models`.
`unit/memberships/test_workspace_storage_unread.py` pins it.
- A hot-path membership read is one SQL statement (`fetch_membership_row`,
`list_membership_rows`). `integration/memberships/test_hot_path_query_counts_postgres.py`
pins the counts.
- Revocation takes effect at the next decision. It can take up to `AGENTA_CACHE_TTL`
(300 s) in two known cases: the Redis invalidation fails, or a read that started before
the mutation committed refills the old entry after the invalidation.
Access layer invariants (run principal):
- Every run has an explicit `RunEnvelope` (principal, authority, run context, sender,
approver). Routers get it from `request_run_envelope(request)`; durable work builds it
with `stored_work_envelope`, `child_envelope` or `resumed_envelope`. No code builds a
human principal from a bare user id.
- There is no implicit human fallback. `check_action_access` needs request access and
raises without it (`RequestAccessMissing`) or for another user
(`RequestAccessMismatch`). Code that checks another human, or runs without request
access (callbacks exempt from the auth middleware, workers), calls
`AccessService.check_human` or `decide` explicitly.
- `AccessService.decide` runs on every call. Nothing caches a decision. Only the
principal record (`fetch_principal`), the membership (`resolve_membership`) and the
project's organization are cached, and their writers invalidate them.
- Cached access DTOs give every `Optional` field a `None` default: the cache drops
`None` fields when it serializes.
- Durable work (queued inputs, continuations, interactions, schedules, subscriptions,
evaluation runs, channel turns, child runs) stores its envelope or principal and
rechecks it with `require` (or `decide`) before it runs. A refusal is a terminal
settlement (failed delivery and paused trigger, failed evaluation run, obsolete
command, cancelled interaction), never a retry.
- Mint a run credential (`sign_secret_token`) only after the decision allows. The minted
envelope copies the caller's; it never upgrades a principal.
- New agent-visible failures use `AgentError` (`access_denied`, `missing_principal`,
`missing_envelope`).
- Access logs carry ids and kinds only, never a token, a header, raw claims or an
envelope.
- `integration/access/test_run_principal_query_counts_postgres.py` pins the hot paths at
zero access-decision statements on warm caches.
Access layer invariants (agent and session sharing):
- Every agent or session access calls `AccessService.decide` with a `ResourceRef`, and
lists pass a read scope in SQL. List queries take a required read scope; there is no
unrestricted scope.
- Every point where a run starts or resumes calls `RunAdmission` (`admit`, `admit_turn` or
`admit_request`) with the stored or request envelope.
- `AccessService` requires a resource policy; every composition root passes
`SharingService`.
- Never treat a null owner as "not migrated".
- Audit events carry identifiers, never credential values.
Organization-scoped decisions and organization credentials (M5):
- An organization subscription login or custom secret (`ORGANIZATION_RUNTIME_KINDS`, never in
`ORGANIZATION_GATEWAY_KINDS`) are the only organization values that leave the API: only to
the run credential, from `GET /secrets/{slug}?scope=organization`, through
`organization_secret_for_runtime`. The runner uses a login like a project login; the
sandbox gets a custom secret as a variable.
- Organization-scoped actions are decided only by `decide_organization`
(`oss/src/core/access/organizations.py`). It allows a browser-session human acting as
itself, with no run envelope, whose organization role is `owner` or `admin`. It reads
the member row uncached and nothing else, so the project organization-owner shortcut
never applies.
- An agent names its connection by scope and slug (`project`, the default, or
`organization`). Nothing is inherited or overridden: a project slug reads project rows
only, no connection means the project default, and an organization connection resolves
only by `VaultService.get_organization_connection` (or `get_available_organization_secret_by_id`
for a bound reference), in the gateway namespace `organization`. `SecretsResolver` is built
per plane; only the LLM plane answers an organization connection. The `/secrets/` routes
serve both owners. The caller states the owner (`scope`, default `project`); with
`organization` the route runs `decide_organization`, otherwise it is the project route as
before. `PUT /secrets/{id}/projects` addresses organization rows only. `GET /secrets/?include_organization=true` adds the organization
rows available to the project without values (also for the `secret-resolve` grant).
- A project's vault lives in ONE cache entry (`effective_secrets`, keyed on the full
project id), filled by one statement (`SecretsDAO.load_effective`), with the project's
rows and the available organization rows. Do not add a second cache read for
organization rows. Project writes invalidate it with the project-wide invalidation;
organization writes delete each project's entry after the commit.
- Project and organization rows are two slug spaces; no write checks one against the
other. Secrets DAO errors are raised `from None` as domain errors: the SQL parameters of
a secrets write hold the plaintext value.
- `integration/secrets/test_organization_secrets_query_counts_postgres.py` pins the vault
and gateway hot paths in five cache modes (warm, cold leader, cold follower, cache
disabled, Redis failing). It needs a reachable Redis: run it with a throwaway one
(`REDIS_URI_VOLATILE=redis://127.0.0.1:<port>/0`), because the stack's Redis is not
published.
Primary references:
- `api/entrypoints/routers.py`
- `api/ee/src/main.py`
### Where to add new API code
- Add new domain features in:
- `api/oss/src/apis/fastapi/<domain>/`
- `api/oss/src/core/<domain>/`
- `api/oss/src/dbs/postgres/<domain>/`
- Avoid adding net-new features to legacy paths:
- `api/oss/src/routers/*`
- `api/oss/src/services/*`
### Standard domain folder structure
For a new domain, follow this shape:
- API layer: `api/oss/src/apis/fastapi/<domain>/`
- `router.py`: route registration + handlers
- `models.py`: request/response schemas
- `utils.py`: parsing/merge/normalization helpers
- Core layer: `api/oss/src/core/<domain>/`
- `dtos.py` or `types.py`: domain data contracts
- `interfaces.py`: DAO/service contracts when needed
- `service.py`: business orchestration
- DB layer: `api/oss/src/dbs/postgres/<domain>/`
- `dbes.py`: SQLAlchemy entities
- `dbas.py`: shared mixins (when needed)
- `dao.py`: Postgres implementation
- `mappings.py`: DTO <-> DBE mapping
Example to copy:
- `api/oss/src/apis/fastapi/workflows/`
- `api/oss/src/core/workflows/`
- `api/oss/src/dbs/postgres/workflows/`
### Database schema migrations (which alembic chain)
New tables/columns are alembic revisions. The legacy `core/` and `tracing/` chains are
**parked** (frozen at `park00000000`) — never add revisions there. Author against the
active chains:
- **Shared schema** (the default — runs in both OSS and EE): add the revision **once** in
`oss/databases/postgres/migrations/core_oss/` (or `tracing_oss/`). EE ships and runs the
`oss/` tree, so do **not** copy it into the EE tree.
- **EE-only schema** (genuinely edition-divergent): `ee/databases/postgres/migrations/core_ee/`
(or `tracing_ee/`).
Full model (version tables, fresh-install/upgrade/OSS→EE-switch flows, FK and replay-skew
rules): `docs/designs/oss-ee-convergence/migration-chains-and-edition-switch.md`. For
alembic mechanics and docker commands, see the parked `core/README.md` — but always run
those commands from the appropriate active chain working directory (`core_oss`/`tracing_oss`
or `core_ee`/`tracing_ee`), not the parked `core/`/`tracing/` dirs.
### Migration gate
A deploy runs `alembic upgrade head` in a Helm pre-upgrade hook, before any new pod
starts, and the api runs two replicas with `maxUnavailable=0`. The old code therefore
runs against the new schema for the length of every rollout. A failed deploy rolls the
application back. It never rolls the database back.
CI scans the revisions a pull request adds and fails the **Migration gate** check on a
destructive or write-blocking operation in the `upgrade()` path. `downgrade()` is not
scanned, because the hook never runs it.
- **Fails the check**: `drop_table`, `drop_column`, `rename_table`, a column rename or
type change through `alter_column`, a tightening to `nullable=False` with no
`server_default`, `add_column` of a `NOT NULL` column with no `server_default`, and the
raw-SQL equivalents in `op.execute(...)` or `conn.execute(...)`.
- **Reported, does not fail**: `create_index` without `postgresql_concurrently=True`,
`drop_index`, `drop_constraint`, and `add_column` of a `NOT NULL` column that does
carry a `server_default`.
- **Not flagged**: anything scoped to a table the same revision creates, and a
constraint or index dropped and created again under the same name.
To clear a flagged revision, prefer the expand and contract pattern. Add the new shape in
one release. Write to both shapes until every pod runs the new code. Remove the old shape
in a later release.
If the change must land as it is, ask a reviewer to apply the
`migration:destructive-approved` label. The check then reports the findings and passes.
The label records that someone accepted the rollout risk.
Run the gate locally before you push:
```bash
uv run api/scripts/migration_gate.py <revision.py> # the files you added
uv run api/scripts/migration_gate.py --all # audit the whole tree
```
### Layering and dependency direction
Required direction:
- Router -> Service -> DAO Interface -> DAO Implementation -> DB
Rules:
- Core services depend on interfaces (`*DAOInterface`), not concrete DB implementations.
- Wire concrete dependencies in `api/entrypoints/*` only.
- Keep DTO/DBE mapping in `dbs/postgres/*/mappings.py`.
- Do not return DBE objects from router/service contracts.
- As much as possible, define appropriate service exceptions (avoid leaking database
exceptions).
### Endpoint design conventions
Use consistent endpoint shapes across domains:
- `POST /query` for filtering/search with payload support.
- `POST /{id}/archive` and `POST /{id}/unarchive` for lifecycle state transitions.
- For revisioned resources, expose:
- `/revisions/retrieve`
- `/revisions/commit`
- `/revisions/log`
Request/response conventions:
- Define explicit request/response models in `models.py`.
- Response envelopes should include `count` plus payload (`item`/`items` style).
- Set explicit `operation_id` on routes.
Query conventions:
- Parse params via `Depends(...)` and optionally parse request body JSON.
- Merge params + body into one query object in `utils.py`.
- Use cursor pagination via `Windowing`, not page-number pagination.
References:
- `api/oss/src/apis/fastapi/workflows/router.py`
- `api/oss/src/apis/fastapi/workflows/utils.py`
- `api/oss/src/apis/fastapi/shared/utils.py`
- `api/oss/src/dbs/postgres/shared/utils.py`
### Git-style Artifact/Variant/Revision pattern
When a resource needs commit/history semantics, use the shared Git pattern instead of
inventing a custom one.
Use this pattern when you need:
- revision history and auditability
- latest vs specific revision retrieval
- revision logs
- variant forks/lineage
Core contracts:
- `api/oss/src/core/git/interfaces.py`
- `api/oss/src/core/git/dtos.py`
Postgres implementation:
- `api/oss/src/dbs/postgres/git/dao.py`
- `api/oss/src/dbs/postgres/git/dbas.py`
Domain DBE examples:
- `api/oss/src/dbs/postgres/workflows/dbes.py`
- `api/oss/src/dbs/postgres/queries/dbes.py`
- `api/oss/src/dbs/postgres/testsets/dbes.py`
Service examples:
- `api/oss/src/core/workflows/service.py`
- `api/oss/src/core/queries/service.py`
- `api/oss/src/core/testsets/service.py`
Reuse pattern example:
- Evaluators reuse workflow persistence and can preserve IDs through
`workflow_id=evaluator_id`.
- Reference: `api/oss/src/core/evaluators/service.py`
### Scope, lifecycle, and archival rules
- Always enforce tenant scope (`project_id` minimum) in DAO reads and writes.
- For revisioned entities, prefer archive/unarchive (`deleted_at`, `deleted_by_id`) over
hard deletes.
- Respect `include_archived` in query paths.
References:
- `api/oss/src/dbs/postgres/shared/dbas.py`
- `api/oss/src/dbs/postgres/git/dao.py`
### Migration and compatibility rules
Migration should preserve compatibility while moving to new APIs.
- Keep old and new routes running in parallel until migration is complete.
- New stack commonly ships under `/*` while old endpoints remain mounted.
- Prefer data compatibility adapters over breaking payload changes.
- Preserve old IDs/shape when continuity is required.
- If old storage temporarily carries new payload shape, mark it explicitly.
Concrete examples:
- Dual mounting and deprecations: `api/entrypoints/routers.py`
- Legacy app storage marker (`WORKFLOW_MARKER_KEY`):
`api/oss/src/core/applications/service.py`
- Legacy dedup key normalization (`__dedup_id__` <-> `testcase_dedup_id`):
`api/oss/src/apis/fastapi/testsets/router.py`
### Router and function style conventions
Router style:
- Register routes inside router class `__init__` using `self.router.add_api_route(...)`.
- Use `@intercept_exceptions()` at the route boundary.
- Use `@suppress_exceptions(...)` only for controlled defaults.
Function signature style:
- Prefer keyword-only parameters using `*`.
- Use grouped sections in signatures/calls with `#` separators for readability.
Example:
```python
async def create_workflow(
self,
*,
project_id: UUID,
user_id: UUID,
#
workflow_create: WorkflowCreate,
#
workflow_id: Optional[UUID] = None,
) -> Optional[Workflow]:
...
```
### Domain-level exceptions
1. **Define domain exceptions in the core layer** (`core/{domain}/types.py` or
`core/{domain}/dtos.py`) — never raise `HTTPException` from services or DAOs.
2. **Catch domain exceptions at the API boundary** — in the router or via a decorator —
and convert them to HTTP responses.
3. **Use a base exception per domain** so callers can catch broadly or narrowly.
4. **Include structured context** (not just a message string) so the router can build
rich HTTP error responses.
**Agent-actionable errors:** Expected domain failures exposed to an agent MUST use
`{code, message, retryable, next_step?, details?}`. `code` is a stable lower-snake-case
semantic cause; `message` is a concise diagnosis. `retryable` means the unchanged request
may safely succeed if repeated later. `retryable: true` requires one imperative
`next_step`; `next_step` may also accompany `retryable: false` when the caller must change
its request or state. Put every error-specific field in `details`; do not add other
top-level fields or nested `reason` envelopes. A code has one public HTTP status, but
handler-mode domain failures return HTTP 200 with `STATUS_CODE_ERROR` and the same
envelope in `ToolResult.content`. Core code raises typed exceptions; API and tool-handler
boundaries select the transport.
**Scope:** the envelope covers failures the PLATFORM authors. Third-party tool output
passes through as-is, symmetrically on success and failure, so a gateway tool's own failure
reason still reaches the model: that reason is what lets it correct a bad argument. On the
`/tools/call` seam this means every platform HANDLER emits the envelope, while the gateway
and workflow-tool arms carry their upstream's shape. Both arrive as errors. Do not read the
rule as "every `STATUS_CODE_ERROR` carries an envelope", and do not delete a consumer's
non-envelope path on that assumption.
The distinction that is easy to get wrong: `retryable` is about replaying the SAME request
unchanged, not about whether the caller has a way forward. A payload the caller must
correct is `retryable: false` WITH a `next_step`. Marking it retryable tells a small model
to send the identical bytes again, which cannot work and burns a turn.
**Example 1 — Folder exceptions (best example of the full pattern):**
Definition in `api/oss/src/core/folders/types.py`:
```python
class FolderNameInvalid(Exception):
def __init__(self, message: str = "Folder name contains invalid characters."):
self.message = message
super().__init__(message)
class FolderPathConflict(Exception):
def __init__(self, message: str = "A folder with this path already exists."):
self.message = message
super().__init__(message)
```
Raised in service `api/oss/src/core/folders/service.py`:
```python
def _validate_folder_name(name: Optional[str]) -> None:
if not name or not fullmatch(r"[\w -]+", name):
raise FolderNameInvalid()
```
Caught in router via decorator `api/oss/src/apis/fastapi/folders/router.py`:
```python
def handle_folder_exceptions():
def decorator(func):
@wraps(func)
async def wrapper(*args, **kwargs):
try:
return await func(*args, **kwargs)
except FolderNameInvalid as e:
raise FolderNameInvalidException(message=e.message) from e
except FolderPathConflict as e:
raise FolderPathConflictException(message=e.message) from e
return wrapper
return decorator
```
**Example 2 — Filtering exception (inline catch in router):**
Definition in `api/oss/src/core/tracing/dtos.py`:
```python
class FilteringException(Exception):
pass
```
Caught in `api/oss/src/apis/fastapi/tracing/router.py`:
```python
except FilteringException as e:
raise HTTPException(status_code=400, detail=str(e)) from e
```
**Example 3 — EE organization exceptions (base class hierarchy):**
Definition in `api/ee/src/core/organizations/exceptions.py`:
```python
class OrganizationError(Exception):
"""Base exception for organization-related errors."""
pass
class OrganizationSlugConflictError(OrganizationError):
def __init__(self, slug: str, message: str = None):
self.slug = slug
self.message = message or f"Organization slug '{slug}' is already in use."
super().__init__(self.message)
class OrganizationNotFoundError(OrganizationError):
def __init__(self, organization_id: str, message: str = None):
self.organization_id = organization_id
self.message = message or f"Organization '{organization_id}' not found."
super().__init__(self.message)
```
Anti-patterns:
- Do NOT return error dicts like `{"_error": True, "status_code": 502, "detail": "..."}`
from services or clients.
- Do NOT raise `HTTPException` from core services — that couples the domain to HTTP.
- Do NOT use bare `Exception` or `ValueError` for domain errors when a typed exception
would be clearer.
### Typed DTO returns from services and clients
1. **Service methods must return typed DTOs** (Pydantic `BaseModel` subclasses), not raw
dicts, tuples, or `Any`.
2. **HTTP clients should return a response DTO**, not
`Tuple[Optional[Any], Optional[str]]`.
3. **Define DTOs in `core/{domain}/dtos.py`** alongside the domain's other data contracts.
4. **Use `Optional[DTO]` for missing entities**, `List[DTO]` for collections.
**Example — Workflow service returns typed DTOs:**
```python
# core/workflows/dtos.py
class Workflow(Artifact):
pass
class WorkflowVariant(Variant):
pass
# core/workflows/service.py
async def create_workflow(self, *, project_id, user_id, workflow_create) -> Optional[Workflow]:
artifact = await self.workflows_dao.create_artifact(...)
if not artifact:
return None
return Workflow(**artifact.model_dump(mode="json"))
```
**Example — HTTP client returns a DTO instead of a raw tuple:**
```python
# Instead of:
async def invoke(...) -> Tuple[Optional[Any], Optional[str]]:
return data, trace_id # untyped
# Do:
class InvokeResponse(BaseModel):
data: Any
trace_id: Optional[str] = None
async def invoke(...) -> InvokeResponse:
return InvokeResponse(data=data, trace_id=trace_id)
```
Anti-patterns:
- Do NOT return raw dicts from service methods.
- Do NOT return tuples like `(data, trace_id)` — use a named DTO.
- Do NOT use `Dict[str, Any]` as a return type when you can define a proper model.
### Migration seams (do not copy for net-new code)
These exist during transition but should not be copied into new implementations:
- Core importing API routers/models directly
- `api/oss/src/core/invocations/service.py`
- `api/oss/src/core/annotations/service.py`
- Core importing legacy `db_manager` in new modules
- `api/oss/src/core/workflows/service.py`
- `api/oss/src/core/testsets/service.py`
- `api/oss/src/core/evaluations/service.py`
Preferred fix for new work:
- introduce/extend core interfaces and adapters
- keep strict layer boundaries