CONTRIBUTING.md
Rama main · réplica de lectura del repositorio original.
# Contributing to Maple
This guide covers local development for the full Maple stack: app services, Postgres,
ClickHouse, and the OTel collector pipeline.
For a single-binary experience without Docker, see [docs/local-mode.md](docs/local-mode.md)
(`maple start`).
## Prerequisites
Install the pinned toolchain (recommended via [mise](https://mise.en.dev)):
| Tool | Used for |
| ------ | ------------------------------------ |
| Bun | JS/TS apps, scripts, tests |
| Node | Some scripts (e.g. mobile) |
| Rust | `apps/ingest` (OTLP gateway) |
| Docker | Postgres, ClickHouse, OTel collector |
First-time bootstrap:
```bash
curl https://mise.run | sh
echo 'eval "$(mise activate zsh)"' >> ~/.zshrc # or bash/fish equivalent
mise trust
mise run setup # installs tools, bun deps, copies .env.example → .env.local, portless CA
```
Without mise, the same steps manually:
```bash
bun install
cp .env.example .env.local
npx portless trust # only needed if you use `bun dev` / portless HTTPS URLs
```
## Architecture (local dev)
```text
Browser → web (3471) → api (3472) → ClickHouse (8123)
↓
OTLP clients → ingest (3473/3474) → OTel collector (4318) → ClickHouse
↓
Postgres (5499) app state (issues, keys, dashboards, …)
```
The development Docker stack (`docker-compose.development.yml`) runs the data plane.
Application processes run on the host (see [Running application services](#running-application-services)).
## Environment file
All services read secrets and overrides from **`.env.local`** at the repo root (gitignored).
Start from the template:
```bash
cp .env.example .env.local
```
### Generate required secrets
Several services share these keys. Generate them once and paste into `.env.local`:
```bash
# AES-256-GCM key for encrypting private ingest keys at rest (must be base64 of exactly 32 bytes)
openssl rand -base64 32
# HMAC key for ingest-key lookup hashes (any non-empty secret; hex is convenient)
openssl rand -hex 32
```
Set the outputs as:
```dotenv
MAPLE_INGEST_KEY_ENCRYPTION_KEY=<output of first command>
MAPLE_INGEST_KEY_LOOKUP_HMAC_KEY=<output of second command>
```
### Recommended baseline for ClickHouse local dev
These values align with `docker-compose.development.yml` and self-hosted auth:
```dotenv
# Warehouse — route API queries to local ClickHouse instead of Tinybird Cloud
CLICKHOUSE_URL=http://localhost:8123
CLICKHOUSE_PROVIDER=clickhouse
CLICKHOUSE_USER=maple
CLICKHOUSE_PASSWORD=maple
CLICKHOUSE_DATABASE=default
# App database: nothing to set. `alchemy dev` binds MAPLE_DB to the docker Postgres on 5499;
# see db:migrate:local below.
# Auth — self-hosted mode (no Clerk account required)
MAPLE_AUTH_MODE=self_hosted
MAPLE_ROOT_PASSWORD=change-me
MAPLE_DEFAULT_ORG_ID=default
# Ingest — forward OTLP to the local collector (which writes ClickHouse)
INGEST_WRITE_MODE=forward
INGEST_FORWARD_OTLP_ENDPOINT=http://127.0.0.1:4318
INGEST_PORT=3474
# Single-tenant ingest keys (no Postgres key store required)
MAPLE_SELF_HOSTED_MODE=single_tenant
MAPLE_ORG_ID_OVERRIDE=default
# Internal service auth (must match across api and scraper)
INTERNAL_SERVICE_TOKEN=dev-internal-service-token
SD_INTERNAL_TOKEN=dev-internal-service-token
# Web → API/ingest URLs when running raw ports (not portless)
VITE_API_BASE_URL=http://localhost:3472
VITE_MAPLE_AUTH_MODE=self_hosted
```
`TINYBIRD_HOST` and `TINYBIRD_TOKEN` remain in `.env.example` because the API validates
them at startup even when `CLICKHOUSE_URL` is set. Placeholder values from the template are
fine for this stack.
> **Clerk mode:** set `MAPLE_AUTH_MODE=clerk` and provide `CLERK_SECRET_KEY`,
> `CLERK_PUBLISHABLE_KEY`, plus the matching `VITE_*` overrides. Test credentials for the
> hosted dev org are documented in [CLAUDE.md](CLAUDE.md).
## Start infrastructure (Docker)
From the repo root:
```bash
docker compose -f docker-compose.development.yml up -d
```
This starts:
| Service | Ports | Purpose |
| ---------------- | ----------------------- | -------------------------------------------- |
| `postgres` | `5499 → 5432` | App DB for `alchemy dev` (Hyperdrive origin) |
| `clickhouse` | `8123`, `9000` | Telemetry warehouse |
| `otel-collector` | `4317`, `4318`, `13133` | OTLP ingest → ClickHouse via mapleexporter |
The collector reads `.env.local` and uses `MAPLE_CLICKHOUSE_PASSWORD=maple` from compose
overrides. Ensure ClickHouse credentials in `.env.local` match (`maple` / `maple`).
Wait for health checks, then apply schema and migrations:
```bash
# Postgres (Drizzle migrations for app state)
bun run db:migrate:local
# ClickHouse (telemetry tables + materialized views)
bun run --cwd packages/clickhouse-cli start apply \
--url=http://localhost:8123 \
--user=maple \
--password=maple \
--database=default
```
Stop infrastructure:
```bash
docker compose -f docker-compose.development.yml down
```
Postgres data persists in the `postgres-data` volume; ClickHouse in `clickhouse-data`.
Remove volumes with `down -v` for a clean slate.
## Running application services
`bun dev` runs every app below as one `alchemy dev` stack behind `https://<app>.localhost`,
and `bun dev api web` runs a subset — see [All at once](#all-at-once). The per-app sections
describe what each one needs and how to run it alone on its raw port.
### 1. Workers: API, alerting, electric-sync
```bash
bun dev api alerting electric-sync
```
The three Cloudflare Workers are served by alchemy's local runtime from the same
`alchemy.run.ts` that deploys them, reachable at `https://api.localhost`,
`https://alerting.localhost` and `https://electric-sync.localhost` through the portless
proxy (branch-prefixed in a linked worktree). Ports are sticky per app and never need
writing down; `bun dev` prints them.
The variables below are the API's:
| Variable | Required | Notes |
| ----------------------------------------------------------------- | --------------------- | ---------------------------------------------------------------------------------------------------- |
| `MAPLE_INGEST_KEY_ENCRYPTION_KEY` | yes | `openssl rand -base64 32` |
| `MAPLE_INGEST_KEY_LOOKUP_HMAC_KEY` | yes | `openssl rand -hex 32` |
| `MAPLE_AUTH_MODE` | yes | `self_hosted` or `clerk` |
| `MAPLE_ROOT_PASSWORD` | yes\* | \*Required when `MAPLE_AUTH_MODE=self_hosted` |
| `MAPLE_DEFAULT_ORG_ID` | yes | Default `default`; must match ingest org override |
| `TINYBIRD_HOST` | yes | Placeholder OK when using `CLICKHOUSE_URL` |
| `TINYBIRD_TOKEN` | yes | Placeholder OK when using `CLICKHOUSE_URL` |
| `TINYBIRD_SIGNING_KEY` / `TINYBIRD_WORKSPACE_ID` | with Tinybird raw SQL | Explicit JWT signing configuration; never derived from `TINYBIRD_TOKEN` |
| `TINYBIRD_RAW_SQL_JWT_RPS_LIMIT` | optional | Positive integer; Tinybird-enforced raw-SQL ceiling with a separate bucket per org |
| `CLICKHOUSE_URL` | recommended | `http://localhost:8123` for local ClickHouse stack |
| `CLICKHOUSE_PROVIDER` | optional | `tinybird` (default) or `clickhouse`; set `clickhouse` for env-level vanilla/self-managed ClickHouse |
| `CLICKHOUSE_USER` / `CLICKHOUSE_PASSWORD` / `CLICKHOUSE_DATABASE` | with CH | Match docker-compose (`maple` / `maple` / `default`) |
| `INTERNAL_SERVICE_TOKEN` | recommended | Internal MCP tool calls; shared with the scraper |
| `SD_INTERNAL_TOKEN` | optional | Prometheus scraper internal API auth |
| `CLERK_*` | clerk mode | See `.env.example` |
Loads env via `alchemy dev --env-file .env.local`; a changed variable needs a restart.
Requires docker Postgres running (`bun run db:migrate:local`).
### 2. Web (`apps/web`)
```bash
bun dev web # in the stack: https://web.localhost
bun --filter=@maple/web dev # alone, raw port
```
Default raw URL: `http://localhost:3471`
| Variable | Required | Notes |
| ----------------------- | ---------- | ------------------------------------------------------------- |
| `VITE_API_BASE_URL` | yes | `http://localhost:3472` (or portless `https://api.localhost`) |
| `VITE_MAPLE_AUTH_MODE` | yes | Mirror `MAPLE_AUTH_MODE` |
| `VITE_INGEST_URL` | optional | Defaults derived; set if ingest port differs |
| `VITE_CLERK_*` | clerk mode | Publishable key + sign-in URLs |
| `VITE_MAPLE_INGEST_KEY` | optional | Browser self-telemetry via ingest |
Sign in (self-hosted): use the org/user you create with `MAPLE_ROOT_PASSWORD`.
Clerk dev login: see [CLAUDE.md](CLAUDE.md).
### 3. Ingest (`apps/ingest`)
```bash
bun dev ingest # in the stack: https://ingest.localhost
bun --filter=@maple/ingest dev # alone, raw port
```
Default port: `3473` (`INGEST_PORT` / `PORT` override). `.env.example` uses `3474`; pick one
port and keep `VITE_*` / `MAPLE_INGEST_PUBLIC_URL` consistent.
| Variable | Required | Notes |
| ---------------------------------- | -------------- | ----------------------------------------------------------------------- |
| `INGEST_FORWARD_OTLP_ENDPOINT` | yes | `http://127.0.0.1:4318` for local collector |
| `INGEST_WRITE_MODE` | recommended | `forward` for ClickHouse stack (default `tinybird`) |
| `MAPLE_INTERNAL_ORG_ID` | yes | Org the gateway files its OWN telemetry under; no default |
| `MAPLE_INGEST_KEY_ENCRYPTION_KEY` | yes\* | \*Required for postgres key store / ClickHouse direct path |
| `MAPLE_INGEST_KEY_DECRYPTION_KEYS` | no | Older keys kept for decryption during a rotation; same value as API |
| `MAPLE_INGEST_KEY_LOOKUP_HMAC_KEY` | yes | Same value as API |
| `MAPLE_SELF_HOSTED_MODE` | recommended | `single_tenant` → static key store (no DB) |
| `MAPLE_ORG_ID_OVERRIDE` | with static | Must match `MAPLE_DEFAULT_ORG_ID` |
| `MAPLE_PG_URL` | postgres store | `postgres://maple:maple@localhost:5499/maple` if not using static store |
| `TINYBIRD_HOST` / `TINYBIRD_TOKEN` | tinybird mode | When `INGEST_WRITE_MODE=tinybird` or `dual` |
| `INGEST_PORT` | optional | Default from port / env |
| `INGEST_REQUIRE_TLS` | optional | `false` locally |
Sources `../../.env.local` automatically. Requires Rust toolchain.
### 4. Scraper (`apps/scraper`)
Optional — polls the API for Prometheus targets and forwards metrics through ingest.
```bash
cd apps/scraper && bun dev
```
Default health port: `3475`
| Variable | Required | Notes |
| --------------------- | -------- | ------------------------------------------------------------- |
| `MAPLE_API_URL` | optional | Default `http://127.0.0.1:3472` |
| `MAPLE_INGEST_URL` | optional | Default `http://127.0.0.1:3474` |
| `SD_INTERNAL_TOKEN` | optional | Default `maple-sd-dev-token`; match API's `SD_INTERNAL_TOKEN` |
| `SCRAPER_CONCURRENCY` | optional | Default `10` |
| `PORT` | optional | Health endpoint, default `3475` |
Loads `../../.env.local` via `bun --env-file`.
### 5. AI chat and triage (`apps/ai`)
`/chat` and `/investigations/*` are forwarded by `apps/api` to the `apps/ai` Worker over a service
binding: the `ChatSession` Durable Object there owns each transcript and runs the agent turn.
Run both under `alchemy dev` (`bun dev api ai`), because a plain `bun` process has no Durable
Object namespace.
The agent runs on OpenRouter by default and needs `OPENROUTER_API_KEY`. Set
`MAPLE_LLM_PROVIDER=workers-ai` to run it on the Workers AI `AI` binding instead, which needs a
Cloudflare account with Workers AI through an alchemy profile (`bunx alchemy login`). The default
model ids live in `apps/ai/src/platform/Llm.ts`. Override them per provider with
`MAPLE_TRIAGE_MODEL_OPENROUTER` or `MAPLE_TRIAGE_MODEL_WORKERS_AI`.
> **`MAPLE_ORG_ID_OVERRIDE` breaks the investigation chat.** It pins every API request to
> one org, but the browser addresses the session as `<clerk org>:<tab>`. With the
> override set to anything other than your Clerk organization's id, the page watches a
> session the API never seeded, and the transcript stays empty forever. Unset it,
> or set it to your Clerk org id, when working on chat.
In Clerk mode the pin admits a user who is not a member of the pinned org only when
`MAPLE_ORG_ID_OVERRIDE_ADMITS_NONMEMBERS=true` and `MAPLE_ENVIRONMENT=development`. `bun dev`
sets the opt-in for you; set it yourself when you run the API outside the dev stack.
When an autonomous investigation fails to start, the `investigations.error` column names it:
| `error` value | Cause |
| ---------------------------------------------------------------- | ---------------------------------------------------------------------------------- |
| `agent_unavailable: …` | No `ChatSession` binding: `apps/ai` isn't running under `alchemy dev` with the API |
| `start_failed: a turn is already running for this investigation` | The session is busy; retry |
| `diagnosis_timeout: …` | The turn ran but never called `submit_diagnosis` within 15 minutes |
### All at once
```bash
bun dev # everything
bun dev api web # a subset
```
One `alchemy dev` stack: the Workers (api, alerting, electric-sync) on alchemy's local
runtime, everything else (web, landing, local-ui, ingest, scraper) as child processes of the
same stack, each behind `https://<app>.localhost` (branch-prefixed in a linked worktree).
Ctrl-C stops all of it. Logs share the one terminal. See `docs/infra.md` for how the ports
and routes are handed out.
## Verify the stack
1. Open `http://localhost:3471` (or `https://web.localhost` with portless).
2. Sign in (self-hosted root password or Clerk test user).
3. Send test OTLP to `http://localhost:3474/v1/traces` (or your ingest port) with a
well-formed ingest key, or rely on static-key mode (any `maple_pk_*` key resolves to
`MAPLE_ORG_ID_OVERRIDE`).
4. Confirm data in the traces/logs UI after the collector batch flushes (~seconds).
Health checks:
- OTel collector: `http://localhost:13133`
- Scraper: `http://localhost:3475/health`
- API: `http://localhost:3472/health`
## Tests and quality
```bash
bun run test # Vitest via turbo, not `bun test`
bun typecheck
bun run format # oxfmt + oxlint --fix
```
Vitest uses embedded PGlite — no Docker Postgres required for unit tests.
## Troubleshooting
| Symptom | Check |
| ------------------------ | ------------------------------------------------------------------------------------------ |
| API fails on startup | Missing `MAPLE_INGEST_KEY_*`, `MAPLE_ROOT_PASSWORD`, or `TINYBIRD_*` |
| Ingest 401 on OTLP | Static key store: use `maple_pk_…` prefix; postgres store: seed keys via API |
| Empty traces UI | ClickHouse schema applied? Collector running? `INGEST_WRITE_MODE=forward`? |
| API DB errors | `docker compose … up -d postgres` + `bun run db:migrate:local` |
| Ingest/API HMAC mismatch | `MAPLE_INGEST_KEY_LOOKUP_HMAC_KEY` must be identical; compare startup fingerprints in logs |
| Chat MCP tools fail | `INTERNAL_SERVICE_TOKEN` mismatch or API not running |
## Further reading
- [docs/persistence.md](docs/persistence.md) — Postgres migrations
- [docs/self-hosted-clickhouse.md](docs/self-hosted-clickhouse.md) — ClickHouse BYO + schema
- [docs/local-mode.md](docs/local-mode.md) — single-binary local mode
- [.env.example](.env.example): full env reference with optional integrations (GitHub App, Autumn, email)