docs/conformance.md

Rama main · réplica de lectura del repositorio original.

# Running the conformance suites

The conformance framework (`openspec/changes/archive/2026-10-17-add-conformance-framework`) holds the rebuild to its
contracts. Scenario files in `mods/conformance/scenarios` script one side of a contract (session
protocol, Twilio protocol, voice SDK, media-control, Autopilot) and say what the other side must
send back. Any implementation, in any language, passes or fails the same files. The format is in
`mods/conformance/SCENARIOS.md`; the fake providers are in `mods/conformance/PROVIDERS.md`.

## Layers

| Layer | Plays | Under test | Needs |
| --- | --- | --- | --- |
| Roomr node | Real Roomr nodes in roomr-fake (`@fonoster/roomr/fake`), fake STT, TTS and LLM | AppBridge, the apiserver's placement | NATS, roomr-fake (downloaded by `npm install`) |
| Real stack | Caller bot through Routr, real AppBridge and node | Everything together | Routr, node, appbridge (nightly, not yet automated: conformance task 2.5 and 3.11) |

## Quick start

NATS is the message bus for AppBridge and Roomr node suites. Use a throwaway server and never
point tests at a shared one.

```sh
docker run -d --name conformance-nats -p 4222:4222 nats:2.11.8
export CONFORMANCE_NATS_URL=nats://127.0.0.1:4222 LOGS_LEVEL=none
npm install && npm run build
```

### Fake node (no containers other than NATS)

```sh
npx mocha --exit --timeout 30000 \
  mods/appbridge/test/conformance.test.ts \
  mods/appbridge/test/integration.test.ts \
  mods/appbridge/test/crash.test.ts \
  mods/autopilot/test/session/fullstack.integration.test.ts
```

`mods/conformance/test` checks every scenario file against the schema and the OpenSpec specs, the
runner, the fakes and the SDK scenarios, and runs without NATS (the media-control and Roomr gRPC
suites need it):

```sh
npx mocha 'mods/conformance/test/**/*.test.ts' --exit --timeout 30000
```

### The Roomr node

The node itself is tested in [fonoster/roomr](https://github.com/fonoster/roomr): unit tests,
end-to-end tests against NATS and Routr, and a contract suite run against the real node and
roomr-fake. Fonoster's release gate requires the pinned Roomr release to have passed them
(`qa-proofs.yaml`). Here, the suites run real nodes in roomr-fake: `npm install` downloads its
binary for the pinned `@fonoster/roomr` (set `ROOMR_FAKE_BIN` to use another one; without a
binary, the `fonoster/roomr-fake` image runs with host networking).

### Real stack

Real SIP calls need Routr and the whole platform up (the dev container's `npm run dev`, or
`ops/rebuild/stack up`). The caller bot
(`sipcaller` in the `fonoster/roomr` image) places the calls; the nightly end-to-end suite is still
to be wired (conformance tasks 2.5 and 3.11). Until then, `mods/appbridge/test/gonode.test.ts`
(AppBridge through a Roomr node) is the closest automated check.

### Timing, load, turn-taking and Pipecat (nightly)

```sh
CONFORMANCE_TIMING_RUNS=200 CONFORMANCE_REPORT_DIR=reports npx mocha --timeout 0 mods/appbridge/test/timing.test.ts
npx tsx mods/appbridge/scripts/scale.ts --calls 100 --seconds 60
npx tsx mods/appbridge/scripts/replay.ts
python -m pip install -r python/pipecat-golden/requirements.txt
PIPECAT_PYTHON=python npx mocha --timeout 60000 mods/appbridge/test/pipecat.test.ts
```

The pass bars (T1 at 50 ms p95, and so on) hold only on dedicated runners. Shared CI runners
report trends.

## Python SDK and runner

```sh
cd python
uv venv .venv --python 3.12
uv pip install --python .venv/bin/python -e fonoster-conformance -e "fonoster-voice[test]"
.venv/bin/python -m pytest fonoster-conformance/tests fonoster-voice/tests
```

## Running the scenarios from a third-party SDK

Your SDK is under test when the scenario's `role` is `appbridge`: the runner plays AppBridge and dials
your app. For each `voice-sdk` scenario you ship one fixture app per name in
`mods/conformance/scenarios/voice-sdk/FIXTURES.md` (for example `say-hello`, `gather-echo`) and the
runner starts it.

Use the Python package as a library, or port its runner (`runner.py`) to your language:

```python
import asyncio
from fonoster_conformance import FakeAppBridge, load_scenarios, run_scenario
from fonoster.voice import VoiceServer   # replace with your SDK

SECRET = "sdk-conformance"

async def main():
    for scenario in load_scenarios("voice-sdk"):
        fixture = FIXTURES[scenario["fixture"]]   # the app for this fixture, written in your SDK
        server = await VoiceServer(port=0, host="127.0.0.1", signing_secret=SECRET).listen(fixture)
        try:
            appbridge = FakeAppBridge(signing_secret=SECRET)
            result = await run_scenario(scenario, lambda: appbridge.dial(f"ws://127.0.0.1:{server.port}/"))
            print(scenario["name"], result.status, result.reason or "")
        finally:
            await server.close()

asyncio.run(main())
```

(`python/fonoster-voice/tests/test_conformance.py` is the complete working example, including every
fixture.) Rules for a runner in a new language:

1. Load scenarios from the published `@fonoster/conformance` package of the same version and
   validate each against `schema/scenario.schema.json`. Do not keep private copies. The Python
   runner reads the directory from `FONOSTER_CONFORMANCE_DIR`, defaulting to this repository.
2. Implement the matcher exactly as in `SCENARIOS.md`.
3. Fail, never skip silently, anything the runner does not understand.
4. Sign the connection with `X-Fonoster-Timestamp` and `X-Fonoster-Signature`.

## CI

`.github/workflows/conformance.yaml`:

| Job | When | Runs |
| --- | --- | --- |
| `suites` | Pull requests touching a contract, nightly, manual | Go node, appbridge, SDK, media-control and Autopilot suites on NATS |
| `python` | Same | Python SDK and runner |
| `nightly` | Nightly and manual (`workflow_dispatch`) | Timing, load, turn-taking replay and the Pipecat golden test; reports are uploaded as the `conformance-reports` artifact |

Live provider tests run by hand with your own provider keys (never put them in the repository):
`DEEPGRAM_API_KEY=… ELEVENLABS_API_KEY=… npm run live-check -w @fonoster/appbridge`. Still to add:
dedicated runners for the nightly job (task 4.2).