README.md
Rama main · réplica de lectura del repositorio original.
# AssemblyAI Self-Hosted Services
Docker Compose stacks for running AssemblyAI transcription services on your own
infrastructure. The stacks are organized by **service** (the API you talk to)
and, within each service, by **model**.
## Choosing a service and model
| Service | Directory | API | Models | GPU requirement |
|---------|-----------|-----|--------|-----------------|
| **Streaming** | [`streaming/`](streaming/) | WebSocket, real-time | Universal English + Multilingual | NVIDIA T4+ per ASR container |
| **Streaming** | [`streaming/`](streaming/) | WebSocket, real-time | Universal-3.6 Pro | NVIDIA L40S, RTX PRO 4500, or RTX PRO 6000 (preferred) |
| **Sync** | [`sync/`](sync/) | Synchronous HTTP, full-file | Universal-3.6 Pro | NVIDIA L40S, RTX PRO 4500, or RTX PRO 6000 (preferred) |
| **Diarization** | [`diarization/`](diarization/) | Synchronous HTTP, full-file | Speaker diarization | NVIDIA T4 or newer, 16 GB+ VRAM |
- **Streaming** transcribes a live audio stream over a WebSocket connection. One
stack serves multiple models; the client selects the model per session. See
[`streaming/docker/README.md`](streaming/docker/README.md).
- **Sync** transcribes a complete file in a single HTTP request/response (audio
≤ 20 min by default). It is self-contained — a single GPU container plus the
license-and-usage-proxy, no load balancer. See [`sync/docker/README.md`](sync/docker/README.md).
- **Diarization** adds speaker labels to a transcript (e.g. the sync response).
See [`diarization/docker/README.md`](diarization/docker/README.md).
[`examples/`](examples/) combines services: [`examples/sync-diarization/`](examples/sync-diarization/)
runs sync and diarization with one license-and-usage-proxy.
Each stack is self-contained under `<service>/docker/`: its compose file(s),
`.env.example`, example client, and `README.md` live together. Run compose
commands from inside that directory.
Each stack can also run on serverless GPUs instead of hardware you manage. Every
stack is a self-contained Modal App deployed with a single `modal deploy` — see
[`sync/modal/`](sync/modal/) and
[`streaming/modal/`](streaming/modal/).
## Repository layout
```
.
├── diarization/
│ └── docker/ # compose stack (speaker diarization)
├── examples/
│ └── sync-diarization/ # sync + diarization with one proxy
├── streaming/
│ ├── docker/ # compose stacks (Universal English/Multilingual, Universal-3.6 Pro)
│ └── modal/ # serverless-GPU packages
└── sync/
├── docker/ # compose stack (Universal-3.6 Pro)
└── modal/ # serverless-GPU package
```
## Prerequisites (all services)
1. **AssemblyAI license** valid for the self-hosted product (the `license.jwt` file).
2. **Docker & Docker Compose**.
3. **GPU support**: NVIDIA Container Toolkit for GPU-enabled containers.
4. **AWS access**: credentials that can pull images from AssemblyAI's ECR.
### GPU runtime setup
All GPU images ship CUDA 13 runtime libraries, so the host needs an NVIDIA
driver that supports CUDA 13 (R580 series or newer). Verify the driver:
```bash
nvidia-smi
```
Install the [NVIDIA Container Toolkit](https://docs.nvidia.com/datacenter/cloud-native/container-toolkit/install-guide.html),
then verify the Docker runtime has GPU access:
```bash
docker run --rm --gpus all nvidia/cuda:13.0.1-base-ubuntu24.04 nvidia-smi
```
### ECR authentication
```bash
aws ecr get-login-password --region us-west-2 \
| docker login --username AWS --password-stdin 344839248844.dkr.ecr.us-west-2.amazonaws.com
```
### License file
Place your AssemblyAI `license.jwt` in the directory of the stack you are
running (`streaming/docker/`, `sync/docker/` or `diarization/docker/`), or point the `LICENSE_FILE_PATH`
environment variable in that stack's compose file at your license file's location.
If your license uses usage-based billing, also set `USAGE_TRACKING_API_KEY` to any API key from your
AssemblyAI dashboard: in `.env` for Docker, or in the `aai-license` secret for Modal. Without it, the
license-and-usage-proxy shuts down at startup (see [Usage reporting](#usage-reporting)).
## Shared component: license-and-usage-proxy
Every service runs the **license-and-usage-proxy** alongside it. It validates
your license and, for usage-based licenses, reports usage to AssemblyAI. Its
configuration and operational behavior are the same regardless of service.
### Usage reporting
The proxy supports two billing modes based on your license:
- **Flat billing** — usage tracking is disabled; no extra configuration needed.
- **Usage-based billing** — the proxy reports usage to AssemblyAI's usage
tracker. Set `USAGE_TRACKING_API_KEY` (any key from the AssemblyAI dashboard)
for the `license-and-usage-proxy` service.
**Behavior with usage-based billing:**
- At startup the proxy validates connectivity by registering with
https://usage-tracker.assemblyai.com. If validation fails, the proxy shuts down.
- Usage is batched and reported every few seconds, with automatic retries.
- If https://usage-tracker.assemblyai.com stays unreachable and all retries fail
(after 5–60 minutes), the proxy terminates itself as a fail-safe to protect
usage-data integrity. Your orchestrator should replace the container.
- If the in-memory usage queue exceeds 1000 items, the proxy logs a warning
suggesting you upscale.
### License status endpoint
`GET /v1/status` reports the live license-validation state:
```json
{
"state": "Ready | Connected | TrustBased | Failed",
"last_successful_checkin": "2025-01-01T12:00:00.000000Z",
"trust_expiration": "2025-01-05T12:00:00.000000Z"
}
```
- `Ready` — initial state before any validation has occurred.
- `Connected` — last validation check succeeded.
- `TrustBased` — last validation failed but is within the trust-window grace
period; services keep serving.
- `Failed` — last validation failed and the trust window expired; serving
containers shut down.
`last_successful_checkin` and `trust_expiration` are ISO 8601 timestamps (null
until the first successful validation).
### Production recommendations (license-and-usage-proxy)
- **Deployment**: gradual rollouts; alert on service restarts.
- **Resources**: 1 CPU + ≥ 2 GB RAM per container; prefer more small containers
over fewer large ones.
- **Monitoring**: alert on `/v1/status` transitions to `TrustBased` (warning)
and `Failed` (critical). For usage-based billing, also monitor usage-reporting
warnings and restarts.
- **Dependencies**: requires a valid license mounted on the container
filesystem (set `LICENSE_FILE_PATH`). For usage-based billing, also requires
connectivity to https://usage-tracker.assemblyai.com at startup.
- **Availability**: run a few containers behind a load balancer.
## Changelog
### v1.2.0
#### Diarization v0.2.1 (2026-11-02)
- Bump `self-hosted-diarization` to `release-v0.2.1`.
#### Diarization — New Self-Hosted Stack (added 2026-10-14)
- **New stack** — [`diarization/docker/`](diarization/docker/) adds speaker labels
to a transcript (`self-hosted-diarization:release-v0.2.1`). Use it with
`self-hosted-streaming-license-and-usage-proxy:release-v1.3.0`.
- **Sync + diarization example** — [`examples/sync-diarization/`](examples/sync-diarization/)
runs sync and diarization with one license-and-usage-proxy and one license
file (added 2026-10-21).
- **Sync project name** — `sync/docker/docker-compose.universal-3-6-pro.yml`
now sets the Compose project name `sync` (it was `docker`), so the sync and
diarization stacks do not share a proxy. Stop a running sync stack before you
pull this change, or stop it after with
`docker compose -p docker -f docker-compose.universal-3-6-pro.yml down`.
#### Streaming — Universal-3.5 Pro stack removed (2026-10-14)
The Universal-3.5 Pro streaming stack is superseded by the Universal-3.6 Pro
stack added on 2026-10-06 (below). If you are still on it:
- **Compose file removed** — `streaming/docker/docker-compose.universal-3-5-pro.yml`
is gone; run `streaming/docker/docker-compose.universal-3-6-pro.yml` instead.
Bring the old stack `down` before bringing the new one `up`.
- **Image env var** — `STREAMING_ASR_UNIVERSAL_3_5_PRO_IMAGE` is no longer read;
set `STREAMING_ASR_UNIVERSAL_3_6_PRO_IMAGE` to
`344839248844.dkr.ecr.us-west-2.amazonaws.com/self-hosted-streaming-asr-universal-3-6-pro:release-v1.2.0`
(see `streaming/docker/.env.example`).
- **`speech_model` value** — clients must connect with
`speech_model=universal-3-6-pro`. `nginx_streaming_asr.conf` no longer routes
`universal-3-5-pro`, so sessions requesting it fail.
- **Modal** — `streaming/modal/modal_app_universal_3_5_pro.py` is removed; deploy
`modal_app_universal_3_6_pro.py` (app `aai-streaming-u36pro`) and stop the old
`aai-streaming-u3pro` app.
- Connection parameters, messages, mode names, and GPU requirements are
unchanged. The Universal-3.5 Pro streaming features listed in this release
carry over to Universal-3.6 Pro.
#### Sync — Universal-3.5 Pro replaced by Universal-3.6 Pro (added 2026-10-14)
The sync stack now serves **Universal-3.6 Pro** from
`self-hosted-sync-asr-universal-3-6-pro:release-v1.2.0`. If you are upgrading
from the Universal-3.5 Pro sync stack:
- **Compose file renamed** — `sync/docker/docker-compose.universal-3-5-pro.yml`
is now `sync/docker/docker-compose.universal-3-6-pro.yml`. Bring the old
stack `down` before bringing the new one `up`.
- **Image changed** — point `SYNC_API_IMAGE` at
`self-hosted-sync-asr-universal-3-6-pro:release-v1.2.0` (the variable name is
unchanged). The `release-v1.2.0` `self-hosted-streaming-license-and-usage-proxy`
image is unchanged.
- **Drop-in for clients** — Same routes, request and response formats, and
config options; no client change is needed. Usage reports are stamped
`universal-3-6-pro`.
- **20-minute audio by default** — The stack now accepts audio up to 20
minutes and request bodies up to 256 MB by default
(`MAX_AUDIO_DURATION_MS=1200000`, `MAX_REQUEST_BYTES=268435456`, up from
120 s / 40 MB). Both remain configurable in `.env`; set the old values to
keep the previous limits.
- **Modal** — `sync/modal/modal_app.py` deploys Universal-3.6 Pro under the
same app name (`aai-sync-u3pro`), so `modal deploy` upgrades an existing
deployment in place. The image repository changed, so that deploy pulls and
converts a new image; refresh the `aai-ecr-credentials` secret first if its
session credentials have expired.
#### Universal-3.6 Pro — New Streaming Stack (added 2026-10-06)
- **New stack** — `streaming/docker/docker-compose.universal-3-6-pro.yml` serves
Universal-3.6 Pro from `self-hosted-streaming-asr-universal-3-6-pro:release-v1.2.0`
(`STREAMING_ASR_UNIVERSAL_3_6_PRO_IMAGE`). Clients connect with
`speech_model=universal-3-6-pro`; `nginx_streaming_asr.conf` routes it to the
new `streaming-asr-universal-3-6-pro` backend. A Modal package,
`streaming/modal/modal_app_universal_3_6_pro.py`, is included.
- **Drop-in for Universal-3.5 Pro** — Same connection parameters, messages,
modes, and GPU requirements. It runs with the existing `release-v1.2.0`
`self-hosted-streaming-api` and `self-hosted-streaming-license-and-usage-proxy`
images; nothing else needs upgrading. To switch, bring the Universal-3.5 Pro
stack `down`, bring the Universal-3.6 Pro stack `up`, and change the clients'
`speech_model`.
- **`max_accuracy` retuned for the new model** — Compared with Universal-3.5
Pro, a turn that ends mid-entity is held for half as long, and an ordinary
sentence waits about 0.1 s more silence before it ends. `min_latency` and
`balanced` are unchanged.
- **No stray non-CJK output on streams pinned to CJK languages** — When
`language_codes` lists only `zh`, `yue`, `ja`, or `ko`, a decoded segment with
letters but no CJK characters is dropped. Segments that contain any CJK
character are kept whole, so Latin words inside CJK speech survive.
#### Streaming API — New Features
- **`acknowledge_silence`** — Opt in to receive a `Silence` message about once
per second while no speech is being transcribed, so clients can tell a quiet
room apart from a stalled connection (Universal-3.5 Pro).
- **Heartbeat on Universal English and Multilingual** — The `session_heartbeat`
opt-in now works on all three models.
#### Streaming API — Fixes and Improvements
- **Webhook hardening** — `webhook_url` targets on link-local or cloud-metadata
addresses are rejected, and delivery no longer follows redirects.
#### Universal-3.5 Pro (streaming)
- **Entity-aware endpointing in all modes** — A turn that ends mid-number,
mid-spelling or mid-address is held until it completes, while ordinary
sentences still end at the mode's minimum silence.
- **Retuned mode thresholds** — The three modes now differ mainly in how long
they wait for an entity to complete; `max_accuracy` ends ordinary sentences
sooner than before.
- **Repetition-loop guardrails** — Runaway decodes that repeat a phrase dozens
of times are collapsed and capped.
- **Speaker tags stripped** — Bracketed speaker tags no longer leak into
transcript words.
- **13 more languages for `language_codes`** — Korean, Russian, Catalan,
Galician, Romanian, Estonian, Persian, Cantonese, Afrikaans, Marathi, Zulu,
Xhosa and Norwegian Nynorsk.
- **No CJK characters on streams pinned to other languages** — Stray Chinese,
Japanese or Korean characters are dropped when `language_codes` names only
non-CJK languages.
- **Multilingual sentence-final punctuation** — Turns in Japanese, Chinese,
Arabic, Urdu and Hindi end on their native punctuation instead of waiting for
the silence timeout.
- **More robust under load** — Timed-out inference requests are aborted, so
overload degrades gracefully instead of failing sessions.
- **Working container health check** — The compose file's gRPC health probe now
reports the container healthy.
#### Universal English and Multilingual ASR
No model change.
- **CUDA 13 runtime** — The host NVIDIA driver must now support CUDA 13 (R580
or newer); T4 and newer GPUs remain supported.
- **Heartbeat support** — See Streaming API above.
- **Clean exit on fatal worker errors** — The container exits instead of
hanging, so the restart policy takes over.
#### Sync — New Features
Shipped in `self-hosted-sync-asr-u3-pro:release-v1.1.0` (2026-09-11) and
carried forward.
- **Compressed audio** — `/transcribe` accepts AAC, MP3, M4A, OGG, WebM and
FLAC in addition to WAV and raw PCM.
- **Streamed uploads** — `POST /v1/transcribe/live` transcribes while the audio
is still uploading (WAV/PCM only).
- **Word timestamps** — Set `"timestamps": true` for per-word `start`/`end`;
requires `SYNC_BFA_ALIGNMENT_ENABLED=true` on the container.
- **13 more languages** — The same list as streaming above.
- **Higher config limits** — Longer `prompt`, `keyterms_prompt` and
`conversation_context` are accepted.
- **`GET /warm`** — Pre-warms a connection ahead of a latency-sensitive
request.
- **`/v1` route prefix** — Routes are served at `/v1/...`; unprefixed paths
continue to work.
- **`keyterms_prompt`** — Preferred name for the keyterms list; `keyterms` and
`word_boost` remain accepted as aliases.
#### Sync — Fixes and Improvements
- Self-hosted usage reports are now accepted end to end; on v1.0.0 they were
rejected, which on usage-based licenses also crash-looped the proxy.
- Guardrails against runaway decodes and hallucinated segments.
- Bracketed speaker tags no longer leak into transcripts.
- Faster large uploads and faster prompt trimming.
#### Sync — Behavior changes
- Words carry `start`/`end` only when `"timestamps": true` is requested.
- A caller-supplied `prompt` is added to the language directive instead of
replacing it, and an explicit `language_code: "en"` now steers the model;
output can differ for these requests.
- More than 100 keyterms are rejected with `400`.
#### License-and-usage-proxy
- No functional changes; rebuilt so every image shares one tag.
#### Also new since v1.0.0
Shipped in the `release-v1.0.1` images but not previously listed:
- **`session_heartbeat`** — Opt in to a periodic `Heartbeat` message
(Universal-3.5 Pro).
- **`encoding=aac`** — AAC audio is accepted alongside PCM and Opus.
- **End-of-turn details on `Turn`** — Final turns carry `silence_before_eot_ms`
and `turn_end_reason` (Universal-3.5 Pro).
- **Language detection covers every supported language code.**
- **Faster finals** — Speculative decoding roughly halves time-to-final after a
`ForceEndpoint` with no change in accuracy (Universal-3.5 Pro).
- **Partials every second by default** — Send `continuous_partials=false` for
the slower cadence (Universal-3.5 Pro).
- **English-first start for unsteered streams** — Prevents a first short turn
from committing in the wrong language (Universal-3.5 Pro).
- **`VLLM_ATTENTION_BACKEND` override** — For GPUs the bundled kernels do not
cover, such as NVIDIA B200 (Universal-3.5 Pro, Sync).
- **Bounded session lifetime** — A session whose client stopped sending closes
within 60 s, and ASR streams that end without a completion signal are
retried.
### v1.0.0
#### Repository restructure
The repository is now organized by **service**: [`streaming/`](streaming/)
(WebSocket, real-time) and [`sync/`](sync/) (synchronous HTTP, full-file). Each
service directory is self-contained — compose files, `.env.example`, example
client, and README live together. Compose files moved accordingly (e.g.
`docker-compose.yml` → `streaming/docker-compose.english-multilang.yml`).
#### Sync — New Self-Hosted Service (NEW)
This release introduces the **Sync self-hosted service**
(`sync/docker-compose.universal-3-5-pro.yml`), serving the Universal-3.5 Pro
model. It transcribes a complete audio file (≤ 120 s) in a single
`POST /transcribe` request/response — a single GPU container plus the
license-and-usage-proxy, no load balancer. It exposes `GET /readyz` (200 once
the model is warm) for readiness probes. See [`sync/docker/README.md`](sync/docker/README.md).
#### Streaming — U3 Pro replaced by Universal-3.5 Pro (BREAKING)
The U3 Pro stack introduced in v0.6.0 is replaced by the **Universal-3.5 Pro
stack** (`streaming/docker-compose.universal-3-5-pro.yml`). If you are
upgrading from the v0.6.0 U3 Pro stack:
- **`speech_model` value changed** — clients must connect with
`speech_model=universal-3-5-pro`. The previous value `u3-rt-pro` is no longer
routed by the load balancer and sessions requesting it will fail.
- **Image env var renamed** — `STREAMING_ASR_U3PRO_IMAGE` is now
`STREAMING_ASR_UNIVERSAL_3_5_PRO_IMAGE`, pointing at the
`self-hosted-streaming-asr-universal-3-5-pro` image.
- **Compose file renamed** — `docker-compose.u3pro.yml` is now
`streaming/docker-compose.universal-3-5-pro.yml`.
- **Hardware recommendations updated** — tested on NVIDIA L40S (48 GB) and
RTX PRO 6000 Blackwell (96 GB); the model weights use ~11 GB of VRAM, the
rest becomes KV cache and sets max concurrency.
#### Images
`release-v1.0.1` is published for `self-hosted-streaming-api` (adds the
peer-aborted-handshake logging fix) and `self-hosted-streaming-asr-universal-3-5-pro`;
`release-v1.0.0` for `self-hosted-streaming-license-and-usage-proxy` and
`self-hosted-sync-asr-u3-pro`.
The English and Multilingual ASR images are unchanged since v0.6.0 — keep
`STREAMING_ASR_ENGLISH_IMAGE` and `STREAMING_ASR_MULTILANG_IMAGE` at
`release-v0.6.0` (see `streaming/.env.example`).
### v0.6.0
#### U3 Pro — New Self-Hosted Stack (NEW)
This release introduces the **U3 Pro self-hosted stack** (`docker-compose.u3pro.yml`), which serves the U3 Pro streaming model. U3 Pro delivers significant improvements over the universal English model on complex entities, short utterances, and end-of-turn (EOT) latency, and is targeted at voice agent scenarios.
Hardware: NVIDIA L4 / A10 / A100 / L40S / H100 (24 GB+ VRAM).
Highlights of U3 Pro behavior delivered with this release:
- **New transcription prompt** ("Transcribe verbatim with standard punctuation. Include filler words and incomplete utterances.") — 22% reduction in voice-agent hallucinations, 10% WER and 29% short-utterance error-rate reduction on voice-agent traffic, 5% improvement on medical, and improved EP F1.
- **Continuous partials during long turns** — partials are emitted incrementally instead of being delayed; turns now stitch up to 60s instead of hard-cutting at 16s/32s.
- **Early partial at 750ms** of detected speech for faster UI feedback.
#### Streaming API — New Features
- **`continuous_partials` query parameter** — clients can opt into continuous partials during long turns.
- **Structured logging** — both the U3 Pro ASR server and the universal ASR server now honor `USE_STRUCTURED_LOGGING`, matching the streaming-api behavior.
#### Other Improvements
- Various logging and metrics improvements across the streaming-api and ASR services.
- Bug fixes and stability improvements.
### v0.5.0
#### English ASR Model
A new English model is released, which produces already-formatted outputs directly and delivers large quality gains on digits, telephony, medical, and CI segments:
- **34% improvement on digit sequence error rate (DSER)**
- **17% improvement on telephony WER**
- **12% average improvement on medical WER**
- **10% average improvement on CI segments WER**
- **~2.4% absolute F1 score improvement on keyterms prompting**
- **Significantly improved timestamp accuracy** — resolves overlapping and zero-duration word issues
#### Multilingual ASR Model
- **~70% absolute improvement in timestamp accuracy** — fixes overlapping words and zero-duration word bugs
#### Streaming API — New Features
- **Error and Warning WebSocket message types** — Dedicated message types that let clients distinguish actionable errors from non-fatal warnings without relying on close codes.
- **Configuration echoed in SessionBegins** — The `SessionBegins` message now includes the resolved session configuration so clients can verify applied settings.
- **Explicit speech-model selection** — Clients explicitly select the speech model at session start.
#### Streaming API — Fixes and Improvements
- **More specific WebSocket close codes** for session termination scenarios, making client-side error handling more precise.
- **Improved `word_finalized` events** — All word finalizations are emitted (not only the last word of a turn).
#### Other Improvements
- Various logging, metrics, and observability improvements across the streaming-api and ASR services.
- Bug fixes and stability improvements.
### v0.4.0
#### English ASR Model
Major improvements to short utterance handling and hallucination reduction:
- **100% reduction in hallucinations**
- **12.8% improvement on short utterances** - Better performance for voice agent use cases
- **7.39% improvement on digit sequence error rate**
- **1.75% improvement on proper nouns**
- **0.46% improvement on CI segments**
- **0.39% improvement on accented speech**
#### Multilingual ASR Model
- **Context biasing support** - Customers can now use context biasing (model-based biasing) with the multilingual model
#### Other Improvements
- Increased concurrent session handling per container, leading to reduced deployment costs
- Improved observability for the license-and-usage-proxy service
- Various bug fixes and stability improvements