docs/voice-and-multimodal-design.md
Rama main · réplica de lectura del repositorio original.
# Voice & Multimodal Output Design
> Status: design draft. Nothing in this doc has shipped. Captures the
> architectural shape for adding voice input/output to agents and the broader
> "agent generated a file in the sandbox — now what?" problem. Supersedes the
> "audio/video routing" deferred item in
> [`file-attachments-design.md`](file-attachments-design.md).
## Why
Two motivating use cases, one architectural gap behind them:
1. **Voice input.** A user sends a voice note on Telegram / WeChat / Signal /
WhatsApp. Today the channel adapter either drops it or surfaces an opaque
file URL the model cannot use. We want the agent to receive the transcript
transparently and optionally still see the original audio.
2. **Voice output.** When the user prefers voice, the agent's final reply
should arrive as a native voice message on platforms that support one
(Telegram `sendVoice`, WhatsApp audio, Signal voice note), falling back to
a file attachment elsewhere.
3. **The bigger gap behind both.** Inbound attachments are well-modelled
today; outbound is text-only. `ChannelOutbound.send` and `session_send`
have no attachment surface. So even when the agent runs a sandbox tool
that produces `report.png`, `summary.mp3`, or `analysis.pdf`, the only way
to deliver it back is a URL pasted into text. Voice output is a special
case of this gap; solving outbound media solves both.
This doc proposes a two-phase sequence:
- **Phase 1 (voice-only):** configuration, inbound STT, outbound TTS for
`text_final`. Synthesised audio is delivered through a channel-direct
path (Telegram `sendVoice` etc.); no protocol change to
`ChannelOutbound.send`. Ships voice end-to-end with the smallest blast
radius.
- **Phase 2 (general outbound media):** generalize the outbound surface
(`ChannelOutbound.send` gains `attachments?`, `ChannelManifest` gains
`outboundCapabilities`, sandbox-generated files become attachment refs).
Voice output retroactively re-routes through this surface; image / PDF
output from sandbox tools rides the same dispatch.
Phase 1 is what the rest of this doc focuses on. Phase 2 is sketched at
the end as the eventual destination.
## Non-goals
- **Talk Mode** (always-on full-duplex streaming, VAD, barge-in). That is a
separate architectural change — different transport, different latency
envelope, different per-channel APIs. Deferred.
- **Real-time streaming TTS.** Synthesis happens on `text_final`, not on
`text_delta`. Adding per-delta synthesis is possible later without
protocol changes; the first cut is one synthesis call per turn.
- **Multimodal model routing for audio/video input.** Inbound voice is
transcribed at the channel layer and injected as text. The model sees a
transcript, not raw audio (the audio is still attached for tools to
inspect, but it does not go to the LLM). Sending audio bytes directly to
multimodal models is a follow-up tracked in
[`file-attachments-design.md`](file-attachments-design.md).
- **Synchronous TTS in the request path.** TTS may take seconds; it must
not block the agent loop. The send pipeline already supports
`backgroundTasks`/`sideEffects` — outbound media synthesis runs there.
- **Multi-provider chain / local-model fallbacks.** Phase 1 supports
exactly one provider — ElevenLabs — for both STT and TTS. The internal
interfaces leave room for a chain later, but no chain code, no
faster-whisper bundle, no Piper. One provider, one API key, one set of
failure modes. Add providers only when we have a concrete reason to.
## How OpenClaw and Hermes Agent solve it
Two reference points; they converged on the same shape.
- **Provider chain, not single provider.** Hermes default STT chain is
`local-faster-whisper > Groq > OpenAI > Mistral > xAI`; TTS chain spans
ten providers with the local Piper engine as the zero-API-key default.
OpenClaw 2026.4.25 ships 14 TTS providers with a similar fallback chain.
On size / timeout / auth failure the next provider runs automatically.
- **Local-first.** Both default to bundled local models (faster-whisper,
Piper) so voice works with zero API keys. Cloud providers are the
upgrade path, not the baseline.
- **Per-channel native rendering.** Voice notes on Telegram / WhatsApp /
Discord / Slack / Signal are transcribed *at the channel adapter*, not at
the gateway. Outbound, the channel decides between native voice message
and audio-file attachment based on what the platform supports.
- **Per-agent / per-chat overrides.** OpenClaw 2026.4.25 added chat-scoped
auto-TTS, personas, and per-agent overrides. Layered config, not a single
global switch.
We adopt **per-channel rendering** (transcribe inbound at the channel,
synthesize outbound at the channel) and **per-agent overrides**. We do
**not** adopt the provider-chain / local-first defaults — phase 1 ships
with one provider (ElevenLabs). The interfaces below leave room to grow
into a chain later, but the runtime wiring is direct. Talk Mode is out
of scope.
## Current state
Relevant scaffolding already exists:
- **`SessionAttachment`** (`packages/protocol/src/index.ts:65`) — id-shape
and url-passthrough, mime type, SHA, sandbox path. Already used inbound;
the type itself is direction-agnostic.
- **Attachment materialization** —
`AgentRunner.materializeAttachmentToSandbox`
(`apps/agent/src/agent-runner.ts:386`) writes bytes under
`<agentHome>/.openhermit/attachments/<sessionId>/<id>/<name>`. The reverse
flow — register an existing sandbox file as an attachment — does not yet
exist but the storage shape is the same.
- **Event taxonomy** — `OutboundEvent` already separates `text_final`,
`thinking_*`, `tool_*`, `agent_start/end`
(`packages/protocol/src/index.ts:217`). "Only TTS the final text" is
already a subscription decision, not a parsing problem.
- **`ChannelManifest`** (`packages/protocol/src/index.ts:464`) — no
`outboundCapabilities` field today. Channels cannot declare which
modalities they can render.
- **`ChannelOutbound.send`** (`packages/protocol/src/index.ts:294`) —
text-only:
`send({ sessionId, to, text, actions })`. No `attachments` parameter.
- **`session_send`** (`apps/agent/src/tools/session.ts:257`) — text-only
for the same reason.
- **Per-agent secret store** — already used by channel plugins for API
keys; the natural home for `OPENAI_API_KEY`, `ELEVENLABS_API_KEY`, etc.
## Phase 1 — Design
Three subsystems, one package, no protocol change to `ChannelOutbound`:
```text
┌─────────────────────────────────────────────────────────────┐
│ packages/voice │
│ • STT provider (single, per-agent) │
│ • TTS provider (single, per-agent) │
│ • per-mime codec selection (audio/ogg vs audio/mp4 …) │
└─────────────────────────────────────────────────────────────┘
▲ ▲
│ used by │ used by
│ │
┌───────┴────────────────┐ ┌──────────┴───────────────────┐
│ Channel adapter │ │ Channel adapter │
│ (inbound path) │ │ (outbound path) │
│ voice msg → STT → text│ │ text_final → gate → TTS → │
│ → postMessage │ │ channel-native voice send │
└────────────────────────┘ └──────────────────────────────┘
```
### 1. Configuration
Per-agent. STT and TTS configured independently — one model per
direction, no arrays. Persisted via the existing `agents.config_json`
column — no new table.
```jsonc
{
"voice": {
"stt": {
"provider": "elevenlabs",
"model_id": "scribe_v1" // optional; provider default if omitted
},
"tts": {
"provider": "elevenlabs",
"voice_id": "JBFqnCBsd6RMkjVDRZzb", // optional; ElevenLabs default voice if omitted
"model_id": "eleven_multilingual_v2", // optional
"speed": 1.0 // optional
}
}
}
```
- **`provider`** — only `"elevenlabs"` is accepted in phase 1, for both
STT and TTS. Validation rejects anything else with a clear error so the
field stays a useful extension point. Configuring them separately
reflects the eventual reality (you may want ElevenLabs for TTS quality
but Whisper / Groq for cheap STT) — phase 1 just happens to point both
at the same vendor.
- **`tts.voice_id`** — ElevenLabs voice catalog id. Owner-selectable so
different agents (or the same agent across personas) can sound distinct.
- **API key** — `ELEVENLABS_API_KEY` in the agent's secret store (already
supported by the secret resolver). One key serves both directions today.
- **CLI** mirrors the existing pattern, e.g.
`hermit config --agent <id> set voice.tts.voice_id = JBFqnCBsd6RMkjVDRZzb`.
Each direction is independently opt-in. An agent may configure only
`voice.stt` (accept voice notes, reply in text) or only `voice.tts`
(reply by voice when triggered, ignore inbound voice). Omitting both
disables voice entirely.
**Per-session toggle.** Independent of agent config, the owner / user can
flip voice-out on or off for a specific session via a channel command
(`/voice on`, `/voice off`). Persisted on `session.metadata.voice_out`.
The TTS gate (below) reads this. Without an explicit toggle, the gate
falls back to modality mirroring (inbound voice ⇒ reply voice, inbound
text ⇒ reply text).
### 2. `packages/voice` — shared provider package
A small package exposing two interfaces and exactly one implementation of
each (ElevenLabs). The interfaces are the extension point; the runtime
just constructs the ElevenLabs adapter from agent config.
```ts
// packages/voice/src/index.ts (sketch)
export interface SttProvider {
readonly name: string;
transcribe(input: {
url?: string;
bytes?: Uint8Array;
mimeType: string;
languageHint?: string;
}): Promise<{ text: string; durationMs?: number; provider: string }>;
}
export interface TtsProvider {
readonly name: string;
/**
* Synthesise `text` into the requested output container. Callers
* (channel adapters) pass the codec their platform needs for native
* voice messages.
*/
synthesize(input: {
text: string;
voiceId?: string;
modelId?: string;
speed?: number;
outputMimeType: string; // e.g. 'audio/mp3', 'audio/ogg;codecs=opus'
}): Promise<{ bytes: Uint8Array; mimeType: string; provider: string }>;
}
export const createElevenLabsStt = (cfg: { apiKey: string }): SttProvider => /* … */;
export const createElevenLabsTts = (cfg: { apiKey: string }): TtsProvider => /* … */;
/**
* Build the voice providers for an agent. Each direction is built
* independently from its own `voice.stt` / `voice.tts` block; either may
* be undefined if the agent only opted into one direction. Throws if
* `provider` is anything other than `'elevenlabs'`, or if the required
* API key is missing.
*/
export const createVoiceForAgent = (
agentConfig: AgentConfig,
secrets: SecretResolver,
): { stt?: SttProvider; tts?: TtsProvider } => /* … */;
```
The chain helper / fallback logic is **deliberately not implemented**.
When (if) we add a second provider, we add it as another constructor and
a chain combinator — the channel-adapter call sites don't change because
they only see `SttProvider` / `TtsProvider`.
ElevenLabs covers both STT (Scribe v1) and TTS, so phase 1 ships with a
single API key and a single billing surface.
### 3. Inbound STT (channel-side transcription)
Channel adapters that receive voice notes transcribe before calling
`postMessage`. The agent sees text; the original audio is still attached
so tools can re-listen if needed.
```ts
// apps/channels/telegram/src/bridge.ts (sketch)
async onVoiceMessage(msg) {
if (!this.voice) return /* fall through to text-only behaviour */;
const audioUrl = await this.bot.getFileLink(msg.voice.file_id);
const result = await this.voice.stt.transcribe({
url: audioUrl,
mimeType: 'audio/ogg',
});
await agent.postMessage(sessionId, {
text: result.text,
attachments: [{ type: 'file', url: audioUrl, mimeType: 'audio/ogg' }],
metadata: {
source: 'voice_note',
transcript_provider: result.provider,
voice_duration_ms: result.durationMs,
},
});
}
```
If the agent has no `voice` config, `this.voice` is undefined and the
adapter falls through to whatever it does today (drop the message, or
surface it as an opaque attachment). Voice input is strictly opt-in.
The `source: 'voice_note'` metadata is the signal the TTS gate uses
(below) to default to voice-out reply.
### 4. Outbound TTS — when to do it
This is the design decision the user flagged. The interceptor must not
TTS every assistant turn — that would be expensive, often annoying
(spoken code blocks, multi-paragraph summaries), and miss the point of
"voice when it makes sense".
The gate runs once per `text_final`, in order. Any predicate firing
**skips** TTS for that turn:
| # | Predicate | Default |
|---|-----------|---------|
| 1 | Session has `voice_out` disabled in `session.metadata` | skip if `false` |
| 2 | The channel doesn't declare voice support in its adapter | skip |
| 3 | The triggering user message was *not* a voice note **and** the user hasn't explicitly turned voice on | skip |
| 4 | `text_final.text` is empty / whitespace only | skip |
| 5 | `text_final.text` exceeds `voice.tts.maxChars` (default 600) | skip + fall back to text |
| 6 | `text_final.text` contains a fenced code block or table | skip + fall back to text |
| 7 | `text_final.text` is a tool-affordance message (approval prompt with inline actions) | skip — the buttons are the point |
Default behaviour ("modality mirroring"): if the inbound was a voice
note, reply with voice; if the inbound was text, reply with text. The
user opts out of this by setting voice-out off, or opts in to "always
voice" via `/voice on` (which flips the predicate 3 to "always allow"
for the session).
When the gate passes:
```ts
// channel adapter, in the event subscription
on('text_final', async (evt) => {
if (!this.voice || !gate.shouldSpeak(evt, session)) {
return await this.sendText(evt.text); // existing path
}
const cfg = this.agent.config.voice!;
const audio = await this.voice.tts.synthesize({
text: evt.text,
voiceId: cfg.voice_id,
modelId: cfg.model_id,
speed: cfg.speed,
outputMimeType: this.platform.preferredVoiceMime, // 'audio/ogg;codecs=opus' on TG
});
await this.platform.sendVoice(to, audio.bytes);
// Optional: also send the text for accessibility / search.
if (session.metadata.voice_keep_text) await this.sendText(evt.text);
});
```
`text_delta` / `thinking_*` / `tool_*` are simply not subscribed to —
the event taxonomy already does the filtering for us.
### Timing — when in the event stream
TTS runs on `text_final`, not on `text_delta`:
- `text_delta` arrives token-by-token; per-delta synthesis multiplies cost
and breaks prosody (no sentence-level context).
- `text_final` is one synthesis per turn. Latency penalty of waiting for
the full response is acceptable for the first cut — voice replies are
inherently slower than text and users expect a brief delay.
- Streaming TTS (sentence-level chunking on `text_delta`) is a follow-up
optimisation tracked in open questions, not phase 1.
Synthesis runs on `session.backgroundTasks` (the same hook the central
scheduler awaits before tearing down `ephemeral` sessions, per the recent
schedule-session fix). This keeps it off the agent's blocking loop while
still letting the scheduler wait for it to complete.
### Phase-1 outbound path
No change to `ChannelOutbound.send`. The TTS interceptor lives entirely
inside the channel adapter — it has the synthesised bytes in hand and
calls the platform API directly (`bot.sendVoice` on Telegram,
`messages.send` with `audio/ogg` body on WhatsApp Business, etc.). The
adapter wires its own event subscription against the runner's outbound
event stream.
The protocol-level `attachments` field on `ChannelOutbound.send` is
deferred to phase 2.
## Phase 2 — Generalize outbound media (sketch)
Once voice ships, the same pattern wants to handle sandbox-generated
images, PDFs, and arbitrary audio. Phase 2 generalizes the outbound
surface so the channel-direct path used in phase 1 becomes the general
case:
- Extend `ChannelOutbound.send` to accept `attachments?: SessionAttachment[]`.
- Add `outboundCapabilities?: { mimeTypes: string[]; voiceMimeTypes?: string[] }`
to `ChannelManifest` so the dispatcher knows what each channel can
render natively.
- Add sandbox-side attachment registration (a sandbox tool, or implicit
resolution of `{ sandboxPath }` refs inside `session_send`) so an agent
that writes `/sandbox/report.png` can ship it through `session_send`
with no special-casing.
- Re-route phase-1 TTS through this surface: the synthesised audio
becomes an attachment ref, `ChannelOutbound.send` dispatches it. The
channel-direct path remains for adapters that haven't migrated yet.
Phase 2 is not blocked by phase 1, but it's a much larger change
(protocol contract, every channel adapter, agent outbound tool surface)
and the user-visible win — voice — lands sooner if phase 1 ships first.
## Migration plan
Phase 1 is one PR per concern, ordered for early integration:
1. **`packages/voice` + config plumbing.** Provider interfaces, the
ElevenLabs STT and TTS implementations, the agent-config schema entry
(`voice.stt.*`, `voice.tts.*`), `createVoiceForAgent` wiring against
the secret resolver. No channel integration yet — verified by unit
tests against a stubbed ElevenLabs HTTP fixture.
2. **Telegram inbound STT.** Wire `voice.stt` into the Telegram adapter's
voice-message handler. Verifies the package end-to-end on the first
channel.
3. **Telegram outbound TTS.** Adds `voice.tts` + the gate-predicate
evaluation + channel-direct `sendVoice`. Modality-mirroring default
on; `/voice on/off` per-session override.
4. **Rollout to remaining channels** (WeChat, WhatsApp, Signal, Discord
where applicable), one PR each — all reuse `packages/voice` and the
same gate logic.
Talk Mode (full-duplex streaming) is a separate, larger initiative tracked
in [`pending-decisions.md`](pending-decisions.md) and is *not* unlocked by
this work — Talk Mode needs a different transport, different latency
profile, and per-channel duplex APIs (Telegram voice chats, WeChat
voip-style channels) we don't carry today.
## Open questions
- **Audio attachments and the LLM.** Today inbound attachments are
materialized into the sandbox but only images are inlined to multimodal
models. Should audio be inlined for models that accept it (Claude, GPT-4o
audio), or always go through transcript-first? Defer; current proposal is
transcript-first regardless, since the channel already has a transcript
in hand from STT and inlining audio would double the prompt cost.
- **Voice cloning / personas.** OpenClaw ships persona-keyed voices. We
could mirror via `agent.config.voice.persona`, but voice cloning has
legal / consent surface that needs operator review before defaulting on.
Deferred.
- **Cost accounting.** TTS bills per character; STT bills per second. The
existing usage telemetry doesn't track audio. Adding it is a small
schema change but worth surfacing before users discover the bill.
- **Where does `attachment_register` live?** Sandbox tool vs. a runner
method invoked by `session_send`? Slight preference for runner method
(`session_send({ text, attachments: [{ sandboxPath }] })` would internally
register), to avoid adding a new tool surface the model must learn.