proposals/2026-08-02-voice-control.md

Rama master · réplica de lectura del repositorio original.

# "Okay Nabu, play Led Zeppelin" - voice control through Home Assistant

**Goal** - a spoken request to a Home Assistant voice satellite searches the PearTune library
and starts it playing on a speaker in the house.

**Tier** - T3. It creates a way to make the library play that is NOT a paired device holding a
grant, which is an auth gate, and it adds an HTTP surface to the host. Everything else here is
small; that one thing is why this needs a proposal.

## Why

Tim asked for it on 2026-08-01, as the follow-on to speaker playback: "it would ask the
PearTune host to make a queue/playlist". He has the hardware - a Home Assistant Voice PE with
the "Okay Nabu" wake word already configured (`select.home_assistant_voice_091831_wake_word`).

It is also much cheaper now than it was then. Phase 1 of the speaker work gave the host a
loopback HTTP server and the ability to start playback on any Home Assistant speaker, so voice
is "route a sentence to what already exists" rather than "build a media path".

### The finding that shapes the whole design

**Home Assistant's built-in media intent cannot be used.** `HassMediaSearchAndPlay`
(`media_player/intent.py`, tag 2026.7.4) matches only entities advertising:

```python
features=MediaPlayerEntityFeature.SEARCH_MEDIA | MediaPlayerEntityFeature.PLAY_MEDIA
```

Measured on Tim's install: the Voice PE reports `1200653` and the Cast speakers `152461` /
`152511`. **None of them has `SEARCH_MEDIA` (4194304).** So no amount of exposing PearTune as a
media source would make "play Led Zeppelin on the man cave speaker" work - the intent would
never match the speaker in the first place.

That rules out the approach I would otherwise have proposed, which was to register PearTune as
a `media_source` and let HA's own intent do the rest.

**So we do the search ourselves.** Our own sentence, our own intent, our own search, and the
speaker is only ever asked to `play_media` - which every speaker in the house does support.
This has a side benefit: the answer comes from PearTune's index, so it can match the library's
actual contents rather than whatever a generic matcher would guess at.

## Scope

### Phase 1 - config on the Home Assistant side, one endpoint on the host

Nothing to install in Home Assistant: `custom_sentences`, `intent_script` and `rest_command`
are all stock YAML. That keeps the first cut small enough to prove the idea end to end.

**Host.**

- A `POST /voice/play` route on the **existing loopback cast-audio server**
  (`host/cast.js`), NOT on the dashboard. The dashboard binds `0.0.0.0` in a container, so
  putting a "make the library play" endpoint there would publish it to the LAN. The cast
  server is already `127.0.0.1`-only and already the audio origin.
- Body: `{ token, query, entityId }`. It searches via the adapter's existing
  `search({ q, limit })`, takes the best hit, and starts it exactly the way a phone does -
  through `casts.play()`, so the capability token, the live-grant re-read and the revoke teeth
  all apply unchanged.
- A **voice token**, generated by the host and shown in the dashboard's Speakers panel for the
  operator to paste into their `rest_command`. Stored beside the HA token in `speakers.json`
  and treated as a secret the same way. Rotating it revokes voice.
- Off by default. No token, no route.

**Home Assistant** (documented, not shipped):
`custom_sentences/en/peartune.yaml` for "play {query}" and "play {query} on {area}", an
`intent_script` that calls a `rest_command`, and the `rest_command` itself pointing at
`http://127.0.0.1:<port>/voice/play`.

### WHOSE authority is this? The part that makes it T3

Every existing way to make this library play is a device holding a grant the operator issued
and can revoke. **Voice is not that.** A person speaking in the room holds no grant, and the
request arrives from Home Assistant rather than over Noise.

Proposed answer, and it wants Tim's agreement rather than my assumption:

- Voice acts as **the operator**, not as any person or device. It is the operator's own Home
  Assistant, on the operator's own box, reachable only from that box's loopback.
- It gets a **synthetic device key** so `casts.play()` works unchanged and, more importantly,
  so a voice-started cast appears in the dashboard and can be stopped like any other.
- **Revocation is the token**, not a grant: clearing or rotating it in the dashboard kills
  voice immediately. That is a different shape of revoke from every other one in this app, and
  the asymmetry should be stated in the UI rather than left implicit.
- **Anyone who can speak in the house can play music.** That is the same trust boundary a
  smart speaker already has, but it is worth saying out loud: this is not authentication, it
  is physical presence.

### LLM voice agents (Tim, 2026-08-02)

Some people wire Home Assistant to an LLM as the conversation agent. That matters here,
because **an LLM agent never sees `custom_sentences`** - it decides for itself what to call.
A sentence-only design would silently stop working for those users.

Checked in `helpers/llm.py` (2026.7.4) rather than assumed. `AssistAPI._async_get_tools`
builds its tool list from `intent.async_get(hass)` with no filter on where the intent came
from, so **an `intent_script` intent is offered to the LLM automatically**, with its
`slot_schema` becoming the tool parameters:

```python
intent_handlers = [h for h in intent.async_get(self.hass) if h.intent_type not in ignore_intents]
tools = [IntentTool(self.cached_slugify(h.intent_type), h) for h in intent_handlers]
```

A `description` is optional (`h.description or f"Execute Home Assistant {name} intent"`) but
is what the model reads to decide when to use the tool, so the documented config always
writes one.

So the design holds for both, and an LLM agent is arguably the BETTER fit: it passes a
free-form query, which suits a search endpoint far more than fixed sentence templates do.
Untested against a real LLM agent - Tim raised it as worth testing at some point, and it is
in TODO.

### Not in scope

- **Requesting music that is not in the library.** "Play X" when X is absent should say so, not
  file a request. The request feature exists and is a different intent.
- Spoken answers beyond a short confirmation.
- Queue manipulation by voice ("skip", "shuffle this"), which HA's own `HassMediaNext` /
  `HassMediaPause` intents may already cover once a cast is playing, since those need only the
  features the speakers DO have. Worth testing before building anything.
- **Phase 2, a packaged custom component.** Config-only works for Tim today but asks anyone
  else to hand-edit `configuration.yaml`. A `custom_components/peartune` integration would make
  it installable, and could register a PearTune `media_player` entity that DOES advertise
  `SEARCH_MEDIA` so the built-in intent works. Only worth it if voice proves useful.

## Compat

- Purely additive and off by default: no token means the route is not registered.
- No wire-protocol change, no new media methods, nothing new on the P2P side at all.
- One new field in `speakers.json`, absent on every existing host, treated as a secret.
- No change to the phone app.

## Verify

1. `npm run verify` green.
2. Unit: the route refuses a missing, wrong or rotated token; refuses when voice is disabled;
   returns a clean "not found" for a query that matches nothing.
3. **The endpoint is loopback-only** - `curl` from the Umbrel's LAN address must fail while
   `127.0.0.1` works. Same test the cast audio route already has.
4. On hardware: say "Okay Nabu, play Led Zeppelin" and hear it start on the speaker.
5. Rotate the token in the dashboard and confirm the same sentence now does nothing.
6. A voice-started cast still dies on revoke and on host shutdown, like any other.

## Rollback

Off by default and additive; clearing the token disables it with no restart. Reverting the
commit removes the route. Nothing persisted outlives it except one config field.

## Open questions

1. **Is "acts as the operator" the right answer**, or should voice be tied to a specific person
   so play counts and history attribute correctly? Attributing to nobody is simplest and
   probably right, but it means voice plays do not show up in anyone's history.
2. **Which speaker, when the sentence does not say?** A default target configured in the
   dashboard is the obvious answer. Using the satellite that heard the request would be
   better, but the request as it reaches the host does not carry that unless we pass it.
3. **How much of the sentence do we match?** "Play Led Zeppelin" is an artist, "play Rock and
   Roll" a track, "play the Destiny soundtrack" an album. The adapters' `search()` already
   spans all three; the question is what to do when the best hit is the wrong KIND.
4. Should a voice play REPLACE what is on that speaker, or refuse if it is already casting for
   someone else?