CHANGELOG.md

Rama main · réplica de lectura del repositorio original.

# Changelog

One version number covers the server image (`ghcr.io/pauserratgutierrez/sluice`) and the SDK (`@pauserratgutierrez/sluice-js`).

## 0.12.0

The SDK reports a refused stream by the code the server sent (breaking for code that matched `http_503`), and stops on a refusal that sending the request again would not change.

### Changed

- **Breaking, SDK: a refused stream reports the server's code.** The client's `onError` got `http_<status>` for every refusal of `POST /stream`, with the response body as its message: `too_many_streams`, `not_replicating`, `server_shutdown` and `session_check_unavailable` all read `http_503`, and a bad token and a signed-out session both `http_401`. It now gets the `error` and `message` the server sent, as control requests already did. A response that is not Sluice's, such as a proxy's error page, is still `http_<status>`, with the status text as its message.
  - Opening the stream is retried after a `5xx`, `408` or `429` only. Any other `4xx` stops the client, as `401` and `403` did; a `400 bad_request`, which a stream request over 1 MiB gets, was retried forever. A control request's error is retryable on the same statuses, `408` included.

## 0.11.0

In issuer mode a shape could receive changes while its holds were checked, and was refused only after; it now receives nothing before they are confirmed, and the holds of a request are checked in one round trip. Sluice warns before its slot is lost, naming the transaction that holds it, and tells clients when changes arrive late. The resume buffer keeps 30 s of every table within a share of memory (breaking for its configuration), so busy tables resume too, and the reader does half the work per change. Tested with PostgreSQL 18.6.

### Fixed

- **Issuer mode: a shape no longer receives changes while its holds are checked.** Sluice registered a shape, then checked that its hold rows exist. A change committed during the check reached the shape even when a hold turned out missing and the shape was refused.
  - On a stream being opened the change went out right after `ready`; on `POST /subscribe`, at once.
  - The check is there to catch a hold row deleted between the issuer's verdict and the watch. It takes a database round trip, and in a reconnect of every stream it waited seconds for a pool connection.
  - A shape is now registered only after its holds were found, and only while its watch has not been cut since; a change dispatched during the check is before the shape's `ready`, as any change before a subscribe is.
  - Present since issuer mode was added (0.1.4).
- On PostgreSQL 16, the slot's state in `/diagnostics` and `sluice_slot_retained_bytes` stayed empty: their query read `invalidation_reason`, a column PostgreSQL 17 added. It now reads it as the reader does, through `to_jsonb`.
- **Two text values are compared as text.** The in-process evaluator compared two text values as numbers when both read as one, so `'10.0'` equaled `'10'` and `'9'` sorted below `'10'`, where PostgreSQL compares two text operands as text. Filters and Tier B predicates on text columns holding numbers could decide otherwise than PostgreSQL. A number against text that reads as one still compares numerically.

### Changed

- Issuer mode: **the holds of a request are checked in one round trip**, each distinct hold once however many shapes name it. A stream opening, `POST /subscribe` and `POST /token` made one `EXISTS` per hold of each shape, one after another. The load test's streams, whose 7 shapes are held by the same row, made 7 per open and per renewal, about 40,000 in a reconnect of 5,770 streams; they now make one each. A missing hold row still refuses only the shapes it holds.
- **Breaking, configuration: the resume buffer keeps changes by time, within a share of memory.** It kept the last `SLUICE_RING_EVENTS` (4,096) changes of each table, which on a busy table reached back seconds: about 2 s at 2,300 changes a second, and in the load test's 8-hour run 44% of the resumes of a table taking 337 a second came back `resumed: false`, each a client reading its state again.
  - It now keeps every table's changes for `SLUICE_RING_MAX_AGE` (30 s; removed on the catalog tick after, so 30–60 s), within `SLUICE_RING_MEMORY` (new; default `0.1`) of the Go memory limit, 64 MiB when none is known. Past that budget it removes the oldest change of any table, so every busy table reaches back as far as the others, and it keeps each table's newest transaction while anything else is left.
  - `SLUICE_RING_EVENTS` and `SLUICE_RING_MAX_BYTES` are gone. `SLUICE_RING_MEMORY=0` disables the buffer, as `SLUICE_RING_EVENTS=0` did, and needs `SLUICE_SNAPSHOT_ENABLED=false`.
  - The buffer's memory counts toward admission like the rest of the live heap. Measured on 1 CPU and 512 MB with a table taking 1,150 changes a second: 30 s of buffer held 17 MiB, and admission let in 4,900 streams of the load test where a buffer of 2 MiB let in 5,770; with 60 s, 34 MiB and 4,460.
  - In a reconnect of every one of those streams, all of them resumed, where the table taking 1,150 changes a second had read its state again; the node was full again within 10 s of the cut, as before.
  - Changes to the revocation tables are no longer kept: no shape may be on them.
- **The reader does half the work per change.** One message of the load test's largest mix (12,000 streams of 7 shapes, each shape held by its member's row), dispatched on one core: 9.4 µs per change before, 4.1 after, with 31 allocations instead of 68 (`BenchmarkRoomMessage`).
  - A change is checked once against each distinct hold filter: the shapes of a stream share one, which was checked once per shape.
  - The primary key is read only when a Tier C probe needs it, not for every change.
  - Columns are read without copying their text, routing keys are matched as the text PostgreSQL sent, and each relation's metrics are looked up once.
  - A resumed shape's replay reads its table's changes from its position on, not the whole buffer, and passes over the changes that are not for it by a hash of their routing columns kept in the buffer (32 bytes per change), without reading their tuples: from a buffer of 4,096 changes, 850 µs before, 18 after (`BenchmarkReplay`). Without that, a reconnect of 4,500 streams that all resumed a table taking 1,150 changes a second spent half its CPU reading the buffer, and live changes waited seconds behind it.

### Added

- **Warnings before the slot is lost.** A logical slot keeps the WAL from its restart point, which the oldest transaction open in the database holds back, whatever tables it wrote. On PostgreSQL 18.6, with the slot consumed to 208 bytes from the WAL end, a transaction left open after inserting one row into a table outside the publication lost a slot bounded at 64 MB after 60 MB of other writes; `wal_status` went from `reserved` to `lost` in one checkpoint. Sluice then stops, and every client reads its state again.
  - `sluice_slot_safe_wal_bytes`: the slot's `safe_wal_size`, read each `SLUICE_CATALOG_REFRESH`; absent while `max_slot_wal_keep_size` is -1.
  - Once the slot retains half of the WAL it may keep, the process reading it logs `replication slot nears max_slot_wal_keep_size` (a warning; an error from 80%) and `/diagnostics` reports `slot_wal_retention`, with the cause: the oldest open transaction (a session's pid or a prepared transaction, readable by any role), or how much of it Sluice has not confirmed yet.
- **Clients learn when changes arrive late.** When the reader's lag reaches `SLUICE_LAG_DEGRADED` (5 s), every stream gets a `delay` event with `lag_ms`, and another with `lag_ms: 0` when the lag falls below half of it; a stream opened meanwhile has `lag_ms` in its `ready`. Before, a client could not tell late changes from none: heartbeats keep arriving while the reader is behind. The SDK reports it as a new connection status, `delayed`, between `open` states; a `switch` over `ConnectionStatus` that lists every case needs it.
- New metrics: `sluice_resumes_total{result}` (`resumed` or `not_resumed`), and `sluice_resume_buffer_seconds`, how far back the buffer covers every table.

## 0.10.0

A stream opened while the replication slot was being created could miss changes; a process now refuses new streams until it replicates. In issuer mode every shape of a request goes to the issuer in one call (breaking for issuers), and after a reconnect of every stream the streams that fit are back within seconds. Tested with PostgreSQL 18.6.

### Fixed

- **A stream opened while the slot was being created no longer misses changes.** The process opened streams before its reader had created the slot. Creating it waits for every transaction that had an xid when it started, and a transaction that commits before the slot is ready is never decoded.
  - A stream opened during that wait got its `ready`, and the SDK's `onLive`, with `wal_lsn` `0/0`. A client that then read its state, as it should after `resumed: false`, or that asked for an initial snapshot, missed every change committed between that read and the end of the wait, until its next read.
  - That happened at the first start, and when a lost slot is created again, just as every client reconnects.
  - On PostgreSQL 18.6, with a transaction holding the creation for 3 s, a row committed after the `ready` of two streams, one with an initial snapshot, reached neither. A row committed once replication had started reached both.

### Changed

- **Breaking, issuer mode: Sluice asks the issuer about every shape of a request in one call.** A stream opening, `POST /subscribe` and `POST /token` each make one call, instead of one per shape. An issuer must change to the new format.
  - The request's `requested` is a list, one entry per shape with the client's label as `sub`; the response is `{"results": [...]}`, one result per requested shape, in order, each what a response used to be (`allow`, `shape`, `holds`).
  - A result with `allow: false` denies its shape only. A call that gets no verdict (unreachable, a timeout, a `5xx`, `408` or `429`, an unreadable body, or `results` that do not answer each shape) leaves every shape of the call `issuer_unavailable`; a redirect or another `4xx` denies them all.
  - Before: a stream with 7 shapes made 7 calls, 4 at a time, and `POST /token` renewed them in turn, one after the other. In a reconnect of every stream, the calls queued at the issuer, timed out, and were retried.
  - Measured on 1 CPU and 512 MB in a reconnect of all of a full node's 5,775 streams, with a test issuer charging 5–20 ms per call plus 2 ms per shape: no `issuer_unavailable` (938 before), a third less CPU in the reconnect (31 s against 44), opens at p50 0.24 s instead of 2.7 s, and message p99 18 ms instead of 95 ms on the streams already back.
- **After a reconnect of every stream, the streams that fit come back.** A reconnect lifts the memory admission measures: streams still being set up, and the garbage of setting them up, counted live because it was allocated during a collection. 0.9 stopped short, at 84% within 30 s and 96% after 4 minutes, and advised capping `SLUICE_MAX_STREAMS` at about 85% of what admission let in; that advice is withdrawn.
  - Up to the most streams admitted below `SLUICE_ADMIT_MEMORY`, for 5 minutes after the node last held that many, the projection may go a fifth of the way from the threshold to the Go memory limit: 0.6 at the default 0.5. Past that count the threshold applies, and past that ceiling a flood is still refused.
  - The 5-point hysteresis and the 10 s hold are gone: admission refuses while the projection is past its ceiling.
  - Measured on 1 CPU and 512 MB, issuer mode, with the single issuer call: all 5,773 streams were back within 10 s of a cut, the clients that had them within 14.3 s, with no `issuer_unavailable` and Go memory at its peak before the cut (442 of 461 MiB). With one call per shape the same change was not reliable: the issuer's queue lifted memory past the limit in one of two runs.
  - A node no client opened a stream on for 5 minutes before a reconnect refills with the threshold alone.
- **New streams are refused with `503 not_replicating` and a `Retry-After` of 1 to 5 s until the process starts replicating**, both while it creates the slot and while it stands by for the reader lock.
  - A process standing by used to accept streams and deliver nothing until it took over.
  - Once replication has started, streams keep opening while the reader reconnects: it resumes from the slot's confirmed position and loses nothing.
  - The SDK retries it as it retries `too_many_streams`. Refusals are logged as `streams refused` with `code=not_replicating`, at most once a second, and counted in `sluice_streams_refused_total{reason="not_replicating"}`.

### Added

- When creating the slot has taken 10 s, and every minute after, the log says what it waits for: `creating the replication slot is waiting for transactions open when it started`, with `waited` and `blocking_pids`.

## 0.9.1

Creating the replication slot no longer holds back `VACUUM` on every table. Tested with PostgreSQL 18.6.

### Fixed

- **Creating the replication slot no longer holds back `VACUUM` on every table.** Sluice asked PostgreSQL for an exported snapshot it never read. While the slot was being created, which waits for every transaction open when it started, `VACUUM` could then neither remove nor freeze, in any table, rows newer than the oldest of those transactions. Sluice now asks for none (`NOEXPORT_SNAPSHOT`), so only the catalogs are held, as for any logical slot. The slot is created at the first start and when `SLUICE_SLOT_RECREATE` replaces it; it starts at the same point as before. On PostgreSQL 18.6, a `VACUUM (FREEZE)` run while the slot was being created froze 0 of 863 pages of a table written just before; it now freezes 863 of 863.

## 0.9.0

Streams admitted by memory, hold checks that no longer depend on filter order or hold up `/token`, shared filters, and the reader's lag in seconds.

### Changed

- **`SLUICE_MAX_STREAMS=0` changes meaning, and is the new default.** It used to refuse every stream, since no stream count is below 0. It now means no fixed cap. The default was `50000`.
  - New streams are admitted by memory instead. Once the memory a collection cannot give back, plus what the streams opened since the last collection are expected to hold, reaches `SLUICE_ADMIT_MEMORY` (new; default `0.5`) of the Go memory limit, new streams are refused with `503 too_many_streams` and a `Retry-After` of 1 to 5 seconds. That memory is the live heap, the stack space goroutines are using and the runtime's own structures; free and fragmented heap, which later allocations reuse, is left out. Streams are admitted again 5 points below the threshold, or once the projection has stayed below it for 10 s: a reconnect storm's replays and garbage can lift that memory past the threshold for a few collections, and it can then settle just above the 5 points.
  - Opens in progress count too: each holds a reservation until the hub counts it, so the simultaneous opens of a reconnect storm cannot all pass on the same number of streams.
  - Streams closed since the last collection are taken off the projection, down to 20% of the measured floor, so that after a mass disconnect the memory the departed streams held does not refuse their reconnections until the next collection. When admission still refuses with far fewer streams than that collection saw, it forces one, at most every 5 s.
  - Capacity is decided before the token is verified. Refusals are logged once a second with their count (`streams refused`) instead of once per request.
  - A positive `SLUICE_MAX_STREAMS` still caps the count besides, with the same count (open and opening) and the same `Retry-After`.
  - With no memory limit known and `SLUICE_MAX_STREAMS=0`, nothing bounds streams, and the log warns at startup.
  - Why `0.5`: with 6,000 streams offered to 1 CPU and 512 MB, `0.5` admitted 5,767, about 42 KiB each of memory a collection cannot free, and delivered messages at p95 9 ms, with RSS peaking at 452 MiB. A threshold of `0.85` on the live heap alone admitted 7,215 of 12,000, but the collector then ran without pause and messages arrived 30–60 s late. On 0.8.x, with a cap above what fit, the process was killed for memory 11 times in 15 minutes.
  - After a reconnect of every stream, a node admission filled comes back to 72–79% within 30 s, 84–92% within a minute and 96% within 4 minutes (three runs), not all the way. While the reconnect runs, a collection counts its allocations as live, since they were made during the collection, and lifts the measure to 0.51–0.56 at 2,700–4,900 streams. After it, the node sits at the threshold with 96% of the streams: at 5,465 streams the live heap was 191–195 MiB, against 189–193 MiB at 5,777 before the cut. With `SLUICE_MAX_STREAMS` at 85% of what admission let in (4,900), 96% were back within 30 s and every stream within 55 s.
- Issuer mode: **a hold is listed under whichever of its equally good columns has the shortest lists.** Sluice counts, per hold table, how many holds share each value of each column. 0.8.1 took the first column in the filter, so a membership written scope first (`project_id=eq.42,user_id=eq.<sub>`) checked each change to a member's row against every hold in the project. The order of a hold's columns no longer matters, and holds already listed move at their next `/token`.
- A hold check that could not be made now drops the shape with the retryable `hold_check_unavailable` and a `retry_after_ms`, at subscribe and at `POST /token`. That covers a request cancelled or timed out, or a database failure. It was `shape_not_authorized`, which clients do not retry, so a `/token` that timed out under load lost its shapes for good. A hold row that does not exist is still `shape_not_authorized`.
- Calls to the shape issuer and to channel hooks carry an `Idempotency-Key` header. Idle connections to them close after `SLUICE_ISSUER_IDLE_TIMEOUT` (new; default 4 s; Go's default was 90 s).
  - Before: an endpoint that closed an idle keep-alive connection just as Sluice sent a request on it made that call fail, and the shape was refused or revoked with `issuer_unavailable`. Node closes idle connections after 5 s, Bun after 10 s.
  - Now: the request is re-sent on a new connection.

### Added

- The reader's lag in seconds: how long ago the transaction being dispatched committed, which is how late every change arrives. It is `sluice_reader_lag_seconds`, and `reader_lag_seconds` in `/readyz`.
  - Past `SLUICE_LAG_DEGRADED` (default 5 s), `/readyz` adds `"degraded": "reader_lag"` and stays `200`.
  - Past `SLUICE_READY_MAX_LAG` (unset by default), it answers `503`. Set this only with more than one node behind the balancer.
- Metrics:
  - `sluice_streams_refused_total{reason}` and `sluice_admission_memory_share`;
  - `sluice_memory_floor_bytes` and `sluice_heap_live_bytes`;
  - `sluice_index_size{index,kind}`, the sizes of the hold and subscription indexes, set at each scrape.
- A build with the `pprof` tag (`go build -tags pprof ./cmd/sluice`) serves Go's profiling handlers on `SLUICE_PPROF_ADDR`, apart from the public listener. Release images are built without it and do not change.
- Load test (`apps/loadtest`): production-shaped profiles — `rooms`, `bigtxn`, `reconnect`, `calibrate` and `ramp` — against an issuer-mode Sluice with hook channels, run with a production container's limits, with per-kind latency, CPU per change and per event, memory against `GOMEMLIMIT`, replication lag and resume-buffer coverage in every report.

### Fixed

- **A resumed shape could get changes out of commit order.** Its replay ran off the request while the reader went on delivering, so a change committed during the replay could arrive ahead of older ones being replayed. A client that upserts by key could then apply an older row over a newer one. The load test counted 279 such jumps in a room's message sequence after 4,255 resumes, in a reconnect of 5,610 streams on 1 CPU. Changes committed during a replay now reach the shape after it. Snapshots are unchanged: their rows, and the replay that follows them, are still sent while live changes arrive.
- A shape installed while the reader was dispatching a change could miss that change: it was routed before the shape existed and buffered after the shape's replay had read the buffer. Changes are now buffered before they are routed.
- A replay delivered inserts and deletes to shapes whose `ops` left them out. It keeps to the shape's `ops` now, as live delivery does.
- A channel join the hook gave no verdict on (a timeout, a `5xx`, an unreadable body) was refused with `channel_not_authorized`, which clients do not retry, so the channel was lost until the client joined again. In the same reconnect, 405 clients lost their room's channel this way. It is now `channel_hook_unavailable`, retryable, with a `retry_after_ms` of 1 to 5 s and never before the cached answer expires. A denial is still `channel_not_authorized`.

### Performance

- Identical filters are parsed once and shared, and collected with the last subscription or hold that uses one. They are keyed by the relation and its columns, so they are shared across catalog refreshes too. Parsed filters were 44% of the live heap at 12,000 streams.
- The events one transaction brings a stream are written together. The writer is woken at the commit, or at once when a quarter of its queue fills. Once the transaction has been dispatching for 5 ms, it is woken at the next row the reader dispatches, delivered or not.
- Hold checks no longer hold up stream opens and `POST /token`. Candidates are evaluated with no lock held, and the lock is taken only to remove a hold that broke.
- A resume copies from the buffer only the changes routed to its shape (by its routing key), not every change to the table since its position. Thousands of subscriptions resume at once after a cut; each copied about 250 changes for a room's messages and held them while it waited for room in its queue. In the same reconnect at the admission threshold, the streams back after a minute went from 84–88% to 92%.

### For operators

- If you set `SLUICE_MAX_STREAMS=0`, it now admits streams instead of refusing them all.
- Give the container a memory limit (or set `GOMEMLIMIT`) so that automatic admission has something to measure against. Watch `sluice_streams_refused_total` and `sluice_reader_lag_seconds`.
- To have every stream back within a minute of a reconnect, cap streams (`SLUICE_MAX_STREAMS`) at about 85% of what admission lets in under your load. Admission alone runs a full node at its threshold, where a reconnect leaves about a tenth waiting for minutes.

### For issuers

- The same request may reach you twice, with the same `Idempotency-Key`, when the first one went out on a connection you were closing. Answer it as any other.
- The order of a hold's columns no longer matters.

### For clients

- The SDK waits for the longer of a `429` or `503`'s `Retry-After` (plus up to a tenth at random) and its own backoff step before opening the stream again. Before, it applied only its backoff, which could come back after a few milliseconds; a refusal that lasts is now retried less and less often.
- `resumed: false` can follow a short disconnect too, for a table busy enough that the resume buffer, which counts every change to it, reaches back only seconds. The READMEs say so now, with measured coverage.
- `hold_check_unavailable` is a new retryable code, handled like `issuer_unavailable`.
- `channel_hook_unavailable` is too, for a channel join the hook gave no verdict on. The SDK's channel asks again after `retry_after_ms` and can send once the join is accepted. A channel that a retryable refusal ended did not recover before.

### Measured

The load test's rooms workload: 7 shapes and a hook channel per stream, membership holds, a message every 15 s per room updating every member's row, PostgreSQL 18.6.

| | 0.8.1 (measured) | 0.9.0 candidate (measured) |
| --- | --- | --- |
| 12,000 streams, 2,320 changes/s, 1 CPU, holds member first: CPU per 1,000 changes | 105 ms | 103 ms |
| Holds written room first | 365 ms per 1,000 changes (1,500 streams, 1 CPU) | the same as member first (microbenchmark below) |
| Live heap at 12,000 streams | 546–572 MiB (46–48 KiB per stream) | 419 MiB (36 KiB per stream) |
| Go memory at 12,000 streams | 1,291 MiB | 1,027 MiB |
| More streams offered than fit in 1 CPU and 512 MB | 12,000 offered, cap 15,000: killed for memory 11 times | 6,000 offered: 5,767 admitted, message p95 9 ms, no kill, RSS peak 452 MiB |
| A member's update checked against holds (microbenchmark, 400 rooms × 28 streams × 7 holds) | 107 µs room first, 3.8 µs member first | 4.8 µs, 4.3 µs |
| `/token` (`Replace`) while the reader checks holds back to back (microbenchmark, Linux) | mean 12 µs, p99.9 2.1 ms (with item 2 only) | mean 2.4 µs, p99.9 0.35 ms |

0.8.2 keeps each subscription's request, about 550 bytes per shape, which 0.9.0's shared filters more than pay back.

## 0.8.2

Issuer mode: `POST /token` no longer grows each shape's filter.

### Fixed

- Each `POST /token` asked the issuer about a shape's effective filter instead of the one the client subscribed with, and narrowed that again with the grant, so every renewal added the grant's terms to the filter once more (`project_id=eq.42,project_id=eq.42,…`). Affected: issuer mode (`SLUICE_SHAPE_ORACLE=issuer`) whenever clients renew their token on an open stream; the more renewals and the longer the process runs, the worse. The RLS oracle was not affected. The process's memory grew by about 320 bytes per shape and renewal, and the cost of matching each change to the table grew with it. The issuer is now asked about the shape as the client requested it, as at subscribe.

  Measured in a load test that renewed every token every 2 minutes (12,000 streams of 7 shapes, 1 CPU, 10 minutes): the live heap grew 13.7 MiB a minute on 0.8.1 and 0.9 MiB a minute with this fix, and `/token` p95 fell from 528 to 289 ms. What remains of that growth is being looked into. Keeping the client's request costs about 550 bytes per shape.

- For the same reason, a grant that widened at a `/token` (dropping a term it had required) left the shape as narrow as before. The shape now follows the grant.

### For issuers

- On `refresh`, `requested` is now what the client subscribed with (`filter`, `columns` and `ops`), as on `subscribe`. It was the effective shape: the previous grant's filter, repeated once per earlier renewal, followed by the client's, and only the granted columns.

### For clients

- Nothing changes. The SDK has no changes: `@pauserratgutierrez/sluice-js` 0.8.2 is 0.8.1's code, published under the server's version.

## 0.8.1

Issuer mode: hold watches that were never removed, and grants dropped by updates that could not affect them.

### Fixed

- A hold filter with two columns that both lead an index and are in the replica identity, such as a membership `project_id=eq.42,user_id=eq.<sub>`, was indexed under one of them and, about half the time, looked up under the other when it was removed. Every `POST /token` (which re-grants each shape) and every stream that closed could leave a watch behind. The process's memory grew with them for as long as it ran, and so did the CPU spent on each change to the hold table, which checked every one left on its value. In a load test with such holds (1,500 streams, 337 changes/s, `/token` every 15 minutes), the live heap grew 39% in 29 minutes and CPU per change 31% (see Measured).
- Such a leftover watch could drop a live shape with `shape_not_authorized`: when the row of an earlier hold of the same subscription was deleted, or updated out of its filter, though the hold of the current grant still existed. An issuer that holds each grant on a row it replaces at every `/token` would see it. Only the watch currently registered for a subscription can cut it now; a leftover one is removed and cuts nothing.
- With `REPLICA IDENTITY DEFAULT` or `USING INDEX` on a hold table, an `UPDATE` that changed no column of the replica identity (which arrives without the old row) dropped with `shape_not_authorized` every grant held by another row sharing a value with it: updating any column of one member's row dropped the grants of the other members of the same project, or the member's grants held by their other projects, depending on which column the holds were routed by. Such an `UPDATE` cuts nothing now: a hold reads only replica-identity columns, so none of them changed. `REPLICA IDENTITY FULL` was not affected.

### Changed

- A hold may read only columns of its table's replica identity, which Sluice checked when granting it and on each catalog refresh (`SLUICE_CATALOG_REFRESH`, 30 s). When a table's replica identity stops covering a hold's columns (for example `ALTER TABLE ... REPLICA IDENTITY DEFAULT` while a hold filters a column outside the primary key), each such shape is now dropped as soon as the reader sees the table's new definition in the WAL, before it dispatches another change, instead of at the next catalog refresh; the first time it sees the table after a restart counts too. A grant or a `POST /token` re-grant with such a hold, made before the catalog catches up, is refused the same way. The client gets an `error` event with `code: "shape_not_authorized"`, the subscription's `sub`, and a message naming the columns and the remedy (a unique index over them and `REPLICA IDENTITY USING INDEX`); the stream and its other subscriptions continue. The log has one `hold filters read columns outside the replica identity; their shapes were dropped` warning per table, with the count. `/diagnostics` lists no `hold_replica_identity` finding for them, since they are no longer installed.
- Among columns that are equally good routing keys (leading an index and in the replica identity), shapes and holds are routed by the first in the filter, where either could be picked before.

### For issuers

- Put the most selective column first in hold filters: in a membership, the member before the scope (`user_id=eq.<sub>,project_id=eq.42`). A change to one member's row is then checked against that member's holds, not every hold in the project. In the same load test, where each message updates every member's row of its room, this order cut Sluice's CPU per change by 75–80% (1,500 streams at 337 changes/s on 1 CPU; 12,000 streams at 2,320 changes/s on 2 CPUs, where message latency p95 fell from 305 ms to 6 ms).

### For clients

- Nothing changes. The SDK has no changes: `@pauserratgutierrez/sluice-js` 0.8.1 is 0.8.0's code, published under the server's version.

### Measured

The load test's rooms workload (`apps/loadtest`): 7 shapes and a hook channel per stream, membership holds written member first (room first on 0.8.0, which keyed them on either column at random), a message every 15 s per room that updates every member's row, PostgreSQL 18.6, `POST /token` every 15 minutes. CPU per 1,000 changes over the first 10 minutes of the steady phase:

| | 0.8.0 | 0.8.1 |
| --- | --- | --- |
| 1,500 streams, 337 changes/s, 1 CPU, 512 MB | 365–387 ms (0.13 cores) | 104 ms with `REPLICA IDENTITY FULL`, 91 ms with `DEFAULT` (0.03 cores) |
| 12,000 streams, 2,320 changes/s, 1 CPU, 4 GB | 499 ms; falls behind (about 1,950 changes/s dispatched, fewer as it runs) | 105 ms with `FULL`, 108 ms with `DEFAULT` (0.25 cores; message p95 10 ms) |
| 12,000 streams, 2,320 changes/s, 2 CPUs, 4 GB | 535 ms (1.24 cores; message p95 305 ms) | 105 ms (0.24 cores; message p95 6 ms) |

- 12,000 such streams fit 1 CPU but not 512 MB. They need about 1.3 GiB of Go memory, and in a 512 MB container the process was killed for memory 11 times in 15 minutes.
- Memory per stream (Go memory, the figure `GOMEMLIMIT` bounds):
  - live heap ~44 KiB idle and ~48 KiB under that load;
  - about twice that between collections: ~93 KiB idle (15,000 streams in 1,359 MiB), 110–120 KiB under load.
- Extrapolated from those, with the live heap kept under 40% of `GOMEMLIMIT` (room for the heap to double between collections and for a burst of reconnections, whose cost is not yet measured): about 4,000 such streams per 512 MB container and 8,000 per GB.
- Growth of the live heap at a constant load:
  - 1,500 streams: flat (72 → 74 MiB over 10 minutes).
  - 12,000 streams: down from about 15 MiB a minute on 0.8.0 to 2–3 MiB a minute (545 → 572 MiB over 10 minutes, with about 800 `POST /token` a minute).

  A possible residual retention per `/token` is being looked into.

## 0.8.0

Request IDs: Sluice joins the trace of the requests it serves.

### Added

- Every HTTP request has an ID: the value of `SLUICE_REQUEST_ID_HEADER` (default `X-Request-ID`) when it is 1 to 128 visible ASCII characters, else a new UUID. It is sent in the same header to the shape issuer and to channel hooks on the calls made for the request, and logged as `request_id` on every line about the request. Responses and events do not carry it.
- Log lines for requests, each with `request_id`: `request refused` for every error response, `subscription refused` for each refused shape or channel (at most 20 per request, then `more subscriptions refused`), and `subscription revoked` for each shape or hook channel `POST /token` drops. Refusals that are not the caller's doing (no verdict from the issuer or a hook, a `5xx`, a node-wide limit) are warnings.
- `stream ended`, when the server ends a stream (not when the client disconnects), under the ID of the request that opened it. A line about a later request on a stream adds `stream_id` and that ID as `stream_request_id`. A lost slot still ends streams with `server_shutdown`; their lines add `cause: replication_stopped`.
- `session revoked; its streams were closed` and `user banned; their streams were closed`, with `commit_lsn`, when a revocation read from the slot closes streams. They have no request ID.
- `SLUICE_REQUEST_ID_HEADER`. Startup refuses `Authorization`, `Proxy-Authorization` and `Cookie`.

### Changed

- `session lookup failed` is now `request refused` with `code: session_check_unavailable`, the error in `err`.

### For issuers and hook endpoints

- Calls made for a request (a subscribe, a join, `POST /token`) carry its ID in `SLUICE_REQUEST_ID_HEADER`; log it to join Sluice's lines. Hook re-checks on the `SLUICE_CATALOG_REFRESH` tick serve no request and carry none. Treat it as a label only: unless a gateway overwrites the header, the client chose it.

### For operators

- Set the header at the gateway on every request, overwriting any value a client sent, and log it there: one ID then follows a request through the gateway, Sluice, the issuer and the hooks.
- More lines: one per refused request and refused subscription, and one per stream when the process stops.

### For clients

- Nothing changes: requests, responses and events are the same.

## 0.7.0

### Fixed

- A replication slot invalidated on PostgreSQL 18 (`max_slot_wal_keep_size` exceeded, `idle_replication_slot_timeout`), or dropped while Sluice ran (a database restore), left the reader retrying forever: streams stayed open with heartbeats only and no error. PostgreSQL 18 reports an invalidated slot with a message Sluice did not recognise. After a failed stream the reader now reads the slot from `pg_replication_slots`, and when it can no longer stream every change the process exits with an error, ending every stream with `server_shutdown`.
- SDK: a shape that had received no change before its stream dropped reconnected without a resume position, so the changes made while it was disconnected were lost without notice, on every server restart among other times. Shapes now resume from the position the server gives when they go live.
- An issuer timeout, network error or `5xx` refused the shape as `shape_not_authorized`, which the SDK forgets: a brief issuer outage during a reconnect wave or a `/token` ejected live subscriptions for good. It is now the retryable `issuer_unavailable`.
- `TRUNCATE` of a hold table did not cut the shapes it held. They are now dropped with `shape_not_authorized`.

### Changed

- **Breaking (protocol):** whether a resume is covered is in the subscription result, as `resumed: true | false`, instead of an asynchronous `resume_too_old` error. `resume_too_old` is now only sent when the buffer no longer reaches an initial snapshot's floor. Clients other than the SDK that send `resume` must read `resumed`.
- `wal_lsn` in `ready` is taken before the subscriptions are installed, and `/subscribe` returns one too: resuming from it replays whatever those subscriptions missed.
- An issuer `4xx` other than `408` and `429`, a redirect and `"allow": false` still deny. A `5xx`, `408`, `429`, timeout, network error or unreadable body is `issuer_unavailable`, with the issuer's `Retry-After` (at most 60 s) or a delay spread between 1 and 5 s as `retry_after_ms`.
- `POST /token` in issuer mode drops a shape the issuer gives no verdict on with `issuer_unavailable` (it was `shape_not_authorized`), and counts it in `revoked_subscriptions`. A new token never keeps a grant made for the previous one.
- The resume buffer grows with each table's traffic up to `SLUICE_RING_EVENTS`, instead of reserving every slot on the first change, and is bounded in bytes by `SLUICE_RING_MAX_BYTES` across every table, evicting the oldest change first.
- The replication slot is created by the process that takes the reader lock, after taking it; a process standing by no longer creates it.
- With `SLUICE_REVOCATION_ENABLED=true`, the revocation tables (`SLUICE_REVOCATION_SESSIONS_TABLE`, `SLUICE_REVOCATION_USERS_TABLE`) are exempt from issuer mode's startup check for `SELECT` on every published table: they are published only for their changes. A shape on one of them is refused with `shape_not_authorized`.
- SDK: a subscription refused with a retryable error stays registered and is asked for again after `retry_after_ms`. A subscription error with `retry_after_ms` means the server removed it and the client subscribes again. The client no longer forgets a position on `resume_too_old`.
- Harness: the Sluice containers' healthcheck is `-readycheck`.

### Added

- SDK: `onLive` on shapes, called each time the server installs the subscription, with `{ sub, reason: 'subscribed' | 'resubscribed', resumed, walLsn }`. It fires once the shape is live, so a read made from it cannot miss a change. `resumed` is true only when the server replays everything since the previous stream. Type `LiveEvent`.
- `issuer_unavailable` subscription error: retryable, with `retry_after_ms`.
- `SLUICE_RING_EVENTS=0` disables the resume buffer; it needs `SLUICE_SNAPSHOT_ENABLED=false`. Every resume is then `resumed: false`.
- `SLUICE_RING_MAX_BYTES` (default `67108864`, 64 MiB).
- `SLUICE_SLOT_RECREATE` (default `false`). With `true`, an invalidated slot is dropped and recreated at startup instead of stopping the process, never while another process holds it; it logs a warning and counts `sluice_slot_recreated_total`.
- `SLUICE_REVOCATION_SESSION_LOOKUP` (default `false`; needs `SLUICE_REVOCATION_ENABLED=true`). `POST /stream` and `POST /token` check that the token's session row still exists, so a session signed out before a restart cannot reconnect: `401` when it is gone, `503 session_check_unavailable` when the lookup fails. Found sessions are trusted for 30 s.
- `sluice -readycheck` probes `/readyz`.
- Metrics `sluice_slot_recreated_total` and `sluice_resume_buffer_bytes`; `result="unavailable"` on `sluice_authz_resolutions_total` and `sluice_authz_lease_refreshes_total`.

### For clients

- Register `onLive` before `subscribe()` and read the current state there unless `e.resumed`. The first `onLive` (`reason: 'subscribed'`) replaces a read made after subscribing; it runs before `subscribe()` resolves.
- `issuer_unavailable` is retryable and handled by the SDK: `subscribe()` resolves with `ok: false` and `onError` receives it with `retryable: true`, but the subscription stays registered and `onLive` follows once it is live. Treat only `retryable: false` errors as the end of a subscription.
- `POST /stream` and `POST /token` may answer `503 session_check_unavailable` with the session lookup on; the SDK retries the stream (`http_503`), and `setAuth` throws a retryable error.

### For issuers

- Deny with `200 {"allow": false}` or a `4xx`. A `5xx`, `408` or `429` is an outage: the client is told to try again, after your `Retry-After` when you send one.

### For operators

- With `SLUICE_REVOCATION_SESSION_LOOKUP=true`: `GRANT SELECT (id) ON <sessions table> TO <authz role>;`. Startup refuses without it. No table `SELECT` is needed, in issuer mode either. The `id` column must be `uuid` or text holding lowercase ids. If the table has row-level security and the role does not bypass it, add a `SELECT` policy for the role; startup warns (`session_lookup_rls`) when none applies.
- Health: use `sluice -readycheck` (`/readyz`) for a single-process deployment. Docker Compose does not restart unhealthy containers; Sluice exits on its own when its slot is gone, and needs a restart policy.
- An invalidated slot now stops the process at startup until it is dropped, or until `SLUICE_SLOT_RECREATE=true` replaces it.
- Size `SLUICE_MAX_STREAMS` to the memory limit (about 8 000 five-shape streams in 512 MB, measured with 0.4.0).

## 0.6.0

### Added

- `SLUICE_JWT_SESSION_CLAIM` (default `session_id`) names the claim that carries the session id, so session revocation works with identity services other than GoTrue (Better Auth: `sid`). The session id is lowercased on both sides of the comparison.
- `SLUICE_JWT_REQUIRE_ROLE` (default `true`). With `false`, tokens without a `role` are accepted and the claim is ignored even when present; the identity's role is empty, `SLUICE_ALLOWED_ROLES` is unused, and startup checks no roles. Only accepted with `SLUICE_SHAPE_ORACLE=issuer`. In that mode no JWT is `service_role`, so `/admin/jwks/refresh` and `/diagnostics` are unreachable with a token; `/admin/shapes/drop` still takes the issuer bearer.
- `SLUICE_REVOCATION_USERS_BAN_COLUMN` (default `banned_until`).

### Changed

- **Breaking:** `SLUICE_REVOCATION_USERS_TABLE` has no default. Unset, there is no user-level revocation. A GoTrue deployment that relies on bans closing streams must now set `SLUICE_REVOCATION_USERS_TABLE=auth.users`.

### For operators

- Upgrading with `SLUICE_REVOCATION_ENABLED=true` and GoTrue bans: set `SLUICE_REVOCATION_USERS_TABLE=auth.users`. Nothing else changes for GoTrue tokens.
- The issuer and hook payloads now carry `"role": ""` for tokens accepted without a role.

## 0.5.0

### Fixed

- SDK: a client stopped by a refused stream (`401`, `403`) reported `reconnecting` although it no longer retried. It now reports `closed`; `setAuth` reopens it.
- SDK: after the server ended a stream, the status stayed `open` while the client waited to reconnect, for up to `retry_after_ms` after `server_shutdown`.

### Changed

- SDK: `reconnecting` is reported only when the stream drops or an attempt to open it fails. Replacing a stream the server ended (`token_expired`, `server_shutdown`) is `connecting`.

### Added

- SDK: `idle` connection status, for a client with nothing subscribed or a hidden tab. It is also the status of a new client, which reported `closed` before.

### For clients

- A `switch` over `ConnectionStatus` needs an `idle` case. A "connection lost" notice can show `reconnecting` as is, without a delay to hide planned reconnects.

## 0.4.0

### Fixed

- A ban that had run out still banned. GoTrue leaves `banned_until` set when a timed ban ends, and any later `UPDATE` of that user (a sign-in sets `last_sign_in_at`) closed their streams with `user_banned` and refused their tokens for two hours. A ban is now in force only while `banned_until` is in the future, is enforced until then (at most two hours), and an `UPDATE` that clears it lifts it.
- A snapshot or resume larger than the stream queue closed the stream with `stream_lagging`: its rows were queued faster than any client reads them. Snapshot rows, `snapshot_end` and replays now wait for room, filling at most half the queue so live changes keep the other half.
- Sluice never exited on `SIGTERM`: closing the database pool waited forever for the connection holding the reader lock, so every stop ended in a `SIGKILL`. It now exits in well under a second.
- Shutdown left every stream open until `SLUICE_SHUTDOWN_GRACE` ran out, so a container stop with a shorter timeout killed the process with no event sent. Streams now end at once with `server_shutdown` (see Added), and `/readyz` reports `503` while draining.
- A resume replay that outlived its subscription kept delivering to it; it now stops, and follows a `/token` rebind of filter and columns.

### Changed

- The stream queue grows as events arrive, up to `SLUICE_STREAM_QUEUE`, so an idle stream holds no buffer (it reserved about 12 KiB before). Everything queued is written together and flushed once, instead of one write and flush per event.
- A change is projected and encoded to JSON once per distinct column list, and shared by every subscriber with that projection, instead of once per subscriber.
- Shapes in one `/stream` or `/subscribe` request are resolved concurrently (up to four at once) and installed in request order, with the same limits.
- Column privileges are cached until the next `SLUICE_CATALOG_REFRESH` tick, so a reconnect wave costs one query per table and role instead of one per shape. A `GRANT` or `REVOKE` reaches new subscriptions within one tick.
- The issuer and hook HTTP clients keep up to 64 idle connections per host (Go's default is 2).
- A Tier C probe takes two round trips instead of four, and every probe and Tier B cross-check is bounded by `SLUICE_TIER_C_TIMEOUT`; one that runs out of time withholds the change.
- Unless `GOMEMLIMIT` is set, the Go heap's soft limit is 90% of the container's cgroup memory limit.
- Keep-alive connections idle between requests are closed after five minutes.
- SDK: `subscribe()` calls made in the same tick are sent in one `/subscribe`.

### Added

- `server_shutdown` stream error, with `retry_after_ms` drawn from `SLUICE_RECONNECT_SPREAD` (default `10s`) so clients reconnect spread out. `POST /stream` answers `503 server_shutdown` while shutting down.
- `SLUICE_TIER_C_TIMEOUT` (default `1s`).
- SDK: a stream-scoped error's `retry_after_ms` sets the delay before the next reconnect, and is exposed as `SluiceError.retryAfterMs`.

### For clients

- `error` events may carry `retry_after_ms`. The SDK honors it; other clients should wait that long before reconnecting.

## 0.3.0

Hook channel joins are now leases on their verdict.

### Changed

- A stream that joined a `hook` channel is asked about again once the verdict expires, on the next `SLUICE_CATALOG_REFRESH` tick, and on `POST /token`. A denial removes the join with a `channel_not_authorized` error for that subscription; before, a join lasted as long as the stream. A re-check that gets no verdict (timeout, 5xx, unreadable body, 401) keeps the join and retries on the next tick.
- `ttl: 0` in a hook response means the verdict is not cached, for allows and denials alike: every join asks, and a joined stream is asked again on every tick. Before, `0` meant "use `SLUICE_CHANNEL_HOOK_TTL`". Omit `ttl` to get that default.
- A verdict's validity is shortened at random by up to a fifth, so joins made together do not all expire on the same tick.
- `POST /token` counts revoked hook channels in `revoked_subscriptions`.
- SDK: when the server removes a channel join, or refuses it when a reconnect resends it, `Channel.onError` receives the error, `send` throws, `channel.presence` is emptied, and `subscribe()` may be called again, including from inside `onError`. Before, the channel kept reporting itself subscribed.
- SDK: a subscription ended by a non-retryable error is forgotten before its error handler runs.

### Added

- Metric `sluice_channel_hook_rechecks_total{result="held|revoked|unavailable"}`.

### For hook endpoints

- Each joined verdict (channel and session) now costs one call per `max(ttl, SLUICE_CATALOG_REFRESH)` while the join lasts; with `ttl: 0`, one per tick. A session's streams on the same channel share one call.
- Return a short `ttl` for grants that can be withdrawn (blocks, removals). A revocation reaches an open stream within the verdict's validity plus one tick.

## 0.2.2

### Fixed

- The catalog refresh ran with PostgreSQL's JIT enabled, and the type query's cost estimate made it compile on every refresh: 0.4 to 1.2 s per refresh, now about 10 ms. The refresh runs in one read-only transaction with `jit = off`.

## 0.2.1

### Changed

- Live changes are encoded as `to_jsonb` encodes them, the same as snapshot rows and PostgREST: ISO 8601 timestamps with `timestamptz` in UTC, arrays as JSON arrays, composite types as objects, domains as their base type. Before, live changes carried PostgreSQL's text output (`2026-09-29 20:24:42+00`, `{a,b}`). The exceptions are listed in [ROADMAP.md](ROADMAP.md#known-limitations).
- Snapshots read columns in text format and share the live encoder, so both paths cannot drift. Snapshots of tables with a column named `t` no longer fail.

## 0.2.0

Published by mistake from the same commit as 0.1.7. Identical to 0.1.7.