CHANGELOG.md
Rama main · réplica de lectura del repositorio original.
# Changelog
All notable changes to Brokoli are documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project loosely follows [Semantic Versioning](https://semver.org/).
Every entry names its owner. With more than one contributor in the
codebase, "who shipped this" is part of the record, not something to
reconstruct from git archaeology.
## [Unreleased]
### Changed
- **A cancelled run no longer calls the pipeline's `on_failure` hook**
-- @hc12r. It called it with `error: cancelled by user`, so a hook that
pages on failure paged for every cancel. A cancel is someone's
decision, not a failure: it calls `on_start` when the run starts and no
failure hook when it ends, as it already raised no failure alert and no
dead-letter entry.
- **Routes grant what the caller's role lists** -- @hc12r. Without a
distribution's permission system, routes decided by role name: `admin`
and `editor` could do everything, `viewer` could read, and every other
role was refused. They now ask the roles table, as
`GET /api/auth/me/permissions` does. So an editor holds what the editor
role lists and no longer manages users, roles, workspace members or
tokens, edits settings, exports the audit log or reads secret variable
values; a viewer may now read workspaces, the audit log and Git sync
status, which the viewer role lists. A role the server does not know
holds nothing, and the permissions endpoint now says so with an empty
list, where it answered with a viewer's permissions. (#864)
- **Bulk enable and disable need `pipelines.edit`** -- @hc12r. `POST
/api/pipelines/bulk` asked for `pipelines.delete` whatever the action.
Enable and disable now ask for `pipelines.edit`, as editing a pipeline
does, and delete still asks for `pipelines.delete`. (#593)
### Fixed
- **Operator and custom roles may do what they list** -- @hc12r. A person
with the `operator` role, told by `GET /api/auth/me/permissions` that
they may run a pipeline and cancel and resume runs, was refused with
403 by every one of those routes, and a custom role got nothing it
listed, so the UI offered actions the server then refused. (#864)
- **A run recovery fails shows up like any failed run** -- @hc12r. When
a process died mid-run and recovery could not tell how far the run got,
it marked the run failed and said so only in
`brokoli_runs_recovery_failed_total` and an error log line: no
dead-letter entry, no failure alert, no notification. It now adds all
three, once, carrying the recovery reason. (#1013)
- **The UI shows whole numbers past 2^53 exactly** -- @hc12r. The API
sends them exactly, but the UI parsed every response into JavaScript
doubles, so a node's output sample and a pipeline preview showed
`9007199254740993` as `9007199254740992`, and two different ids could
look the same. Those responses now keep such integers as their exact
text; every other number is unchanged. (#1001)
- **Bulk pipeline actions do what the single-pipeline endpoints do** --
@hc12r. `POST /api/pipelines/bulk` counted an action it did not know as
succeeded for every id, and changed nothing. It now answers 400. Its
delete went straight to the store, so pipelines that depended on a
deleted one were left pointing at nothing, and the schedule and audit
log were not updated. A pipeline others depend on is now refused, with
the dependents named, while the rest of the batch goes ahead. A
dependent named in the same request does not block it, in any order.
Enable and disable could change a Git-managed pipeline, recorded no
version and left the schedule stale until the next reconcile. They now
go through the same save as an edit. (#593)
- **A dataset that misses Arrow says why** -- @hc12r. A dataset written
as NDJSON because it could not be written exactly as Arrow (a nested
value, a column mixing types, a column null in every row, or an integer
past 2^53 among fractions) fell back without a word, so it could miss
the faster format on every run and nobody would know. The log now names
the column and the cause, once per cause, and with
`BROKOLI_STREAM_CODEC=arrow` the refusal says the same. (#529)
- **A remote job no worker takes names what it needs** -- @hc12r. It
failed with "timed out waiting for a worker" and no more, which reads
the same whether no worker was running or none advertised a capability
the job requires. The failure now says how long it waited and lists the
capabilities a worker must advertise to take it. (#529)
## [0.22.1] - 2026-11-11
A security patch: v0.22.0 rebuilt with Go 1.26.11. Nothing else changes.
### Security
- **Go 1.26.11** -- @hc12r. GO-2026-6859 (CVE-2026-97054): in Go 1.26.10,
when a request body passed the limit of `http.MaxBytesReader`, which the
API uses for its request size limits, net/http set `Connection: close`
without synchronization, and an HTTP/1 server whose handler read the
body while writing response headers could panic. v0.22.0 was built
with Go 1.26.10. The toolchain pin, CI and release builds move to the
fixed release.
## [0.22.0] - 2026-11-11
A mapping and isolation release. `map` runs a node once per item of an
upstream node, in parallel and with its own status per item, on code,
`source_db`, `task` and `source_api` nodes, within limits an operator
sets. Agents can message people, be messaged back, and subscribe to run
events. User code can run as its tenant's own unprivileged user. And a
long run of fixes keeps every change, token, variable, connection, log and
count inside the workspace it belongs to.
### Changed
These can change what an existing deployment does.
- **Changes are made only in the current workspace** -- @hc12r. Someone
in two workspaces could run, edit, delete or cancel the other
workspace's items by id. Reading by id still works in any workspace a
person belongs to; a change to an item of another workspace now answers
"not found". A workspace token now acts in its own workspace only.
- **A run parameter of 2^53 or more sent as a plain JSON number is refused
with a 400** -- @hc12r. It used to run with a rounded number. Send it
tagged, `{"$bptd": "int64", "value": "..."}`, as the message says.
- **Mapping has ceilings** -- @hc12r. An `expansion.max_instances` above
100000, accepted before, is refused unless `BROKOLI_MAP_MAX_ITEMS` is
raised, and a `map` block on a node type that does not support it is
refused at save instead of being ignored.
- **dbt nodes get the filtered environment** -- @hc12r. A dbt project
that reads a server variable through `env_var()` needs it named in
`BROKOLI_DBT_PASS_ENV`.
- **Task nodes get their pipeline's declared parameters** -- @hc12r. In a
pipeline that declares parameters, a task receives the resolved, typed
values, and the run's other string parameters are no longer passed to
it. A pipeline that declares none passes its string parameters as
before.
- **The engine is non-dumpable** -- @hc12r. Its `/proc` files are readable
only with `CAP_SYS_PTRACE`, so a debugger or profiler attaching to the
server needs that capability.
- **Testing a change request needs edit permission**, not only run
permission -- @hc12r.
- **An archive with two entries for the same file is refused** -- @hc12r.
This applies to task bundles, plugin packages and driver archives.
- **`GET /api/pipelines/summary` answers 500 when it cannot read** --
@hc12r. It answered `200 []`, so an outage looked like an empty
workspace.
- **For Go embedders** -- @hc12r. `api.ValidateOrgAccess` and
`api.ValidateWorkspaceAccess` are removed; use `RequestScope(r)`.
`store.NewSQLiteStore` refuses `:memory:`, the empty path and in-memory
URIs; use a file.
### Added
- **Task interface digests, with expected values for every SDK** --
@hc12r. `pkg/taskinterface` computes the semantic digest of
`task-interface-canonicalization.md` section 5 (`CanonicalRendering`,
`Digest`, `InterfaceDigest`), with the canonical rendering of
`ir-canonicalization.md` section 2, checked against its 35 cross-SDK
vectors. Section 5 now says exactly what is hashed: record fields sorted
by name, enum values in their order, a bundle's `interface_digest` over
the node interface alone (the unknown-rows interface when a task declares
none), and a separate digest for its pipeline parameters. Each
differential vector carries the expected rendering and digest. They agree
with the values the JVM SDK and the Python SDK's renderer compute. (#990)
- **One rule for numbers and key order in canonical IR** -- @hc12r.
`ir-canonicalization.md` claimed Python's `json.dumps` and JavaScript's
`JSON.stringify` write the same bytes. They do not: Python writes `1e-05`
and `1.2345678901234568e+20` where JavaScript and Go write `0.00001` and
`123456789012345680000`, and `JSON.stringify` of a key-sorted object puts
integer-like keys first. So the two SDKs could compute different digests
for the same pipeline. Section 2 now specifies RFC 8785's number layout
and code-point key order for every key, and
`docs/schema/fixtures/ir-canonicalization/rendering.json` holds 35
cross-SDK vectors that both SDKs test byte for byte.
`task-interface-canonicalization.md` names the same number layout. (#982)
- **`map`: a code node runs once per item, in parallel** -- @hc12r. A
`code` node with a `map` block (`over`, `key`, `max_items`,
`max_parallel`, `on_item_failure`) runs once per row of an upstream
node, each row its own instance with its own status, up to
`max_parallel` (default 4) at once on the local server, where the
`expansion` block `.expand()` compiles to ran them one after another.
`key` takes each item's identity from a column, checked for values and
uniqueness, instead of its position. `on_item_failure` chooses between
stopping at the first failure (`fail`), running every item and then
failing (`continue`), or leaving failed items out (`skip`, which still
fails when no item succeeded). `map` on any other node type is refused
at save and when the node runs, instead of being ignored. The
`expansion` block runs as before. First phase of ADR-050 (#967); see
`docs/mapping.md`.
- **`map` on `source_db`: one query per item, values bound** -- @hc12r.
A `source_db` node with a `map` block runs its query once per item of
an upstream node, with `${item.column}` bound like `${param.x}`: as a
driver placeholder, never as SQL text, since item values come from data.
A missing or null column fails that item naming the reference instead of
binding an empty value, `${item...}` in a node without `map` is refused
at save, and the `map.over` edge is the one incoming edge a source may
have. A dry run maps the first 3 items. See `docs/mapping.md`.
- **`map` on `task` and `source_api` nodes** -- @hc12r. A mapped `task`
runs its bundle once per item, the item its one-row input checked
against the task's declared input, locally or as one remote job per
item under the item's own attempt. A task with no input port, or with an
artifact or collection output, is refused. A mapped `source_api` makes
its request once per item, with `${item.column}` encoded for where it
lands: percent-encoded in the url's path (a `.` or `..` segment fails the
item) and query, as text in `params` and `headers` values (a header with
a line break fails the item), and as a typed JSON value in a JSON body.
A reference that could change where the request goes (the url's scheme,
host, port or fragment, a header or parameter name, a text body, any
other setting) is refused at save and at run. `checkpoint_every`,
`"response": "artifact"`, and `requests_per_second` with more than one
item at a time cannot be combined with `map`; with one at a time the
rate holds across items. ADR-050 (#967); see `docs/mapping.md`.
- **Server-wide limits on mapping** -- @hc12r. An operator sets the
item limit a `map` or `expansion` block gets when it declares none
(`BROKOLI_MAP_DEFAULT_MAX_ITEMS`, 1000) and the most it may declare
(`BROKOLI_MAP_MAX_ITEMS`, 100000), and likewise for local parallelism
(`BROKOLI_MAP_DEFAULT_MAX_PARALLEL`, 4; `BROKOLI_MAP_MAX_PARALLEL`, 64).
`BROKOLI_MAP_MAX_REMOTE_PARALLEL` (16) bounds the items of one node in
flight on remote workers. A declaration above a ceiling is refused when
the pipeline is saved and when it runs, naming the setting, the value
and the variable; a collection over the default names the default and
how far a block may raise it. A variable that is not a whole number of
at least 1 is ignored, with a warning in the log of every mapped node.
An `expansion.max_instances` above 100000, accepted before, is now
refused unless the operator raises the ceiling. ADR-050 (#967); see
`docs/mapping.md`.
- **Oracle reads through a native driver, experimental** -- @hc12r. An
Oracle connection can be pinned to the `orkidea` driver build
(Orkidea, an ADBC driver for Oracle that needs no Oracle Instant
Client), and its `source_db` reads then return decimals exactly, as
decimal strings, where the built-in driver turns them into doubles when
that is exact. A `NUMBER` without a declared precision is read as its
exact decimal text: the driver's own default, a 64-bit float, rounds
past about 15 digits, and the built-in path never rounds. Saving a
pinned connection refuses the settings the native driver cannot honour
yet (SID, TLS and native encryption options, and driver options that
name another server, a login or a file on the worker) instead of
connecting without them. Writes to Oracle stay refused on both paths.
Testing a pinned Oracle connection runs `SELECT 1 FROM dual`, since
Oracle before 23ai has no `SELECT` without `FROM`. See
`docs/native-drivers.md`, "Oracle connections".
- **`experimental` driver lifecycle** -- @hc12r. A catalog entry can now
be marked `experimental`: it installs and runs like a supported one and
is shown as experimental on the Drivers page. A server older than this
change refuses a whole catalog that contains the value, so a catalog
should list an experimental release only once the servers that read it
have this change.
- **User code can run as its tenant's own unprivileged user** -- @hc12r.
Set `BROKOLI_USER_CODE_UIDS` to a range of user IDs (for example
`1000000000-1999999999`) and every child that runs user code (code nodes
on the warm pool and on the one-shot and streamed paths, bash, task
nodes, dbt) starts as a user derived from its org and workspace instead
of as the server's user. A server running as root in its container
otherwise runs that code as root: a script could read the data
directory and every other root-only file, write the server's binary,
and read or signal another tenant's code running at the same time. With
the setting, two tenants get different users and neither is root.
Each execution gets a private folder (its HOME and TMPDIR, its script,
its input) handed to that user before it starts; warm code workers are
kept per user as well as per tenant, and remote work orders carry their
tenant so a remote worker separates tenants the same way. What the code
writes is read back without following links out of its folder, and a
worker naming an output path other than the one it was given is refused:
the server reads more than the code may, so a planted link would
otherwise make it read a root-only file on the code's behalf. The server
writes its own files private (umask 077) while the setting is on.
The server checks at startup that it can do this, and refuses to start
rather than run user code as root when it cannot: it needs root's
`CHOWN`, `DAC_OVERRIDE`, `FOWNER`, `KILL`, `SETUID` and `SETGID`, plus
`SYS_RESOURCE` to set the children's resource limits (containers do not
get it by default), and `fs.protected_hardlinks=1` on the host. Unset,
nothing changes, and a server running as root says in its log that user
code runs as root. A bash node's configured `working_dir` and a dbt
project folder are used as they are, so with the setting on they must be
open to the tenant's user; the bash error now says so instead of
"permission denied" from bash itself. Plugins still run as the server's
user.
- **A photo in the sidebar account chip** -- @hc12r. `SidebarAccount`
(`@brokoli/auth`) takes an optional `avatarSrc`: the signed-in person's
photo, shown in place of their initials. The initials come back when it is
absent or fails to load.
- **People reply to agents** (ADR-048 phase 2) -- @hc12r. A person writes
in their conversation with an agent (`POST
/api/messages/conversations/{id}/messages`), and an agent's owner can open
one from the agent's page (`POST /api/agents/{id}/conversation`). The
agent is told through a new trigger event, `message.received`, which goes
to every trigger of the agent that subscribes to it, by webhook or feed,
whatever pipeline the trigger watches and with no cooldown, and carries
the message and who wrote it. Agents read with the MCP tool
`read_messages` (unread messages, then marked read; or one conversation's
thread) and `GET /api/agents/me/messages`, and answer with `send_message`
in the same conversation. People's messages are redacted, cut at 8 KB and
limited to 30 an hour in a conversation. In the UI, opening a message in the Inbox shows
its whole conversation with a box to answer in (Ctrl+Enter sends), and an
agent's page has "Talk to" for its owner; when the agent has no trigger
for messages, the reply says it reads them when it next checks.
- **Agents can message people** (ADR-048 phase 1) -- @hc12r. An agent
sends a message to its owner, a person in its workspace, or the whole
workspace, with the MCP tool `send_message` or `POST
/api/agents/me/messages`: text, a level (`info`, `attention`, `urgent`),
an optional topic, and links to runs, pipelines and change requests that
Brokoli resolves and labels. It needs the new `messages.send` permission
(agent, operator and editor roles; not viewer). Every reader must be able
to see what a message links to, text is redacted like MCP results and
cut at 8 KB, `urgent` goes only to the owner (3 a day), and there are
hourly limits. People read their messages with `GET /api/messages`, a
conversation with `GET /api/messages/conversations/{id}`, mark them read,
and mute an agent's conversation; the owner sees how many of its
messages were muted. Messages are kept 180 days. See
`docs/agent-messages.md`.
In the UI the sidebar's bell becomes the **Inbox**: messages from agents
(with their avatar, level, topic and links; opening one shows it whole
and marks it read, with a switch to mute the agent) and the alerts as
before. Its count is unread alerts plus messages that need you. An
agent's page tells its owner how many messages it sent this week and how
many went to people who muted it.
- **Agent avatars** -- @hc12r. Every agent has one of eight avatars from
the broccoli family (broccoli, romanesco, cauliflower, purple sprouting,
broccolini, Brussels sprout, kohlrabi and seedling), drawn from the
Brokoli mark, each with a face of its own. An agent starts with the one
its id hashes to; its owner can pick another on the agent's page. It
shows in the agents list, on the agent's page, beside "Agents told" on a
run, and in a pipeline's settings. The API returns it as `avatar` and
takes it on `PATCH /api/agents/{id}`; an agent's owner may change it
without `settings.manage_users`. `AgentAvatar` and `AgentAvatarPicker`
are exported for a distribution's own pages.
- **Agent triggers: subscribe an agent to run events** (ADR-047) -- @hc12r.
A trigger names an agent, the events it wants (`run.failed`, which
includes runs a quality check or contract gate stopped, and
`run.blocked`), one pipeline or every pipeline in the workspace, the URL
deliveries go to, and a cooldown (30 minutes by default) so a pipeline
that keeps failing is reported once. Whoever may manage the agent (its
owner, or `settings.manage_users`) manages its triggers:
`/api/agents/{id}/triggers` and `brokoli agents trigger
list|create|update|delete|rotate-secret`. Each trigger has a Standard
Webhooks signing secret, shown once and stored encrypted with the key
that protects connection credentials; rotating it keeps the old one
signing for 24 hours. A trigger is refused when the agent could not see
what it would be told about: its role, within its owner's permissions,
must grant `pipelines.view` and `runs.view`, the pipeline must be in
the trigger's workspace and organization, and a distribution's identity
rules apply as they do to the agent's own requests. The URL must be
allowed by the outbound network policy and carry no credentials.
Events are delivered by the scheduler's leader, which reads the run
event log (so runs on remote workers count), signs each delivery per
Standard Webhooks, and retries anything not answered `2xx` within 10
seconds for 24 hours. The event carries the run, its failed steps and
their types, and its error (redacted as MCP results are), never a
credential: the agent calls back over MCP with its own token. Before
every delivery the agent's access is checked again, and a delivery to
an agent that may no longer see the pipeline is skipped. Each trigger
keeps a delivery log (`brokoli agents trigger deliveries`, kept 30
days) and can redeliver. Set `BROKOLI_PUBLIC_URL` for events to carry
links to the MCP endpoint and the run. See `docs/agent-triggers.md`.
- **Agent triggers: `run.recovered` and test events** (ADR-047 phase 2)
-- @hc12r. `run.recovered` is delivered when a run succeeds after the
pipeline's previous finished run failed, so an agent learns that the fix
it proposed, once applied, worked. `brokoli agents trigger test` (`POST
.../triggers/{id}/test`) delivers a `run.failed` marked `"test": true`
to check a receiver. The cooldown is now per event type, so a delivered
failure does not hide the recovery after it, and test events do not
count toward it.
- **Agent triggers: a feed for agents without a URL** (ADR-047 phase 2)
-- @hc12r. A trigger with `"delivery": "feed"` (`brokoli agents trigger
create --feed`) keeps its events for the agent, which reads them with
its own token: `GET /api/agents/me/events?after=<cursor>`, or the new
`next_events` MCP tool. Passing the cursor back acknowledges what was
read, so an event the agent did not get to comes back. Access is
checked when the agent reads; unread events expire after 7 days. A
feed trigger has no URL and no signing secret.
- **A reference triage agent** (`examples/agents/triage`, ADR-047) --
@hc12r. One Python file that does the whole loop: it receives signed
deliveries (`serve`) or reads the agent's feed (`poll`, with `--once`
for a scheduled job), asks Brokoli why the run failed (`explain_run`,
`get_pipeline`), asks a model whether the cause is in the pipeline's
definition, and if it is, validates the fix and proposes it with a
title and a description for the reviewer. Otherwise it reports what it
found and proposes nothing. It never applies a change, proposes nothing
more while its own proposal for the pipeline waits, and acknowledges a
feed event only after triaging it. The model can be Claude (the
default), any model behind an OpenAI-compatible API (`--provider openai
--base-url ...`: OpenAI, Azure OpenAI, Gemini, Mistral, DeepSeek, Groq,
OpenRouter, or Ollama, vLLM and LM Studio locally), or a program of your
own that reads the request on stdin. Every answer is checked against the
decision's shape before the agent acts on it. It tells its owner what it did
with `send_message`: a proposal as `info` with the change request
linked, a cause outside the definition as `attention` (`--no-messages`
turns this off). Written to (`message.received`), it answers each person
once with what it last found.
- **An Agents page** (ADR-047 phase 3) -- @hc12r. Agents now have a page
of their own in the UI: create one, issue and revoke its tokens (with the
command to connect a client), and manage its triggers: make a webhook or
feed trigger, choose its events, pipelines and cooldown, switch it off,
send a test event, make a new signing secret (shown once), and read its
delivery log with every attempt and a button to send one again. Before,
triggers could only be managed with the CLI or the API. A distribution
with its own agents page can embed the triggers panel (`AgentTriggers`).
A run's page names the agents told about it, where each delivery stands,
and the change request each then proposed, linked; a pipeline's settings
list the triggers that watch it.
- **See which agents were told about a run, and which triggers watch a
pipeline** (ADR-047 phase 3) -- @hc12r. `GET /api/runs/{id}/agents`
lists the agents a run was delivered to, where each delivery stands,
and the change request each agent then proposed to the pipeline, if
any. `GET /api/pipelines/{id}/triggers` lists the triggers on a
pipeline and on its whole workspace. Both answer anyone who may see the
pipeline; a webhook's URL is shown only to whoever may manage the agent.
Neither lists an agent the caller may not know of (a distribution that
scopes agents to a tenant decides).
`GET /api/agents/trigger-events` lists the event types a trigger may
subscribe to, with what each means. A distribution can add its own:
`api.RegisterTriggerEvent` while the program starts, then
`api.RaiseTriggerEvent` when one happens, and it is delivered as run
events are (once per occurrence, cooldown, access checked at delivery).
### Removed
- **`api.ValidateOrgAccess` and `api.ValidateWorkspaceAccess`** --
@hc12r. Tenancy is the request's scope now (ADR-049): `scopedStore(s, r)`
for reads and actions, `RequestScope(r)` for the rule itself. The first
checked the organization only; the second was never called. A test now
fails when a handler calls the raw store for pipelines, runs, alerts or
dead letters outside a short list of known sites, which may only shrink.
### Fixed
- **Bundles and packages with two dots in a file name extract** -- @hc12r.
The archive extractor refused any entry whose name contained `..`, so a
task bundle with a resource such as `notes..txt` was refused on upload as
if it escaped the extraction root. Only a whole `..` path segment is
refused now, with absolute paths and volume names as before. It also let a
second entry with the name of an earlier file replace it on disk: a
bundle holding a user's own `manifest.json` next to the real one failed on
upload with an empty format error, and `ReadManifest` read the other one.
An archive with two entries for the same file is now refused with a
message naming it. This applies to task bundles, plugin packages and
driver archives. (#999)
- **Cloning a pipeline created in the UI works** -- @hc12r. A UI-created
pipeline gets a `pipeline_id` slug from its name, and the slug is unique
within its workspace, so the clone, which copied the slug, failed with a
500 every time. A clone now gets `<slug>-copy`, then `-copy-2` and so on,
and a 409 if all of those are taken. It no longer copies the original's
webhook token, which is a secret, and it is always a UI pipeline, even
when the original is managed in Git, where a copy would refuse every
edit. (#590)
- **The pipeline summary counts every run, and reports a failed read** --
@hc12r. `GET /api/pipelines/summary` counted each pipeline's runs over its
newest 200 only, so a busy pipeline was undercounted, and the UI carried
a footnote saying so (#624). It now counts in the database, over every
run that started, as the dashboard does, and reads only the five newest
runs for the last run and the run history. It also answered `200 []`
when the pipeline list failed, and showed a pipeline as never run when
its runs could not be read, so an outage looked like an empty workspace.
A failed read now answers 500 (#591). The UI's 200-run caveats are gone.
- **The login screen no longer offers a password form the server refuses**
-- @hc12r. `GET /api/auth/methods` always said `"password": true`, even
when an integration had turned password login off through
`PasswordLoginEnabledFunc`, so the screen showed a form whose every
attempt was refused. It now applies the login handler's own check, and a
test holds the two together. (#594)
- **The task bundle spec describes the bytes core writes** -- @hc12r.
`task-bundle-v2-canonicalization.md` said the manifest is written with
sorted keys, and that entries are ordered by path. `Assemble`, which
`brokoli bundle jvm` uses, writes the manifest as compact JSON in a fixed
key order, with Go's escaping, and writes directories before
`manifest.json` and the files. A second builder written from the spec
could never match core's bundles. The spec now states the exact archive:
the gzip header, the tar headers (including how a path over 100 bytes or
outside ASCII is written: split into ustar's prefix field, or behind a PAX
header), entry order and modes, the manifest's key order and escaping. It also says that two builders match on the
uncompressed tar, not on the deflate bytes. Tests pin every rule. Nothing
about the bundles core writes changes. (#986)
- **A short password no longer mangles error messages** -- @hc12r.
Credential redaction replaced every occurrence of the password's text, so
a one-letter password turned "supported" into "suxxxxxxxxxxorted" and
left the operator without the reason a connection failed. A password is
now removed wherever it stands as a token of its own, at any length,
including inside a quoted DSN, and never inside another word. It is
still never weakened by length, since a short password left in a run log
would be worse than an over-redacted message. (#780)
- **A whole `float64` written without `.0` is a valid `float64`** -- @hc12r.
JSON does not tell `5` from `5.0`, and task outputs keep integers exact,
so a whole value written as `5` (JavaScript's `JSON.stringify(5.0)`, and
RFC 8785) reached validation as an `int64` and was refused with
`expected float64, got int64`. A Node task returning `5.0` for a
`float64` field failed its own output contract. An integer is now a
`float64` when a `float64` holds it exactly, up to 2^53 in magnitude, and
bounds apply to it as to the equal `float64`. Beyond 2^53 it is refused.
(#983)
- **Node previews keep whole numbers above 2^53 exact** -- @hc12r.
`GET /api/runs/{id}/nodes/{node}/preview` decoded the stored rows into
float64 on both SQLite and Postgres, so `9007199254740993` came back as
`9007199254740992` although the preview was stored exactly. Integers now
read back as int64, as a run's own rows have them. The Postgres store also
ignored a preview it could not decode; it now reports the error. The UI
still rounds such values when it parses the response, tracked in #1001.
(#965)
- **Task nodes receive their declared parameters, typed and with defaults**
-- @hc12r. A `task` node's keyword arguments were the run's string
parameters, locally and on remote workers, so a declared `float64`
reached the task as text, and a declared default never reached it. A
pipeline that declares parameters now passes its tasks the resolved
values: typed, with defaults applied, and with `decimal`, `timestamp`,
`duration`, `bytes` and large `int64` values tagged. In such a pipeline,
string run parameters are no longer passed to tasks. A pipeline that
declares no parameters passes its string parameters as before. (#978)
- **A SQLite store refuses a database SQLite makes per connection** --
@hc12r. `NewSQLiteStore(":memory:")` opened a pool of up to 25
connections, each with its own empty database, so tables and writes
vanished between them: a test driving a run through such a store failed 6
runs in 10. `:memory:`, the empty path and in-memory `file:` URIs are now
refused with an error that says to use a file. Nothing in this repository
opened one through the store. (#636)
- **An int64 beyond 2^53 is no longer accepted as a plain number** --
@hc12r. Task interface validation accepted any whole float64 as an
`int64`, so a value a JSON decoder had already rounded (`9007199254740993`
arrives as `9007199254740992`) passed as if it were what was written. A
plain JSON number is now an `int64` only up to 2^53 - 1. Beyond that the
value must be tagged, `{"$bptd": "int64", "value": "..."}`, or come from a
decoder that keeps integers. This applies to run parameters, defaults and
task outputs. A run parameter of 2^53 or more sent as a plain number is
now refused with a 400 naming the tagged form, where it used to run with
a different number. (#979)
- **Two task-node fallbacks say so in the server log** -- @hc12r. A store
that cannot pin a resolved execution record makes every attempt select
its payload afresh, so a retry can run a different payload; that is now
logged once per store type. When a remote task job cannot be tagged with
the runtimes its bundle needs, because the store cannot read bundles, the
bundle is missing, or its manifest is unreadable, the job still carries
only the protocol tag, and that is now logged too, naming the bundle.
Neither changed behaviour before; both were silent. (part of #529)
- **A `source_api` body given as JSON is sent** -- @hc12r. A `body` written
as a JSON object or array in the node's config was dropped, and the
request went out with no body at all. It is now sent as JSON, with
`Content-Type: application/json` unless the node sets its own.
- **Starting a run of a pipeline that no longer validates answers 400** --
@hc12r. A stored pipeline the validator refuses when a run is requested,
for example one declaring more items than a mapping ceiling lowered since
it was saved, was answered with a 500, as if the server were at fault. It
is now the 400, with the same message, that saving it would get.
- **Dependency vulnerabilities** -- @hc12r. `andybalholm/brotli` v1.2.7
(GO-2026-6888) and `klauspost/compress` v1.20.2 (GO-2026-6865, a panic on
two-block frames), and `golang.org/x/net` v0.62.0 (CVE-2026-78661).
- **Pipeline code can no longer read the engine's environment** -- @hc12r.
Code, bash and task nodes run as children of the engine under the same
user. Their own environment was filtered, but `/proc/<engine>/environ`
was not, so a script could read every variable the server started with,
database URL and keys included. The engine now marks itself
non-dumpable, which makes its `/proc` files readable only with
`CAP_SYS_PTRACE`.
- **dbt nodes get the filtered environment** -- @hc12r. They inherited the
server's whole environment, so a dbt project's `env_var()` could read
its database URL and keys. They now get the same allowlisted environment
code nodes do; name a further variable in `BROKOLI_DBT_PASS_ENV`.
- **Warm code workers are not shared across workspaces** -- @hc12r. A warm
worker keeps what the last script left in its process, and the pool
handed it to the next script whichever workspace it belonged to. Warm
workers are now reused only within one organization and workspace.
- **Testing a change request needs edit permission** -- @hc12r. It runs the
proposed code on the server, and needed only run permission, so a role
that may propose and run but not edit could run any code it proposed.
- **Webhook exemption no longer skips the workspace check for other
routes** (ADR-049) -- @hc12r. The workspace check skipped every POST
whose path contained `/webhook`, not only the webhook routes, so a
signed-in member could name a workspace they do not belong to in such a
request (a path parameter whose value is `webhook`, for example) and
have it listed and written there. The exemption now matches the webhook
routes by shape, as authentication already did, and a request scope
computed outside the middleware refuses a workspace the caller is not a
member of. `GET /api/auth/me` reports the workspace only when the caller
belongs to it.
- **Changes are made in the current workspace** (ADR-049) -- @hc12r.
Someone in two workspaces could run, edit, delete, cancel or resolve an
item of the other workspace by id, with the permissions of the one they
were working in: a viewer in one who edits in the other could edit the
first. Reading by id still works in any workspace a person belongs to;
changes are now made only in the current one, and answer "not found"
elsewhere. This covers pipeline edit, delete, rollback and bulk actions,
runs (trigger, backfill, cancel, resume), change requests (propose,
apply, decline, test), dead letters, alert actions and git push.
- **Variables, connections and secret stores of the wrong workspace** --
@hc12r. Working in a second workspace used the first: a variable key or
connection id both had named the first one's (read, overwritten, tested
or deleted from the second), a create without a workspace went into the
first, and the secret stores page listed and changed the first one's.
All three now use the workspace the request acts in.
- **Workspace tokens were not kept to their workspace** -- @hc12r. A
token's `workspace_id` claim was never read, so a token minted for one
workspace acted in every workspace its creator belongs to. It now acts in
that workspace only and sees no other.
- **Workspace isolation for alert actions** (ADR-049 phase 3) -- @hc12r.
Marking an alert read, dismissing, assigning, acknowledging and resolving
it checked the organization only, so a person could act on another
workspace's alert by id. They now answer "not found" for an alert outside
the caller's workspaces. "Mark all read" marks the current workspace's
alerts, not every workspace's in the organization. A run fetched by id
for its plan or instances, and resolving a dead letter, now give the same
"not found" for a run or entry the caller may not see as for one that
does not exist.
- **Deleting a pipeline could delete other workspaces' pipelines**
(ADR-049 phase 3) -- @hc12r. `DELETE /api/pipelines/{id}?resolve=cascade`
deleted every pipeline in the organization that depended on it, in any
workspace, and the refusal without `resolve` listed them by name. A
cascade that would reach a pipeline the caller may not see is now
refused (decouple still works), and the refusal names only the
dependents the caller may see, with a count of the others
(`dependents_elsewhere`).
- **Dependencies on other workspaces' pipelines** (ADR-049 phase 3) --
@hc12r. A pipeline could be made to depend on any pipeline in the
organization, which let it fire on, and report the status of, a pipeline
in a workspace its author does not belong to. A new dependency must now
be on a pipeline the caller may see; dependencies already saved are kept
and do not block editing. A pipeline's dependency view
(`GET /api/pipelines/{id}/deps`) keeps only the id and whether it is
satisfied for an upstream in another workspace, not its name, status or
timing.
- **The scheduler status listed every organization's schedules** --
@hc12r. `GET /api/scheduler/status` returned every scheduled pipeline on
the server, with its name, cron and last and next run, to any signed-in
person. It now lists the schedules of the pipelines the request lists:
the current workspace's.
- **Live run state and logs of other workspaces** (ADR-049 phase 3) --
@hc12r. The live connection (`/api/ws`) let a session watch its
organization's dashboard, which listed every workspace's runs, and any
run of the organization by id, logs included. Where workspaces are kept
apart, the server now publishes a dashboard per workspace
(`dashboard.{org}.{workspace}`), refuses the organization-wide one, and
lets a session watch a run only in a workspace its person belongs to.
The UI watches the workspace's dashboard.
- **Purge deleted every workspace's run history** (ADR-049 phase 3) --
@hc12r. `POST /api/system/purge` deleted the organization's old runs, so
an admin of one workspace deleted the run history and run files of all
of them. Where workspaces are kept apart it deletes the current
workspace's.
- **Dashboard counts and the run calendar were the organization's** --
@hc12r. `GET /api/dashboard` counted every workspace's runs (runs today,
success rate, trends, running run ids, top failing) next to the current
workspace's pipelines, and `GET /api/runs/calendar` counted the
organization's runs per day. Both are the current workspace's where
workspaces are kept apart.
- **Git push could export another organization's pipeline** -- @hc12r.
`POST /api/git/push/{id}` checked the pipeline named in the path, then
handed the path's id on unchanged; given a `pipeline_id`, the export
looked it up again across every organization. It now passes on the id of
the pipeline it checked.
- **Workspace isolation for the rest of the pipeline routes** (ADR-049
phase 2) -- @hc12r. The dead-letter queue (`GET /api/dlq`, a pipeline's
entries and resolving one), a pipeline's dependencies and dependents, the
dependency graph and bulk actions checked the organization only. They now
use the same tenancy scope: another workspace's pipeline is "not found"
(bulk included, which said "access denied" and so told which ids exist),
dependents and the graph name only pipelines the caller may see, and the
dead-letter list is the current workspace's. Every pipeline route that
wrote the organization and workspace checks by hand now makes one call.
- **Workspace isolation for runs and alerts** (ADR-049 phase 1) -- @hc12r.
Run routes (run, logs, events, plan, previews, grid, cancel, resume,
backfill, dry run), `GET /api/runs?started_by=me`, and the alert list and
unread count checked the organization only, so inside one organization a
person reached runs of workspaces they do not belong to, and every
workspace's Home showed the others' runs and failures. They now go
through one tenancy scope, computed by the workspace middleware and
applied by a scoped store: lists show the current workspace, and an item
by id is reachable in any workspace the caller belongs to, "not found"
otherwise. An alert of a deleted pipeline is no longer shown.
- **A log warning could end up in a CLI command's output** -- @hc12r.
Every command tried to open `brok.log` in the working directory when it
started, and outside the allowed data directories printed a warning to
stdout. `brokoli agents token create ... > triage.token` then wrote the
warning into the token file, and the token did not work. Log lines now go
to stderr, never stdout, and the log file is opened only when something
is logged.
- **A new agent trigger was sent failures from before it existed** --
@hc12r. The dispatcher re-reads the last two minutes of run events
(Postgres can commit them out of order), so a trigger created within
two minutes of a failure was delivered that failure. A trigger now hears
only about what happens after it is created.
- **`min` and `max` over dates, timestamps and text returned 0** -- @hc12r.
An aggregate folded only values that parse as numbers, so over a column
of dates (`2026-10-27`) none counted and every group's `max` was `0`,
with no error. When a group has no numeric value, `min` and `max` now
order its values as the `sort` rule does and return the value itself:
the latest date, the earliest timestamp (instants are compared by time,
whatever their zone), the last word. A group with any number keeps the
numeric answer it had, values that are not numbers still skipped, and a
group with no value at all still gives `0`. The output schema follows:
`min`/`max` of a date, time, timestamp or text column keep that type, so
a `create_table` sink makes a column that holds them, and over a column
of unknown type nothing is declared and the type is inferred from the
values. Both the in-memory and the streamed aggregation changed, and the
test that holds them to identical results now covers dates.
## [0.21.0] - 2026-10-31
A performance and correctness release. Row filters, sorting, bulk loads
into Postgres and MySQL and run logging are several times faster; REST
sources retry politely, stop when their run is cancelled and speak HTTP/2;
and the database now records its schema migrations. It also fixes results
that were quietly wrong, among them SQL Server bulk loads that stored
out-of-range integers truncated, and `create_table` sinks that worked only
on a pipeline's first run.
### Changed
These can change what an existing deployment does.
- **Upgrading from 0.20** -- @hc12r. The first start applies schema
migrations and logs each one: the baseline (migration 1) everywhere; on
SQLite, migration 2 rewrites stored timestamps at a fixed width and logs
how many per column; and the index on run start time (migration 3 on
SQLite, 2 on Postgres), during which writes to `runs` wait on Postgres.
Later starts apply none, and 0.20 still starts against an upgraded
database, so a rollback works. Nested objects and arrays are written as
JSON instead of Go's `map[...]` syntax. API request bodies are limited to
4 MiB. A REST response is limited to 512 MiB unless the node sets
`max_response_bytes`; a 4xx other than 408, 425 and 429 is no longer
retried; retry waits are jittered and capped at five minutes, and
`fixed` backoff is honoured. SQL Server bulk loads refuse an integer
outside its column's range, where they stored it truncated, so a load
that wrote wrong numbers now fails and names the row. A `create_table`
sink no longer fails when its table exists. A file path that leaves the
data directories through a symlink is refused. Outbound HTTP uses HTTP/2
where the server offers it. Each is described below.
- **Versioned schema migrations** -- @hc12r. The database now records the
migrations applied to it, in a new `schema_migrations` table, and a start
applies only the ones it lacks. Until now the whole schema setup ran on
every start, with most of its errors ignored. The first start after
upgrading runs that setup one last time, as migration 1, and logs
`applied schema migration 1 (baseline)`; later starts skip it. An older
release still starts against an upgraded database, so a rollback works.
The system roles and the built-in templates are still brought up to date
on every start.
- **Index on run start time** -- @hc12r. Purge, retention, the run calendar
and the dashboard counts filter runs by start time alone, and no index led
with it. Measured on SQLite with a million runs, the 365-day calendar went
from 4.2 s to 1.2 s. Two indexes that duplicated the leading column of
another (`node_runs(run_id)`, `run_events(run_id)`) are dropped, saving a
write per row. On Postgres the index is built during the first start after
upgrading, and writes to `runs` wait until it is done.
- **Row filters are 6-9x faster** -- @hc12r. A `filter_rows` transform
parsed its condition again for every row, and formatted every cell as
text before comparing it. The condition is now parsed once, a string cell
is compared as it is, and an ordered comparison reads an integer or
float cell as a number directly. On 100,000 rows: `amount >= 50` 70 ms to
7.5 ms, `status = s1` 37 ms to 6.3 ms, `status in [...]` 64 ms to 8.2 ms,
each down from 100,000-400,000 allocations to about 20. Which rows a
filter keeps is unchanged; a test checks the new matcher against the old
one across awkward values (NaN, infinities, negative zero, integers past
2^53, float32, nil, dates).
- **Sorting is 1.5-3.6x faster** -- @hc12r. A `sort` transform turned
every number into a 200-bit `big.Float`, and formatted and test-parsed
every value, text included. Numbers now compare as float64 wherever that
cannot change the order: float and integer cells, integer text within
2^53, and plain decimals of up to 15 significant digits, such as `12.50`
from a CSV. A column holding any other number keeps the exact
comparison for all of its numbers. On 100,000 rows, allocations fell
from 400,000-900,000 to 6. The order is unchanged; a test checks it
against the old comparison over values packed where rounding is tight.
- **A running run checks for a cancel without loading its node runs** --
@hc12r. Every running run re-reads its durable cancel flag at each wave
and every five seconds, and did so with the full run lookup, which also
loads every node run. It now reads the flag alone: on SQLite about 30 µs
however large the run, where the full lookup took 0.56 ms at 50 node runs
and 4.4 ms at 500.
- **Creating a table infers column types without parsing every value
every way** -- @hc12r. To choose each column's type, every value was
formatted as text and tried as an integer, a float, a boolean and five
date layouts, each failed parse allocating an error. Typed values are now
classified directly, and text is only tried where it can match: a word
like `customer-000123` skips every parse. On 100,000 rows a text column
went from 181 ms and 2 million allocations to 8 ms and none, typed
columns 4-10x faster; every column gets the type it got before.
- **Run logs are written in batches when nodes log at once** -- @hc12r.
Every log line was its own database transaction, and on SQLite each
commit waits for the disk while holding the lock node updates also need.
Lines that arrive while another is being written now wait and are
stored together in one transaction. A line is still stored by the time
the node's log call returns, and a node's lines keep their order. With 8
nodes logging at once a line costs 0.31 ms instead of 1.4 ms; with 32,
0.09 ms instead of 1.9 ms. A store without batch writes keeps writing
one line at a time.
- **Bulk loads into Postgres and MySQL encode rows 1.9-2.8x faster** --
@hc12r. The COPY and `LOAD DATA` writers render each value into the
row's line directly instead of building a string per value, and integers,
booleans and floats no longer go through `fmt`. The COPY writer also built
a new 6 KB escaper for every value that held a tab, newline, carriage
return or backslash. Measured on 10,000 mixed rows: COPY went from 2.8 to
1.0 ms per thousand rows and from 51,000 allocations to 13, `LOAD DATA`
from 1.8 to 0.95 ms and from 31,000 allocations to 13. The bytes written
are unchanged, held to the old encoders by a test over every value kind.
- **Outbound HTTP uses HTTP/2 where the server offers it** -- @hc12r.
The outbound client's own dialer, the one that enforces the network
policy, had switched off Go's automatic HTTP/2, so REST sources, object
stores, BigQuery, notifications and the rest spoke HTTP/1.1 to every
server, and each request in flight to a host held a connection of its
own. A REST source fetching eight pages at once made eight TLS
handshakes; it now makes one and sends the pages over it. Servers that
do not offer HTTP/2 are spoken to over HTTP/1.1 as before, and the
policy still checks every connection.
- **Dead code removed** -- @hc12r. Eight functions, a type and a field that
nothing called, found by staticcheck. One was a YAML "sanitizer" that was
never wired in, and would have been wrong if it were: it stripped `*`, `&`
and `!!` from every string config value, so `SELECT * FROM t` would have
lost its `*`. No behaviour changes.
### Fixed
- **SQL Server bulk loads no longer corrupt out-of-range integers, and
accept numbers that arrive as text** -- @hc12r. The bulk-copy path wrote
an integer into a narrower column by truncating its bytes: 300 bound for
a `tinyint` was stored as 44, and a value past an `int`'s range wrapped,
where the statement path refuses it as an overflow. It also refused a
string for an integer, float or bit column, and a CSV source hands every
value over as a string, so loading a CSV into SQL Server failed on its
first row whenever a column was numeric, including every table
`create_table` made from that CSV. Out-of-range values are now refused
and text is converted, both the way SQL Server treats the same value on
the statement path, held to that by a test against a live server.
- **A database sink that creates its table works on every run, not only
the first** -- @hc12r. `create_table` is offered as "create the table
if it does not exist", but the table was created with a plain
`CREATE TABLE`, so a pipeline succeeded once and then failed on every
later run with `relation ... already exists`. Postgres, MySQL,
ClickHouse, SQLite, Snowflake and Databricks now use
`CREATE TABLE IF NOT EXISTS`, and SQL Server, which has no such clause,
checks `OBJECT_ID` first. A failed load still rolls a newly created
table back on Postgres.
- **The UI builds on macOS and Windows** -- @Fj460. Two logic files had the
same name as a component apart from case (`queries.ts` and `Queries.tsx`,
`heatmap.ts` and `Heatmap.tsx`), so on a case-insensitive filesystem the
component's import resolved to the wrong file and `npm run build` failed.
- **Nested values are written as JSON** -- @hc12r. A REST source passes
nested objects and arrays through as they are, and the Postgres COPY
writer, the SQL statement writer, MySQL's bulk load and CSV files
rendered them in Go's own syntax: `{"a": 1, "b": [2, 3]}` arrived as
`map[a:1 b:[2 3]]`, which nothing can read back. A `jsonb` column refused
the write; a text column or a CSV file stored it silently. They are now
written as JSON, with `<`, `>` and `&` kept as they are and 64-bit
integers exact.
- **The server closes idle connections** -- @hc12r. With no idle timeout
set, a kept-alive connection that sent nothing more stayed open, with its
goroutine and file descriptor, until the client closed it. Idle
connections now close after 11 minutes, longer than load balancers keep
a backend connection idle (GCP's is 600 seconds), so the proxy still
closes first. Uploads, downloads and event streams are unaffected: there
is still no read or write deadline.
- **File nodes cannot be led out of the data directories by a symlink** --
@hc12r. The check that a path is inside `BROKOLI_DATA_DIRS` read the path
as written, so a symlink inside a data directory pointing elsewhere (for
example `/data/exports -> /etc`) passed it, and the file was then read or
written wherever the link led. Files are now opened through Go's
`os.Root`, which refuses a path that leaves the directory at the moment
it is opened, with no gap between a check and the open. Links that stay
inside a data directory keep working, absolute ones included. The Excel
loader also opened the file a second time by name; it now reads the one
it checked.
- **A preview of parallel steps cannot crash the server** -- @hc12r. The
steps of a preview that run side by side each recorded their result in
one shared map with no lock. Two finishing at the same moment were a data
race, and Go ends the process on concurrent map writes. It was rare
because the store's connection pool usually ordered the writes; with the
window widened, the race detector flags it at once. Results are now
recorded under a lock.
- **Vulnerability preflight distinguishes tool failures from findings** --
@arin30. Preflight reuses only the pinned `govulncheck` version, reports a
missing or unusable scanner as a non-green `SKIP`, and keeps actual
vulnerability findings as failures. Empty or unparseable scanner output is
treated as unavailable; the earlier build stage remains responsible for
reporting source compilation failures.
- **Listing schedules no longer holds up scheduled runs** -- @hc12r. The
schedule list read each pipeline's last run with its own query while
holding the lock a schedule takes when it fires, so with many schedules
every scheduled run waited for the whole list. It now reads all the last
runs in one query, after releasing the lock.
- **API request bodies are bounded** -- @hc12r. Creating or updating a
pipeline, and 21 other endpoints (connections, variables, secret stores,
runs, backfills, templates, users and more), decoded a JSON body of any
size, so one request could make the server buffer as much as a client
cared to send. They now read at most 4 MiB, the size of the largest
pipeline definition the API already accepted in a change request, and
answer 400 past it. Endpoints that set a smaller limit keep theirs.
- **REST page retries skip failures that cannot pass, and honour
Retry-After** -- @hc12r. A paginated `source_api` retried every failed
page, so a 401, 403 or 404 was requested again on every retry before the
node failed with the same error. Only transport errors, 408, 425, 429 and
5xx are retried now. When the server sends `Retry-After` and it is longer
than the configured backoff, the retry waits for it, up to five minutes.
- **A REST response can no longer exhaust memory** -- @hc12r. A
`source_api` read each response body whole, with no limit, so an endpoint
that kept sending ended the process. A response is now capped at 512 MiB,
and a declared length over the cap is refused before anything is read.
A node that expects more sets `max_response_bytes`. An oversized page is
not retried.
- **A failed log write no longer stops the server** -- @hc12r. The shared
logger the loaders and the engine write through handed every write error
to a helper that calls `log.Fatalf`, so a full disk under the log file
ended the process at its next log line. Logging is best effort now; a
line that cannot be written is dropped.
- **Outbound HTTP clients no longer leak connections** -- @hc12r. Webhooks,
alerts, notifications, `sink_api`, waits and the other outbound callers
build a client per request, and its transport had no idle timeout, so
every dropped client kept its connection and two goroutines open until
the remote server closed it (50 requests left 150 goroutines behind).
Idle connections now close after 90 seconds, and a TLS handshake that
never finishes fails after 10 instead of hanging a client that has no
overall timeout.
- **SQLite timestamps sort and compare correctly** -- @hc12r. SQLite stores
timestamps as text and compares them as text, but they were written with
trailing zeros trimmed, so `10:00:05.1` sorted after `10:00:05.123` and
`10:00:05` after both. Run logs could come back out of order, and lease
expiry, due waits and purge cutoffs could misjudge values within the same
second. Every timestamp is now written at a fixed width in UTC, and
migration 2 rewrites the ones already stored; the first start after
upgrading logs how many it rewrote per column. Also fixed: the default
workspace showed a zero creation date, and run provenance stored its time
in yet another format. Postgres was not affected.
- **REST page retries spread out, stop at five minutes, and honour fixed
backoff** -- @hc12r. A failed page waited exactly 100 ms doubled per
attempt, so pages fetched concurrently that failed together (a rate
limit, typically) all retried at the same instant and failed together
again. The wait is now a random point in the upper half of the backoff.
It is also capped at five minutes: with a large `page_max_retries` the
doubling never stopped, and from the 37th retry it overflowed into no
wait at all. `fixed`, offered in the node inspector, was treated as
exponential for page retries and now waits 100 ms every time. A
server's `Retry-After` is still kept exactly.
- **A cancelled run stops retrying pages on remote workers** -- @hc12r.
When a paginated REST source ran its pages on workers, a page waiting
to retry did not notice a cancel. The run was reported cancelled at
once, but the page carried on: it finished its wait, sent the next
attempt to the workers, which fetched the page from the API again, and
repeated this until its retries ran out, up to a minute with the
default exponential backoff and six retries. The wait now ends on a
cancel. Its backoff is also jittered like the in-process page retry,
and `fixed` waits one second instead of backing off exponentially.
- **A cancelled run sends no more work to remote workers** -- @hc12r.
The runner reports a cancel without waiting for the goroutines
dispatching pages, expansion items, partitions or task instances, and
one that reached the dispatcher afterwards still put its job on the
queue: a worker ran it in full, and only then found the run cancelled.
The dispatcher now checks for a cancel first and settles the attempt as
failed without sending anything.
## [0.20.0] - 2026-10-25
A reliability release. A node's timeout now stops the work it times out:
before, Brokoli stopped waiting but the step carried on, and a retry could
write the same rows a second time. Alongside that, fixes for crashes,
races and quietly wrong answers found in an audit of the codebase.
### Changed
These can change what an existing deployment does.
- **Upgrading from 0.19** -- @hc12r. No new migration runs. Node
timeouts are now enforced: a step that times out is cancelled, and a
timed-out attempt that does not stop within the node's timeout (at most
30 seconds) fails the node instead of being retried beside it, so a
step that used to finish past its timeout now fails at it. Cancelled
runs move from `brokoli_pipeline_runs_failed` to the new
`brokoli_pipeline_runs_cancelled`. Integers from REST sources, JSON
files and code nodes are `int64` rather than `float64`. Dedup keeps rows
it used to drop as duplicates, and joins match an integer key with the
same number as a float. Session tokens are accepted only when signed
with HS256. A server whose scheduler cannot start exits. Each is
described below.
- **A server whose scheduler cannot start exits** -- @hc12r. In `all`
and `scheduler` mode, a scheduler that failed to start (its first read
of the pipelines failed) was logged as a warning, and the process went
on serving with no scheduler: nothing ran on a schedule, and only that
log line said so. The process now exits with the error, so a process
supervisor or orchestrator restarts it.
### Fixed
- **Starting a Postgres installation no longer rebuilds the variables
key** -- @hc12r. Every start dropped the `variables` primary key and
added it again, as two statements with their errors ignored. While one
process started, another still serving could fail to save a variable,
since saving relies on that key; and if adding it failed, the table was
left with no key at all, unnoticed. The key is now changed once, in one
transaction, and a start that cannot add it fails and says why.
- **A node that panics fails its run instead of the process** -- @hc12r.
A node's handler runs in a goroutine of its own, outside the recover
that guards the rest of the run, so a panic in it (a bug in a node type
or in a distribution's executor) ended the whole process: the server,
the scheduler and every other run in it. The node now fails with "node
<name> panicked: <value>", the run fails with it, and the stack is
logged.
- **A failed run calls its `on_failure` hook, and `on_node_failure`
works** -- @hc12r. Every failure ended the run through a path that sent
the alert and the notification but not the hook; the code that called
`on_failure` sat after it and was never reached, so only a cancelled run
called it. `on_node_failure`, accepted by the pipeline IR schemas, was
never called at all. Now a node that fails after its retries calls
`on_node_failure` with `node_id`, `node` and `error`, and the failed run
then calls `on_failure` with `error`. A node stopped by a cancel does not
call `on_node_failure`. Errors sent to hooks are redacted, as they are
everywhere else a run's error leaves Brokoli. A failed run also logs
"run finished" like the other outcomes.
- **A quoted SQL name can no longer end its quotes early** -- @hc12r.
Brokoli wrapped table and column names in the database's quotes
without escaping a quote inside them, and relied on every caller
refusing such names first. One path did not: a migrate whose source
and destination are the same Postgres server quoted the column names
of the source query's result as they came, so a column named with a
double quote broke the migration's SQL, and a crafted name could have
added to it. Quotes inside a name are now doubled, the way each
database writes them, and that path checks `dest_table` and
`key_columns` like every other write path.
- **Integers from REST sources, JSON files and code nodes stay exact**
-- @hc12r. These paths decoded every number as a float64, so an id
above 2^53 arrived changed (9007199254740993 became
9007199254740992). A numeric pagination cursor was also sent back
formatted as a float: 1234567 went out as `1.234567e+06`, so the next
page was never found. An integer now decodes to an exact int64, as it
already did once data crossed a node boundary; a number with a
fraction or exponent is still a float64. Code that embeds Brokoli and
type-asserts these values sees `int64` where it saw `float64`.
- **Dedup, join and count_distinct compare values, not their
printouts** -- @hc12r. Dedup keyed rows by printing each value, so
rows that were not duplicates were dropped: the number 1 and the text
"1" print the same, a missing value printed like the text "<nil>",
and a NUL inside one value could line two rows up. A join, the other
way round, never matched an integer id from a database with the same
id read as a float elsewhere, and count_distinct counted 1 and 1.0
twice. All three now compare values: numbers by value whatever type
carries them, text only with text, and a missing value only with a
missing value.
- **Concurrent first requests no longer race on the session-token
secret** -- @hc12r. The secret that signs session tokens was loaded on
first use with no lock. Requests arriving together raced on it, and
with no `BROKOLI_JWT_SECRET` and no secret file each could generate
its own, so a token signed under one failed to verify under another.
Starting the server again in the same process could also replace it
and invalidate tokens already issued. It is now loaded once and kept,
and a token is accepted only when signed with HS256.
- **Reading `/metrics` during runs is no longer a data race** -- @hc12r.
The engine counts runs, attempts and events with atomic updates, and
the metrics endpoint read the same fields plainly. It now reads them
through `Engine.Metrics()`, a snapshot taken atomically, which code
embedding Brokoli should use too.
- **Scheduled, resumed and woken runs can stage task data** -- @hc12r.
Each way a run starts built its runner separately and copied the
engine's settings by hand, and the copies drifted. Only runs started
through the API or by a worker could issue the data capabilities a
remote task node needs for inputs over the inline row cap or for
stored outputs. Scheduled, backfill, webhook and trigger runs,
resumes, and runs woken from a wait were refused with "this server
issues no data capabilities". A woken run also ignored the engine's
native-driver and wait-in-place settings. Every run now gets its
runner from one place.
- **A list that fails partway through says so instead of coming back
short** -- @hc12r. Twenty-two store functions (connections,
variables, workspaces and their members, API tokens, the run calendar,
the dead-letter queue, physical instances) read rows until the driver
stopped and never asked why, so an error mid-list returned a shorter
list as if it were complete. They now return the error.
- **A panic inside a store transaction no longer leaves it open** --
@hc12r. `WithTx` rolled back only when its function returned an error.
A panic left the transaction open, and on SQLite every later write
then failed with "database is locked".
- **A server whose scheduler failed to start shuts down instead of
hanging** -- @hc12r. When the scheduler could not start (its first
read of the pipelines failed), the server logged a warning and carried
on. At shutdown, stopping the scheduler then waited for loops that
had never started, so the process hung until it was killed. Stopping
a scheduler twice also panicked. Both now return.
- **Rate limiters no longer leak a goroutine each** -- @hc12r. Every
rate limiter started a cleanup goroutine that never stopped. A server
builds four, so code that builds servers repeatedly (tests, an
embedding application) leaked four per server. Idle clients are now
forgotten while requests are handled, with no goroutine.
- **A cancelled run is no longer counted as a failure** -- @hc12r. A
cancelled run ended with the same error as a failed one, so the engine
added it to `brokoli_pipeline_runs_failed`, and cancelling runs raised
the metric operators alert on. Cancelled runs are now counted in a new
`brokoli_pipeline_runs_cancelled` (`RunsCancelled`, also in
`Engine.Metrics()`), and the error they return is `engine.ErrRunCancelled`,
with the same message as before.
- **A retry no longer runs alongside the attempt that timed out** --
@hc12r. When a node timed out, Brokoli stopped waiting for it but did
not stop it: a handler that does not watch its deadline kept running,
and the retry started beside it, so a sink or migrate could write
twice. A retry now waits for the timed-out attempt to stop, for up to
the node's timeout (at most 30 seconds). An attempt still running after
that fails the node instead of being retried beside it, with an error
that says so.
- **A sink_db write stops when its attempt times out** -- @hc12r. A
database sink wrote under a context of its own, so a node's timeout
never reached the database: a write blocked on a lock carried on after
its attempt had timed out, and the retry wrote the same rows again
(6 rows for 3 on Postgres and on SQLite, with the run reported as
successful). The write now runs under the attempt's deadline and is
cancelled with it. `engine.ExecuteSQLContext` is the context-aware form
of `ExecuteSQL`.
- **A migrate stops when its attempt times out** -- @hc12r. Same flaw
as the database sink: migrate read and wrote under the run's context,
which a node's timeout never reaches, so a write blocked on a lock
outlived its attempt and the retry migrated the rows again (6 rows for
3 on Postgres). Reads and writes now run under the attempt's deadline,
on every migrate path including partitions and remote workers.
`engine.QueryDatabaseContext` is the context-aware form of
`QueryDatabase`.
- **An API sink stops sending when its attempt times out** -- @hc12r.
`sink_api` sent its batches under the run's context, which a node's
timeout never reaches, so after the node had timed out and failed it
went on sending the remaining batches for as long as the run lasted.
It now sends under the attempt's deadline. A request already received
by the endpoint cannot be taken back; no further batches go out.
- **A notify node gives up when its attempt times out** -- @hc12r. It
posted with no context and only a 10-second client limit, so a slow
endpoint held the attempt past the node's timeout. The request now
carries the attempt's deadline.
- **Database and API sources, dbt and code expansions stop with their
attempt** -- @hc12r. These steps ran under contexts their node's
deadline never reached. A `source_db` query whose attempt had timed out
went on waiting in the database. A `source_api` request ran to the
API's reply, and a paginated fetch could go on to further pages and sat
through every retry wait. A timed-out `dbt run` kept building models.
A code expansion's local instances ignored even cancelling the run.
Each now stops when its attempt times out or is cancelled.
`fetchers.ContextFetcher` and `RESTFetcher.FetchContext` bound a fetch
by a context.
## [0.19.0] - 2026-10-24
Brokoli becomes a runtime for AI agents as well as people (ADR-046): an
MCP server in every binary, agent identities with their own tokens and a
role capped by their owner's, change requests an agent proposes and a
person applies, and MCP clients that connect with OAuth. Also previews
that read only the rows they show, and migrations safe for several
instances starting at once.
### Changed
These can change what an existing deployment does.
- **Upgrading from 0.18** -- @hc12r. Migrations run automatically on
start. On Postgres they now take an advisory lock, so the database URL
must be a direct connection or session pooling, not a
transaction-pooling proxy. The built-in `admin`, `editor` and
`operator` roles gain `pipelines.propose`, and a new built-in `agent`
role exists; custom roles are unchanged. The editor's Preview no longer
writes through outputs or records a run, a Git-managed pipeline can no
longer be rolled back from Brokoli (403), and a missing file under
`/assets/` is a 404 instead of the application page. Each is described
below.
- **`pipeline_id` is unique within a workspace, not an organization**
-- @hc12r. Two workspaces of one organization can each have an
`orders`, so a staging workspace can hold the same pipelines as
production. The per-workspace index is created before the
per-organization one is dropped, so uniqueness never lapses, and
existing data is valid under it. `GetPipelineByPipelineIDInWorkspace`
is the lookup to use; `GetPipelineByPipelineIDInOrg` now answers
`ErrAmbiguousPipelineID` when more than one workspace has the id,
rather than returning either. The git push route looks in the caller's
workspace first.
### Added
- **Brokoli for AI agents over MCP** (ADR-046 phase 1) -- @hc12r.
`brokoli mcp` serves the Model Context Protocol on stdio for an agent
client on your machine, acting on a server with the credential `brokoli
login` stored (or `--api-key` / `BROKOLI_TOKEN`); `POST /api/mcp` serves
it over Streamable HTTP (stateless, JSON) from the server. Eight read-only
tools: `list_pipelines`, `get_pipeline`, `validate_pipeline`,
`get_lineage`, `list_runs`, `get_run`, `get_run_logs` and `explain_run`,
which answers why a run failed in one result (the failing nodes with
their errors and last log lines, the run's error lines, the last
successful run before it and the pipeline versions saved since). Every
tool is a call to the REST API as the caller, through the same router
and permission checks, so an agent sees what its account sees; tools
the caller lacks the permission for are not listed. Every result is
redacted (credentials by name, URL passwords, bearer and basic
credentials in text) and bounded (pages of at most 100, at most 1,000
log lines, long text cut). See `docs/mcp.md`.
- **Agent identities** (ADR-046 phase 2) -- @hc12r. An agent is an
account for an AI agent: a role (the new built-in `agent` role by
default: view pipelines and runs, run pipelines; never `admin`), an
owner, and tokens of its own (`bka_...`, stored as a hash, 90 days by
default and at most 365, revocable one by one, last use recorded). Its
permissions are its role's intersected with its owner's on every
request, never including reading secrets or managing users, roles,
members or tokens. It cannot sign in or manage agents. Runs it starts
are attributed to it (`triggered_by.kind` `agent`, with `on_behalf_of`
when its owner issued the token for themselves) and audit entries carry
`actor_kind`, `owner_id`, `on_behalf_of` and `token_name`. Managed with
`/api/agents` and `brokoli agents`; agents are not listed among users.
Over MCP the agent is told who it acts as, and is warned (as is
`brokoli mcp`) when it uses the server's static API key. See
`docs/agents.md`.
- **Change requests** (ADR-046 phase 3) -- @hc12r. A proposed new
version of a pipeline that does not run until a person applies it:
`POST /api/pipelines/{id}/changes` records the whole definition, the
version it was made against and its validation; the diff (pipeline
fields, nodes added, removed and changed, connections) is computed on
read. Applying (`pipelines.edit`, people only) is exactly a save, with
the version message `change request: <title>`; a request whose pipeline
changed since is out of date and refused; declining takes a reason, and
the author can withdraw. New permission `pipelines.propose` (admin,
editor, operator and agent roles). The MCP tools `propose_change` and
`list_change_requests`; the pipeline page shows open requests with the
diff and Apply / Decline. Pipelines deployed from a repository take
requests only through a distribution's `api.ChangeRequestDeliverFunc`.
See `docs/change-requests.md`.
- **Agents run pipelines over MCP** (ADR-046 phase 4) -- @hc12r.
`trigger_run` runs a saved pipeline, with params and declared
parameters, and with `wait_seconds` (up to 45) answers the run's
outcome; `dry_run_pipeline` previews one, writing nothing; `cancel_run`
stops a run (`runs.cancel`, which the `agent` role does not have). The
acting tools are marked as such in their annotations, so a client asks
the person first, and a run started this way is recorded as the agent's.
- **Connect an MCP client with OAuth** (ADR-046 phase 5) -- @hc12r. A
hosted client that speaks the MCP authorization specification connects
to `/api/mcp` with only its URL: it discovers the authorization server
(`/.well-known/oauth-protected-resource`, `/.well-known/oauth-authorization-server`),
registers itself, and sends the person to Brokoli's approval page; the
person approves and the client gets tokens. PKCE (S256) is required;
access tokens last an hour and are renewed with rotating refresh tokens;
a connection acts as the person who approved it and can be revoked.
`BROKOLI_PUBLIC_URL` names the server's public address for discovery.
- **Connect an MCP client as one of your agents, and see your connected
apps** (ADR-046 phase 5b) -- @hc12r. The approval page offers the
person's enabled agents besides themselves; a connection made as an
agent acts exactly as one of its tokens (its role, capped by the
owner's, attributed to the agent), and ends when the agent is disabled
or changes owner. **Settings, Connected apps** lists each connection
(application, identity, connected, last used) and disconnects it at
once (`GET /api/oauth/connections`, `DELETE /api/oauth/connections/{id}`;
persons only). The `ConnectedApps` component is exported from
`@brokoli/auth`.
- **A review page for each change request, and testing a change before
applying it** -- @hc12r. A proposed change opens on its own page
(`/pipelines/{id}/changes/{changeId}`, linked from the pipeline's page
and from the editor): its description rendered as Markdown; a card per
node it adds, changes or removes, with scripts and SQL as line diffs;
the pipeline's graph with what changes marked; and a **Test** that
previews the current and proposed pipelines side by side on 10 rows,
showing step by step the columns, row counts and values that differ.
`POST /api/pipelines/{id}/changes/{changeId}/test` answers both
previews. A test runs the proposed code, so only a person with
`pipelines.run` starts one; an agent is refused. `propose_change` tells
agents the description is Markdown and what to put in it.
- **`GET /api/changes`: the workspace's change requests** -- @hc12r.
Across its pipelines, newest first (at most 100), each with
`pipeline_name`, filtered by `status`; only pipelines the caller may see.
For a person's list of what to review, wherever it was proposed.
- **`/api/auth/me` names the workspace** -- @hc12r. The answer adds
`workspace_id`, and `workspace_name` when it is known: the workspace
the request acts in. The MCP server says it in its instructions ("You
act in Brokoli as agent triage-bot, on behalf of ada, in the workspace
"Data Platform" (id ...)"), since an agent acts in one workspace and
may not list workspaces, and could not tell which one it was in.
- **A shorter sidebar, and Home** -- @hc12r. The community sidebar goes
from eleven items to seven, grouped by what a person is doing: Home and
Pipelines; Observe (Dashboard, Lineage); Connect (Connections, Secrets &
variables, Integrations); with Settings and Help at the bottom. Pages
that belong together share one item and a tab strip (Dashboard and
Schedule; Lineage and Pipeline dependencies; Variables and Secret
stores; Plugins and Drivers), each keeping its own address, so links
keep working. Home, where sign-in now lands, lists what needs you: the
pipelines whose latest run failed and the changes waiting for a
decision, counted next to Home in the sidebar. Search still finds every
page by its own name, with the shortcut letters it had. The tab strip
is a shared component (`SectionTabs` in `@brokoli/shell`).
- **`api.AgentHooks`** -- @hc12r. A distribution takes part in agent
identities: `Created` (an error disables the agent again and is the
answer, with `AgentHookError`'s status), `Updated`, `Disabled`, and
`Visible`, which hides agents the caller may not see from the list and
every per-agent route. Where a distribution resolves a request's
permissions (`RequestIdentityFunc`), an agent's stand as resolved, less
what no agent may hold: the distribution binds them to the owner in its
own roles.
- **`api.AgentIdentitiesEnabled`** -- @hc12r. A distribution whose
tenancy agents do not take part in yet sets it to false: `/api/agents`
then answers 404 and no agent token authenticates. Agents belong to no
organization in core.
- **MCP extensions for distributions** -- @hc12r. `mcpserver.Register`
adds a distribution's tools (listed by the permission each needs,
answering through the same caller, redacted and bounded like core's,
with `AddReadTool`, `Get` and `Esc`) and facts to `explain_run`, which
appear under `context`; a failing one is reported in `could_not_check`
instead of failing the explanation.
- **`api.SetChangeRequestDelivery`** -- @hc12r. A distribution that
delivers a change request elsewhere (a pull request opened after the
request is recorded) records where it went once it knows; only an open
request's delivery changes. With `SetChangeRequestOutcome`, a delivered
request shows its link on the review page and settles when it is merged
or closed there.
- **Signed webhook paths a distribution can register** -- @hc12r.
`api.RegisterSignedWebhookPath(path)` exempts one exact POST path from
session and API-key authentication, for a route whose handler verifies
the request itself (a webhook signed with a shared secret). Matched like
the built-in webhook exemptions: the exact string the router dispatches
on, so no prefix, trailing segment, encoded separator or other method
passes.
- **Queued jobs that are not pipeline runs** -- @hc12r. `RunJob` has a
`Kind`; empty is a pipeline run, as every job was before. A distribution
registers a handler per kind in `Registry.JobHandlers`, and a worker
runs a job of that kind with it, in a worker slot, renewing its claim
as a run does: acknowledged when the handler returns nil, returned to
the queue when it returns an error. A job whose kind the worker has no
handler for goes back to the queue for one that does.
### Fixed
- **A run exists when its id is returned** -- @hc12r. On the in-process
path, `POST /api/pipelines/{id}/run` answered 202 with the run's id
before the run was recorded: the runner wrote it from its goroutine, so
a client asking for the run at once (an MCP client's `trigger_run`
waiting for the outcome, say) could get 404. The run is now recorded
before the id is handed out, as the job-queue path already did.
- **An agent's name is unique where it is seen, not across the server**
-- @hc12r. Agent names were usernames, unique server-wide, so a
distribution with organizations refused a second organization's
`triage-bot` ("an account named triage-bot exists already"), telling it
that another tenant had one. A new agent's name is now its display name,
unique within a scope a distribution sets (`AgentHooks.Scope`; the whole
server by default, where people's usernames are taken too), and its
username is an internal `agent-<id>`. Existing agents keep their names.
An agent refused by `AgentHooks.Created` is deleted instead of disabled,
so its name is not lost for good.
- **A preview read every row of its sources to keep ten** -- @hc12r. The
editor's Preview and a change request's test held a whole source in the
server's memory: previewing a pipeline over millions of rows took as
long as a run and could exhaust the server. Sources now stop at the
rows a preview keeps: a CSV is read until it has them, a database query
is cancelled once it has them, a paginated API stops fetching pages, and
a Python step that `emit()`s stops once it has emitted them. Such a step
could not be previewed at all before ("emit() requires file-mode
output"); it now runs as a streamed run does.
- **Preview wrote through output nodes and recorded a run** (#595) --
@hc12r. The editor's Preview (`POST /api/pipelines/{id}/dry-run`) ran
the pipeline as a run: a `sink_file` overwrote its file with the
preview's rows, other outputs and notifications acted, hooks were
called, and a run was left in the pipeline's history. A preview now
runs in a scratch store that is thrown away, holds back every step that
may act outside Brokoli (outputs, `notify`, `migrate`, `bash`, `dbt`,
`task`, `wait`, and nodes declaring the `sink` capability) and what
follows it, reporting them as `held` and `not_run` with the reason, and
calls no hook. Steps run in the server's own process, and a preview
stops after 90 seconds.
- **A worker waits out an unreachable native driver catalog instead of
exiting** (#869) -- @hc12r. Installing the managed driver set tried each
download once, so a DNS or network blip while a worker started made it
exit and restart. A failure to reach the catalog is now retried with
backoff (2 seconds, doubling to 30) until the install's own deadline;
a digest mismatch or a release not in the catalog still stops it at
once, and a worker missing a declared driver still takes no jobs.
- **Pipelines built from code are laid out instead of stacked** -- @hc12r.
The canonical IR a repository deploys leaves node positions out on
purpose (layout is not a change to the pipeline), so such pipelines
were stored with every node at the origin and drawn on top of each
other everywhere but the editor. A pipeline whose nodes all sit at one
position is now laid out when created or saved
(`engine.ArrangeStacked`, for distributions that store pipelines
themselves too), and the runs page's graph lays out one already stored
that way. Positions no longer count as a graph change, so pausing or
renaming an old pipeline still skips full validation when only its
layout moved.
- **A tab opened before an upgrade gets the new version instead of a
developer error** -- @hc12r. Pages are loaded on demand, and a tab
opened before an upgrade asks for files of the old build. The server
answered a missing file under `/assets/` with the application page
(200, `text/html`), so the browser failed with "error loading
dynamically imported module" and React Router showed its developer
screen. Missing assets now get a 404 (never cached), and a page that
fails to render shows `ErrorScreen` from `@brokoli/ui`: for a module of
an earlier build it reloads once to get the new one (not again within
30 seconds, so a broken build cannot loop), otherwise it says what
failed and offers a reload. The community application uses it on every
route, inside the shell where the shell rendered.
- **The editor draws a pipeline compiled from code laid out, and moving
nodes no longer leaves a Git-managed pipeline "unsaved"** -- @hc12r. A
pipeline deployed from code arrives with every node at one position;
the editor drew them on top of each other, and laying them out to see
them counted as an edit, so after saving through `gitSave` the editor
still said "Unsaved changes". Stacked nodes are now laid out as part of
loading, and for a pipeline saved through `gitSave` (whose layout is
not saved anywhere) moving or laying out nodes does not count as a
change.
- **Rollback refuses Git-managed pipelines** (#854) -- @hc12r. Editing a
pipeline whose source is Git was already refused, but rolling it back to
an earlier version was not: the pipeline then ran something its
repository did not say, until the next sync silently undid it. A
rollback now returns 403 with the same advice as an edit, to revert the
change in the repository.
- **Instances starting together no longer crash migrating one database**
(#849, #852) -- @hc12r. On Postgres, the processes of an installation
that start at the same time all migrated the database at once. A first
install restarted repeatedly before settling (five processes, fourteen
restarts in a rehearsal), failing on `duplicate key value violates unique
constraint "pg_type_typname_nsp_index"` or on two processes seeding the
same template, and an upgrade adding an index could fail the same way.
Migrations now run one process at a time under a Postgres advisory lock;
the others wait, then find everything in place. They need a direct
connection or session pooling, not a transaction-pooling proxy.
- **Starting before Postgres accepts connections no longer exits** (#852)
-- @hc12r. A refused connection is retried for up to 60 seconds, so
processes that start alongside their database wait for it instead of
restarting. A wrong password or database name still fails at once.
The wait is exported as `store.WaitForPostgres` (#853), so code that
opens its own pool at startup waits the same way instead of coming up
without its feature.
### Security
- **Go 1.26.10** -- @hc12r. GO-2026-6809: crypto/x509 did not distrust
compromised intermediate certificates, reachable from every outbound
TLS connection. The toolchain pin, CI and release builds move to the
fixed release.
## [0.18.17] - 2026-10-17
### Changed
- **`pipeline_id` is unique within an organization** (#846) -- @hc12r.
It was unique across the whole server, so in a multi-tenant deployment
one tenant's create was refused for a name another tenant used. The
unique index is now `(org_id, pipeline_id)`; opening a database creates
it before dropping the old one. `GetPipelineByPipelineIDInOrg` looks one
up within an organization. Single-tenant servers behave as before.
- **`conn_id` is unique within a workspace** (#847) -- @hc12r. Same issue
for connections. Uniqueness is now `(workspace_id, conn_id)`: Postgres
drops the inline constraint, and SQLite rebuilds an older `connections`
table once, keeping every row. The store gains `GetConnectionIn` and
`DeleteConnectionIn`; runs resolve a `conn_id` in their pipeline's
workspace; connection pool budgets are keyed by workspace and name; the
connection routes act on the connection in the caller's own workspace.
## [0.18.16] - 2026-10-17
### Security
- **`./uploads` is no longer served without sign-in** (#844) -- @hc12r.
`GET /uploads/{filename}` served any file in `./uploads` to anyone: the
route sat outside `/api`, so no authentication ran, and nothing tied a
file to its owner. Core writes nothing there; the route is removed.
## [0.18.15] - 2026-10-17
### Security
- **Scope checks fail closed and answer alike** (#840) -- @hc12r. Run and
pipeline routes checked the caller's organization only when the pipeline
could be read, so a failed read skipped the check, and a missing
pipeline answered differently from another organization's. They now
refuse anything short of a pipeline the caller can see, with the same
404. `POST /api/git/push/{id}` exports only a pipeline the caller's
organization can see, and `GET /api/git/status` takes `gitsync.view`.
Dependency lookups for a pipeline with no organization are scoped to the
caller's organization. Blob capability refusals no longer say whether
the attempt exists. Finished-run events keep their organization in the
live state, and the cancel and wait-timeout events carry one.
### Fixed
- **The installer test no longer reports a phantom install** (#842) --
@hc12r. The harness reused install directories through `$RANDOM`, so a
refusal check could find an earlier run's binary.
## [0.18.14] - 2026-10-17
### Security
- **No admin creates or resets an account above its own role** (#837) --
@hc12r. `POST /api/auth/users` accepted role `superadmin` from any
caller holding role `admin`; only a superadmin may create one now, and
anyone else gets 403 with no account created. `POST
/api/auth/admin-reset-password` no longer lets an admin reset a
superadmin's password, and in multi-tenant mode, with the new
`UserOrganizationsFunc` hook set, resets only a user whose every
organization is the caller's current one (it compared primary
organizations only).
- **Variables and connections are created only in the request's own
workspace** (#838) -- @hc12r. `POST /api/variables`, `PUT
/api/variables/{key}` and `POST /api/connections` honoured a
`workspace_id` from the body as given; a named one must now be the
workspace the request was authorized for (403 otherwise), and an
organization member who belongs to no workspace can no longer create
into `default`. `PUT /api/variables/{key}` also wrote the body's key,
ignoring the path; the two must match.
- **Server-wide operations ask for permissions of their own** (#839) --
@hc12r. Plugin install and remove take the new `plugins.manage`, and the
notification channels (read and write) take the new
`notifications.manage`, instead of `settings.edit`. The built-in admin
role holds both; custom roles that relied on `settings.edit` for these
need the new permission. Also: `POST /api/system/purge` is scoped to the
caller's organization whenever there is one (an organization whose id
is `default` purged every tenant's runs); `POST /api/test-connection`
takes `connections.test`; `POST /api/pipelines/import` always assigns a
new id and logs store errors instead of returning them; `GET /api/audit`
and `/api/audit/{id}` take `audit.view`, and the list keeps only the
caller's organization's entries.
## [0.18.13] - 2026-10-17
### Fixed
- **SQLite connections wait for a busy writer instead of failing** (#835,
fixes #834) -- @hc12r. A `sqlite://` URI or a bare `.db`/`.sqlite` path
was opened with SQLite's default busy timeout of 0, so a writer that
found the file locked failed at once with `database is locked
(SQLITE_BUSY)`. A partitioned migrate into one SQLite file, whose ranges
write in parallel, failed this way on most runs (4 of 10 succeeded at
`max_parallel` 6). Both forms now get `_pragma=busy_timeout(5000)`, the
value the metadata store already uses, unless the URI sets its own
`busy_timeout`. Nothing else about the database is changed.
## [0.18.12] - 2026-10-14
### Changed
- **Template curation asks for `templates.manage`** (#832) -- @hc12r.
Creating, changing and deleting pipeline templates was gated on the
role claim being `admin` or `superadmin` and nothing else. Templates
are shared by the whole instance, so these routes now go through the
same strict permission gate as plugin install, system purge and native
drivers. Without a permission extension nothing changes: only `admin`
and `superadmin` may curate templates. With one, the extension decides
who holds `templates.manage`; a role named `admin` is no longer enough
on its own. Listing templates still needs no permission.
## [0.18.11] - 2026-10-14
### Changed
- **`RequestIdentityFunc` can resolve a request's permissions** (#830) --
@hc12r. The hook now returns an `IdentityRefinement`: the claims and a
refusal as before, and optionally the request's permissions. When it
resolves them, `PermissionChecker.HasPermission`, the open-source
permission gate and `/api/auth/me/permissions` read them instead of
looking the role claim up in the roles table, whose one namespace for
the whole instance had no room for roles defined per organization. An
empty, non-nil list grants nothing; with none, the role claim decides
as before. `api.RequestPermissions` exposes the list. The hook's
signature changes; it was added in 0.18.10.
## [0.18.10] - 2026-10-13
### Added
- **`RequestIdentityFunc`** (#827) -- @hc12r. An embedding application
can refine the identity of every request that carries one, after its
organization and workspace are resolved and before any handler runs:
return the claims the rest of the request sees, or refuse it with a
status and message. Every permission check and handler reads the
returned claims, so memberships kept by the application take effect on
the next request instead of when the token expires. Unset, nothing
changes; anonymous and open-mode requests never reach it.
### Fixed
- **API tokens on Postgres** (#828) -- @hc12r. `GetAPITokenByHash`
scanned the nullable `last_used_at` into a `time.Time`, and nothing sets
that column, so on Postgres every API token was refused with
`401 invalid token`. SQLite was unaffected.
## [0.18.9] - 2026-10-11
### Added
- **Managed native drivers** (#825) -- @hc12r. `BROKOLI_MANAGED_DRIVERS`
declares the driver set by exact identity (name, version, library
sha256). A deployment that sets it is managed: `GET /api/drivers` and
`/api/drivers/catalog` list the declared set whatever the answering
process has installed, install and remove through the API return `403`
for every caller, and `serve --mode worker` installs every declared
build it lacks from `BROKOLI_DRIVER_INDEX` at startup, checks each
library against its declared digest, and does not start without them.
Any non-empty value is managed, even one that does not parse.
`/api/capabilities` reports `native_adbc_drivers.managed`; the Drivers
page then lists the provided drivers with nothing to install or remove.
`Manager.EnsureManaged` and `InstallFromCatalogAt` take the index URL
explicitly.
## [0.18.8] - 2026-10-11
### Added
- **Park before dispatch** (#822) -- @hc12r.
`Engine.ParkBeforeDispatch(run, pipe, params)` checks a pipeline's wait
nodes that have nothing upstream (`file_exists` excepted) on the
dispatching process, before the run is handed to another, and parks the
run on the first unmet one, so it holds no slot anywhere while it waits.
Config is resolved as a runner resolves it; a failing check fails the
run, as a runner's first check does.
- **`Engine.WakeHandoff`** (#822) -- @hc12r. Offered every run the
parked-wait watcher wakes, after the claim: taking it means it does not
run here, declining lets it run here as before, an error fails it.
## [0.18.7] - 2026-10-11
### Added
- **Waits can wait in place where a run cannot park** (#821) -- @hc12r.
`Engine.WaitInPlace` makes an unmet wait keep checking its condition in
the process running it, every `poll_interval`, until it is met, the run is
cancelled or `timeout` passes, ending with the parked wait's timeout
message. It is for a process that executes runs for another process's
store, which cannot park a run; the run keeps its slot while it waits.
Off by default. A wait node's attempt timeout covers the wait's own
`timeout` in this mode, and a retried wait keeps its first deadline.
- **`PipelineWaitChecker`** (#821) -- @hc12r. A store that cannot look
pipelines up can answer a `pipeline` wait itself; "pipeline not found"
still covers both a missing pipeline and another organization's.
- **`RunOptions.RunID`** (#821) -- @hc12r. Creates the run with a given
ID, so a process running a run another process created can cancel it by
that ID.
## [0.18.6] - 2026-10-10
### Security
- **Bash node limits hold for every process the command starts** (#819,
closes #818) -- @hc12r. The CPU, file-size and open-files limits, and the
address-space limit a node asks for without a cgroup, were set on the
shell after it had started. A command that forks at once, as any pipeline
does, could start children before they landed, and those children ran
with no limits (a CI run allocated 300 MB under a 100 MiB cap). The shell
now sets the limits on itself before the command runs, clamped to the
server's own limits as before; a limit it cannot set stops the node and
says so. Linux only, as before: other platforms apply none of these
limits.
## [0.18.5] - 2026-10-10
### Fixed
- **A node stops with the reason when its connection's credentials cannot
be resolved by the store** (#816) -- @hc12r. A store that resolves
connections on the caller's behalf can find a connection and fail to
resolve its credentials (for example an `env://` reference not allowed
where it is resolved). The resolver read that as a missing connection,
logged a warning, and ran the node without credentials, so it failed
later with "source_db node requires 'uri' config". A store now marks the
case with `store.ErrCredentialsUnresolved`, and the node fails with the
store's reason.
- **A `pipeline` wait says when it cannot be checked** (#816) -- @hc12r.
Where the store cannot look pipelines up, the wait answered "pipeline not
found" since v0.18.1, for an upstream that existed. It now says the
process cannot check the condition. A missing pipeline, or one in another
organization, still reads "not found".
## [0.18.4] - 2026-10-10
### Security
- **excelize past six more HIGH advisories** (#814) -- @hc12r.
CVE-2026-107211 to -107216, all reachable from a workbook a pipeline
reads: an unchecked pivot-cache field index, an unbounded row number in
`Rows.Columns`, a nil dereference in `GetSlicers`, panics in `Decrypt` on
a malformed OLE container, an attacker-sized allocation in `extractPart`,
and mutually-referencing array formulas. No release contains the fixes
yet; this pins upstream commit `6258dce`. Go's vulnerability database
does not list them yet, so `govulncheck` does not report them.
## [0.18.3] - 2026-10-10
### Security
- **Go 1.26.9 and `golang.org/x/net` v0.60.0** (#812) -- @hc12r. Thirteen
standard-library advisories (GO-2026-6599 to GO-2026-6617) in the
net/http HTTP/1 and HTTP/2 server and client, html/template, crypto/tls,
net/textproto and os, five of them also in `golang.org/x/net`. Release
binaries and images are built with the fixed toolchain. The vulnerability
allowlist is empty: the excelize advisory it waived is no longer reported.
### Fixed
- **Legacy `.xls` workbooks are refused with a clear message** (#811) --
@hc12r. The Excel reader handles `.xlsx` only; a binary `.xls`
(Excel 97-2003) or password-protected workbook used to fail with
"unsupported workbook file format". It is now recognised by its signature
and refused with a message to save it as `.xlsx`. A file that is `.xlsx`
inside but named `.xls` still reads.
- **File extensions match without case** (#811) -- @hc12r. `DATA.CSV` or
`REPORT.XLSX` failed as unsupported on the batch path while the streaming
path accepted them, so the same file ran or failed depending on whether
the run streamed. An unsupported or missing extension now names the
supported formats.
- **Cancelling a run right after it parks** (#811) -- @hc12r. A cancel sent
in the moment between a run parking at a wait node and its runner
unregistering was reported as delivered while the run stayed `waiting`;
a cancel in the same moment after a run finished reported success. Both
now take the store-side path: the waiting run is cancelled, the finished
one reports that it already completed.
## [0.18.2] - 2026-10-08
### Security
- **excelize past three HIGH advisories** (#809) -- @hc12r. Three
advisories were published on 2026-10-08 against
`github.com/xuri/excelize/v2` v2.11.0, which the Excel loader reads
workbooks with: an integer overflow in column-name parsing
(GHSA-c85p-xxjj-2r75), an unbounded spin count when decrypting an
agile-encrypted workbook, and an unbounded `<col max>` attribute, all
reachable from a workbook a pipeline reads. No release contains the fixes
yet; this pins upstream commit `696050f`, which has all three.
## [0.18.1] - 2026-10-08
### Fixed
- **A `pipeline` wait stays inside its organization** (#807) -- @hc12r.
The target pipeline must belong to the waiting run's organization;
otherwise the wait fails with "pipeline not found", as for a pipeline
that does not exist. It used to report the latest run status of any
pipeline by ID.
- **A `file_exists` wait is held to the data directories** (#807) --
@hc12r, as every file node is (`BROKOLI_DATA_DIRS`). It used to check
any path on the worker.
- **A waiting run can be cancelled** (#807) -- @hc12r. A run parked at a
wait node used to stay waiting until it woke or timed out: the cancel API
answered 404, or 200 with no effect where cancels are broadcast. It is
now cancelled and its park removed; if it woke at the same moment, the
cancel is applied to it as a running run.
### For code embedding the engine
- `store.WaitingRunCanceller` is a new optional store capability
(`CancelWaitingRun`); the built-in SQLite and Postgres stores implement
it. A store without it keeps the previous behaviour.
## [0.18.0] - 2026-10-07
MySQL and ClickHouse connections can read through native ADBC drivers, and
a driver says which connection types can use it.
### Added
- **MySQL and ClickHouse connections can be pinned to a native driver**
(#805) -- @hc12r. A `source_db` node reading such a connection runs
through the pinned build in the isolated worker, as Flight SQL,
PostgreSQL and SQLite already do.
- MySQL uses the `mysql` driver: `mysql://host:port/schema`, with the
connection's driver options as query parameters and credentials passed
as options, never in the URI.
- ClickHouse uses the `clickhouse` driver, which speaks ClickHouse's HTTP
interface: `http://host:8123/`, or `https://host:8443/` when the
connection sets `"secure": true`. A pin on the native-protocol ports
9000/9440 is refused, saying why.
See `docs/native-drivers.md`.
- **Which connection types can use a driver** (#805) -- @hc12r. The
driver APIs report `usable_by`, and saving a connection pinned to a driver
that does not serve its type is refused (the run checks again). The
Drivers page shows it, and no longer offers to install a driver no
connection can use yet; the connection form chooses drivers by it.
### Fixed
- **Arrow decimals other than 128-bit decode** (#805) -- @hc12r. Only
`decimal128` was read; a `decimal32`, `decimal64` or `decimal256` column,
which the MySQL driver returns for `DECIMAL(10,2)`, failed the run after
its rows were read. All four widths decode exactly.
## [0.17.2] - 2026-10-07
### Fixed
- **A native source no longer warns that it has no driver** (#803) --
@hc12r. A `source_db` node reading a connection through a native ADBC
driver logged, on every successful run, that its connection type "has no
database driver in this build" and that "the run will fail". Such a
connection is read through its pinned driver; the warning is now only
given for a type that genuinely has none.
## [0.17.1] - 2026-10-06
### Fixed
- **System roles now follow the code on every start** (#801) -- @hc12r.
The admin, editor, operator and viewer roles were written to the database
only on the first run, so an existing deployment kept the permissions of
the version it was installed with. A permission added since --
`secret_stores.manage` (v0.17.0) or `drivers.manage` -- was held by no
stored role: a permission check reading the stored role refused it to
admins, and the UI hid the matching controls. Every start now adds any
missing system role and brings each one's permissions to the code's
definition. Custom roles are untouched.
## [0.17.0] - 2026-10-06
Native ADBC drivers: a `source_db` node can read through a native driver --
Flight SQL always, PostgreSQL and SQLite when a connection is pinned to one --
inside an isolated, short-lived worker process. Partitioned `migrate` runs key
ranges in parallel. Secret store changes have their own permission. **Two
changes need attention before upgrading**; they are listed first.
### Changed -- read before upgrading
- **Changing a secret store takes `secret_stores.manage`** (#792) -- @hc12r.
Creating, editing and deleting stores used to take the connection
permissions; viewing and testing still do. The built-in editor and admin
roles hold it, so nothing changes for them; a custom role that managed
stores needs it added.
- **New permission `drivers.manage`** (#799) -- @hc12r. Installing and
removing native drivers takes it, and only the built-in admin role holds
it: a driver is native code every worker on the host loads, for every
workspace. A distribution with tenants should map it to its platform
operators, never to a tenant's administrators.
### Added
- **Native ADBC drivers** (#799) -- @hc12r. Driver builds are installed
into the driver directory (`BROKOLI_DRIVER_DIR`) from an archive and its
digest (`brokoli drivers install <archive> --sha256 ...`) or from a
curated catalog (`brokoli drivers install <name> --catalog`, or the new
Drivers page). A connection pins one exact build by its library digest
(`driver_identity`); a run reads through that build in a fresh
`worker task` child that looks the identity up and verifies the library
immediately before loading it, then streams Arrow straight into the
artifact store. Works on the streaming path, the batch path and dry
runs; Test runs `SELECT 1` through the pinned driver. New connection type
`flightsql`. Runs are routed by `native-adbc:*` capability tags to
workers that have the build. See `docs/native-drivers.md`.
- Loading a driver needs a binary built with cgo and `-tags adbc`. **The
standard release binaries cannot run native drivers**; they can manage
drivers and pinned connections, and say so where it matters.
- There is **no default catalog**: catalog installs are available once an
operator sets `BROKOLI_DRIVER_INDEX`, since the catalog is not signed.
- The worker gets an allowlisted environment, never the server's
(`BROKOLI_NATIVE_PASS_ENV` widens it), and limits itself with
`BROKOLI_NATIVE_MEMORY_MB`, `BROKOLI_NATIVE_CPU_SECONDS` and
`BROKOLI_NATIVE_OPEN_FILES`. A Flight SQL host is checked against the
outbound policy before the driver is given it.
- Run parameters (`${param.x}`) in a native source's query are refused
for now rather than sent as literal text.
- **Partitioned `migrate`** (#793) -- @hc12r. A `partition` setting
(`column`, `strategy` numeric or date, `boundaries`, `max_parallel`)
splits a migrate into key ranges run in parallel, each with durable
status. It requires `mode: upsert` with the partition column among
`key_columns`, so a retried partition cannot duplicate rows. Postgres to
Postgres on the same endpoint merges server-side through a staging table;
across databases it streams in bounded batches. Measured at about 2.25x
(same endpoint, 8 partitions) and 2.47x (separate databases, 4
partitions) a single stream.
- ADR-044 (Arrow transport and a composable runtime) and ADR-045 (optional
native worker capabilities), both proposed (#796, #797) -- @hc12r.
### Fixed
- **A secret variable read without an encryption key is still reported
secret** (#792) -- @hc12r. A worker without the key returned the stored
value flagged as a plain variable, so it could reach logs unmasked.
### For code embedding the engine
- `extensions.CapabilityJobQueue.DequeueForCapabilities(advertised func() []string)`
is a new optional queue capability; the worker reads its capabilities
each time a job is considered. The in-memory queue implements it.
- `extensions.WorkOrder.Migrate` (`MigrateWorkOrder`) carries one migrate
partition, as connection references, never credentials (#793).
- `Engine.NativeADBCWorker` runs native sources; `NativeWorkerLauncher` is
the isolated implementation. An embedder whose binary does not register
core's `worker task` command sets its `Executable`.
- `codeexec.AllowlistedEnv(passVar, extra...)` is the environment filter
code nodes and the native worker share.
- `models.DriverIdentity` and `models.Connection.DriverIdentity` are new;
connections gained a `driver_identity` column (added automatically).
## [0.16.0] - 2026-10-03
Secret stores (ADR-041): a workspace can keep its credentials in its own
secret manager (HashiCorp Vault, OpenBao, AWS Secrets Manager or SSM), and a
connection refers to them with `secret://<store>/<path>#<field>`. The value is
fetched where the node runs and never written down. Resolved credentials are
now masked in everything a run records. **One change needs attention before
upgrading**; it is listed first.
### Changed -- read before upgrading
- **Credential references are checked when a connection is saved** (#783,
closes #781) -- @hc12r. `password_ref` and `extra_ref` used to be accepted
whatever they held, and a typo surfaced as a failed run. Saving now refuses,
with a 400 naming the field: an unknown scheme, a malformed `vault://` or
`k8s://` body, an `env://` name that is one of the server's own secrets, a
client-made `encrypted://` value, and a `secret://` reference to a store
that does not exist in the connection's workspace or whose `#field` does not
suit the provider. Nothing is fetched on save, and the operator allowlists
are still enforced where a reference resolves, not on save. A reference
already stored is not re-checked.
- For code embedding the engine: `secrets.Resolver.Resolve` takes a
`secrets.Scope` (#784). `Chain.Resolve(ctx, ref)` keeps its signature;
`Chain.ResolveIn(ctx, scope, ref)` is new.
### Added
- **Secret stores** (#787, #788) -- @hc12r. A store is a named,
workspace-scoped connection to a secret manager, managed under
`/api/secret-stores` (create, edit, delete, test) and on a new Secret
stores page. A connection's password, its whole extra document, or any
single value inside extra can be a `secret://` reference; the connection
form offers "From a secret store" for the password. Stores authenticate
with the machine's own identity (`ambient`, refused where
`BROKOLI_SECRET_STORE_AMBIENT=deny`), a short-lived OIDC token naming the
store, run and node (`oidc`), or a stored token (`token`, encrypted, never
returned). Values are cached for one run and dropped when it ends; a store
test reports the version and field names, never the value; a store a
connection refers to cannot be deleted or renamed. A distribution can add
providers through `extensions.Registry.SecretStoreProviders`. See
`docs/secret-stores.md`.
- **`vault` provider** for HashiCorp Vault and OpenBao (#789) -- @hc12r.
KV v2 with versions; token, JWT/OIDC and Kubernetes login; https required
except for a loopback dev server; redirects kept on the configured server;
session tokens revoked after use. See `docs/secret-store-vault.md`.
- **`aws_secrets_manager` and `aws_ssm` providers** (#790) -- @hc12r.
Secrets Manager secrets are single values or, when JSON objects, named
fields; SSM parameters are decrypted single values. OIDC uses
`AssumeRoleWithWebIdentity` with a session named after the run;
`ambient` uses the machine's credential chain. A store cannot set its own
endpoint. See `docs/secret-store-aws.md`.
### Security
- **Resolved credentials are redacted from what a run records** (#784,
closes #782) -- @hc12r. A connection's password and the credential-like
values of its extra settings (keys, tokens, auth headers, with the token
after a `Bearer` scheme on its own) are masked as `[redacted]` in node
logs, node and run errors, events, alerts and notifications, the DLQ and
recorded SQL. Before this, a target that echoed a credential back put it
verbatim in the run's error and every log line. Values under 8 bytes are
not masked. Remote paginated `source_api` pages mask their own errors.
- **The connection API says which references it shows** (#783, closes
#755) -- @hc12r. `encrypted://` stays masked; `env://`, `vault://`,
`k8s://` and `secret://` references are returned, as the location of a
credential a user needs to see to edit it. The code comments now agree.
### Docs
- ADR-041 is accepted (#785).
## [0.15.0] - 2026-10-02
Four new connectors -- Google Cloud Storage for file nodes, and
Snowflake, Oracle and Databricks for reading -- and a Bash operator
node. It also closes two ways a role that may only start runs could
change what a pipeline executes. **Two changes need attention before
upgrading**; they are listed first.
### Changed -- read before upgrading
- **Run parameters are bound in source SQL, not written into it** (#778,
closes #774) -- @hc12r. A `${param.*}` reference in a `source_db`
query or a `migrate` `source_query` was pasted into the SQL text, so
anyone allowed to start a run (`pipelines.run`, which the built-in
operator role has without `pipelines.edit`) could change the SQL it
executed. It is now replaced by the driver's placeholder and the value
passed separately. `'${param.x}'` binds as text and a bare
`${param.x}` as a number. **These now fail, with a message saying
what to write instead:** a reference inside a longer literal
(`LIKE '%${param.x}%'`, refused at validation), one used as a table or
column name, and a bare one whose value is not a number. A query that
uses a run parameter is no longer pushed down into a same-server
write. `${var.*}`, `${interval.*}` and `${run.*}` are unchanged.
- **Code-node scripts no longer substitute `${param.*}`** (#775) --
@hc12r. The same escalation in Python and TypeScript source: a run
parameter could change the code a node ran. Scripts read parameters
from `params` (`params["x"]`, `params.x`), as they always could. A
script that contains `${param.x}` now fails validation with that
advice. Other references still substitute.
- For code embedding the engine: `QueryDatabase`, `QueryBigQuery` and
`DryRunBigQuery` take trailing bind arguments (#778); existing calls
compile unchanged.
### Added
- **Google Cloud Storage for file nodes** (#772) -- @hc12r.
`source_file` and `sink_file` read and write objects in a `gcs`
connection's bucket, authenticating with OIDC workload identity
federation, a service-account key, or the machine's identity where
allowed. A failed write leaves no object behind, downloads are capped
and pinned to the measured generation, and the API endpoint cannot be
set from connection data. See `docs/gcs-file-delivery.md`.
- **Snowflake, read-only** (#769) -- @hc12r. Queries through a guarded
driver: only five URI options are accepted, password authentication
only, `PUT`/`GET` file transfer refused, the native minicore library
never loaded (`-tags minicore_disabled` plus `SF_DISABLE_MINICORE`),
and errors never repeat the password. Writes are refused by name. See
`docs/snowflake.md`.
- **Oracle, read-only** (#770) -- @hc12r. Through the pure-Go `go-ora`
driver: numbers decode exactly, passwords are redacted from driver
errors, and writes are refused by name. Tested against a real Oracle
Free server in CI. See `docs/oracle.md`.
- **Databricks SQL, read-only** (#771) -- @hc12r. Through a guarded
driver that validates the host and HTTP path, never repeats the token,
and sends every request through the outbound network policy. Writes
are refused by name. See `docs/databricks.md`.
- **Bash operator node** (#773) -- @hc12r. Runs a shell command on the
worker and passes its input through. It runs in its own process group,
so cancellation stops everything it started; output is logged with
bounds; values reach the command only through `env`, never by
substitution into the command. Trusted-worker execution, like a code
node. See `docs/bash-operator.md`.
- **Memory limit for bash nodes** (#777) -- @hc12r. Enforced through a
cgroup v2 leaf per run under `BROKOLI_BASH_CGROUP`, which counts real
memory so a JVM or Node started from the command works; an address
space limit only when the node sets `max_memory_mb`; otherwise the log
says the limit is not enforced. On Kubernetes the pod's memory limit
remains the ceiling.
- **`BROKOLI_DISABLED_NODE_TYPES`** (#775) -- @hc12r. Refuses node types
such as `bash,code,task` at validation and again on the worker, and
lists them in `GET /api/capabilities` as `disabled_node_types`. The
editor leaves them out of the node palette and marks existing nodes of
a refused type (#776). See `docs/code-execution-policy.md`.
### Fixed
- **The editor's file-node Location field offers Cloud Storage** (#772)
and the source-database forms offer Snowflake, Oracle and Databricks
as sources only (#769, #770, #771) -- @hc12r.
## [0.14.0] - 2026-09-28
Three new backends -- Azure Blob Storage for file nodes, native bulk
writes to SQL Server, and BigQuery -- and short-lived OIDC credentials
for reaching a customer's cloud without a stored key. It also closes
four ways a credential could be used wrongly or leak. **Three changes
need attention before upgrading**; they are listed first.
### Changed -- read before upgrading
- **The server refuses to start without a valid encryption key** (#757)
-- @hc12r. When `BROKOLI_ENCRYPTION_KEY` was set but invalid (not
base64, or not 32 bytes), the server logged one warning and ran with a
key of 32 zero bytes: credentials saved under the real key stopped
decrypting, and new ones were encrypted under a key anyone can
reproduce. A key file that existed but was unreadable or short was
silently replaced with a new key, losing every stored credential. Each
is now a startup error naming what to fix. **Before upgrading, grep
the server logs for `could not load encryption key`**: a deployment
that logs it has been running on the zero key and will not start until
the key is fixed. Key files longer than 32 bytes are still read as
their first 32 bytes, as before.
- **A credential reference that cannot be resolved fails the node**
(#758) -- @hc12r. A `password_ref` or `extra_ref` (`env://`,
`vault://`, `k8s://`, `encrypted://`) that failed to resolve was logged
to the server and skipped, and the node ran with an empty password, or
with the ciphertext as the password. The run log now says which
reference failed and why, and the node stops before it runs. Runs that
were silently proceeding on an empty credential will now fail at the
step that caused it.
- **Remote `source_api` pages carry the connection by reference** (#760)
-- @hc12r. A paginated page dispatched to a worker used to carry the
connection's password and auth headers in its work order. It now
carries the connection's ID and the pipeline's workspace, and the
worker resolves the connection itself. **Workers from before this
release fail those pages** with "source URL is required" (never an
unauthenticated request), so upgrade workers with the server. Code
that runs work orders directly should call
`ExecuteInstanceWorkOrderResolving` / `ExecuteInstanceJobResolving`
with a connection resolver.
- For code embedding the engine: `ConnectionResolver.Resolve`,
`ResolveIn`, `ResolveWithWarnings` and `ResolveWithWarningsIn` gain an
`error` return (#758).
### Added
- **Azure Blob Storage for `source_file` and `sink_file`** (#750, #687)
-- @hc12r. Through an `azure_blob` connection, with an account key or
a SAS token, on the batch and streaming paths. A failed write leaves
nothing behind, and the connection test authenticates for real. Every
remote file location now sits behind one transport interface, so the
next one is a single file. See `docs/azure-blob-file-delivery.md`.
- **SQL Server bulk writes** (#761, #681) -- @hc12r. `sink_db` and
`migrate` append and overwrite through SQL Server's native TDS bulk
copy: a million rows in about 12 seconds instead of about 3 minutes.
Columns are matched under the database's collation, as statements
are. Upsert is refused by name. Test connection uses the compiled
driver.
- **BigQuery** (#762, #764, #765, #766, #767, #685) -- @hc12r. A native
backend (ADR-042): `source_db` queries as query jobs, `sink_db` append
and overwrite as load jobs, which are free and atomic per job. Every
query is capped at 10 GiB billed by default, columns are discovered by
a free dry run, and jobs are labelled with the run. Authentication by
service-account key, by the machine's own identity, or by OIDC
workload identity federation with no stored key; an optional
`billing_project` pays for the jobs. Upsert is refused by name. See
`docs/bigquery.md`.
- **OIDC token sources and Google workload identity federation** (#763)
-- @hc12r. `BROKOLI_OIDC_TOKEN_FILES` serves tokens a platform writes
to disk, such as Kubernetes projected service-account tokens.
`BROKOLI_SECRET_STORE_AMBIENT=deny` switches off the machine's own
identity on servers that run pipelines for several teams; tokens read
from disk are that identity and follow the switch. See
`docs/workload-identity.md`.
- **Test connection resolves credential references** (#759) -- @hc12r,
through the same path a run uses. A connection whose credentials come
from `env://`, `vault://` or `k8s://` used to be tested with none. The
result notes when references were resolved on the server rather than
where runs resolve them.
- **ADR-041, external secret stores** (#756) and **ADR-042, native
BigQuery backend** (#762) -- @hc12r.
### Fixed
- **File nodes could only pick SFTP connections** (#750) -- @hc12r. The
Location field offered SFTP alone, so an S3 connection was never
selectable in the UI though the engine supported it.
- **SQL Server statement writes over 1,000 rows** (#761) -- @hc12r. SQL
Server refuses an `INSERT` with more than 1,000 row values; batches are
now capped there.
## [0.13.1] - 2026-09-26
Fixes two defects in 0.13.0, both found by validating 0.13.0 live on a
cluster before deploying it. Upgrade if you run 0.13.0.
### Fixed
- **Numeric contract rules gave wrong answers on integer data** (#748,
Tnsor-Labs/actually-fine#8) -- @hc12r. `range`, `type: number` and
`type: integer` breached every value that was not a 64-bit float,
reporting "value is outside the allowed range" for 3 in [0, 5]. That is
every integer column from `source_db`, and any spilled or streamed
data, where whole numbers come back as 64-bit integers to stay exact.
Under `quarantine` the rows were dropped without explanation, and the
same pipeline cleared different rows depending on whether its data had
been spilled. The contract engine now treats every numeric type as a
number and decides `integer` on the value.
- **Unrunnable transform rules are now actually refused on save** (#747)
-- @hc12r. 0.13.0 described this and did not do it: the check was wired
into the per-node detail validation, not the path saving a pipeline
takes. **The behaviour change listed under 0.13.0 takes effect in this
release.** The same split ran the other way for contracts, which were
refused on save but not flagged in the per-node view; both checks now
run on both paths, and a test asserts the two validators agree.
## [0.13.0] - 2026-09-26
### Added
- **`contract_gate` node** (#741, #742) -- @hc12r. Validates rows against
a versioned data contract from
[actually-fine](https://github.com/Tnsor-Labs/actually-fine), evaluating
the contract's own IR rather than translating it into the older
`quality_check` rule model. Cleared rows pass downstream, quarantined
rows are excluded, and a `reject` or `halt` breach fails the node after
its evidence has been written to the run log. The gate **streams**: it
is stream-eligible like a transform, so a `source_db -> contract_gate ->
sink_db` pipeline keeps flowing by reference and never materialises the
dataset. The contract engine checks record by record and a halt stops
at the record that caused it. See `docs/contract-gate.md`.
- **Guided contract editor in the UI** (#744) -- @hc12r. Contract Gate
appears under Processing in the Node Library with a form for contract
metadata and rules, predicate-specific fields for all nine predicates,
and optional JSON paste or `.json` upload. A rule's kind (`record` or
`stream`) is derived from its predicate rather than chosen, because
every predicate mandates exactly one and the other is always refused.
- **`run.lineage_incomplete` run event** (#734) -- @hc12r. Recorded when
a lineage profile or provenance write does not land, naming which half
of the record is missing and why. Informational: it never changes a
run's status.
- **Node.js runtime documentation and a TypeScript example** (#520) --
@dvd233.
### Changed
These can change what an existing deployment does.
- **Transform rules are validated when a pipeline is saved** (#740) --
@hc12r. Thirteen requirements that were enforced only at execution --
`sort` without `columns`, `aggregate` without `group_by`, `add_column`
without a name or expression, `project` without projections, and so on
-- are now save-time **errors**, as are unknown rule types and rules
with no type. **A pipeline carrying such a rule, which previously saved
and then failed on the last step of every run, will now be refused on
save.** Nothing that could ever have run is affected: each of these is
wrong under any input. Checks that depend on the data, such as whether
a named column exists, remain at execution time.
- **Contract gate contracts are validated when a pipeline is saved**
(#744) -- @hc12r. An invalid contract, for example a `unique` predicate
on a `record` rule, is refused on save rather than failing at run time.
- **Outbound HTTP requests identify as Brokoli** (#739) -- @hc12r. Every
request a pipeline makes -- `source_api`, `sink_api`, webhooks, wait
nodes, alerts, notifications -- now sends `User-Agent:
brokoli/<version> (+https://github.com/Tnsor-Labs/brokoli)` unless the
node sets its own. Previously they arrived as `Go-http-client/1.1`.
Operators who filter or allowlist on the agent string will see the new
value.
- **A recovered run that fails reports its own failure** (#736) --
@hc12r. A run put back on the queue by startup recovery carries a note
saying why while it is pending. If it then fails, its error is now the
failure that ended it, not that note. The note is not lost: it stays in
the `run.recovery_requeued` event.
### Fixed
- **A failed run persisted with an empty error** (#736) -- @hc12r. Every
terminal path wrote the reason into the run's terminal event and none
wrote it onto the run, so `runs.error` stayed empty and the run view
had nothing to show. Runs executed in-process by the core engine were
affected on every failure. The stored row now agrees with what the run's
own events rebuild to.
- **Endpoints rejected pipeline requests with 406** (#739) -- @hc12r.
Many endpoints refuse the Go default agent outright. Measured against
one public API, the same request answered 406 as `Go-http-client/1.1`
and 200 with an identifying agent. The failure surfaced as an opaque
`HTTP request failed: status code 406` pointing at the user's URL.
- **A run could succeed with its lineage silently missing** (#734) --
@hc12r. When a store could not write a lineage profile or provenance
record, the only trace was a debug log line per node, or for provenance
a line in the process log that was never associated with the run. The
gap is now logged against the run -- a store that cannot record lineage
says so once as a warning, a real write failure is an error -- and
recorded as `run.lineage_incomplete`.
### Development
Contributor-facing; no effect on a running deployment.
- **The license gate could not fail CI** (#743) -- @hc12r. The step
piped the scan into `tee` without `pipefail`, so it took `tee`'s exit
code and passed while the scan was refusing a dependency. The gate's
own tests now run the workflow step's command and assert it propagates
a failing scan.
- **`preflight.sh` could not pass** (#735) -- @hc12r. It gave the engine
package a 15 minute budget where CI gives it 25, and its UI stage
discarded the result of `npm run check` while failing on a Svelte
baseline file removed in the React migration.
- **CI's S3 test service moved to SeaweedFS** (#731) -- @hc12r. The
MinIO images it used were withdrawn from public registries.
## [0.12.1] - 2026-09-22
### Fixed
- **A binary built from source served a blank page and reported
success** (#728) -- @hc12r. `web/dist/index.html` was a real built page
committed to the repository, while `.gitignore` excluded
`web/dist/assets/`. Both arrived on 2026-04-07 in one commit that set
out to solve a real problem, `//go:embed` needing a file to match so
module consumers could compile; committing a built page answered it,
and shipped a broken one. The published module therefore carried a page
asking the browser for two asset files that were not in it, and any
binary compiled without running `build-ui.sh` embedded that page,
logged `Serving embedded UI`, and served a shell whose script and
stylesheet 404. Nothing said the UI was missing.
This reached three real paths. `go install
github.com/Tnsor-Labs/brokoli@latest` produced such a binary, because
the module contains the committed `index.html` and none of the assets.
The README's own "From Source" steps did too: `cd ui && npm run build`
writes `ui/apps/community/dist`, and only `build-ui.sh` places the
result on the embed target, so following the documented instructions
gave a blank page. And a contributor building locally hit the same
thing, which is the opposite of what CONTRIBUTING promises about
running the whole product on a laptop.
Release tarballs and container images were never affected: the release
workflow builds the UI first and refuses to publish without it.
Now nothing under `web/dist` is committed. `web/dist/.gitkeep` is
tracked only so `//go:embed all:dist` compiles in a checkout that has
not built the UI, and `build-ui.sh` restores it after it wipes the
directory, so a build leaves the tree clean. A binary with no bundle
serves `web/unbuilt/index.html`, a self-contained page that says the
UI was not built into it and gives the command, and the server logs a
warning instead of announcing success. `scripts/check-ui-bundle.sh`
verifies that a built bundle carries every asset its `index.html`
references, and refuses a page that references nothing at all;
`build-ui.sh` and the release workflow both call it. The README now
documents `build-ui.sh` as the source build step.
## [0.12.0] - 2026-09-22
> **Behaviour change:** a database node handed a Snowflake connection is
> now refused by name. Snowflake has no driver in this build; previously
> the connection built a URI that reached a driver which could not serve
> it, and the failure named neither the connection nor the real reason.
> The connection type itself still exists and still stores credentials.
> **Behaviour change:** `count_distinct` in the bounded-memory streaming
> aggregate path now matches batch execution, including null handling.
> A pipeline that aggregated a large dataset through the streaming path
> may report a different, and correct, distinct count than it did before.
### Added
- **Dataset schemas are a first-class contract** (#691, #694 to #699,
#701, #703 to #720) -- @hc12r. A pipeline can declare what a dataset
looks like, in a portable descriptor (`brokoli.dataset-schema/v1`,
published at `docs/schema/dataset-schema-v1.json` with positive and
negative conformance fixtures) built on the ADR-032 BPTD types:
ordered columns, scalar and decimal types with precision and scale,
nullability, enum, array, map and record descriptors, and a
`additional_columns` policy of `closed` or open.
The declaration is enforced at runtime, for materialized and streamed
datasets alike. A node whose output is missing a declared column now
fails naming the columns that are missing; under a `closed` schema,
undeclared columns and a changed column order fail too. This is the
point of the feature: before it, a source that quietly stopped
returning a column produced a narrower dataset and a green run.
Declared schemas propagate. Joins carry column types through the
collision planner, transforms carry renamed, dropped, added and
aggregated fields, code-node `output_schema` survives dynamic
expansion, and dataset map, dataset filter, condition and wait nodes
pass schemas through. A union of incompatible inputs is refused rather
than guessed. Decimal declarations map to the engine's exact decimal
type, so precision and scale survive into sink table creation.
The editor gained a schema editor on file, API and database sources,
column completion and missing-column validation in transforms, quality
checks, aggregates and database sink upsert keys, join collision
controls with a preview of the declared output columns, runtime schema
snapshots (columns, types, null percentages) on a node run, and failed
node runs that name the missing column or join key and list what was
available.
- **Explicit join collision policy** (#691, #694, #706, #707) --
@hc12r. When both sides of a join carry the same column name, the
policy is now stated rather than implied: `prefix` (the default, and
what the engine did before), `error`, or `alias` with stable
right-side aliases and recursive prefix avoidance. One output-schema
planner now serves both execution and column lineage, so what the
editor previews and what the run produces come from the same code.
When both upstream schemas are declared, join keys are validated for
presence and type compatibility, so a typo fails instead of being
hidden by the runtime's unknown-column fallback.
- **Native filter, project and aggregate operators** (#692, #700 to
#702, #708, #722) -- @hc12r. Language-neutral relational operators
with a versioned predicate and `case_when` evaluator, carried through
both SDKs. Filters push down into SQL only where the predicate is
provably equivalent, and unsafe shapes are refused rather than
approximated. Expression output types are inferred from declared
input types, including decimals, and stay unknown when a result is
mixed or unsupported.
- **API execution profiles** (#693, #721) -- @hc12r. A `source_api`
node can carry a versioned execution-profile marker
(`docs/schema/execution-profile-v1.json`) beside a fully expanded
policy: timeout, retries and backoff, requests per second, maximum
concurrency, retry scope and checkpoint interval. Core deliberately
does not prescribe profile names, so an organization can define its
own as long as it emits the same shape with every value explicit; a
profile that leaves any of them implicit is rejected, because a
profile is only portable when it carries the whole policy. A `strict`
profile also refuses concurrency against a sequential pagination
strategy (`cursor`, `next_link`, `link_header`), which cannot honour
it. Planner explanations show the effective policy.
- **SQL Server connector** (#681, #688, #724) -- @hc12r. The pure-Go
`go-mssqldb` driver is compiled in and registered, connection tests go
through `DetectDriver` and an authenticated ping rather than a port
probe, and SQL Server is usable by database nodes again. Cross
compilation stays clean (`CGO_ENABLED=0` for linux/amd64 and
windows/arm64), and CI runs an environment-gated smoke test against a
real server.
- **Customer-owned S3 for file nodes** (#538, #688, #725, #726) --
@hc12r. `source_file` and `sink_file` can read and write through an S3
connection, with endpoint options for S3-compatible providers and
optional `session_token` temporary credentials, which the shared
artifact store accepts as well. Documented in `docs/s3-file-delivery.md`
with Python and TypeScript examples, and covered by MinIO integration
tests that run through the real runner in CI.
- **The installer verifies what it downloaded** (#689) -- @hc12r.
`install.sh` checks the archive against the release's `checksums.txt`
and refuses on a mismatch, a missing entry, a missing checksums file,
or no SHA-256 tool available. Before this it installed whatever it
received.
### Changed
- **The connection catalogue no longer claims coverage it does not
have** (#680, #723) -- @hc12r. `IsDatabase` is now separate from
`BuildsURI`, because a transport such as HTTP, SFTP or S3 has a URI
without being something a database node can use. Redshift and
ClickHouse connection tests run through their real drivers instead of
checking that a port accepts a TCP connection, unsupported database
types are refused by name, and Snowflake is out of the node-usable
catalogue until it has a driver. Tracked in #688.
- **ADR-016 no longer claims plugin upgrade in place shipped** (#679)
-- @hc12r. It was never built; #511 tracks it.
### Fixed
- **`count_distinct` disagreed with itself between execution paths**
(#692, #722) -- @hc12r. The bounded-memory streaming aggregate counted
distinct values differently from batch execution, including how it
treated nulls and canonical value identity. Same pipeline, same data,
different answer depending on which path the dataset took.
- **Decimal precision was lost on the way to a sink** (#704, #705) --
@hc12r. A decimal declared with precision and scale reached the
engine's generic numeric type, so a created table did not carry the
exact type that was asked for.
- **Schemas did not survive code-node expansion** (#703) -- @hc12r. A
declared `output_schema` was dropped when a code node expanded
dynamically, so downstream nodes saw an undeclared dataset.
- **`install.sh` could not install anything on Alpine** (#690) --
@hc12r. busybox `wget`, which is the `wget` on Alpine, rejects
`--max-redirect` and `--show-progress`; both are gone, and the
`Location:` header is matched by field rather than line start.
`scripts/install_test.sh` now runs the real installer offline under
dash, bash and busybox against a fake release with stubbed `curl` and
`wget`, 25 checks, wired into the Security workflow.
## [0.11.32] - 2026-09-18
> **Breaking change for out-of-tree extensions:**
> `extensions.PlatformProvider.StartServices` gained a variadic tail,
> `StartServices(s interface{}, engine ...interface{})`, matching the
> convention `RegisterRoutes` already documents. An implementation must
> add the parameter to compile; it may ignore it, and a provider must
> tolerate an empty tail because an older core passes nothing.
### Fixed
- **A run dispatched to a remote worker could still be executed twice,
in the one process most likely to do it.** v0.11.31 gave recovery a
way to ask an extension "is somebody else already running this?"
(`Engine.ExternalRunClaim`), but the only hook that carried the engine
to an extension was `RegisterRoutes` — and `RegisterRoutes` is reached
only from `api.NewServer`. `--mode scheduler` builds
`api.NewMinimalServer` instead, while still running the recovery
sweep, so the sweep ran with no claim installed. Measured in
production under a burst of about twenty concurrent runs: four runs
executed twice, leaving sixteen duplicated `node_runs` rows, with the
runs reporting success. `StartServices` now receives the engine,
because `shouldStartPlatformServices` gates exactly the two modes that
run recovery (`all` and `scheduler`), making it the one hook both of
them pass through. (#676) -- @hc12r
- **Every pod restart ran one recovery pass with no extension hooks
installed.** Recovery runs in two places: a periodic leader sweep, and
a startup sweep in `serve` that is unconditional in every mode. The
startup sweep ran before platform services were started, so it swept
before any extension had been given the engine — and a rolling deploy
is exactly when runs are in flight on remote workers. Platform
services now start first. (#676) -- @hc12r
- **A re-queued run that then succeeded still showed a recovery error.**
Recovery records why it put an interrupted run back on the queue in
`runs.error`, and nothing cleared it again, so a run that executed a
second time and succeeded kept an error describing the execution that
was interrupted. The runner now clears it on the success path only;
the `run.recovery_requeued` event still carries the message, so the
audit trail is unchanged, and a failed run keeps its error. (#675)
-- @hc12r
### Known gap
- `--mode api` also runs the startup recovery sweep, and platform
services are deliberately not started on api replicas (that would run
the trial checker, usage collector and SLA checker on every one of
them), so an api pod still boot-sweeps without extension hooks.
Closing it needs either a hook-only attach point separate from
starting services, or not sweeping outside the mode that owns
recovery. Tracked rather than rushed. -- @hc12r
## [0.11.31] - 2026-09-17
> **Behaviour change for crash recovery:** a run is now left alone for
> its first 30 seconds before recovery will adopt it
> (`Engine.RecoveryMinRunAge`). A run whose process genuinely died
> therefore stays non-terminal for up to that much longer before anyone
> takes it over. The trade is deliberate: recovering a dead run later is
> a delay, while adopting one that is still starting runs the pipeline
> twice and reports no failure at all.
>
> **Note for anyone running the workflows themselves:** the Security
> workflow no longer calls `govulncheck ./...` directly. It runs
> `scripts/check-vulns.sh`, which reaches the same verdict but allows
> audited exceptions in `security/vuln-allowlist.json`. The script needs
> `jq`, and it runs its own test suite as a separate step beforehand.
### Fixed
- **A run dispatched to a remote worker could be executed twice.**
Recovery exists to adopt runs left behind by a dead process, and it had
no way to tell one of those from a run that had only just started
somewhere it cannot see. In production a run executing on a remote pool
worker was requeued to an in-cluster worker 350ms after the remote
worker began its first node, because that worker's first event had not
reached the server yet, and both then ran the pipeline through to the
end. The run simply succeeded twice: nothing reported a failure, and a
pipeline writing to a database or an API would have written everything
twice. Recovery now leaves a run alone until it is at least
`Engine.RecoveryMinRunAge` old, 30 seconds by default, which is one
full reclaim sweep interval plus an order of magnitude over the
observed first-event lag. It also gained `Engine.ExternalRunClaim`, an
optional hook answering whether a run is held by an executor this
process cannot see; it is nil here, so behaviour in this repository is
unchanged, and a hook that panics counts as a held claim, since
declining to recover is loud and reversible while assuming nobody owns
a live run is neither. A run with no start time is not covered by the
age guard, because nothing has claimed it yet. (#672) -- @hc12r
### Security
- **excelize moved from v2.10.0 to v2.11.0**, clearing GO-2026-6453
(unbounded memory allocation in the streaming row reader), which is
reachable from `ExcelLoader.Load`. The sibling advisory GO-2026-6452
(panic via a negative shared-string index) has no upstream fix: its
record lists every released version as affected. Its named vector does
not reproduce on v2.11.0, where the library returns `invalid shared
string index -1` as an ordinary error, so it is recorded as an audited
exception rather than an observed crash. As defence in depth against
the symbols this repository does not exercise, the Excel loader now
runs the excelize parse calls inside a recover, so a malformed
spreadsheet fails its own node instead of panicking the process that
hosts every other run. Errors excelize returns normally are untouched,
and a panic in Brokoli's own code still crashes loudly. (#673) -- @hc12r
### Changed
- **The vulnerability gate allows audited exceptions, and expires them.**
`scripts/check-vulns.sh` replaces the bare `govulncheck ./...` in the
Security workflow. It keeps the same verdict, failing on anything this
repository's own code calls, but an advisory with no upstream fix can
be listed in `security/vuln-allowlist.json` with a reason, a mitigation
and a reviewer. Two properties stop that becoming a suppression: an
allowlisted advisory that later gains a fixed version fails the build,
so an exception cannot outlive the thing that justified it, and
everything waived is printed with its reasoning on every run. Empty,
unparseable or config-less scanner output fails rather than reading as
clean, which is the vacuity that once let a security gate here pass
while inspecting nothing. govulncheck stays pinned to `@latest`, since
catching newly published advisories against pinned dependencies is the
whole point. (#673) -- @hc12r
## [0.11.30] - 2026-09-17
> **Behaviour change for webhook senders:** a webhook request with
> `Content-Type: application/json` now has its body read. A malformed
> JSON body answers 400 and starts no run, where it used to be ignored
> and the run started anyway. A body over 64 KiB answers 413, and
> `parameters` that fail validation against the pipeline's declared
> parameters answer 400. Only typed `parameters` are accepted, never the
> untyped `params` map. Requests with any other content type still have
> their body ignored and behave exactly as before.
>
> **Behaviour change for variable references containing a pipe:**
> `|` now introduces a filter inside `${...}`. A reference such as
> `${param.x|y}`, which used to resolve to an empty string because no
> param key is literally named `x|y`, now stays visibly unresolved in
> the output.
>
> **Interface change for out-of-tree `store.Store` implementations:**
> `store.PreviewStore` takes and returns a `store.NodePreview`, which
> carries `Truncated` and `TotalRows`. Every implementation outside this
> repository has to update in lockstep.
### Added
- **Date filters for timestamp variables.** A file or table can be named
after the day it covers without a second variable:
`${interval.start|date:YYYYMMDD}` renders `20240314`, and
`${interval.start|shift:-1d|date:YYYY-MM-DD}` names the day before.
Filters go inside the existing braces, separated by `|` and applied
left to right. `date:` takes the tokens `YYYY YY MM DD HH mm ss` and
copies every other byte, so `MM` is the month and `mm` the minute; a
literal may not contain the letters `Y M D H S`, which stops a typo
such as `yyyy-mm-dd` from rendering a real-looking wrong date.
`shift:` takes a signed count of `s m h d w` and chains for compound
offsets; there are no months or years, since neither is a fixed
duration. Filters apply to `interval.start`, `interval.end` and
`run.started_at`, and change the format, never the clock or zone. Any
filter the resolver cannot satisfy leaves the whole reference visible
rather than guessing a timestamp, and a filtered `${interval.*}` still
counts as slice-scoped for backfill. ADR-028 records the formatting
helpers as taken. (#664) -- @hc12r
- **The SQL a node ran is recorded with each attempt.** A templated query
used to be opaque after the fact; the statement is now stored with its
variables already substituted, as an `attempt.query` run event whose
payload field is `statement`, read through the existing
`GET /api/runs/{id}/events`. No schema change. What is recorded is SQL
someone wrote: a source's query (batch and streamed), a Migrate
node's source query, SQL a `sql_generate` node forwards into a sink,
and the query a pushed-down segment reads with. The composed
`INSERT ... SELECT` a pushed-down write executes is recorded too,
because it embeds the author's query and is the only record of what
that path ran. SQL the engine builds for itself is not: a sink's
generated `INSERT ... VALUES` records a
`-- [brokoli] generated write not recorded:` note in its place, bulk
loads (PostgreSQL COPY, MySQL LOAD DATA, ClickHouse batch append)
record a `-- [brokoli] no SQL statement:` note, and an overwrite's
`DELETE` or `TRUNCATE` is not recorded, so a sink never shows an
unexplained empty panel. A statement is recorded before it executes, so
the one that failed is kept. Values from `${secret.*}` and encrypted
`${var.*}` are masked as `[redacted]`; if such a value appears in the
statement and is shorter than 8 bytes, the whole statement is withheld
with a note saying why,
since masking something that short would shred the SQL without proving
the secret was gone. A connection string is never recorded. Each
statement is capped at 64 KiB with a visible
`-- [brokoli] statement truncated: N of M bytes recorded` marker. A
single attempt can record several statements. (#665, #669) -- @hc12r
- **Re-run a run from a chosen node.**
`POST /api/runs/{id}/resume` accepts an optional
`{"from_node": "<node id>"}`: that node and everything downstream of it
run again, and the work the earlier run already did upstream is reused.
It appends a new run linked back through `resumed_from_run_id` and a
`run.resumed_from_node` event, and leaves the original run exactly as
it was, rather than clearing and mutating it. Unlike a plain resume it
accepts a run that succeeded as well as one that failed or was
cancelled, since re-running one branch after fixing its query is the
ordinary reason to reach for it. A run still running, pending or
waiting is refused because its outcomes are still being written, a
blocked or skipped run is refused because it never executed a node, and
a node absent from the pipeline version the run executed is refused by
name. A body that cannot be read as JSON answers 400 instead of quietly
falling back to a plain resume. With no body the endpoint behaves
exactly as before, and it uses the existing `runs.resume` permission.
(#666) -- @hc12r
- **The run UI for all three.** Templatable fields (file paths, database
tables, API urls and bodies) show a live preview of `${...}` date
filters with the specific reason for any refusal, and warn that a
manual run has no interval. The run page has a SQL tab listing the
recorded statements grouped by node and attempt, with highlighting and
a copy button, showing exactly what the recorder stored. A selected
node offers "Re-run from here", whose confirmation names how many
downstream nodes will run again, gated on `runs.resume` and a settled
run status. The truncated-preview banner
now falls back to the node's produced row count when the stored total
is unknown, so it no longer contradicts the header above it.
(#668, #670) -- @hc12r
### Fixed
- **Webhooks ignored their request body.** The handler checked the token
and the rate limit but never read the body, so parameters sent with a
webhook were silently dropped. A JSON body is now read, bounded to
64 KiB, and its `parameters` are passed to the run as typed run
parameters validated against the pipeline's declarations, so a webhook
can only set what the pipeline declares, with its type checked. The
body is decoded only for `application/json`, so senders posting form
or plain-text payloads keep working. Unknown fields in the body are
deliberately tolerated, since webhook senders post payloads of their
own. (#533) -- @MrBeldum
- **Node previews declare when the stored sample is truncated**, with
`truncated` and `total_rows` on the API and in both stores, and the
run UI shows a truncation banner (same Callout pattern as the
dependency map). Existing previews whose sample sits exactly at the
50-row cap are backfilled as truncated with unknown total.
**Interface change:** exported `store.PreviewStore` / `store.NodePreview`
gains `Truncated` and `TotalRows` (`*int64`); every out-of-tree
`store.Store` implementor must update in lockstep. (#659) -- @MrBeldum
## [0.11.29] - 2026-09-16
> **Behaviour change for multi-workspace organizations:** pipelines are
> now scoped to a workspace, both in what a list shows and in what an id
> can reach. An organization whose people all share one workspace sees
> no difference. An organization that put pipelines in separate
> workspaces will find that the pipelines page, the dashboard and its
> counts show only the current workspace, and that a member of workspace
> A can no longer open, export, clone, run a plan for, roll back or
> delete a pipeline in workspace B. Reaching another workspace's
> pipeline by id answers 404 rather than 403, so an id you cannot see
> does not announce that it exists.
### Fixed
- **Every workspace showed every pipeline in the organization.** The
pipelines list, the pipeline summary views and the dashboard all
branched on organization first and then ignored the workspace
entirely, so creating a second workspace and switching to it showed
the same pipelines as the first, and the dashboard counted them all.
Only the community fallback had ever filtered by workspace. The three
paths now scope to the caller's current workspace. A session that has
not chosen a workspace resolves to one the caller belongs to rather
than filtering on `default` and showing an empty page, and a member of
no workspace sees nothing rather than everything. The organization
scoped reads stay organization wide where that is the right question:
a plan limit counts an organization's pipelines wherever they sit.
(#662) -- @hc12r
- **A pipeline could be reached by id from another workspace.** Ten
handlers that take a pipeline id (get, update, delete, export,
validate, plan, list versions, rollback, clone, validate nodes)
checked the caller's organization and nothing else, so inside one
organization a member of any workspace could read and change a
pipeline belonging to another. Export and clone mattered most: both
return a whole pipeline definition. Connections and variables already
resolved the caller's workspaces and were never affected. The check
resolves the full set of workspaces the caller belongs to, rather than
the single workspace a request is filed under, so somebody who belongs
to two workspaces keeps access to their own work in both.
(#661) -- @hc12r
- **Built-in pipeline templates never reached an existing database.**
Seeding ran only when `pipeline_templates` was empty, so a template
added to the built-in set after an install's first migrate could not
arrive, and no upgrade delivered it: a database seeded with two
starters still offered two while the product shipped four. Each
database now records which built-ins it has been offered, in a
`pipeline_templates.seeded_ids` setting, and each migrate inserts only
the ones missing from that record. A template an administrator deleted
stays deleted, and an edited one is still never overwritten. One
deliberate consequence: a built-in deleted before this change returns
once, because nothing recorded that it had ever been offered.
(#660) -- @hc12r
### Changed
- **Light is the default theme** for anyone who has not chosen one.
Somebody who picked dark keeps it, since that choice is stored. The
community shell applies light before first paint, along with the
`theme-color` meta, so there is no flash of the wrong theme on load.
The toggle and its persistence are unchanged. (#657) -- @hc12r
## [0.11.28] - 2026-09-15
> **Behaviour change for deployments that reach services over Tailscale
> or another carrier-grade NAT range:** the outbound policy now blocks
> `100.64.0.0/10` by default. Allow it with
> `BROKOLI_OUTBOUND_ALLOW_CIDRS=100.64.0.0/10` (or a narrower range).
> **Behaviour change for deployments using `vault://` or `k8s://`
> credential references:** they now resolve only what the operator
> lists, in `BROKOLI_SECRET_VAULT_ALLOW` (path prefixes) and
> `BROKOLI_SECRET_K8S_ALLOW` (`namespace/secret` names). An unlisted
> reference fails with a message naming the setting. See
> `docs/secret-references.md`.
### Changed
- **The outbound network policy blocks more internal destinations**, for
every pipeline connector, webhook and SFTP connection:
- `0.0.0.0/8` (only `0.0.0.0` itself was blocked), `100.64.0.0/10`,
`192.0.0.0/24` and local-use NAT64 `64:ff9b:1::/48`, blocked like
the private ranges and opened the same way;
- an IPv6 address that routes to an IPv4 one (NAT64 `64:ff9b::/96`,
6to4 `2002::/16`, IPv4-compatible `::a.b.c.d`) is judged as that
IPv4 address, so `64:ff9b::a9fe:a9fe` no longer reaches the cloud
metadata endpoint;
- cloud metadata endpoints (`169.254.169.254`, `169.254.170.2`,
`fd00:ec2::254`, `100.100.100.200`) stay blocked under
`BROKOLI_OUTBOUND_ALLOW_PRIVATE=true` and under allowlisted ranges
that merely contain them; only a CIDR naming the exact address opens
one. (#653) -- @hc12r
- **`vault://` and `k8s://` references are denied by default**, as
`env://` already was. Each read with the server's own credentials,
which reach far more than one connection should, including the
server's own secrets, and a connection is something a workspace editor
can create and point at a server of their choosing. Vault paths
containing `%`, `?` or a backslash are refused, so an encoded `..`
cannot walk out of an allowed prefix. New guide:
`docs/secret-references.md`. (#654) -- @hc12r
- **Building Brokoli from source needs Go 1.26** (`go.mod`'s `go`
directive moves from 1.25.0 to 1.26.0). `golang.org/x/crypto` v0.56.0,
which carries the SSH fixes below, declares `go 1.26.0`, as does every
later release. CI and the release builds already use Go 1.26.6.
(#650) -- @hc12r
- **Changing a connection's type, host or port requires entering its
password again**, for every connection type, and its extra settings
too unless they are a database's driver options (`sslmode` and the
like). Stored secrets cannot be read back, and they are no longer
carried over to a different server, where the next test or run would
have sent them. The update is refused with a message saying which to
enter. (#650) -- @hc12r
### Added
- **Deliver files to, and collect files from, SFTP servers** (ADR-040).
`source_file` and `sink_file` take a `conn_id` naming an `sftp`
connection; `path` is then a path on that server, relative to the
connection's base directory unless absolute. Every format the file
nodes handle works remotely, on the batch and the streamed path.
Guide: `docs/sftp-file-delivery.md`. (#650) -- @hc12r
- **The server's host key must match** the connection's `host_key`
(a `SHA256:` fingerprint, a public key line or a `known_hosts`
line). A mismatch or a missing key refuses to connect and names the
key the server presented. `insecure_skip_host_key_check` skips the
check and logs a warning on every connection.
- **Delivery is atomic.** A file is uploaded under a hidden temporary
name and renamed into place once complete, so a partner never picks
up a half-written file and a failed run leaves the previous file as
it was. Replacement uses `posix-rename@openssh.com`; a server without
it gets remove-then-rename, and the run log says so.
- Password or private key authentication (with passphrase), from the
connection's encrypted settings.
- SSH connections go through the same outbound network policy as HTTP
connectors: private and loopback servers need
`BROKOLI_OUTBOUND_ALLOW_CIDRS` or `BROKOLI_OUTBOUND_ALLOW_PRIVATE`.
- **A connection's base directory is a boundary:** with one set, no
path can leave it, absolute or relative; `..` is always refused.
- Temporary names are unguessable and created exclusively, so nothing
planted on a shared server can redirect a delivery.
- Only algorithms Go's SSH library does not classify as insecure are
offered; SHA-1 key exchange and DSA host keys are refused.
- A transfer fails after two minutes with no data moving (keepalives
keep a merely quiet connection up), and a node timeout or cancelled
run ends a transfer immediately.
- The editor's preview never delivers: a remote `sink_file` does not
connect during a dry run.
- A remote file is its own lineage asset, `sftp://<conn_id>/<path>`.
- Downloads are capped at the connection's `max_download_bytes`
(10 GiB by default), against both the declared size and what arrives.
- A `known_hosts` line marked `@revoked` or `@cert-authority` is
refused as a host key; a configured key line pins negotiation to
that key's type.
- The file node forms have a **Location** field for the connection.
- `golang.org/x/crypto` moves to v0.56.0, which fixes two denial of
service bugs in its SSH channel handling (GO-2026-6354,
GO-2026-6355) that this is the first code to reach.
### Fixed
- **JSON sources keep the file's column order**, the same on every run.
Columns came from ranging over a Go map, so a JSON file or API
response feeding a CSV sink wrote its columns in a different order
from one run to the next. They now follow each key's first appearance
in the document, across every page of a paginated API response.
(#655) -- @hc12r
- **A pipeline can only use connections from its own workspace.**
Connections were looked up by `conn_id` alone, and `conn_id` is unique
across every workspace, so a pipeline could name another workspace's
connection and run with its credentials. A run now resolves a
`conn_id` only within its pipeline's workspace, for every node type
that takes one; a connection from elsewhere is reported as not found,
in the same words as a missing one. Single-workspace installs are
unaffected: pipelines and connections both default to `default`.
(#652) -- @hc12r
- **Testing an `sftp` connection now signs in.** It opened a TCP
connection, read the SSH banner and reported success, so a wrong
password or an unknown server tested green. It now checks what a run
needs: the network policy, the host key, authentication, the SFTP
subsystem and the base directory. An unknown host key fails with the
key the server presented. (#650) -- @hc12r
- The connections page said `sftp` and `s3` connections were usable by
nodes when no node read either. `sftp` now is; `s3` is marked as not
usable, which is true. (#650) -- @hc12r
## [0.11.27] - 2026-09-15
> **Behaviour change:** the `sort` transform now orders numeric columns
> as numbers. It compared every value as text, so ascending `2, 10` came
> out `10, 2`. Pipelines that sort a numeric column will produce a
> different row order. Mixed columns order numbers, then text, then empty
> values; descending is the exact reverse.
### Changed
- **The embedded UI is now the React app** (#645). `build-ui.sh` builds
the community app into `web/dist`, the same embed target as before, so
the binary serves it with no change to how it is built or run. The
lineage page renders each column edge's `evidence` and `mapping_reason`,
and the reason a node's columns cannot be traced. The last Svelte
commit is tagged `ui-svelte-final`; restoring it is a checkout of that
tag. The Svelte end-to-end browser tests were dropped with it, and
React end-to-end tests have not been added yet. -- @hc12r
### Added
- `format` on each stored dataset in `GET /api/runs/{id}/provenance`
(`ndjson` or `arrow-ipc`), recorded with the digest because the two
only mean something together: the same rows stored in two formats have
two digests. (#643) -- @hc12r
### Fixed
- **`attested` column edges could not appear when a node's input was
stored as NDJSON.** A source's output takes its format from
`BROKOLI_STREAM_CODEC` or its first batch, while a node's output took
Arrow whenever the whole dataset allowed, so a node that changed
nothing stored the same rows in a different format and no digest could
match. A node with a single stored input now stores its output in that
input's format. Where the upstream is NDJSON, the node's output stays
NDJSON rather than upgrading to Arrow. (#643) -- @hc12r
- **The `sort` transform ordered numeric columns as text** (#642).
Ascending `2, 10` came out `10, 2`, and `9.5` sorted after `10.5`,
whether the numbers were typed or held as text, as a CSV column often
is. Numbers now compare exactly as numbers (integers beyond a float's
precision included), then text, then empty values; descending is the
exact reverse, which puts empty values first as Postgres does. The sort
is stable. (#644) -- @hc12r
## [0.11.26] - 2026-09-15
### Added
- **Column edges a run proved are marked `attested`** on
`GET /api/lineage`. An identity edge (the same column, from a node's
single input) is promoted from `declared` when the run its profile came
from stored that input and the node's output with the same digest: the
bytes did not change, so the column passed through untouched. Two
inputs, a missing digest on either side, unequal digests, or a renamed
or derived column all stay `declared`. The edge's `mapping_reason`
names the run. (#639) -- @hc12r
### Fixed
- **The provenance record left out the output digest for most nodes.**
`GET /api/runs/{id}/provenance` recorded a node's output before the
engine stored it, so a node that returns rows in memory (a quality
check, a transform, most processing nodes) showed no digest for an
output that was in fact stored, while the node consuming it showed
the digest for the same bytes. Sources were unaffected, which is why
the v0.11.25 tests did not see it. (#637) -- @hc12r
## [0.11.25] - 2026-09-15
> **Breaking:** `confidence` is gone from every entry in `column_edges` on
> `GET /api/lineage`, replaced by `evidence`. A client that reads it as a
> number will find it absent.
### Changed
- **Column lineage is derived from the pipeline definition instead of
guessed from column names** (ADR-039). Every node type now declares how
its output columns map to its inputs, or declares that it cannot say.
(00d376f) -- @hc12r
What changes in the graph:
- A **derived column now names what it derives from**. `total = price *
qty` previously produced no edge at all, because the name match found
nothing called `total` upstream.
- A **coincidental name match no longer produces an edge.** Two
unrelated datasets that both had `id` used to be joined by one.
- **No edges are drawn through a node that runs user code.** A code,
dataset_map, dataset_filter, task or dbt node reports that it cannot
trace columns, and why. The graph shows the reason.
- `mapping_reason` now states the derivation in the pipeline's own
terms -- `price * qty`, `renamed from qty`, `join key: id matched
against cust_id`. It was the constant string
`observed column name match`.
**Node and dataset lineage is unchanged and stays complete**, including
through the nodes above. Which node produced which dataset is always
knowable; which field produced which field is not, and the graph now
distinguishes the two.
- `evidence` replaces `confidence` on each column edge: `declared`,
`attested`, `parsed` or `inferred`. Filter to `declared` and `attested`
for facts only. `confidence` was the literal `0.7` on every edge ever
produced, which the UI rendered as "70% inferred". (00d376f) -- @hc12r
### Added
- `GET /api/runs/{id}/provenance` returns what each node in a run actually
consumed and produced: the upstream datasets it read, the dataset it
wrote, row counts, observed columns, and a `sha256` digest where the
dataset was stored. Written for every run and deleted with the run.
(#633) -- @hc12r
A digest is present only when the dataset went through the artifact
store. Small datasets pass between nodes in memory and are recorded
with row counts and columns but no digest; treat an absent digest as
"not checkable", never as "unchanged".
- `columns_opaque` and `opaque_reason` on a lineage node, so a node with
no column edges says why rather than looking unfinished. (00d376f)
-- @hc12r
- `models.AllNodeTypes` and `models.IsKnownNodeType`, one canonical list
for the gates that iterate node types. (00d376f) -- @hc12r
- `Engine.RunPipelineAsyncLocalOpts`, the in-process run path with
`RunOptions`. The in-process path returns the run ID before the run row
exists, so attribution written against that ID straight away races the
run and, since #632, is refused. Pass it as `TriggeredBy` instead; the
runner records it after creating the run. (#634) -- @hc12r
### Fixed
- **Run attribution outlived its run, permanently.** The `run_attribution`
table had no foreign key and its cleanup method was called from nowhere,
so every row written since the feature shipped survived the run's purge
and accumulated forever. It now cascades with the run, the same as every
other per-run table. Existing rows whose run is already gone are cleared
on the next boot. (#632) -- @hc12r
### Upgrading
1. **Replace `confidence` with `evidence`** wherever you read
`column_edges`. There is no numeric equivalent; the value was constant.
2. **Expect fewer column edges**, and none at all for a pipeline built on
code nodes. The edges that are gone were name matches with nothing
behind them. Check `columns_opaque` on a node to tell "no lineage" from
"lineage not traceable here".
3. **A new node type must declare its column lineage** or declare itself
opaque, or `TestEveryNodeTypeDeclaresItsColumnLineage` fails.
## [0.11.24] - 2026-09-14
> **Breaking:** `extensions.OpenLineageEmitter`'s three methods take two new
> arguments. Out-of-tree implementations must be updated.
>
> **Security:** five endpoints could return or modify another organization's
> data. Upgrade if you run more than one.
### Security
Each of these was reachable by any signed-in user. All are @hc12r.
- `GET /api/audit/{id}` returned **any** audit entry among the newest 500,
from any organization, and required no permission. Entries carry the
acting user and the before and after values of a change. Now
feature-gated, permission-checked and scoped to the caller's
organization. (#625)
- `POST /api/pipelines/{id}/dlq/{dlqId}/resolve` resolved the entry named
in the URL without checking it belonged to the pipeline, so any
dead-letter entry in the deployment could be marked resolved. (#628)
- `GET /api/pipelines/dependency-graph` returned every pipeline in the
deployment, across all organizations, when the caller's organization
could not be resolved. (#628)
- `GET /api/connections?page=N` returned the `encrypted://` credential
reference that the unpaged list masks. (#628)
- `POST /api/pipelines/import` honoured a `workspace_id` in the request
body, so a pipeline could be placed in a workspace the caller does not
belong to. (#628)
### Added
- **Lineage events reach an OpenLineage catalogue.** Setting
`BROKOLI_OPENLINEAGE_URL` previously produced no events at all. Runs now
emit `START` before execution and `COMPLETE` or `FAIL` after, each
carrying the datasets the run reads and writes. (#627) -- @hc12r
### Fixed
- `limit` on `GET /api/alerts` and `GET /api/dlq` is capped at 1000 and
defaults to 100. It previously reached the database unbounded. (#628)
-- @hc12r
### Documentation
- **ADR-039** proposes deriving column lineage from the IR instead of
matching column names, with explicit evidence levels. (#626) -- @hc12r
### Upgrading
1. **If you implement `extensions.OpenLineageEmitter`**, add
`inputs, outputs []LineageDataset` to all three methods.
2. **If you call `GET /api/audit/{id}`**, it now requires the `audit`
feature and returns only your organization's entries. Entries stored
without a tenant are no longer visible to anyone.
3. **If you pass `limit` above 1000** to the alert or dead-letter lists,
you now receive 1000 rows.
## [0.11.23] - 2026-09-14
### Changed
- The platform provider now receives the encryption key config alongside
the engine, so an enterprise build stores its own secrets under the same
key core uses rather than deriving one from the environment. No action
for OSS deployments. (#622) -- @hc12r
## [0.11.22] - 2026-09-14
### Added
- **Take ownership of a failure.** Alerts carry an assignee, an
acknowledgement and a resolution, via `POST /api/alerts/{id}/assign`,
`/acknowledge` and `/resolve`, with `?state=` and `?assignee=me` filters
on the list. Acknowledging an unowned incident also assigns it.
(#619) -- @hc12r
### Fixed
- **Alert read state was shared by the whole organization.** One person
marking an alert read marked it read for everyone, and the unread count
was the organization's rather than the reader's. Read state is now per
person. (#619) -- @hc12r
- Runs started by a dependency fan-out or a resume recorded nothing about
what started them, and the dashboard's recent-activity list omitted the
field entirely. (#620) -- @hc12r
### Upgrading
- `store.Store` gains seven alert methods; the five organization-wide ones
remain for deployments with no authentication.
- Read state moves to a new `alert_reads` table, created on boot.
**Alerts marked read before upgrading stay read for everyone**, because
the old column is still the fallback.
- `?state=` now rejects an unrecognised value instead of ignoring it, and
`?assignee=` accepts only `me`.
## [0.11.21] - 2026-09-14
### Added
- **Runs record what started them, and who.** Every run carries
`triggered_by` with a kind -- `user`, `schedule`, `webhook`,
`dependency`, `backfill`, `api_token` or `retry` -- plus the person's id
and name where one was involved. Previously only scheduled runs were
distinguishable. (#617) -- @hc12r
- `GET /api/runs?started_by=me` lists the caller's own recent runs across
pipelines. (#617) -- @hc12r
### Upgrading
- `store.Store` gains four attribution methods.
- Attribution lives in a new `run_attribution` table, created on boot.
- `triggered_by` is **absent** for runs created before upgrading. Absent
means "not recorded", not "nobody started it"; clients should render the
two differently.
## [0.11.20] - 2026-09-14
Every headline figure on the dashboard was wrong in some way. Two were
wrong in a direction that reads as reassurance.
### Fixed
All @hc12r.
- **Success rate counted runs that had not finished**, so a pipeline's
rate fell while its own runs were in flight and a cancelled run counted
as a failure. The denominator is now finished runs, and an empty window
reports `null` rather than `100`. (#606)
- **Every dashboard count stopped at 200 runs per pipeline.** A pipeline
on a one-minute schedule produces 1,440 runs a day and reported 200; the
seven-day chart showed a decline that was an artifact of the cap.
Counting now happens in the database. (#608)
- **Sub-second runs reported a duration of zero**, because timestamps were
truncated to whole seconds. The same truncation could make a pipeline's
"most recent run" name one that was not the latest. (#607)
- **The seven-day trend chart bucketed UTC dates into local-day keys.**
East of UTC the first hours of each day landed on the previous one; west
of UTC, late-evening runs vanished from the chart. (#609)
- **"Top failing pipelines" had no time window**, on a panel whose
neighbours are all 24-hour figures. (#610)
- **The run calendar answered differently on SQLite and Postgres** and
silently substituted 90 days for any `days` value it could not honour.
(#611)
### Added
- `PUT /api/auth/me/profile` sets display name, email and a new avatar
URL. An account created with only a username previously had no way to
fill these in. (#604) -- @hc12r
### Upgrading
1. **`success_rate_24h` can now be `null`** when nothing finished in the
window. Clients reading it as a number must handle null.
2. New response fields: `runs_24h_finished`, `top_failing_window_hours`,
`recent_runs_sample_size`, and `duration_ms` on each recent run.
Recent-run timestamps now carry fractional seconds.
3. `GET /api/runs/calendar` returns **every** day in the window, including
empty ones, and rejects a `days` value outside 1..365 instead of
substituting 90. Days are UTC.
4. `store.Store` gains three aggregate methods.
## [0.11.19] - 2026-09-14
Access-control hardening. Four of these let a pipeline author or a
workspace editor reach things they should not, and one of them is the
control plane's own encryption key.
### Fixed
- **Pipeline templating could read the server's environment** (#597) --
@hc12r. `NewVariableContext` copied the whole of `os.Environ()` into the
variable context and `${env.NAME}` returned any key from it, so an
author could read `BROKOLI_ENCRYPTION_KEY`, `BROKOLI_JWT_SECRET` or
`BROKOLI_DB_URL`. Lineage resolves variables without executing
anything, so a value could surface from a graph request alone.
`${env.*}` is now deny-by-default against
`BROKOLI_PIPELINE_ENV_ALLOW`, with four names refused even when listed.
A refused reference stays visible rather than resolving to empty.
- **A code node could read them directly** (#598) -- @hc12r.
`pkg/codeexec.WorkerEnv()` decides what environment author code sees,
and the pooled executor always used it. The one-shot and streamed
executors passed `os.Environ()` straight through, so whether a script
could read the deployment's secrets depended on which of two
interchangeable executors ran it -- the exposure moved with
`BROKOLI_CODE_POOL`, which nobody would think of as a security setting.
- **An `env://` reference resolved any server variable** (#599) --
@hc12r. A connection whose `password_ref` was `env://BROKOLI_JWT_SECRET`
handed back the signing secret, and a connection is something a
workspace editor can create. Now deny-by-default against
`BROKOLI_SECRET_ENV_ALLOW`, refused by name rather than resolving to
empty.
The three mechanisms above are gated separately, because "what may a
pipeline author interpolate" is not "what may hold a connection
credential". They share one floor: `pkg/secrets.AlwaysDeniedEnvName`.
- **Saving a secret variable without retyping it encrypted it twice**
(#600) -- @hc12r. The masked-value branch restored the stored
ciphertext, and the encrypt step could not tell that from a freshly
typed secret, so it encrypted it again. A pipeline decrypted once and
received ciphertext; nothing failed loudly, and the only recovery was
to type the secret in again.
- **Variables were global, not per workspace** (#601) -- @hc12r. The
table keyed on `(key)` alone, so one workspace's save overwrote
another's and a read returned whichever had written last. Variables
hold secrets. The key is now `(workspace_id, key)`, and `GetVariable`
and `DeleteVariable` take a workspace so the compiler catches a caller
that omits it.
- **The invite routes skipped authentication instead of making it
optional** (#602) -- @hc12r. The middleware skipped `/api/invites/`
before parsing any token, so a signed-in person reached the accept
handler with no claims and got 401 with a valid session. Anyone signing
in with GitHub, Google or Keycloak has no password to fall back on, so
for them accepting an invite was impossible.
### Changed
- **`AnnounceWorkerTrustAssumptions` is exported** (#596) -- @hc12r. Two
worker binaries, and only one could say what it holds. The API-only
shape -- the one the line exists to distinguish -- could not print
"holds no control-plane secrets".
### Upgrading
`${env.*}` and `env://` are now deny-by-default. A deployment relying on
either sets `BROKOLI_PIPELINE_ENV_ALLOW` or `BROKOLI_SECRET_ENV_ALLOW`
with the names it uses; a refused reference says which list it wanted.
Nothing in the documentation or examples used either, so most
deployments need no change.
## [0.11.18] - 2026-09-13
### Added
- **A worker that cannot reach the database can still be told what a
connection is** (#587, #588) -- @hc12r. Groundwork for the API-only
worker: a worker holding no database and no encryption key must still
connect to the data source its nodes name, so something has to resolve
that connection for it. `ConnectionResolver.ResolveConnectionByID`
resolves credential refs to plaintext where the encryption key already
legitimately lives, so only the result crosses the wire and not the key
that opens every other credential in the deployment.
It answers "what is this connection", not "may you have it"; the
authorisation is the caller's, and is deliberately not in core.
- **A shared-store worker declares what it holds at startup** (#587) --
@hc12r. `--mode worker` connects straight to the database, and where a
signing secret is also set it can mint any session including an
administrative one. That is a reasonable trade inside one trust
boundary and the wrong shape anywhere else. It was also the quieter of
the two worker shapes, indistinguishable at a glance from one that
holds nothing, so the trade was being made by operators who had never
been told they were making it. Each secret is now named with what it
grants; a worker holding none says so. Values are never logged.
### Fixed
- **A worker that cannot resolve connections said the connection was
missing** (#587) -- @hc12r. `ConnectionResolver` reported every
`GetConnection` failure as `conn_id "x" not found` and then returned
the config unchanged, so the node ran with its credentials unresolved
and failed somewhere less obvious. That is right for a genuinely
missing connection, where a node may carry inline fields as a fallback,
and wrong for a store that cannot look connections up at all: the
operator was told something about their data that was really a property
of their deployment.
New `store.ErrUnsupported` separates "cannot perform this" from
"performed it and found nothing", which were indistinguishable at a
call site and call for opposite responses.
## [0.11.17] - 2026-09-13
### Fixed
- **The SQL artifact cap measured the blob, not the row it writes**
(#585) -- @hc12r. `WriteArtifactRef` checks `ref.SizeBytes` against
`BROKOLI_SQL_ARTIFACT_MAX_BYTES` before converting. Since v0.11.15 the
bytes that reach the column are the NDJSON conversion of that blob, and
for a compact format those are not the same number: the blob is Arrow,
the column is NDJSON, and the gap widens with how compressible the data
is. Measured at 2.2x on a 20,000-row dataset, a 1.73 MB blob against a
3.75 MB column value.
So a ref comfortably under the cap could write a row well over it,
which is the one thing the cap exists to prevent. Its own error text
explains why the limit matters -- an artifact is a single column value,
written as one statement parameter -- and that reasoning applies to the
converted bytes rather than the blob.
Found by watching a fleet degrade rather than by reading the code: nine
streamed runs had written 238 MB of the artifacts table, more than 577
task runs had. The cheap pre-check stays, because rejecting an
oversized blob without reading it is still right; the converted bytes
are now checked too, against the same limit and with the same named
error.
## [0.11.16] - 2026-09-13
### Added
- **The streamed codec is named in the node log** (#583) -- @hc12r.
v0.11.15 gave the streaming write path two codecs, chosen per stream
from the first batch, and nothing said which one ran. That matters
because the fallback is deliberate: `arrowEncodableSchema` declines any
dataset it cannot represent exactly and NDJSON takes over, so a
pipeline can quietly lose the faster path and look identical from the
outside.
```
Streamed 300000 rows, 5 columns from stream.csv (11.3 MB)
by reference as arrow-ipc (never materialized)
```
The streamed `source_file`, `source_db` and `transform` lines all carry
it. The name is the same string the ref carries and the same one
`BROKOLI_STREAM_CODEC` accepts, so an operator who reads it can act on
it; a test asserts that correspondence rather than leaving it to
coincidence.
## [0.11.15] - 2026-09-13
### Fixed
- **A truncated dataset decoded as a shorter one, and reported success**
(#576) -- @hc12r. `DecodeNDJSON`, `ReadNDJSON` and `ReadColumnarBinary`
each stopped at the first row they could not parse and returned the
rows so far with a nil error. A truncated or corrupted blob therefore
read back as a smaller dataset that looked fine: a resumed node
restored half its own output and the run was marked successful.
Already known, and written into a test comment rather than fixed:
"DecodeNDJSON silently truncates at a bad line; the batch reader must
not". The streaming reader was made loud for exactly this reason; the
materialising readers never were. Each now names the row it stopped
at, which separates "byte one is not JSON" from "row 4,000,001 was cut
off mid-write".
- **An Arrow artifact read back empty** (#577) -- @hc12r. `spill()` has
chosen Arrow for any uniformly typed dataset since #521, and two
artifact stores could not read one back.
`SQLArtifactStore.WriteArtifactRef` copied the blob's bytes verbatim
into a TEXT column that `ReadArtifact` decodes as NDJSON, so binary
Arrow IPC went in and an empty dataset came out, with no error, for any
node whose output happened to be typed. That is the store every
distributed deployment uses. `LocalDiskArtifactStore.ReadArtifact`
refused any format but NDJSON outright, so such a run could not be
replayed from its own artifact.
### Added
- **The streaming write path can produce Arrow** (#578, #330) -- @hc12r.
`PutStream` and `streamTransformToRef` wrote NDJSON unconditionally,
so the largest outputs in the product, the ones that by construction
never fit in memory, were the only ones that never got the faster
codec. ADR-033 section 8 calls Arrow "the preferred optional dataset
transport"; on the write side nothing had implemented it.
The codec is inferred from the first batch and settled before a byte is
written, because the media type labelling the blob is an argument to
`Put` and by the time a byte exists it is too late to label it. A
stream whose first batch is not exactly representable is NDJSON from
the start, which is what it would have been anyway. A later batch that
disagrees fails by name, since the schema is already on the wire and
widening it would change the data.
Measured on #330's own pipeline, 1,000,000 rows of 81 MB CSV through
`source_file` to `sink_file`: **21.1s to 9.0s, 2.3x**. The NDJSON arm
reproduces that issue's own baseline (21.1s against its reported 22s),
which is what makes the two comparable. On the codec alone,
write-then-read is 3.58x with 3.18x fewer allocations and 1.97x smaller
on the wire.
`BROKOLI_STREAM_CODEC` selects it: `ndjson` restores the previous
behaviour byte for byte, `arrow` requires Arrow and refuses to fall
back, which is what lets a measurement prove which codec ran.
- **A column holding both integers and floats can be Arrow** (#578) --
@hc12r. `arrowEncodableSchema` refused that mixture, which is what
every decoded dataset looks like: `numericValue` returns `int64` for a
whole number and `float64` otherwise, deliberately, so both codecs
agree. A float column with some whole values therefore came back mixed
and re-encoding it fell back to NDJSON forever after, so a transform
reading Arrow could never write Arrow. Float64 holds both and the
decoder narrows the whole ones back; integers a widening would round
are refused by checking each one rather than by a magnitude bound.
## [0.11.14] - 2026-09-13
### Fixed
- **A staged task input was written where the worker could not read it**
(#572, #574) -- @hc12r. An input above the inline row cap is staged in
a blob store and fetched by the claiming worker through the control
plane. Staging went through `SQLArtifactStore.Blobs()`, which is local
disk, so in a distributed deployment the bytes landed on one pod and
the fetch looked on another: 404, after a capability had been minted
and a work order dispatched, from a component that had done nothing
wrong.
The store's own comment recorded the assumption that had stopped
holding, that spill "has no cross-pod visibility requirement". True of
spill, false of task staging, and both were using one store.
ADR-038 separates them. Spill keeps local disk. Anything another
process reads goes to a store declared cross-pod, and staging now
refuses by name, saying what to configure, rather than succeeding and
failing three hops later.
### Added
- **`S3Store` can be configured** (#574) -- @hc12r. It has been able to
hold blobs since #268 and to resolve them by digest since #561, but
`NewS3Store` had zero non-test callers, so object storage was code
nothing ran. `BROKOLI_BLOB_S3_BUCKET` and its credentials now build
it, and it becomes the cross-pod store above. A configured bucket that
turns out unusable logs loudly and still yields no store, so an
operator's mistake cannot read as an ordinary single-node install.
## [0.11.13] - 2026-09-13
A correctness release. Two of these are silent-wrong-answer defects and
one is a permission gate that was open by default.
### Security
- **The open-source permission fallback was allow-by-default** (#527,
#566) -- @hc12r. `requirePerm`'s fallback denied exactly one case, a
viewer attempting a write, and every other role fell through to the
handler. Any role the build does not recognise passed every gated
route: a typo, a value from a newer build, a role an operator
invented, and the empty string included. It is deny-by-default now
over the known role vocabulary. Enterprise deployments were never
affected, since Team's RBAC replaces this path entirely.
### Fixed
- **A spilled Arrow dataset reached a code node as NDJSON** (#567) --
@hc12r. `stageRefToNDJSONFile` copied a referenced dataset byte for
byte into a file named `.ndjson` and handed it to the wrapper, with no
check on the ref's format. Feeding Arrow to the NDJSON decoder returns
**zero rows**, reported as success rather than as an error.
Newly reachable in v0.11.11, which made Arrow accept integer columns:
ordinary tables with an id started spilling as Arrow. **If a code node
of yours received an empty dataset from an upstream node on v0.11.11
or v0.11.12, this is why.** The function had no test; it has three
now.
- **A worker could not fetch a staged task input, third layer** (#565)
-- @hc12r. `JWTAuth` refuses a request with no session, and a worker
holds none. It runs before the workspace gate v0.11.12 fixed, so that
fix was never reached. Three middlewares sit in front of the
data-plane blob routes and each was exempted separately, months apart,
each fix uncovering the next. A test now drives the assembled chain
and asserts the request arrives, rather than that one middleware
permits it.
- **A duplicate `pipeline_id` is 409, not 500** (#570) -- @hc12r. It
answered 500 with the driver's own constraint text, which a client can
act on in no way, and it is what an ordinary re-run of a deploy script
produces since the id is derived from the name.
### Changed
- **Arrow writes bounded record batches** (#569) -- @hc12r. The encoder
built the whole dataset into one record, so the reader's "one record
batch becomes one batch of rows" contract resolved to the entire
dataset: an Arrow reference materialised on read however carefully the
reader was written. Rows now flush every 1000, matching the NDJSON
batch size.
The reference streaming path had never been benchmarked, so the
numbers behind every "6-8x" claim in this file were measured on a
different path. Measured now, reading a spilled dataset back through
`OpenBatches`:
| rows | codec | ns/op | allocs/op |
| --- | --- | --- | --- |
| 10,000 | ndjson | 134,760,542 | 299,776 |
| 10,000 | arrow | 17,965,713 | 70,353 |
| 100,000 | ndjson | 1,029,709,527 | 2,999,956 |
| 100,000 | arrow | 164,124,398 | 705,303 |
7.5x and 6.3x faster, 4.3x fewer allocations, 1.48x smaller on the
wire.
- **A goroutine budget that scaled with nothing** (#568) -- @hc12r. The
parked-waits test asserted a fixed margin of 20 while parking 250
runs, and failed CI at 21 on a change that touched no engine code.
`runtime.NumGoroutine` is process-global, so neighbouring tests land
inside a fixed margin. The budget scales with the park count now.
## [0.11.12] - 2026-09-12
### Fixed
- **A worker could not fetch a staged task input on a multi-tenant
deployment** (#563) -- @hc12r. A worker holds no session and belongs
to no user, so `WorkspaceMiddleware` had no workspace to resolve for
it and answered 401 "authentication required" before `blobAuth` could
resolve its identity. Every reference-based task input, which is every
input above the 10,000-row inline cap, failed that way.
The rejecting branch only runs when a workspace resolver is installed,
so single-tenant deployments and every in-process test were
unaffected, which is why v0.11.11 shipped with it. The data-plane blob
routes are now exempt from workspace resolution, matched on segment
shape rather than a path prefix so the exemption cannot be reached by
burying "blobs" in another route.
This is the defect #528 set out to close, surviving one layer up.
Enterprise needs its own half of the same path; a deployment wants
both.
## [0.11.11] - 2026-09-12
### Security
- **A webhook no longer says which pipelines exist** (#542, #551) --
@team-humaki. An unauthenticated caller got 404, 403 or 401 depending
on whether the pipeline was missing, had no webhook configured, or had
one with a different token. Three answers to a question the caller was
not authorised to ask, so anyone could enumerate pipeline ids from
outside. All three are now the same 404 with the same body, and the
real reason goes to the server log with the id quoted, so a newline in
a URL cannot forge a log line.
- **An anonymous caller can no longer consume a pipeline's webhook
rate-limit slot** (#534, #541) -- @hc12r. The limiter was claimed
before the token was checked, on the raw id from the URL, so a
stranger who knew only a pipeline id could hold its slot: their own
request was rejected, but only after stamping the limiter, and the
legitimate sender then got 429. A pipeline could be silently stopped
from outside. The slot is claimed only after the caller has proved it
may trigger that pipeline, and entries for ids matching no pipeline
are no longer recorded at all.
- **Rate limiting counts a client, not a connection** (#539, #540) --
@hc12r. The bucket key was `RemoteAddr`, which includes the ephemeral
source port, so every new TCP connection got its own fresh budget.
A caller opening a connection per request was never limited, and the
visitor map grew per connection rather than per client.
- **The outbound analyzer catches `http.DefaultTransport`** (#266, #531)
-- @AghastyGD. The last standard-library path that could issue a
request without passing through `pkg/netguard`. Includes an audit of
every `AllowLoopback: true` site, both of which are deliberate, and a
regression test for the capability store's exception that ADR-022 had
been claiming existed.
- **A worker can authenticate to the blob endpoints, and only those**
(#528) -- @hc12r. An opaque device token is accepted on the data-plane
blob routes and nowhere else, so a remote worker can fetch a staged
input without holding a session.
### Fixed
- **Arrow decoding was corrupting 64-bit integers** (#560) -- @hc12r.
Every integer was converted through `float64` before being narrowed
back, so anything past 2^53 arrived altered and unflagged:
`9007199254740993` decoded as `...992`, and `MaxInt64` decoded as a
float64 rather than an integer at all. Both Arrow readers share that
code, so it reached task datasets from a harness and spilled node
outputs alike.
**If a task of yours handles 64-bit identifiers and either side chose
Arrow, its results may have been wrong.** This is the fourth
appearance of one defect class, after v0.11.6, v0.11.7 and v0.11.8;
the file's own header had claimed integers arrive "as whole float64s",
which stopped being true when the NDJSON path gained a decoder that
holds them.
- **`sink_file` with `format: sql` wrote JSON** (#545, #546) -- @hc12r.
It named the file `.sql` and put JSON in it. It now generates real
INSERT statements with a dialect and an optional CREATE TABLE.
- **Generated DDL declares BIGINT past int32** (#547, #548) -- @hc12r.
A column holding values beyond 2^31 was declared INTEGER, so the
script it produced would not load.
- **Deleting a node no longer breaks the editor** (#550) -- @hc12r.
Removing a node left the canvas bound to a stale index, so the wrong
node's properties were shown and the last node could not be deleted.
- **A database node keeps its connection** (#544) -- @hc12r. Selecting a
stored connection dispatched two updates in the same tick and the
second landed without `conn_id`, so the panel snapped back to the
manual URI field and the save dropped the connection.
- **Test Connection tests the connection the node uses** (#557) --
@hc12r. It only ever read `config.uri`, so the one configuration
carrying working credentials was the one it refused to test. A stored
connection is now tested by id on the server, where the credentials
are.
### Added
- **Arrow engages on tables with an integer column** (#521, #560) --
@hc12r. The encoder accepted only string, float64 and bool, so a
single integer column sent the whole dataset down the NDJSON path.
Essentially every real table has an integer id, and the fallback is
silent, so the measured 6.8x decode and 4.5x encode applied to
datasets that mostly do not exist. Integers map to Arrow Int64 now.
The conservative rule is unchanged: a column mixing an integer with a
float still declines, as do nested values, slices and all-null
columns.
- **Draft pipelines** (#107, #554) -- @hc12r. Fail-closed persistence
meant a pipeline had to be executable to be saved, so there was no way
to start one from scratch and finish it later. A draft skips
executable validation on save and in exchange cannot run; publishing
applies the full validation, and a published pipeline cannot return to
draft (disable it instead).
- **Schedules in plain language** (#552, #553) -- @hc12r. "every weekday
at 9am" compiles to `0 9 * * 1-5`, with the cron and the next few
occurrences shown before saving. A closed grammar: anything it does
not understand is refused with a suggestion rather than guessed at,
and cron is still accepted directly.
- **`S3Store` resolves a blob by digest** (#561) -- @hc12r. The blob
endpoint turns a capability's namespace and digest into a reference
through `DigestResolver`, which only the local-disk store implemented.
A deployment configured for object storage therefore had a working
store that could not serve a single capability read.
- **An editor toolbar that scales** (#555, #556) -- @hc12r. Twenty-one
controls on one row broke twice in two days. The schedule moved into a
popover, the long tail into an overflow menu, Run and Publish became
one contextual action, and every editor action is reachable from the
command palette the app already had.
- **A local Postgres fixture** (#549) -- @hc12r. `scripts/devdata`
brings up Postgres with 100k rows chosen to be awkward: embedded
newlines and quotes, unicode, nulls, values past 2^31 and 2^53.
### Changed
- **The Security workflow runs on pushes to main and on demand** (#536,
#543) -- @hc12r. It only ran on pull requests, so main itself was
never scanned.
- **Two tests stopped depending on their surroundings** (#558, #559) --
@hc12r. One reached for `select.first()` on the page, which stopped
meaning the connection picker once the toolbar gained a timezone
picker, and took main red when both landed. The other required a dial
to a private address to fail, which is a fact about the network the
suite runs on rather than about the code.
## [0.11.10] - 2026-09-11
A security release. Upgrade if you run Brokoli with authentication
enabled and expose it to any network you do not fully trust. Every
released version before this one is affected.
### Security
- **Authentication could be skipped on most POST routes by putting
"/webhook" in the path** (GHSA-jxjf-p7pv-22m9) -- @hc12r. The webhook
trigger routes carry their own token rather than a session, so both
auth middlewares exempted them. The exemption asked whether the
request path merely *contained* `/webhook`:
```go
strings.Contains(r.URL.Path, "/webhook") && r.Method == "POST"
```
`r.URL.Path` is the decoded path, while the router dispatches on
`RawPath` when it is set. Two shapes satisfied the test while being
routed somewhere else entirely: a path parameter whose value is the
literal word `webhook` (`POST /api/pipelines/webhook/backfill`), and a
percent-encoded separator that grows a `/webhook` segment the router
never sees (`POST /api/pipelines/p123%2Fwebhook/backfill`).
Getting past the middlewares mattered because the open-source
permission gate then allowed the request through as well. It read the
resulting absence of claims as "open mode, no users created yet",
which is also exactly what an unauthenticated request looks like.
`HasPermission` had already been corrected for that same ambiguity and
consults JWTAuth's explicit marker instead; its sibling in
`RegisterRoutes` had not. The two together let a request reach a
handler with no authentication, no permission check, and no
organisation scoping.
Be accurate about the practical reach: no route in the shipped table
could be driven to read or change data, because nearly every affected
handler resolves its target using the same path parameter that carried
the bypass, and that lookup then fails. The exception is close enough
to matter, though. `dlqResolveHandler` uses its pipeline parameter
only to confirm the pipeline exists and then resolves a dead-letter
entry named by a second parameter, so a pipeline whose id happened to
be `webhook` would have made that reachable unauthenticated. The
containment was a property of which routes exist, not of any control.
Both halves are fixed. `isWebhookTriggerRequest` matches on method and
path shape, alongside the capabilities and observability helpers that
already worked that way, and reads the same string the router will
dispatch on so the two cannot disagree. The permission fallback moved
to `fallbackPermissionMiddleware`, where it now requires an explicit
open-mode marker and can be driven directly by a test rather than only
through the whole router.
Both fixes were confirmed by watching the new tests fail against the
old behaviour, not merely pass against the new one.
## [0.11.9] - 2026-09-10
A welcome to @harlanljones, whose first contribution is in this release.
They picked up a good first issue, found that its central claim was
wrong, said so, and fixed the code correctly instead of following the
instruction. That is exactly the instinct we want, and the issue is
better for it.
### Fixed
- **Remote task dispatch now actually works.** (#519, #522, #525) --
@hc12r. It never did in `--mode worker`, which is the only mode a
distributed deployment runs. Three separate defects, each hiding the
next, all found by standing up a fleet with one worker pool per
runtime and watching a JVM task fail on a pod with no JDK:
- **A task work order was rejected on arrival** (#519). Its identity
was validated as requiring a non-empty `InstanceKey`, but a task node
has no expansion semantics: it dispatches one job whose identity is
the whole-node attempt, `(RunID, NodeID, "", Attempt)`. That empty
key is the same convention the execution-attempt store and the blob
endpoint already use. The worker rejected and acked the job, the
dispatcher waited out its full timeout, and the run failed with
"timed out waiting for a worker", a message naming the symptom and
hiding the cause.
- **A worker's runner had no capability issuer** (#522). Two functions
build a Runner from an Engine, and the issuer was added to one of
them. The worker path silently lacked it, so a task input over the
inline row cap was refused rather than staged by reference. The
engine had none either, because the wiring sat on the API-only code
path.
- **A worker presented no identity to the blob endpoint** (#525).
The endpoint requires both a caller identity and a capability, and
the client sent only the capability, so every reference-based fetch
was rejected with 401 before the capability was read. The credential
is an opaque token, never a JWT.
None of these were reachable in process, so unit tests and CI stayed
green throughout while the feature could not run where it was needed.
- **A fixable HIGH advisory in a transitive dependency** (#524) --
@hc12r. `google.golang.org/grpc` moves to 1.83.2 for CVE-2026-84445,
reached through OpenTelemetry's OTLP exporter. Real exposure here is
essentially nil, since the advisory concerns gRPC-Go xDS servers and
Brokoli runs none, but the image scan gate fails on fixable findings
and the bump is a patch release.
### Added
- **A task's data plane reaches remote workers.** (#509, #510, #517) --
@hc12r. A capability-authenticated blob endpoint (ADR-033 section 6),
input staged by reference when a dataset is too large to ride inside a
work order, and artifact and collection outputs written from a remote
worker. A write grant names the namespace rather than the object,
because the store is content addressed and a task's output has no
identity until it exists. That relaxation is refused on a read, refused
alongside a pinned checksum, and confined to the route that names no
object.
### Changed
- **NDJSON helpers no longer claim to be Arrow.** (#516) --
@harlanljones. `WriteArrowJSON` and friends wrote NDJSON, and the
`TransferArrow` constant named a mode that never existed. Harmless
while nothing used Arrow, actively misleading once real Arrow IPC
landed beside them. The issue wrongly suggested the constants were
unused and could be deleted; they drive the file-mode switch in
`codenode.go`, and this change correctly renames rather than removes
them.
- **CONTRIBUTING explains why to contribute and how to build.** (#515) --
@hc12r. There were no setup instructions at all, and the build order
trap was undocumented: the UI must be built before the binary, or the
embedded assets are stale.
- **Build artifacts are no longer tracked.** (#523) -- @hc12r. A 12MB
`brokolisql` binary was committed. It is an output of this repo and
rebuilds from source.
## [0.11.8] - 2026-09-08
### Fixed
- **64-bit integers now survive a task end to end** (#492, #496, #504) --
@hc12r. Three separate places were losing them, and the last one was
the engine itself:
- **Node tasks now carry them** instead of refusing them. v0.11.7 made
the harness refuse a value JavaScript cannot represent, which beat
corrupting it but left Node unable to process ordinary bigint keys.
It now handles the literal before `JSON.parse` can destroy it and
hands the task a `BigInt`, with the reverse on output. Works on the
supported Node range; no runtime floor change.
- **The engine was corrupting them regardless of adapter.**
`readTaskResult` decoded the result manifest with a plain
`json.Unmarshal`, so every number became a `float64` -- which cannot
hold an integer above 2^53. Python, Node and JVM all emitted
`9007199254740993` correctly and the pipeline received `...992`.
This is the same defect as v0.11.6's dataset-decode fix, surviving
on a second path.
**If a task of yours handles 64-bit identifiers, its results may have
been wrong.** Re-run anything whose output depended on one.
- **The code-worker socket path could collide** (#449, #505) -- @hc12r.
It was built from `time.Now().UnixNano()`, which looks unique and is
not: two workers spawned in the same clock tick got the same path and
the second died with `bind: address already in use`. Each worker now
gets a kernel-guaranteed unique directory, so the outcome no longer
depends on clock granularity.
### Added
- **`brokoli bundle jvm`** (#507) -- @hc12r. Packages compiled JVM
bytecode into a task bundle a server can run:
```
brokoli bundle jvm --classes build/classes/java/main --entrypoint com.example.tasks.Transforms#dailyRollup --name daily-rollup
```
Until now the runtime could run a JVM task bundle and nothing could
produce one. Because the adapter loads **bytecode**, one command serves
Java, Kotlin, Scala and Groovy alike, and adds no dependency to your
project. A class compiled from a language other than Java needs that
language's runtime at load time, so pass its jar with `--jar` and the
bundle carries it.
- **A second JVM language is verified, not assumed** (#504) -- @hc12r. A
Groovy task runs through the adapter unchanged, which is what makes
"every JVM language" a tested claim. A missing language runtime now
reports a `contract_violation` naming the jar to add, rather than a
`platform` failure implying the server is broken.
### Changed
- **The engine package has its own CI job** (#504) -- @hc12r. It was
sharing one 20-minute budget with a UI build, a Playwright install and
a second full pass for coverage, and its per-package timeout kept
firing -- naming whichever innocent test held the baton. It does not
import the web package, so its own job skips the UI pipeline entirely.
No test was shortened or skipped to achieve this.
- **ADR-037 proposes editor integration** (#506) -- @hc12r, scoped around
moving deploy-time preflight into the editor rather than around
visualisation.
## [0.11.7] - 2026-09-08
### Added
- **A third task runtime class: the JVM** (ADR-036, #498, #499, #500) --
@hc12r. A `task-bundle/v2` declaring a `jvm` payload runs through the
same engine, dispatch path and result contract a python or node task
uses, with only the adapter differing. Java tasks are ordinary public
static methods; the fully-qualified class and method travel in the
manifest's existing `entrypoint.module`/`.symbol`, so no new bundle
fields were needed, and the classpath is derived from the bundle's own
file list rather than declared a second time -- which means it is
covered by the bundle digest for free.
**Requires a JDK, not only a JRE**, and Java 17 or newer. The adapter
compiles its harness once per (harness version, JDK) into a host cache
and reuses the classes, because measurement rejected the alternative:
launching Java source directly costs 1.07-1.16s per attempt against
0.07s precompiled, and task nodes get a fresh child per attempt. A
host without `javac`, or below the version floor, is refused **by name
at validation** rather than discovered mid-run.
- **One conformance suite across all three adapters** (ADR-033 section
19, #502) -- @hc12r. Python, Node and JVM are now held to the same
seven behaviours rather than to three unrelated test suites: scalar
round-trip, 64-bit integer fidelity, dataset-by-reference, artifact
media type, collection item keys, declared-kind-is-authoritative, and
`user_code` on raise. Where an adapter legitimately differs the
difference is declared with its reason, so "this runtime cannot do
that" is asserted rather than left as silence.
### Fixed
- **A task's artifact and collection outputs work on a dispatched task
node** (#493) -- @hc12r. They worked locally and failed with "this
server has no artifact blob store to hold it" when the node was
dispatched to an in-process instance worker -- blaming configuration
for absent plumbing, since the worker was holding a perfectly usable
blob store at the call site and passing nothing. A claimant genuinely
without one still gets the same honest refusal.
- **The Node task harness no longer silently corrupts 64-bit integers**
(#492, #494) -- @hc12r. `JSON.parse` turned `9007199254740993` into
`...992` on every input row of every Node task -- the same class of
defect #479 fixed on the Go side, which JavaScript cannot fix the same
way because it has no wider number type. The harness now detects an
out-of-range integer **from the raw text, before parsing**, and
refuses by name.
**Behaviour change**: a Node task that was receiving a silently wrong
id now fails instead. That is the intent -- an id quietly off by one
is worse than a refused run -- but a pipeline that appeared to work may
now stop. Python and JVM tasks handle such values exactly and are
unaffected.
### Changed
- **Python bytecode is no longer tracked** (#495) -- @hc12r. Five `.pyc`
files were committed, including two pytest caches for two different
pytest versions.
- **The Python harness's 64-bit fidelity is now asserted, not assumed**
(#497) -- @hc12r, and a scheduler test that failed for two minutes a
day was fixed (#501): at 04:59 UTC both `0 5 * * *` and `*/2 * * * *`
next fire at 05:00:00, so a "did the tighter schedule take effect"
assertion written as a comparison was false while the code was right.
## [0.11.6] - 2026-09-08
### Fixed
- **A run's declared parameters now actually reach the task that
declared them** (#487, #490) -- @hc12r. ADR-032 section 7 reads as
though declaring a parameter causes its value to arrive. It did not.
A submitted value was validated against its declaration, recorded on
the run row, and then read exactly once -- to persist it. Node
execution never consulted it, so a run triggered with `threshold=0.9`
validated, recorded 0.9, and executed with the task's own default:
**a green run and a wrong answer**, with nothing anywhere reporting a
problem.
A task now sees its own `parameters` binding alongside the existing
`params`. Deliberately separate rather than merged, because
`params` values are always strings and declared ones keep their
declared type -- a merge would have delivered a declared `0.9` as
`"0.9"`, handing the task a *wrong type* rather than a missing value.
Remote dispatch carries them too, additively, so an older claimant is
unaffected. The new **`task-parameters-v1`** capability is what lets
an SDK refuse a server that would accept the declaration and then
ignore the values; `task-interface-v1` only ever meant the server
accepts the IR.
- **The license-compliance gate was inspecting zero dependencies**
(#484) -- @hc12r. Both CI and `preflight.sh` ran `go-licenses report
./... || true` and grepped the output for GPL/AGPL/SSPL/EUPL.
`go-licenses` cannot load a Go 1.24+ standard library: it exits
fatally having printed nothing, `|| true` swallowed the status, and
the grep then searched an empty file. The gate had reported "no
forbidden licenses found" for its entire existence while checking
**none of the 88 dependencies**. Replaced with a tested classifier
that treats an empty scan and an unclassifiable license as failures,
and identifies a license by the one it names *first* -- MPL-2.0 names
GPL/LGPL/AGPL in its own body, and the BSD family never names itself.
- **A gosec verdict from a scan that inspected nothing is now refused**
(#488) -- @hc12r. The baseline comparison treated zero findings as the
best possible outcome, so a run that scanned no code would have passed
while suggesting the baseline drop to 0. The file count and the
absence of Go build errors are now asserted before the verdict is
trusted.
- **Task code no longer inherits the engine's environment** (#481) --
@hc12r.
- **Port-aware edges are refused rather than silently rewired** (#482)
-- @hc12r. `from_port`/`to_port` were dropped and the edge routed as
if unported, producing a wrong pipeline rather than a refused one.
- **64-bit integers survive task dataset decoding** (#479) -- @hc12r.
`9007199254740993` decoded to `...992`.
### Added
- **Arrow IPC datasets end to end** (#475, #477, #478) -- @hc12r.
Measured 6-8x faster than NDJSON on the read path. Spilled datasets
are written as Arrow only when the round trip is exact; a mixed-type
column, an all-null column, a nested value or a slice each make it
decline, since an optimization must never become a semantic change.
- **`pkg/datacap`: data-plane capabilities** (#485) -- @hc12r. ADR-033
section 6's opaque, control-plane-issued grants, bound to tenant, run,
attempt, fencing generation, direction, object and expiry. The
foundation for remote task input and output; nothing calls it yet.
- **A task cannot reach the engine's file descriptors** (#483) --
@hc12r. ADR-033 gate 6's remaining third, after secrets and state.
### Changed
- **A finished remote instance is noticed in ~25ms, not up to a second**
(#473) -- @hc12r.
- **CI no longer re-downloads its dependencies every run** (#476) --
@hc12r.
- **SQLite driver: modernc.org/sqlite 1.37.1 -> 1.53.0** (#474) --
@hc12r.
- **ADR-012 and ADR-017 define `CommitAttemptResult`** (#489) --
@hc12r. The multi-port publication operation ADR-033 section 7
required before the remote task data plane can be built: one fenced
compare-and-swap publishing every output port's manifest and settling
the attempt together, all ports or none.
- **ADR-036 proposes the JVM runtime class and SDK** (#486) -- @hc12r.
Measurement drove the design: JEP 330 source launch costs 1.07-1.16s
per attempt against 0.07s precompiled, so the adapter compiles once
and caches. Requires a JDK rather than only a JRE, which is recorded
as the decision's real cost.
- **ADR-033's acceptance gates split by distribution** (#480) -- @hc12r.
## [0.11.5] - 2026-09-07
### Added
- **`task` nodes can emit artifacts and collections** (#470, #471) --
@hc12r. Completes the four value kinds ADR-032 section 6 defines: as
of v0.11.4 a task could emit a scalar or a dataset, and the other two
were refused by name. An **artifact** output keeps its bytes opaque --
they move into the content-addressed blob store and the node's output
becomes the reference, in the same four-column
uri/media_type/size_bytes/checksum row a `source_api` artifact
response already produces, so downstream has one representation to
understand rather than two. Media type comes from the result
manifest and is checked against the port's declared `media_types`.
With no blob store configured an artifact output is refused by name
rather than inlined, since inlining would defeat the kind entirely
and can be enormous.
A **collection** output becomes one row per separately addressable
item, each row in the shape that item's own kind already produces --
so a collection of artifacts is N reference rows, a collection of
scalars is N value rows, and downstream reads collections with the
machinery it already has. Each item is read by the same reader its
kind uses standalone, so artifact items get identical safe-open and
integrity verification. ADR-032 section 6's rule that duplicate item
keys are invalid (even when duplicate values are allowed) is
enforced, naming both colliding positions.
### Changed
- **`task-result/v1` gained a recursive `items` field** (#471) --
@hc12r. Required for the `collection` kind and only that kind. Each
entry is an output port result in its own right, carrying whatever
its own kind requires, so a collection of artifacts holds per-item
path and checksum. Additive: no existing producer emits collections,
and the dataset, artifact and scalar rules are unchanged.
- **Reading a task-written file is now one code path** (#470) --
@hc12r. `openStagedOutput` holds ADR-033 section 7 rule 6 in a single
place -- beneath/no-follow resolution, regular files only, size
checked against both the server's cap and the manifest's own claim --
and every output kind that reads a staged file goes through it.
## [0.11.4] - 2026-09-07
### Added
- **`task` nodes join the data plane: datasets in and out** (ADR-033
phases 4b/5a/5b, #465, #467) -- @hc12r. A task could previously return
exactly one inline scalar, so nothing downstream could consume real
data from one. Tasks now produce and consume NDJSON datasets (ADR-033
section 8's baseline interoperable format; Arrow IPC remains the named
optional upgrade), which also closes ADR-033's cross-language
acceptance gate: a Python task feeding a Node task and a Node task
feeding a Python task both run through the same backend. Which kind a
port carries is decided by the node's declared interface, never
guessed from a return value's runtime shape. A task declaring an input
port is a consumer and may receive edges; one declaring nothing stays
a source, so existing single-task pipelines are unaffected.
Reading a task-written dataset follows ADR-033 section 7 rule 6: the
file is opened beneath a trusted staging descriptor with no-follow
semantics, and stat, hash and decode all run through that one handle,
so a symlink or replacement race cannot swap the bytes between the
check and the read. Declared size and checksum are verified against
what was actually read; traversal, symlink escape, non-regular files,
oversize, unknown codecs and non-object rows are each refused by name.
- **Runtime-aware placement for task jobs** (#465, with
`brokoli-enterprise` #187) -- @hc12r. A task job now carries a
per-runtime capability tag alongside the protocol tag, so a node
bundle stops being dispatched to a worker that has no node runtime. A
bundle offering a choice of runtimes carries only the protocol tag,
since capability matching is AND-superset and naming either class
would exclude a worker that has the other. New
`taskbundlev2.ReadManifest` lets the dispatcher see what a bundle
offers without paying for a full extraction.
- **Contract validation on both task boundaries** (ADR-032 section 10,
#468) -- @hc12r. Rows crossing into a task are validated before it
starts, and rows coming out before they are committed downstream --
previously only scalar outputs were checked. An input violation
reports separately from an output one, because an input violation is
the upstream's fault and pointing at the consuming task would send an
author to the wrong file. Large datasets are validated by
deterministic sampling chosen by row index, so retries check the same
rows, and a failure states how many rows were actually checked and
under which mode rather than implying the whole dataset was verified.
## [0.11.3] - 2026-09-07
### Added
- **The Node reference adapter: `task` nodes are no longer Python-only**
(ADR-033 phase 4a, #463) -- @hc12r. `pkg/taskharness/nodeharness` is
the second of ADR-033's two required reference adapters, making the
runtime protocol's language-neutrality demonstrated rather than
asserted. A bundle declaring a `node` payload runs through a real
`node` subprocess -- local and remote dispatch alike -- under the same
protocol, failure taxonomy, resource ceilings and output-contract
validation a Python task gets. Three differences are forced by the
language, not chosen: keyword arguments arrive as one object argument,
the result is awaited (so `async` tasks work), and the memory ceiling
comes from V8's `--max-old-space-size` rather than a self-applied
`RLIMIT_AS` -- the same mechanism a TypeScript code node already uses.
`taskbundlev2.SelectPayload` replaces the Python-only selector, taking
the first payload in manifest order whose runtime class the server
supports, so a bundle author controls preference between equivalent
payloads and selection stays deterministic. Placement is not yet
runtime-aware: a node bundle can still be dispatched to a worker
without a node runtime, where it fails with a clear error rather than
anything silent (per-runtime capability tags are tracked as their own
follow-up).
- **Task outputs are validated against their declared contract**
(ADR-032 section 10, #462) -- @hc12r. A task that declares an output
type and returns something else now fails its run instead of flowing
the wrong value downstream. `pkg/taskinterface.ValidationFailure` gives
a runtime contract violation real structure -- direction, port name,
contract path, expected descriptor, checked row count, validation mode
-- rather than a flat string. Failure reports never carry the offending
value's content, only its kind and shape, and suppress even that for a
declaration marked `sensitive`. A node carrying no declared interface
is not second-guessed: absence stays honest and the check is skipped.
## [0.11.2] - 2026-09-07
### Fixed
- **Three dependency CVEs** (#460) -- @hc12r. `golang.org/x/crypto`
v0.54.0 to 0.55.0 (CVE-2026-56854, CRITICAL: ssh auth bypass via
unenforced source-address restrictions); `golang.org/x/mod` v0.37.0 to
0.40.0 (CVE-2026-56864/56865, HIGH: a malicious GOSUMDB could serve
arbitrary module content, and a transparency-log tile verification
bypass) -- this one came in via `pkg/taskruntime`'s semver dependency,
added in v0.11.1; `google.golang.org/grpc` v1.83.0 to 1.83.1
(CVE-2026-84304, HIGH). Caught by `brokoli-enterprise`'s Trivy scan,
which runs on every push there and flagged all three as soon as v0.11.1
landed in its dependency tree -- this repo's own `preflight.sh` doesn't
run Trivy locally and `govulncheck`'s database didn't have these
cataloged yet.
## [0.11.1] - 2026-09-06
### Added
- **Task bundles: a versioned, content-addressed project archive**
(ADR-031, #432, #433) -- @hc12r. `task()` compiles to a `task_bundle`
archive instead of a bare script -- multi-file projects with real
imports, executed through the same warm code-node pool (ADR-029) under
the same resource ceilings. `POST`/`GET /api/task-bundles/{digest}`
serve it content-addressed and org-scoped; a byte-identical re-upload
is a no-op, a colliding digest is a 409, never a silent overwrite.
- **Portable task interfaces** (ADR-032 steps 1-2, #438, #440, #441) --
@hc12r. A task's row/parameter shape is now a typed, canonical schema
(BPTD: named ports, parameter declarations) attached to
`models.Node.Interface`/`models.Pipeline.Parameters`, not an implicit
"whatever columns show up" contract -- IR 2.2, with 13 of 19 built-in
node types declaring their interface out of the box.
- **Graph assignability and typed run parameters** (ADR-032 step 4,
#442-#446) -- @hc12r. Twelve directional compatibility rules
(`pkg/taskinterface`) now gate every edge at validate time, and a
pipeline's typed parameters actually resolve (defaults applied, unknown/
missing-required rejected) at trigger time -- `TriggerRun` returns a
clean 400 on a bad submission instead of a 500. Fixed a real bug found
along the way: `Pipeline.Parameters` was decoded but never persisted by
either store backend, silently dropped on every read.
- **SDK inference for task interfaces** (ADR-032 step 3, #447, #450) --
@hc12r. Python `@task` infers its row/parameter schema from type
annotations, `TypedDict`, and dataclasses; TypeScript gets explicit
schema/parameter builders (no erased-generic reflection). Six
cross-SDK differential fixtures prove equivalent declarations in both
languages normalize to the identical interface.
- **The polyglot task runtime's schema and worker-capability foundation**
(ADR-033 phases 0-1, #452, #453) -- @hc12r. Canonical schemas and
fixtures for `task-bundle/v2`, `task-runtime/v1` (a duplex JSON Lines
protocol between a trusted worker and a task harness), and
`task-result/v1`; a structured worker-capability/placement model. A
`task` node is a recognized IR type from here on, but deliberately
refused at deploy time until the phases below give it somewhere to run.
- **Task nodes execute for real, locally** (ADR-032/033, #454, #455) --
@hc12r. A `task` IR node -- the polyglot runtime this year's ADR-032/033
proposals described -- now actually runs: it fetches a versioned,
content-addressed `task-bundle/v2` archive, extracts and verifies it
(per-file digests, not just presence), and executes it through a new
duplex-protocol harness (`brokoli.task-runtime/v1`, JSON Lines
handshake with structured logs/progress/cancellation) and a Python
reference adapter, instead of a bare script. New
`POST`/`GET /api/task-bundles-v2/{digest}` upload/fetch endpoints
mirror the existing task-bundle/1 contract in a separate,
content-address namespace. Trusted isolation profile, Python payloads
only, for now -- the same staged scope this project's other polyglot
work (TypeScript code nodes, ADR-030) shipped with.
- **Task nodes dispatch to a remote worker fleet** (ADR-033, #456) --
@hc12r. A task node whose deployment has an instance job queue and
execution-attempt store configured (the same condition dynamic
code-node expansion already uses) now runs on a remote worker instead
of in the pipeline engine's own process -- one instance job per node,
reusing the existing remote-dispatch transport dynamic expansion and
source_api pagination already share.
- **Task node retries can no longer silently swap execution
environments** (ADR-033 section 4, #457) -- @hc12r. The exact
payload and execution environment a task attempt resolves (interpreter,
dependency lock, platform) is now pinned on first use and reused by
every later attempt of that node -- a worker-fleet change between
retries can no longer silently change what an already-running attempt
lineage executes. A pinned environment that becomes unavailable fails
named `platform`, not a generic error.
### Changed
- `./preflight.sh` runs ~45% faster (#448) -- @hc12r. The security-scan
stage was oversubscribing every CPU core; scans now share a capped,
persistent build cache instead of losing it (and re-fetching
`govulncheck`/`go-licenses`) on every single run.
## [0.11.0] - 2026-08-30
### Added
- **TypeScript code-node capabilities are advertised honestly** (ADR-030,
#426, A5) -- @hc12r. `/api/capabilities` now reports `code_languages` and
`code_js_wrapper_version`, and adds `code-typescript` only when Node >=20
actually resolves at server startup. Hosts without Node continue to expose
the Python-only capability set, so SDK preflight refuses TypeScript before
persistence rather than allowing a runtime failure.
- **Code nodes run on a warm worker pool** (ADR-029, #419-#423) --
@hc12r. Long-lived Python workers over a framed, versioned Unix-socket
protocol replace spawn-per-invocation: interpreter and pandas/pyarrow
boot are paid once per worker, not once per node run -- measured
71.6ms to 3.3ms per execution (21.7x) on the expansion fan-out shape.
Crash isolation strictly improves (a dying worker costs one execution
and a lazy respawn); cancel/timeout kill only the busy worker.
`BROKOLI_CODE_POOL=0` keeps the legacy spawn path for one release;
Windows stays on it. The wrapper itself is now a real, versioned,
pytest-tested Python library embedded in the binary, and every code
execution's run log records its wrapper version and effective limits.
- **Resource ceilings for code nodes** (ADR-029 P0, #420) -- @hc12r.
`BROKOLI_CODE_MEMORY_MB` / `BROKOLI_CODE_CPU_SECONDS` server defaults
(with `BROKOLI_CODE_MAX_*` clamps) and per-node `max_memory_mb` /
`max_cpu_seconds`, enforced as rlimits. A breach fails that script
with an error naming the limit -- never a bare "signal: killed", and
never the whole pod. Memory is enforced as address space; the
wrapper's pandas/pyarrow transfer acceleration is disabled under a
memory cap (numpy's allocator exits uncatchably from C when its
reservation fails).
- **Script progress is finally consumed** -- @hc12r. The wrapper's
`#PROGRESS:` markers were emitted and discarded for their whole life;
on the pool path they become typed frames that land in the run log as
info lines.
- `/api/capabilities` advertises `code_protocol_version`,
`code_wrapper_version`, and the `code-streaming-emit` execution
feature, closing the gap where an `emit()` script deployed silently
to a server whose wrapper predated the idiom -- @hc12r.
- **TypeScript code nodes actually dispatch** (ADR-030, #428, #429) --
@hc12r. The JavaScript/TypeScript wrapper reaches contract v1 and
code workers dispatch to a Node runner under the same per-run
resource ceilings as Python -- the half that turns the advertised
`code-typescript` capability into an executable one.
- **A language-neutral IR canonicalization and digest spec** (#417) --
@hc12r. Byte-identical IR from any SDK is now a stated property with
the normalization rules written down, so the Python and TypeScript
SDK differential oracle has a concrete contract to test against.
### Changed
- **Code-node scripts no longer inherit the host environment**
(ADR-029's secrets fix) -- @hc12r. Workers start from an allowlist
(`PATH`, `HOME`, `LANG`, `LC_*`, `TZ`, `TMPDIR`). Deployments whose
scripts legitimately read host variables name them in
`BROKOLI_CODE_PASS_ENV` (or set `*` to restore the old behavior).
- User `print()` in a code node can no longer corrupt small-dataset
results: output rides the protocol, prints ride the run log -- @hc12r.
- Code-node tracebacks now carry the user script's own line numbers
(`<code-node>`), and a script containing `%` can no longer corrupt
the wrapper -- @hc12r.
### Fixed
- **Recovery no longer claims runs owned by an active local runner** --
@hc12r. Between code-node dispatch steps a live run holds no
execution lease, and the 20s reclaim sweep could read that lease-free
gap as an orphaned run and mis-schedule it. Recovery now consults the
engine's active-run map and defers any run the local runner owns, so
a multi-node host stops mis-scheduling its own healthy runs.
## [0.10.80] - 2026-08-29
### Fixed
- **Revocable API tokens actually authenticate** (#414) -- @hc12r.
`api_tokens` rows could be created and listed, but nothing on the
request path ever consulted them: presenting one got 401 like any
garbage bearer, so the tokens page minted decorations. The middleware
cannot validate them itself without opinions that belong upstream
(which claims a workspace token implies), so this adds the extension
seam instead -- `api.BearerTokenResolverFunc`, in the
OrgResolverFunc/AuthMethodsFunc pattern the package already uses.
Consulted only after JWT parsing fails, so sessions never pay the
lookup, on both the main request path and the WebSocket path; a nil
func or false return leaves the 401 exactly as it was. Downstream
distributions wire it to the api_tokens hash lookup, which also makes
browser-confirmed device authorization mint revocable brk_ tokens
instead of 24-hour session JWTs.
## [0.10.79] - 2026-08-29
The orchestration wave: the three Airflow-inspired capabilities filed
after the data-intervals arc all land -- connection pools, the grid
view, and deferrable waits -- plus ADR-024's fold finished.
### Added
- **Connection pools** (#408, closes #398) -- @hc12r. A connection
gains `max_concurrent` (0 = unlimited, today's behavior exactly): a
named concurrency budget that nodes, not runs, draw from -- a
ten-node pipeline may touch the warehouse in one node for two
seconds. The slot is held for the duration of the attempt's actual
execution and released by the executing goroutine itself, so a
timed-out-but-still-running handler keeps its slot until it genuinely
stops touching the connection. Waiting blocks (bounded by the run's
own cancellation, not the node timeout) and is visible in the node's
run log; an explicit `pool:` naming no connection fails the node,
because a budget that silently doesn't limit is a disabled control.
PER ENGINE INSTANCE in this release, stated on the field and in the
UI; the store-backed cluster-wide budget is #407. Acceptance ran
against live Postgres, asserted from the server's side: four racing
runs through a 2-slot pool, `pg_stat_activity` sampled every 40ms,
never more than 2 active. Also fixed on the way: the connection
resolver existed only under `serve`, the same silently-missing-
control shape -- `NewEngine` now always constructs it.
- **Grid view** (#409, closes #400) -- @hc12r. The question an operator
actually asks is "how has this pipeline behaved lately", and no list
answers it. `/pipelines/{id}/grid` is the matrix that does: runs as
columns (newest right), nodes as rows in the planner's own dependency
order, one status-colored cell per node-run's LATEST attempt -- a red
row is a broken node, a red column is a bad day, a diagonal is a
deploy. Version boundaries are drawn on the column axis (a behavior
change lining up with a deploy is the diagnosis half the time),
backfill-triggered columns are marked, and clicking a cell deep-links
the runs page with that run expanded. One new store query
(`GridNodeRuns`, the ROW_NUMBER-per-partition shape both stores
already use twice) serves the whole matrix.
- **Deferrable waits** (#411, closes #399) -- @hc12r. A node that waits
-- for a file, an endpoint, a partition, another pipeline -- used to
burn a worker slot for the whole wait. The new `wait` node type parks
instead: when its condition (`file_exists`, `http` through the
netguard-guarded client, `interval_elapsed` pairing with ADR-028, or
`pipeline` subsuming the dependency-gate shape) is not met, the run
flips to the new status `waiting`, a durable park lands in the store,
and the slot frees. One leader-gated watcher babysits every parked
wait and wakes the SAME run when the condition fires; timeouts are
part of the condition and fail naming it. Measured at the issue's own
number: 1,000 parked waits hold 0 run slots and a sub-20 goroutine
delta, and a parked run survives instance restart. Recovery and
startup redispatch deliberately exclude `waiting`: a parked run is
paused, not interrupted work. `deferrable-waits` joins
`supported_execution_features`.
### Changed
- **URI schemes are registry claims** (#410, closes #361) -- @hc12r.
`detectDriver` and `dialectForURI` stop being per-backend tables in
the engine: each registered dialect states the schemes it owns, the
driver that opens them, and the DSN shape that driver wants, and the
unknown-scheme refusal builds its supported list from the claims so
the message cannot drift. What deliberately did not fold is recorded
in ADR-024's update section: snowflake (a driver with no dialect
owner), the sqlite filename suffixes and schemeless Postgres default
(heuristics with no scheme to claim), and BuildURI plus the
connection catalog (they span non-database connectors). Behavior-
preserving by the ADR's own bar: the pre-existing mapping tests pass
unchanged.
## [0.10.78] - 2026-08-29
ADR-028 lands whole: a scheduled run knows the half-open data interval
it covers, and catch-up, backfill and the SDK all speak in those
intervals. The tracking issue (#397) closed with all four phases on
main.
### Added
- **Data intervals** (#402, ADR-028 accepted in #396) -- @hc12r. A
scheduled run carries `[data_interval_start, data_interval_end)` --
the previous tick to the fired tick, stamped at dispatch. In node
config, `${interval.start}` and `${interval.end}` resolve to RFC3339
UTC through the ordinary variable resolver; a run WITHOUT an interval
leaves the reference visibly unresolved instead of silently empty,
because silent emptiness is how bad data happens. A resumed run keeps
its original interval, so the slice a failed run was responsible for
is the slice its resume processes. Scheduled dispatch is unique on
`(pipeline_id, data_interval_start)`, which turns a leader-failover
double-fire into a quiet no-op instead of a duplicate run. The DST
edges are pinned by measurement, not assumption: in a zoned schedule,
spring-forward skips the day's fire entirely and fall-back fires both
occurrences -- the test table documents the library's real behavior.
- **Opt-in per-interval catch-up** (#404) -- @hc12r. The default stays
exactly what it was: after downtime, one catch-up run for the most
recently missed tick. A pipeline that sets the new `catchup` flag (a
real column, declared in the IR schema, a checkbox next to the
schedule field) instead gets every complete interval the schedule
owed, oldest first, sequentially, anchored on the newest
`data_interval_end` on record so the chain continues rather than
being re-derived from wall clocks. The per-pipeline cap (50)
truncates keeping the NEWEST intervals -- an unattended catch-up must
land the pipeline current -- and logs the dropped range; older slices
stay reachable through an explicit backfill.
- **Backfill walks the schedule's own interval grid** (#403) -- @hc12r.
The previous backfill parsed date strings, invented a daily grid
regardless of the pipeline's actual schedule, executed every run
synchronously inside the HTTP handler, and stopped at the first
failure. Replaced end to end: enumeration walks the pipeline's own
schedule, one interval-stamped run per slice, oldest first at
concurrency 1 in the background; a failed slice is counted, not
fatal; ranges past 500 intervals are refused naming the count rather
than silently truncated. A pipeline that references neither
`${interval.*}` nor the grandfathered `${param.date}` is refused
without `force`, because every run would do identical work. New CLI
verb: `brokoli backfill <pipeline-id> --start --end [--force]`.
- **`data_intervals` in `supported_execution_features`** (#405) --
@hc12r. Advertised now that pipelines using it actually run (the
list's honesty rule), and pinned by the capabilities test because the
SDK gate matches the exact string. SDK counterpart shipped as
brokoli-sdk#71: `Pipeline(catch_up=...)` stopped raising, compiles to
the `catchup` field, and deploy preflight refuses servers that
advertise features but not this one -- naming the feature instead of
letting the server's fail-closed decoder 400 opaquely.
### Changed
- **Backfill API answers with a plan, not a run-ID list** (#403) --
@hc12r. `POST /api/pipelines/{id}/backfill` now returns 202 with the
accepted plan (interval count, bounds, concurrency); the old response
required the entire backfill to finish inside the request, which no
real range could survive. The date-only `start_date`/`end_date`
fields keep their historical inclusive-day meaning alongside the new
RFC3339 `start`/`end`, and pipelines built on the old `${param.date}`
convention backfill unchanged -- every backfill run still receives
`params[date]` set to its slice's start date.
- **The write vocabulary moves into `pkg/dbdialect`** (#401, progress
on #361) -- @hc12r. Statement generation's per-backend record --
quoting, literal rules, boolean spellings, type maps, timestamp
layouts, escaping, the CREATE TABLE suffix -- moves out of
engine/sqlgen.go into each backend's own file as a `StatementWriter`
capability, now that the bulk-write equivalence corpora against live
Postgres, MySQL and ClickHouse exist to refactor against. SQLite,
SQL Server and the generic fallback join the registry as a third,
narrowest claim tier: write vocabulary only.
### Fixed
- **Unknown connection schemes are refused by name** (#395, closes
#383) -- @hc12r. detectDriver's default handed any unrecognised URI
to pgx, so `oracle://`, `bigquery://` and `databricks://` -- all
offered by the connection catalog -- failed with a Postgres error
naming the wrong backend. An unmapped scheme is now refused naming
that scheme and listing what this build supports, without echoing the
URI (it carries the password).
- **ClickHouse parses string values the way the server parses
literals** (#394, closes #392) -- @hc12r. A CSV-shaped dataset
appended into a typed ClickHouse table diverged by path: the
statement path's rendered literal `'123'` was parsed server-side into
an Int64 column, while the native batch appender refused. The batch
writer now probes the target's columns once (LIMIT 0, read through
the dialect's own TypeReader) and parses strings headed at integer,
float and boolean columns; Decimal, DateTime and String pass through
to the driver's own parsing. A probe failure degrades to no coercion,
which is where the path stood before.
## [0.10.77] - 2026-08-28
The ClickHouse arc closes (ADR-027 phase 4, #382): dbt runs against
ClickHouse connections, and the pushdown question is answered by a
measurement rather than left open.
### Added
- **dbt on ClickHouse** (#391) -- @hc12r. A dbt node pointed at a
ClickHouse connection generates a `driver: native` profile against the
connection's own host and port -- one connection record, one address,
where dbt-clickhouse's HTTP default would have invented a port the
connection never described. The profile stanza is adapter-aware:
ClickHouse has no database/schema split, so it carries the target
schema (the database dbt builds in) and no `dbname`. Proven end to end
against a live server: a real dbt run through the generated profile,
per-model outcomes recorded, and the built model read back through
Brokoli's own source path -- ADR-025's read-back rule, literally.
Operational note, recorded where it bites: dbt-clickhouse 1.8.9's
native client imports `pkg_resources`, which setuptools 81 removed, and
the resulting failure blames the wrong package ("clickhouse-driver is
not installed" -- it is). dbt environments should pin `setuptools<81`;
Brokoli's own CI does, with the reason in a comment.
### Changed
- **ClickHouse pushdown is refused on measurement, not deferred by
default** (#391, ADR-027) -- @hc12r. The driver reports no row count
for `INSERT ... SELECT` -- verified against a statement that provably
moved 400 rows and still answered zero -- so a pushed-down ClickHouse
segment could never report its write honestly, which ADR-023 makes the
price of eligibility. Two further blockers stand recorded behind it:
no transactions (a half-finished segment cannot roll back or resume
safely) and no locks (concurrent writes to one table cannot
serialize).
The refusal is self-updating: a test runs that exact insert and fails
the day the driver starts reporting a real count, with instructions to
revisit the ADR before anything else changes.
## [0.10.76] - 2026-08-28
ClickHouse arrives as the third database backend (ADR-027): readable,
appendable, overwritable -- and honest about everything it is not. The
capability climbed the same ladder MySQL did, one phase per PR, each
earned by tests against a live server.
Measured, 50,000 4-column rows against the 25.3 LTS container: the native
batch writes at 410-525k rows/sec against 145k for rendered SQL -- on the
benchmark machine, the fastest write path in the product (Postgres COPY
measured 250-460k on the same hardware). The benchmark is committed
(`BenchmarkClickHouseWrite`) so the claim stays checkable.
### Added
- **ClickHouse connections, sources, and migrations out** (#385, #386,
ADR-027 phases 0-1) -- @hc12r. `clickhouse://` connections over the
native protocol (pure-Go driver, so the single-binary release is
untouched), `source_db` reads, and migrations out of ClickHouse that
carry declared column types: `UInt32` widens to a 64-bit destination
column, `Decimal(12,2)` stays exact, `DateTime64` lands as
`timestamptz` because a ClickHouse DateTime is an epoch instant, and a
`Nullable(String)` NULL survives as NULL. Wrapped type names --
`Nullable(...)`, `LowCardinality(...)` -- unwrap before anything is
judged.
The catalog entry follows a new rule with teeth: a connection type
enters the catalog only in the change that makes `source_db` actually
work against it, and the connection test now runs `SELECT 1` through
the driver for every backend rather than stopping at a handshake.
- **ClickHouse sinks: append and overwrite** (#387, #388, phases 2-3) --
@hc12r. Appends go through the native batch protocol; `create_table`
renders real column types with `Nullable(T)` explicit -- ClickHouse
columns are non-nullable by default, the inverse of every other
backend, and a missed wrapper would turn NULLs into zeros -- plus the
engine clause the language requires (`ENGINE = MergeTree ORDER BY
tuple()` by default, `table_engine` in node config to override).
Keyed appends and overwrites are stream-eligible, so a table larger
than worker memory writes with a one-batch resident set.
**What ClickHouse does not promise, stated rather than smoothed
over.** There are no transactions: each committed batch is atomic and
nothing larger is, so a failed load keeps the batches that were
acknowledged, and an interrupted overwrite keeps its truncate. A test
observes exactly that weaker state and will fail if ClickHouse ever
gains something better to use. A resumed run that re-runs a ClickHouse
write warns in its own log that already-committed batches are still in
the table, naming the mitigations. Concurrent overwrites of one table
are unguarded -- ClickHouse has no advisory-lock primitive -- and the
code says so.
**Upsert is refused, not counterfeited.** ClickHouse has no
synchronous merge; `ReplacingMergeTree` deduplicates eventually, at
merge time, which is a different promise than `mode: upsert` makes on
the other backends. The refusal fires in the pipeline editor at
validation time and explains the native alternative. Pushdown is
likewise fully refused -- structurally (the dialect claims no
same-server capability) and observed end to end -- until a
differential corpus earns it.
### Fixed
- **Unknown connection schemes no longer route into the Postgres
driver** for the connection test's purposes: `SELECT 1` through the
real driver replaces ping-only success, so a green connection test now
means the server executes SQL for that user. (The `detectDriver`
default-to-pgx fallthrough itself is tracked in #383.)
### Known limits
- The native batch does not coerce string values into numeric columns
where the rendered-SQL escape hatch would; a CSV-sourced append into a
typed ClickHouse table needs a cast upstream for now. Tracked in #382.
## [0.10.75] - 2026-08-28
The write paths get fast. Creating a destination table no longer costs the
bulk path, and upserts stop being the slowest write in the product --
measured, on 50,000 rows against Postgres: `create_table` sinks went from
60k to ~450k rows/sec, and upserts from ~26k to ~95k.
Two upsert behaviours changed deliberately in the process; both are
documented below and pinned by tests.
### Changed
- **Creating a table no longer forfeits the bulk write path** (#376, #378)
-- @hc12r. `sink_db` wrote 411k rows/sec into a table that existed and
60k when it had to create one, because the `CREATE TABLE` and the rows
were rendered as one SQL string and one statement pinned the whole write
to the statement path. The DDL now executes inside the bulk writer's own
transaction, so the rows go through COPY or LOAD DATA -- and on Postgres
a failed load still rolls the table creation back, which the obvious fix
(run the DDL first) would have quietly lost.
`migrate` never used the bulk writers at all, in any configuration; it
now does, and a 50k-row migration went from 1.07s to 392ms.
- **Upserts are staged through the bulk protocol** (#377, #379, #380) --
@hc12r. Rows bulk-load into a session temp table and one merge moves
them into the target, inside the same transaction where the backend has
one.
On Postgres this is 3.7x on inserts and 2.8x on conflicts at 50k rows --
and the interesting part is why the first attempt was not: the cost was
never only parsing. `INSERT ... ON CONFLICT` pays for per-row
speculative insertion into the unique index, staged or not, so the
shipped merge is an UPDATE join plus an anti-join INSERT under an
`EXCLUSIVE` lock -- measured 3-6x faster than `ON CONFLICT` on the same
staged set, and equal to `MERGE` without requiring Postgres 15.
Concurrent upserts of one table now serialize; readers keep reading
through MVCC.
On MySQL the same staging buys no throughput, and the committed
benchmark says so rather than implying otherwise -- MySQL never had the
speculative-insertion tax. What it buys is memory: upsert was the one
sink mode that could not stream, and its statement path held the rows
plus their rendered SQL text. A keyed upsert is now stream-eligible on
both backends, with a resident set of one batch whatever the table size.
**Behaviour change one: duplicate keys within one dataset now resolve to
"the last row wins", on both backends.** Previously Postgres errored
("ON CONFLICT cannot affect row a second time") only when the duplicates
happened to share a 100-row batch, and applied last-write-wins when they
did not -- the outcome depended on row position. MySQL always applied
last-write-wins. A run that used to fail on Postgres can now succeed;
the old refusal is itself pinned by a test so the difference stays
documented.
**Behaviour change two: `key_columns` must name a unique index, and the
refusal is named.** Postgres previously surfaced the server's generic
"no unique or exclusion constraint" error after rendering the whole
statement; it now refuses before any rows move, naming what was asked
for and what unique indexes the table actually has -- the check MySQL
upserts already had.
Counts: Postgres reports distinct rows merged (in-set duplicates
collapse); MySQL keeps its arithmetic of 1 per insert and 2 per update,
exactly as its statement path always reported.
- **A stale performance claim corrected** (#379) -- @hc12r. The COPY
path's comment rejecting staged upserts as "measured slower (660ms)"
compared staging against COPY-append -- a comparison staging can only
lose -- and never against the statement-path upsert it would replace.
`BenchmarkUpsertWrite` (both backends, three sizes, both load shapes) is
now the standing record.
### Added
- **ADR-027: ClickHouse as the third backend, append-first** (#381,
accepted) -- @hc12r. The direction for the next backend arc: pure-Go
driver so the `CGO_ENABLED=0` release is untouched, bulk append first
because that is both ClickHouse's design center and what this release
made the write path best at, upsert refused by name rather than
counterfeited with `ReplacingMergeTree`, and a catalog rule with teeth
-- a connection type enters the catalog only in the PR that makes
`source_db` actually work against it. Implementation is phased in #382;
the existing advertised-but-dead connection types are #383.
## [0.10.74] - 2026-08-27
A dbt node stops being one opaque success or failure. One invocation now
produces a record per model, failures say which of dbt's five modes
happened, and a model dbt built can be handed straight to a sink without
the rows leaving the database.
Alongside it, a column's declared type now survives the trip to a
destination table instead of being guessed from a sample of its values —
first for `migrate`, now for `sink_db` as well.
### Added
- **dbt reports an outcome per model, from one invocation** (#365, #366,
#367, #368, ADR-025) — @hc12r. Running dbt per model to get per-model
status was measured at 30x slower than one invocation: 665s against 21.8s
on a 100-model project. So Brokoli runs dbt once and reads what it already
writes — `manifest.json` for the build order and `run_results.json` for
what each node did — and reconciles the two into one row per model in
`execution_attempts`, keyed by the model's dbt `unique_id`.
Both artifacts are version-drifting under stable schema numbers, so the
readers decode only the fields Brokoli needs and are proven against two
dbt versions. A model that dbt skipped is distinguished from one that was
never selected, and a skip is reported with the failure that caused it
rather than as a bare "skipped" — the reconciler walks to the nearest
failing ancestor.
- **dbt failures say which of the five modes happened** (#370) — @hc12r.
"dbt failed" left whoever was on call to open the log and work out which.
Each mode was reproduced against real dbt 1.8.9 before being encoded:
a compile error means nothing ran, a model error means the models that
do not depend on it are committed, and a test failure means everything
built and the data is wrong. Those call for different actions.
Neither signal separates them alone — exit 1 covers both a failing model
and a failing test, and a missing `run_results.json` covers both a compile
error and a kill — so the classifier reads the exit status, the artifacts
and the run's own context together. When the artifacts do not support a
specific answer it says so rather than guessing between them.
- **A dbt profile is generated from a Brokoli connection** (#369) — @hc12r.
A dbt node pointed at a `conn_id` no longer needs a `profiles.yml` at all:
Brokoli writes one into a per-run temporary directory (0700, file 0600)
and removes it afterwards. A project that brings its own profile keeps
using it; the generated one never overwrites a real `profiles.yml` or the
default profiles directory.
- **A dbt model can be handed downstream as a reference** (#371, ADR-023) —
@hc12r. Naming an `output_model` makes the dbt node hand its sink a
`TableRef` rather than rows, so a sink on the same connection composes
through the pushdown compiler that already existed and the rows never
enter the engine. Nothing in that machinery had to learn what dbt is.
It refuses by name where it cannot: an ephemeral model has no relation to
reference, and a project authenticating through its own `profiles.yml` is
reaching a server Brokoli cannot name, so composition requires `conn_id`.
- **The dbt node is configurable from the pipeline editor** (#372) — @hc12r.
Project directory, the connection it runs against, and the model it hands
downstream are now first-class fields rather than raw JSON, with the
`target_schema`/`target` precedence stated in the form and a warning when
a project's own profile would win over the chosen connection.
- **ADR-026: check for a Python runtime, do not ship one** (#359, #364) —
@hc12r. dbt-core is Python; the release is a `CGO_ENABLED=0` binary
cross-compiled to six targets. Rather than invent a policy, this applies
the one ADR-016 already set for plugin runtimes: resolve and check, never
provision. dbt must be present and within a declared supported range, and
is refused by name at validation time rather than mid-run — a dbt node
failing halfway through a DAG has already cost whatever ran before it.
**Someone who does not use dbt acquires no Python dependency and sees no
change to their install.** The container image remains an operator's
option, not a requirement.
### Fixed
- **A stale workspace id no longer hides every pipeline behind an empty
200** (#373, #231) — @hc12r. `localStorage` is keyed by origin, not by
instance, so `http://localhost:8088` serves whichever build was last run
there and a workspace id stamped by one instance was sent to the next.
The filed symptom was a blanket 403 in multi-tenant mode. The worse one
was silent: with no tenant resolver there is nothing to check the header
against, so an unrecognised id was used verbatim and every scoped list
came back empty. The product read as brand new — "Build your first
pipeline", stat cards at zero — while the Recent Runs panel beside it,
scoped by org rather than by workspace, listed the pipelines it would not
show.
A stored workspace id is now a hint until a workspace list confirms it,
and an unconfirmed hint is never sent; resolution is awaited before the
app renders, so nothing can race it; and a workspace stored for one user
is dropped when a different user signs in. On the server, a build with no
tenant registry ignores the header entirely rather than filing requests
under a name nobody can list — which closes the same trap for `curl` and
API tokens.
- **A destination table is created from the source's real column types**
(#362, #374, #360, #363) — @hc12r. Type inference rendered each sampled
value with `%v` and tried to parse it, which cannot recover what the
source declared. A `bigint` whose sample fitted in an int32 became `INT`
and the load failed on the first value above 2^31 — after the table
existed. An exact decimal became a float, and a timestamp became text.
`migrate` now reads the source's types directly (#362), and `sink_db`
with `create_table` carries them across the node boundary (#374): a
source records what its columns are, transforms carry it through the same
rules the rows went through, and the sink uses it. Per column — a dataset
where one column was rewritten by a transform still gets exact types for
the other nine, with inference as the fallback for that one.
**What changes for an operator.** Tables created by `create_table` may
differ from the ones previous versions produced, and will match the
source more closely. A column whose type the destination genuinely cannot
hold without losing information — a `timestamptz` landing on MySQL — is
now named and refused before anything is written, where it previously
dropped the zone silently. Nullability is deliberately not carried from a
Postgres source, because pgx does not report it and inventing a `NOT NULL`
the source did not have would fail the load.
- **Three defects on the `create_table` path** (#374) — @hc12r, all found by
the tests written for the change above.
`create_table` was silently broken whenever the sink was on the same
server as its source: that path compiles into an `INSERT INTO` an existing
relation and has no step that creates one, so the run failed on a table
that was not there. It now declines that compilation and creates the table.
A single-row load declared every column `INTEGER` regardless of content —
the inference threshold is four fifths of the sample, which rounds to zero
for one row, so the first test matched unconditionally. A text column came
out `INTEGER` and then failed to insert its own rows.
Postgres rendered an unbounded `text` column as
`VARCHAR(9223372036854775807)`, which the server rejects outright: pgx
reports the length of `text` as the maximum int64 and the renderer had no
upper bound. It now renders `TEXT`. `migrate` could reach this too, for
any text column landing on Postgres.
- **A recovery test no longer races the clock, and its grace period is
per-engine** (#358, #264) — @hc12r. The test needed a 30ms window to still
be open when recovery read it, which a loaded machine could miss. It no
longer depends on the clock at all.
The grace period moved from a package-level variable onto the Engine. A
package-level override is process-wide, so one test's value was observable
by every other engine in the process and no test touching it could run in
parallel with another. Verified over 30 runs under `-race` with six
processes competing for CPU.
## [0.10.73] - 2026-08-25
An interrupted run that wrote nothing now goes back on the queue instead of
failing. With worker autoscaling on, that turns a routine eviction from a
failed pipeline someone has to notice into a non-event.
### Added
- **An interrupted run that provably wrote nothing is re-queued** (#289) —
@hc12r. When a worker dies or is evicted mid-run, recovery could not tell
whether a node's side effect had completed, so it failed the run. That
default was right, and it stays — re-running a pipeline whose sink already
appended half its rows would double those rows.
But it was the right default for the wrong population. Autoscaling evicts
pods, nodes drain, rolling deploys are routine, and many interrupted runs
never reach anything that writes. Those are now re-queued automatically.
The test is not "did a sink run" but "could anything with a durable record
have written outside Brokoli". A node stuck mid-attempt is exactly the
ambiguous case, so if such a node could write, the run still fails.
The rule is a whitelist that defaults to deny. `code` is excluded because
it runs arbitrary user code; `source_api` is excluded because its HTTP
method is configurable, so a "source" can POST; `migrate`, `dbt` and
`notify` write by design; and unknown types, including every plugin node,
get no assumption made about them.
Four guards, each with a test. A deployment with no job queue fails the run
rather than parking it pending forever, which is the stranding fixed in
#216. Durable cancel intent outranks re-queueing. Re-queues are bounded, so
a run that keeps being interrupted becomes visible instead of cycling
silently. And a record for a node the current pipeline version no longer
contains blocks it.
**What changes for an operator:** runs that used to fail with "interrupted
mid-execution and cannot be safely resumed" may now complete on their own.
A new `brokoli_runs_recovery_requeued_total` metric counts them — a rising
count is eviction pressure rather than a fault, but one run cycling through
it repeatedly is a run that cannot finish.
### Security
- **Every outbound HTTP path now uses the sanctioned client, and CI enforces
it** (#320, #266, ADR-022) — @AghastyGD. The S3 connection test built a
bare `http.Client` and issued a request to a host derived from unvalidated
request fields, in the same file that already used the guarded client
correctly for its other connection types. That path, the notification
webhook, alerts, and the engine connector paths are all migrated.
All sixteen outbound call sites now consult the operator's configured
policy (`BROKOLI_OUTBOUND_ALLOW_CIDRS`, `BROKOLI_OUTBOUND_ALLOW_PRIVATE`)
rather than a hardcoded one, so an internal service reachable from one
node type is reachable from all of them.
A type-aware `go/analysis` pass now fails the build on a new direct
`http.Client`, `new(http.Client)`, `http.DefaultClient`, or the
package-level HTTP helpers — resolving through type aliases and aliased
imports, which a text search would miss. Exceptions require a
`//netguard:allow <justification>` directive that must carry a reason and
is consumed by exactly one use, so one comment cannot silently cover a
second violation added beside it later.
### Internal
- **ADR-025 accepted** (#354) — @hc12r. The dbt integration takes dbt's
manifest as its contract and makes a *model*, not a command, the unit of
work, with dbt remaining the compiler and executor. Nothing is implemented;
what is settled is the direction, so the first implementation does not
choose it by accident. Blocked on the Python-packaging decision (#352);
implementation tracked in #353.
- **The engine test binary gets a timeout it can finish inside** (#329) —
@hc12r. The package measured 581s under `-race` against Go's 600s default:
nineteen seconds of headroom, so a loaded runner panicked and named
whichever test held the baton rather than anything at fault. CI and
preflight now pass `-timeout 15m`, chosen against the job's own 20m limit
so a genuine hang still panics with stacks inside the job.
The dbt tests now overlap each other, 84s to 56s, since each spends about
ten seconds in interpreter startup that it waits on rather than computes.
Parallelising 118 further tests was measured and reverted: it bought ten
seconds, because under `-race` with no idle cores the package is CPU-bound,
and it introduced an intermittent failure. Splitting the package is the
durable fix and now has evidence behind it.
## [0.10.72] - 2026-08-25
The `dbt` node could not run dbt. Every command it offered failed before
reading the project, and it shipped that way because nothing ever ran it.
### Fixed
- **The `dbt` node could not execute any dbt command** (#348) — @hc12r. It
appended `--output json` to every invocation. That flag exists only on
`dbt ls`, so `run`, `test`, `build` and `seed` — every command the UI
offers — failed at dbt's option parser before touching the project:
```
$ dbt run --project-dir /tmp --output json
Usage: dbt run [OPTIONS]
Error: No such option '--output'.
```
Verified against dbt-core 1.8.9. `--output json` now goes to `ls` alone;
the other commands get `--log-format json`, which is the flag they have.
It reached production because nothing ever ran it. The only test
referencing dbt substituted a fake executor specifically to avoid needing
a real project on disk, so the wrapper was never executed against the tool
it wraps. The tests added here run real dbt-core against real Postgres,
and were checked against the reintroduced defect to confirm they fail on
it.
Three further defects in the same file go with it:
- `run_results.json` was read relative to the **engine's** working
directory rather than the project's, so it resolved to the wrong path
for every project that is not the process cwd.
- `DBT_PROFILES_DIR` was exported unconditionally, so an unset
`profiles_dir` set it to empty and overrode dbt's own `~/.dbt` default,
leaving a project that relied on that default unable to resolve its
profile.
- The command ran without the attempt's context, so a hung dbt ignored
both the node timeout and a cancelled run. Every other external call in
the engine threads it.
Failures now name the models that failed rather than reporting an exit
code, read from `run_results.json` — which dbt writes even when the
command fails, so it carries strictly more than the exit status. dbt's
JSON log stream is rendered as its human messages, with anything that does
not parse passed through so a stack trace still reaches the run log.
**If a pipeline has a `dbt` node, it has never run.** There is no data to
repair and nothing to re-run — the node failed every time. What changes on
upgrade is that it starts working, so a pipeline that was failing at this
node will now proceed past it.
### Added
- **ADR-025: run dbt projects as they are, and own the orchestration around
them** (#349) — @hc12r. Proposed. Records what the integration should be
rather than a shell wrapper: dbt's manifest as the contract, a **model**
as the unit of work rather than a command, with each model a node carrying
its own status, retry, and lineage, while dbt stays the compiler and
executor. A built model is a `TableRef` in ADR-023's sense, so it composes
with the pushdown compiler and the dialect layer from ADR-024 instead of
duplicating them.
The rejected alternatives are recorded with their reasons, including a
native model layer — architecturally the best fit for what now exists, and
commercially the worst, because it asks a team with a working dbt project
to rebuild rather than migrate. The cost is stated too: dbt-core as a
Python runtime dependency is the first thing that breaks the
single-static-binary property, and that packaging decision needs its own
ADR before implementation rather than during it.
### Internal
- **The dbt node's tests run real dbt in CI** (#348) — @hc12r. A fixture dbt
project (a seed, two models with a genuine `ref` edge, schema tests, and
one model that fails at execution rather than compile time) built by
dbt-core against the Postgres service container. dbt is pinned, because
its CLI and its `run_results.json` schema both drift across minor
versions, and an unpinned upgrade quietly changing behaviour is how this
defect would return. The tests skip rather than fail when dbt is absent,
so a contributor without a Python environment is not blocked.
## [0.10.71] - 2026-08-25
MySQL becomes a backend on the same footing as Postgres. Two of these are
data-correctness fixes on the MySQL write path, and one is a behaviour change
that will fail pipelines which were previously succeeding — read the upsert
item before upgrading.
### Fixed
- **MySQL string literals did not escape backslashes** (#338) — @hc12r.
Values are rendered into SQL text rather than bound as parameters, and the
escaping doubled the quote character while doing nothing about backslash.
MySQL treats a backslash as an escape inside a literal unless
`NO_BACKSLASH_ESCAPES` is set, which is not the default.
So an ordinary Windows path arrived altered:
```
C:\Users\report.csv -> C:Usersreport.csv
```
and a value ending in a backslash escaped the quote meant to close the
literal, after which MySQL parsed the following text as SQL rather than
data. Verified against a live MySQL 8.4 before and after the fix.
Postgres was never affected: `standard_conforming_strings` makes a
backslash an ordinary character there, which is why this survived — the
escaping was correct for the backend it was written against.
**Upgrading does not repair data already written; re-running the pipeline
does.** The rows worth checking are text values containing backslashes.
- **MySQL passwords containing URI metacharacters could not authenticate**
(#338) — @hc12r. The DSN was assembled with `net/url`, which
percent-encodes userinfo, while go-sql-driver reads credentials verbatim.
Any password containing `@ : / ? # %`, a space, or non-ASCII failed with
`Access denied` and nothing pointing at the cause. Credentials are now
carried verbatim; driver options keep their encoding, because those the
driver does unescape.
- **A MySQL overwrite was not atomic** (#344) — @hc12r. `GenerateSQL`'s
contract is that clear-then-insert runs as one transaction. On MySQL,
`TRUNCATE TABLE` implicitly commits, so an overwrite with `truncate: true`
destroyed the previous rows the moment the clear ran — a failure in any
later statement rolled back nothing and left the table empty.
The `truncate` option now degrades to a transactional `DELETE` on MySQL.
It keeps its measured advantage on Postgres, where `TRUNCATE` rolls back.
`create_table: true` has the same MySQL caveat and it is now documented
rather than fixed: `CREATE TABLE` also commits implicitly, so a run that
creates its target and then fails leaves the empty table behind.
- **A driverless connection failed with an error naming nothing** (#346) —
@hc12r. A connection whose type has no compiled-in driver — Oracle is the
live example, offered in the catalog — failed a run with a message naming
neither the connection nor its type, because the sentence explaining it
went to the server's stdout. Resolver warnings now reach the run's node
log, where the pipeline author reads them.
### Changed
- **MySQL upsert now requires `key_columns`, and validates them** (#344) —
@hc12r. **This will fail pipelines that previously succeeded, and that is
the point.**
MySQL's `ON DUPLICATE KEY UPDATE` merges on whichever unique index the
inserted row collides with; the statement cannot name one. `key_columns`
was therefore accepted and silently ignored — rows merged on whatever
index the schema happened to have, which may not be the columns
configured, rewriting rows nobody addressed.
`key_columns` is now required for MySQL upsert, as it already was for
Postgres and SQLite, and is checked against `information_schema` before
the write: the named columns must be exactly a unique index of the target
table, or the run fails naming the indexes the table actually has.
If a table carries additional unique indexes beyond the one named, the run
logs a warning listing them — a row can still collide there and MySQL
offers no way to prevent it, so the check does not pretend otherwise.
A pipeline that starts failing after this upgrade was merging on a key it
did not configure. The fix is to name the real one.
- **MySQL upsert emits the row-alias form, requiring MySQL 8.0.20 or later**
(#344) — @hc12r. `VALUES()` in `ON DUPLICATE KEY UPDATE` has been
deprecated since 8.0.20 (April 2020) and warns on every statement.
Verified against live 8.4, which accepts both and warns only on the old
one. MariaDB has no alias form and is not in the connection catalog.
- **Three MySQL driver options are now accepted** (#344) — @hc12r.
`interpolateParams`, `maxAllowedPacket` and `clientFoundRows`.
`multiStatements` remains deliberately rejected: generated SQL is
interpolated rather than parameter-bound, so allowing multiple statements
per call would turn any escaping defect into arbitrary statement
execution rather than a corrupted value.
### Added
- **Large MySQL writes stream instead of materialising** (#345) — @hc12r.
An append or overwrite into MySQL goes through `LOAD DATA LOCAL INFILE`
fed from an in-process stream: rows travel as data, never as SQL text, and
the worker holds one batch regardless of dataset size. Previously a MySQL
sink always materialised — the bounded-memory property applied to reads on
any driver but writes only on Postgres.
Measured at 1,000,000 rows against MySQL 8.4:
```
LOAD DATA 7.0s ~143k rows/s one batch of memory
INSERT fallback 11.5s ~87k rows/s one batch of memory
statement path 11.0s ~91k rows/s whole dataset + ~60MB of SQL text
```
The statement path's wall time flatters it; its real cost is the memory,
which is what this removes.
The fast path needs `local_infile=1` on the server, which is off by
default in MySQL 8. When the server refuses, Brokoli degrades to batched
`INSERT`s inside the same transaction — same contents, same atomicity,
still one batch of memory. Nothing to configure; the choice is made per
write by asking the server.
- **A same-server MySQL pipeline runs inside the database** (#346) — @hc12r.
Pushdown, previously Postgres-only, now applies to MySQL on the same
terms: every transform rule is proven equivalent by running it both ways
against a live server and comparing, and anything unproven runs in the
engine.
Making the differential corpus shared between backends immediately found a
real bug: the aggregate compiler emitted a bare `GROUP BY`, which is
harmless on Postgres where text equality under a deterministic collation
is byte equality, but MySQL's default collations are case-insensitive and
collapsed `Lisbon` and `lisbon` into one group where the engine keeps two.
Grouping is now byte-wise on both backends; on Postgres the emitted SQL
changed and the results did not.
One difference between backends is documented rather than closed:
`filter_rows` on a boolean column qualifies on MySQL and not on Postgres,
because MySQL's `BOOLEAN` is a `TINYINT` whose driver reports the server's
own `1`, making the comparison numeric on both sides.
- **`sink_api` streams** (#337, #327) — @hc12r. A `sink_api` node consumes
its input in batches rather than materialising it, so posting a large
dataset no longer bounds the run by worker memory. `source_api` cannot
stream and the reason is recorded with it.
### Internal
- **A database backend is described in one place** (#341, #340, #343,
ADR-024) — @hc12r. `pkg/dbdialect` holds what a backend is: quoting,
literal escaping, type classification, the shape probe, and the casts and
collation that make a SQL comparison agree with the Go one. Postgres was
migrated onto it with no behaviour change, and MySQL is the second
implementation. A backend may claim a capability only when a differential
test proves it against the reference path.
One dead parameter turned out to be load-bearing: the pushdown compiler
had threaded a dialect through since it was written, and the only
production caller passed the literal `"postgres"`, correct solely because
an upstream gate refused everything else first.
- **The live-database tests run in CI** (#339) — @hc12r. Postgres and MySQL
service containers, so the differential suites that back every equivalence
claim run on every push rather than on one laptop. They had only ever run
locally.
## [0.10.70] - 2026-08-24
An integer of a million or more could be changed as it moved between nodes,
and on some pipelines that happened silently. That is the item to act on, and
it was found while benchmarking the other one: a pipeline whose source and
sink are on the same server now runs inside the database.
### Fixed
- **Integers stopped surviving a streamed node boundary** (#334) — @hc12r.
Reproduced against the v0.10.69 tag. A `source_db` to `sink_db` copy of a
million rows failed:
```
copy: ERROR: invalid input syntax for type integer: "1e+06"
```
Values moved between nodes as NDJSON, and the decode never asked
`encoding/json` to preserve number syntax, so every number returned as a
`float64`. Rendering one with `%v` is not the text that went in:
```
{"id":1000000} -> float64 -> "1e+06"
{"big":9007199254740993} -> float64 -> "9.007199254740992e+15"
```
The second line is the worse one — the value changed, not its spelling.
The failing case is the lucky one. Against an integer column Postgres
rejects it, which is loud and safe. Against a **text** column it is accepted
and stored, so the pipeline succeeds and the data is wrong; and anything
past 2^53 loses digits wherever it lands. The same encoding backs resume
artifacts and code-node input staging, so a value could change identity
between a run and its resume. Any table with a million rows has ids in this
range, so this is the ordinary shape of the data rather than an edge case.
Fixed at the decode sites rather than in the renderers: the original text is
preserved and values normalise to what the materialising path already
produces. The code node's own output decode is covered too, so a Python
script returning `1000000` no longer has it rendered as `1e+06`.
`copyEscape` also formats floats explicitly instead of falling through to
`%v`. Decoding should make that unreachable, but a renderer that can
silently change a value is not worth leaving on the strength of "should" —
that is exactly how this survived, since the CSV writer had already patched
the symptom for file sinks and database sinks kept it.
**Upgrading does not repair data already written; re-running the pipeline
does.** The runs worth checking are the ones that succeeded against text
columns, not the ones that failed.
One consequence, recorded rather than hidden: JSON cannot distinguish `150`
from `150.0`, and the decoder resolves that towards an integer so a bigint
id survives. A float that happens to be integral therefore comes back as an
integer. Nothing downstream can observe it — the CSV and `COPY` writers
render both identically, aggregation accepts both, and a `number` type
assertion matches both.
### Added
- **A same-server pipeline runs inside the database** (#324, ADR-023) —
@hc12r. Where a `source_db` and a `sink_db` name the same server and the
transform between them can be expressed as SQL, the segment becomes one
`INSERT INTO … SELECT`. The engine issues a statement and reads a row count;
no row crosses the worker.
Measured on 1,000,000 rows between two tables on one Postgres server:
```
read through the engine 12s
run in the database 2s
```
Both wrote identical data — a four-column join matches on every row, with
none unique to either side. The win is time rather than memory; streaming
had already bounded memory. What disappears is parsing and re-encoding a
million rows to move them somewhere they already were.
There is nothing to switch on and no setting on a node. The engine decides,
the way it already decides whether to stream, and reads rows the ordinary
way whenever it cannot prove the two produce the same result.
It refuses by default and the conditions are strict: the same host, port,
database, user and password as written, without DNS resolution; Postgres
only; one consumer per node; `append` or `overwrite`; and every transform
between them already proven equivalent by comparison against a live
database. Two records reaching the same machine under different names are
treated as different servers, and so is the same server reached as two
different roles — running one role's query inside another's write is not a
performance question.
`BROKOLI_DATA_PLANE=interpreted` forces every pipeline back onto the
reading path, for incidents.
A pushed-down segment says so in the run log, naming the nodes the database
absorbed and the rows written. Every node in the segment is credited with
that count: those nodes did produce the rows, they simply never held them,
so reporting zero for a source would answer "how many rows did this read"
with a number true of the engine and false of the pipeline.
- **The SQL compiler the above depends on** (#323, #332) — @hc12r. A
transform's plan compiles to SQL as a second backend beside the Go
executor. Only rules whose two backends can be shown to agree are compiled:
`drop_columns` and `rename_columns`, `filter_rows` on a text or numeric
column, and `count`, `min` and `max` grouped by text columns.
Brokoli's transform semantics are not SQL semantics, which is why the list
is short. A value is compared as its rendered text, a missing value renders
as `<nil>` and is compared like any string, and an ordered comparison is
numeric only when both sides parse as numbers. The generated SQL reproduces
that rather than doing the idiomatic thing — casting to double precision so
the same precision is lost, comparing text under the C collation because Go
compares bytes, and resolving a NULL row to the constant the Go matcher
would produce.
Every rule runs both ways against a live Postgres and the rows are compared;
rules the compiler declines are asserted to be declined, so nothing starts
being pushed down without a passing comparison. `sum` and `avg` are refused
because floating-point addition is not associative and the database may
aggregate in a different order than the engine reads — the same four values
total `1` one way and `0` the other. Grouping by a numeric column is refused
because Brokoli groups on rendered text, so `10.50` and `10.5` are two
groups where SQL sees one.
- **ADR-023** (#333) — @hc12r. Records the decision the above implements:
a pipeline edge carries a reference whose *kind* says where the data is, and
whether the engine needs to read it at all, so that work can stay where the
data already is. Includes the resolutions for staging lifetime, resume
semantics, outbound safety, row counts, and declared validation.
## [0.10.69] - 2026-08-24
Streaming reaches `source_file`, and two bugs that had been hiding behind the
fact that it did not. Both of those were live in v0.10.68: one let a node type
an organisation's plan blocks execute anyway, and one silently discarded the
edits a Python code node made to its rows. Neither announced itself.
### Security
- **A node type blocked by an organisation's plan executed anyway on the
streaming path** (#321) — @hc12r. The gate lived inside `runNodeLogic`. The
streaming path dispatches through `runNodeStreamed`, which never consulted
it, so a gated node ran whenever its upstream produced a reference and the
run reported plain success.
What kept it hidden is that enforcement had become a side effect of a
memory-management decision: the same pipeline was gated below the streaming
threshold and ungated above it, and every node type that learned to stream
widened the hole a little further. `source_db` streaming since v0.10.67 was
already enough to reach it; teaching `source_file` to stream made an
ordinary two-node pipeline take that path, which is how it surfaced.
The check now sits at the one point both execution paths pass through.
Keeping a copy inside each is the arrangement that failed.
### Fixed
- **A code node's edits to its rows were silently discarded when the input
arrived by reference** (#321) — @hc12r. Reproduced against the v0.10.68 tag:
```python
for r in rows:
r["name"] = r["name"].upper()
output_data = {"columns": columns, "rows": rows}
```
produced the original rows, unchanged, and the run succeeded.
`_LazyRows.__iter__` returned a fresh generator on every iteration, parsing
fresh dicts out of the NDJSON file. The script mutated those, they were
dropped at the end of the pass, and the output writer re-read the untouched
file. The harness called that the untouched lazy passthrough, which held
until the object handed back was one the script had written to. It needs no
unusual code to hit — any streamed input feeding the most ordinary Python
idiom there is.
Edits are now journaled to disk: a row written to during a pass is appended
to a journal as the pass moves past it, and the journal becomes the file
later passes read. Rows before the first edit are byte-copied rather than
re-encoded, and a pass abandoned early carries the unread remainder over
verbatim, so the dataset stays complete either way. Only scripts that
actually mutate pay for it; a read-only pass never opens a journal, and
memory stays constant in both cases.
Editing a row after the pass has already moved past it cannot be placed
correctly and now raises, with an explanation of what to do instead, rather
than being dropped. The previous test suite covered four ways to read a
by-reference input and none to write to one, which is how this shipped.
### Added
- **`source_file` streams, so a large CSV no longer kills the worker** (#321)
— @hc12r. Measured on identical pipelines differing only in their source
node, 1,000,000 rows writing a 112 MB CSV under a 512 MiB hard container
limit: `source_file` was OOM-killed after 3 seconds (exit 137) where
`source_db` completed in 17 MiB. Without a hard cap, `source_file` peaked at
653 MiB resident for a 108 MB file, roughly six times the file on disk —
`GOMEMLIMIT` does not prevent it, being a soft target the process grows past
before the cgroup intervenes.
The file is now read into the blob store batch by batch and the node returns
a reference. The same run finishes at 17 MiB with output byte-identical to
the `source_db` run, and the full chain — `source_file` into a `code` node
editing every row into `sink_file` — completes inside the same 512 MiB limit
with all 1,000,000 edits applied.
CSV only, for a structural reason rather than an unfinished one: JSON's
column set is the union of keys across every object in the file and is not
known until the last one is parsed, so a single-pass reader would have to
guess the columns from the first batch or read the whole file to find them.
Formats that cannot stream keep the materialising path.
### Removed
- **`pkg/loaders/streaming_loader.go`** (#321) — @hc12r. 258 lines
implementing exactly this feature, with four passing tests and no caller
outside its own package. It could not be adopted as it stood: it signalled
errors by sending a nil row and discarding the error, so a truncated file
was indistinguishable from a complete one; it stored `""` where the
materialising loader stores `nil` for an empty field, so whether streaming
engaged would have changed the data; it took no context, so a cancelled run
leaked the reader until EOF; and it sent one row per channel operation where
the consumer wants batches. Each of those is now a test on the replacement.
## [0.10.68] - 2026-08-24
A pass over the connection subsystem and the authentication middleware.
Nearly everything here shares one shape: something was true of the system
and nothing said so — a connection that was not encrypted, a credential
that had been destroyed, a request that had never been authenticated, a
build that could not open the database type it offered.
### Security
- **A server could serve every permission-gated route with no
authentication at all** (#316) — @hc12r. `NewServer` mounts the
authentication middleware only `if userStore != nil`, and `serve.go`
left it nil on two paths: a store whose `RawDB()` is not a `*sql.DB`,
silently, and a `NewUserStore` error, behind a `WARNING` it then
continued past. Neither is exotic — `NewUserStore` fails when the users
table cannot be created, which is what a database still accepting no
connections produces, and this control database did it repeatedly under
load while the fix was being written.
With no middleware mounted, no request carries claims, and
`HasPermission` read a missing claim as open mode and returned `true`.
Demonstrated: an unauthenticated request through a `settings.edit` gate
returns `200`. Unlike the count bug in #306 this does not clear when the
database recovers — the middleware is never mounted, so the process
serves everything to everyone for its whole lifetime.
Fixed at both layers, either of which closes it. Open mode is now a mark
`JWTAuth` deliberately sets when it passes an unconfigured system
through, and `HasPermission` allows an unauthenticated request only when
that mark is present; a request that never met the middleware carries no
mark and is denied. Separately, a user store that cannot be built is
fatal — a container that exits gets restarted once the database is
ready, where one that starts without authentication stays that way until
somebody reads the log. A genuinely fresh install still bootstraps, which
is asserted by test alongside the bypass.
-