CHANGELOG.md
Rama main · réplica de lectura del repositorio original.
# Release 2.1.0
## What's New
* [Connect-V2: Sessionless SDK Dial](#connect-v2-sessionless-sdk-dial) - SDKs can dial services without a per-service controller session; the router authorizes the dial locally against the Router Data Model
* [ZAC Bootstrapping CLI](#zac-bootstrapping-cli) - CLI commands to download, configure, and serve the Ziti Admin Console without hand-editing YAML
* [Verify Traffic with ext-jwt-signers](#verify-traffic-with-ext-jwt-signers) - `ziti ops verify traffic` can authenticate with an ext-jwt-signer (OIDC) to test the certless ephemeral-cert path
* [Cluster Quorum Recovery](#cluster_quorum_recovery) - A mechanism for recovering clusters that have irrevocably lost the ability to form a quorum
* [Quickstart Cluster](#quickstart-cluster) - `ziti run quickstart cluster` brings up a multi-node HA cluster in a single command for testing and development and learning
* [Fully Connected Controller Mesh](#fully-connected-controller-mesh) - Controllers now proactively keep the cluster mesh fully connected
* [Controller Advertise Address Updates](#controller-advertise-address-updates) - Changing an HA controller's `advertiseAddress` and restarting now updates the address the cluster and routers use
* [Wildcard OIDC Issuers](#wildcard-oidc-issuers) - Controllers with a wildcard server-certificate SAN can serve OIDC for explicitly allow-listed hostnames
* [Controller Managed Router Configuration](#controller-managed-router-configuration) - (beta) Routers can take configuration from the controller and apply it at runtime, starting with link listeners, dialers and heartbeats, with automatic collection of links the new config leaves stale
* [Multiple LAN Interfaces for tproxy](#multiple-lan-interfaces-for-tproxy) - `lanIf` now accepts a single interface or a list of interfaces
* [Multiple Resolver Addresses for tproxy](#multiple-resolver-addresses-for-tproxy) - `resolver` now accepts a single address or a list of addresses
* [DNS Upstream Query Modes](#dns-upstream-query-modes) - choose how multiple DNS upstreams are queried: parallel fan-out (default) or serial fail-through
* [Controller Read Throughput Under Load](#controller-read-throughput-under-load) - a bbolt upgrade lifts a ceiling on concurrent read transactions that could stall a busy controller
* [Policy Evaluation Without Store Scans](#policy-evaluation-without-store-scans) - creating, updating or deleting a policy or a policy target no longer scans every entity or policy in the store
* [Logging Now Uses slog with an Async Handler](#logging-now-uses-slog-with-an-async-handler) - Logging moves to Go's `log/slog` behind an asynchronous sink; output is unchanged by default, with new flags to tune buffering
* [Build Flags](#build-flags) - A build of the controller can name the build time choices it was made with, and clients can read them from `/version`
* [Security Advisories](#security-advisories) - Eight security advisories, plus the two control-plane certificate validation fixes first released in 2.0.2
## Security Advisories
This release addresses nine security advisories. It also includes the two control-plane certificate and
identity validation fixes first released in 2.0.2 and 1.6.18; anyone upgrading from 2.0.1 or earlier is
picking those up here for the first time. See the linked GitHub Security Advisories for full details, impact,
and affected versions.
* [GHSA-q8g9-jc4c-jp6q](https://github.com/openziti/ziti/security/advisories/GHSA-q8g9-jc4c-jp6q) (High) - The controller buffered the entire body of every
inbound request before any authentication check and with no size cap, so an unauthenticated client could
exhaust controller memory, and crash it, by sending parallel large-body requests to endpoints such as
enrollment.
* [GHSA-j952-6x8x-jmj6](https://github.com/openziti/ziti/security/advisories/GHSA-j952-6x8x-jmj6) (High) - The unauthenticated legacy enrollment path buffered
the request body a second time, allocating twice the memory per request and roughly halving the bandwidth
needed to drive the controller out of memory. Amplifies GHSA-q8g9-jc4c-jp6q.
* [GHSA-p6gx-g438-rjc8](https://github.com/openziti/ziti/security/advisories/GHSA-p6gx-g438-rjc8) (High) - The identity routes handed out the live enrollment
token and JWT of any identity, including admins. The identity detail and list embed the outstanding
enrollment, and the identity enrollments subresource applied no scoping, so a non-admin holding only
identity or enrollment read access could read an admin identity's enrollment secret and redeem it through
the public enrollment endpoints, minting a certificate or password authenticator on that admin identity
and escalating to admin. The top level `/enrollments` routes were already scoped.
* [GHSA-hhm9-wf63-g7qj](https://github.com/openziti/ziti/security/advisories/GHSA-hhm9-wf63-g7qj) (Medium) - When accepting an incoming router-to-router link, a
router verified the dialing router's identity against the whole presented certificate chain instead of the
leaf certificate whose key the TLS handshake proved. An attacker holding enrolled router credentials could
present another router's certificate as filler and be admitted on a link under that router's identity,
letting it intercept, inject, drop, or strand the circuits routed over that link.
* [GHSA-7868-235p-7497](https://github.com/openziti/ziti/security/advisories/GHSA-7868-235p-7497) (Medium) - The controller did not validate the API session
token when creating a circuit via CreateCircuitV3, taking the dialing identity from a router-supplied
header instead. An attacker holding enrolled router credentials could create circuits on behalf of any
identity permitted to dial the service through that router, without that identity having authenticated,
yielding data-plane access under an impersonated identity. The same gap meant expired and revoked API
sessions were not caught at circuit creation.
* [GHSA-4h58-w989-xgg4](https://github.com/openziti/ziti/security/advisories/GHSA-4h58-w989-xgg4) (Medium) - The token-based enrollment endpoint skipped
audience and issuer validation when the request carried a `ziti-token-issuer-id` header, so an attacker
holding any unexpired JWT signed by a configured external JWT signer, even one minted for a different
audience, could enroll a new identity onto the network.
* [GHSA-6v5r-p2wr-q492](https://github.com/openziti/ziti/security/advisories/GHSA-6v5r-p2wr-q492) (Medium) - The current-api-session certificates endpoint
performed an unscoped list, so any authenticated user could read the API session certificates (subject
DNs, fingerprints, and full PEM chains) of all identities, not just their own.
* [GHSA-whjr-3j94-gw3c](https://github.com/openziti/ziti/security/advisories/GHSA-whjr-3j94-gw3c) (Medium) - A JWKS endpoint URL configured on an external JWT
signer was fetched server-side with no timeout, private-range blocking, or allowlist, letting a caller with
external-jwt-signer management access make the controller issue requests to arbitrary internal URLs,
including cloud metadata endpoints (SSRF).
* [GHSA-354c-gpg9-j988](https://github.com/openziti/ziti/security/advisories/GHSA-354c-gpg9-j988) (Low) - With promptOnWake or promptOnUnlock enabled on an MFA
posture check, the edge router dereferenced a nil wake/unlock timestamp while locally evaluating an
authorized client's dial or bind, panicking and crashing the router (data-plane denial of service).
* [GHSA-mrpr-756c-xm47](https://github.com/openziti/ziti/security/advisories/GHSA-mrpr-756c-xm47) (Critical) - Improper peer certificate validation on the controller
cluster mesh, router links, and metrics endpoint. TLS peer checks accepted a connection when any presented
certificate chained to the trusted CA while taking the peer identity from the leaf certificate, allowing a
peer to be admitted under a forged identity without possessing a trusted key. On HA/clustered controllers
this allows joining the controller cluster as an arbitrary controller.
* [GHSA-cc5m-7mhm-xh9f](https://github.com/openziti/ziti/security/advisories/GHSA-cc5m-7mhm-xh9f) (Medium) - Control-channel connections carrying a channel-type header
bypassed router certificate and identity verification, allowing an attacker that can reach the controller
control port to be admitted as an arbitrary router identity and manipulate that router's fabric terminators,
faults, and circuit routing. Impact is limited to router data model metadata (service and identity names) and
control-plane manipulation; it does not by itself grant access to the services the network protects.
* [GHSA-ccq6-vc5v-4ghh](https://github.com/openziti/ziti/security/advisories/GHSA-ccq6-vc5v-4ghh) (Low): The controller and the router
wrote complete service session tokens into structured log fields during edge control operations, several of
them at Error or Info, so the values appeared in logs at default verbosity during ordinary operation. A
service session token does not authenticate a client on its own, because the controller validates it only
against a matching api session token, but logging it placed one half of a credential pair into logging
infrastructure, which carries its own access controls, retention, backups and exports. These log events now
carry the session id, which is not a credential.
## Connect-V2: Sessionless SDK Dial
Ziti SDKs built on sdk-golang v2 can now dial a service without first obtaining a
per-service session token from the controller. With Connect-V2 the router authorizes the
dial locally against its copy of the Router Data Model, the same policy and posture data
the controller already distributes. Dropping the per-dial session round-trip to the
controller lowers dial latency and controller load, most noticeably for identities that
dial many services.
There is nothing to configure. Connect-V2 is negotiated automatically: a router advertises
the capability, an SDK that supports it uses the new path, and older SDKs transparently
continue to use the session-based dial. Policy and posture checks are still enforced on
every dial; they simply run against the Router Data Model on the router instead of during
session creation on the controller.
Connect-V2 requires both a router and an SDK from this release (or newer). Mixed networks
are fine: a dial uses Connect-V2 only when both ends support it, and otherwise falls back
to the session-based path.
## ZAC Bootstrapping CLI
Three new commands make it easy to get the Ziti Admin Console running without manual file editing.
**Download ZAC:**
```
ziti ops console download [version] --location <dir>
```
Downloads a ZAC release from GitHub and extracts it into the given directory. `version` defaults to
`latest`. The target directory must be empty or not yet exist; the command never removes or overwrites
existing files.
**Configure a controller to serve it:**
```
ziti ops console configure <controller-config.yml> --all --location <dir>
```
Edits the controller config YAML in place, adding or updating the `spa` web-listener binding so the
controller serves ZAC from the given directory. Selects listeners with `--all` or `--name`; prompts
interactively when neither is given. The file's comments and structure are preserved.
**Serve it locally:**
If you prefer to serve the console locally rather than have a controller host it, you can serve ZAC
straight from the ziti CLI:
```
ziti run console --location <dir>
ziti run console --version latest
```
Serves the ZAC SPA over HTTPS on `127.0.0.1:8443`. Generates a self-signed cert automatically if none
is supplied. The browser points ZAC at whatever controller you choose inside the console itself.
To serve with your own certificate on a specific address and port:
```
ziti run console --location <dir> --bind-address 0.0.0.0 --port 9443 \
--tls-cert ./server.pem --tls-key ./server.key
```
**Bootstrap it with the quickstart:**
```
ziti run quickstart --zac
```
Adding `--zac` to the quickstart downloads ZAC and configures the controller to serve it in one step,
so the console is available at `https://<ctrl-address>:<ctrl-port>/zac` as soon as the network is up.
The assets install under `<home>/console` by default (override with `--zac-location`), and
`--zac-version` selects the release (defaults to `latest`). Re-running against an existing `--home`
reconciles the binding, so adding `--zac` on a later run enables the console on a network that was first
brought up without it. `ziti run quickstart cluster --zac` installs the assets once and serves the
console from every node.
Two related quickstart re-run fixes ship alongside this:
* Re-running against an already-initialized `--home` now adopts the controller and router ports/addresses
the environment was first created with, instead of falling back to the flag defaults. Previously a
re-run that omitted `--ctrl-port` would wait on the default port while the controller listened on the
original one.
* Re-running against an existing `--home` with a flag that only takes effect when the environment is
first created (for example `--ctrl-port`) now reports that the flag was ignored, rather than silently
dropping it.
## Verify Traffic with ext-jwt-signers
`ziti ops verify traffic` can now authenticate with an ext-jwt-signer (OIDC), so the certless
ephemeral-certificate path can be traffic-tested.
* `--ext-jwt-signer <name>` selects the signer to authenticate with and generate a cert from.
* `--ext-jwt-redirect-url` sets the OIDC redirect URL (default `http://localhost:20314/auth/callback`).
## Cluster Quorum Recovery
A new offline CLI command, `ziti ops cluster recover <controller-config>`, lets
operators rebuild a stuck HA controller cluster after losing quorum. Use it when
enough controllers are permanently gone that `ziti ops cluster add` and
`ziti ops cluster remove` fail with "no leader" — for example, a 2-node cluster
where one node is unrecoverable, or a 3-node cluster that lost two nodes at once.
The command must be run while the surviving controller process is stopped. It
reads the same controller config the controller would, opens the raft data
directory, calls `raft.RecoverCluster` to force the configuration down to a
single local node, and aligns the FSM-tracked member list and snapshot data so
no stale peers leak through on restart. After it succeeds, restart the
controller and add new peers normally with `ziti ops cluster add`.
### End-to-End Encryption (e2ee) Improvements
* Allow hosting-side crypto material to be generated on per connection basis (instead of per terminator)
## Quickstart Cluster
`ziti run quickstart cluster` stands up a multi-node HA controller cluster with a single command, for
testing, learning, and local development. It launches one quickstart child process per node (default 3,
minimum 3, configurable with `--size`), initializes the first
node, joins the rest, and prints a banner once the whole cluster is online listing each node's controller
address, router address, process id, and per-node log file.
Each node runs as its own operating-system process, so you can stop, restart, or attach a debugger to any
single node to explore HA behavior without disturbing the others. The banner prints the exact
`ziti run quickstart` command needed to start each node by hand.
Lifecycle mirrors the single-node quickstart. Pass `--home` for a persistent cluster you can stop and start
again: restarting against an existing `--home` rejoins the existing cluster rather than re-initializing it,
and the nodes start together to re-form a quorum. Omit `--home` to run from a temporary directory that is
removed on a clean shutdown. Pressing Ctrl-C stops every node.
## Fully Connected Controller Mesh
In an HA cluster, controllers form a mesh of channel connections that raft uses to
communicate. Previously we followed the raft library's lead: connections were made as
needed to allow elections, and after that only the leader maintained connections to its
followers. That is enough for raft itself, but it means most nodes have no direct link to
most other nodes.
Now controllers keep the mesh fully connected. This gives better visibility into system
state from any node, not just the leader, and it provides a baseline for building
additional non-raft coordination features on top of the mesh.
Each controller runs a `PeerDialer` that proactively dials every known cluster member and
works to keep the mesh fully connected. Failed dials are retried with exponential backoff,
and when two controllers dial each other at the same time a deterministic tie-break (based
on their SPIFFE IDs) keeps a single connection rather than a redundant pair. The dialer's
state can be inspected with `ziti fabric inspect ctrl-peer-dialer`.
The dialer is tunable via a new optional `cluster.dialer` config section. All values have
defaults, so no configuration change is required:
```yaml
cluster:
dialer:
minRetryInterval: 1s # minimum time between dial retries
maxRetryInterval: 1m # maximum time between dial retries
retryBackoffFactor: 2.0 # multiplier applied to the retry interval after each failure
fastFailureWindow: 30s # if a connection is lost within this window, apply backoff instead of resetting the retry delay
dialTimeout: 10s # maximum time a single dial attempt may run
scanInterval: 30s # period of the full scan that reconciles dial state against membership
queueCheckInterval: 5s # how often expired entries are popped from the retry queue
```
A related `cluster.nonMemberGrace` setting (default `1m`) controls how long a leader will
let a TLS-valid but non-member controller stay connected to the mesh before dropping it.
This gives a controller that is being added to the cluster time to be accepted as a member
before its connection is reaped.
## Controller Advertise Address Updates
In an HA cluster, a controller's `ctrl.options.advertiseAddress` was only read when the cluster was
initialized or the node joined. After that the address lived in the raft configuration, so changing it in
the config file did nothing: other controllers kept dialing the old address, and routers and new enrollment
JWTs kept getting it. The usual way into this is putting a proxy in front of the ctrl listener, on a
different port, after the cluster already exists. A single-node cluster had no way out short of
re-bootstrapping.
The config file is now the source of truth. On startup, if a controller's configured advertise address
differs from the one stored in the cluster, it asks the leader to update it, retrying until the change
lands. The member keeps its voter or non-voter status, and connected routers get the new address through
the usual cluster membership update.
The leader refuses the change if another member already has the address, or if a different controller
answers there. If the leader can't reach the new address at all, it makes the change anyway and logs a
warning, since some controllers only dial out.
Running `ziti agent cluster add` or `ziti fabric cluster add` for an existing member at a new address also
updates its address in place, instead of removing and re-adding it.
The leader must be running this version for the update to happen. An older leader ignores the request and
the stored address stays as it is. The requesting controller logs a warning and keeps retrying, so the
change goes through once the leader is upgraded.
## Controller Managed Router Configuration
Routers can now be configured from the controller. A config is attached to a router the same way one
is attached to a service, the controller distributes it through the Router Data Model, and the router
applies it at runtime without a restart.
**Status: beta.** The data model, distribution, and the first config type are in place and usable.
The shape of the config types may still change. See `doc/design/ctrl-managed-router-config.md` for
the full design.
The pieces below build on each other, in order.
### Config Type Target Field
Config types now have an optional `target` field that indicates what kind of entity the config type
is intended for. Valid values are `"service"`, `"router"`, and `"other"`. The field is set on creation
and is immutable afterward.
The `target` field distinguishes config types meant for services, config types meant for routers,
and config types meant for other purposes, which keeps UIs, APIs, and validation clean.
A database migration sets `target = "service"` on all existing config types. Services and identity
service config overrides now require that referenced configs have a config type with
`target = "service"`.
The CLI has been updated to support the new field:
* `ziti edge create config-type` now accepts a `--target` flag
* `ziti edge list config-types` now shows a `Target` column
### Router Configs
Routers (edge, transit, and fabric) now have a `configs` field that holds a list of config IDs the
router should use, associating routers with configs in the same way services already are.
Validation rules:
* Every config referenced by a router must use a config type with `target = "router"`. Configs
with `target = "service"` (or anything else) are rejected.
* A router may reference at most one config per config type. Attempting to attach two configs of
the same type is rejected with a duplicate-config error naming both configs.
* Deleting a config automatically removes it from the `configs` list of any router that referenced
it, so dangling references are not possible.
The `configs` field is exposed on router create, update, patch, and detail responses across the
edge, transit, and fabric router REST APIs.
The CLI has been updated to support the new field:
* `ziti edge create edge-router` accepts `--config <id>` (repeatable)
* `ziti edge create transit-router` accepts `--config <id>` (repeatable)
* `ziti edge update edge-router` accepts `--config <id>` to replace the router's config list
* `ziti fabric create router` accepts `--config <id>` (repeatable)
* `ziti fabric update router` accepts `--config <id>` to replace the router's config list
### How Routers Consume Configs
**Routers opt in, per config type.** A router accepts nothing from the controller unless its own
config file says so, and it chooses which types it will accept:
```yaml
managedConfig:
allow:
- router.link
```
An entry naming a type family accepts every version of it, so `router.link` covers `router.link.v1`
and any later version. An exact type name works too. The special entry `all` accepts every type:
```yaml
managedConfig:
allow:
- all
```
Omitting the `managedConfig` section, or leaving `allow` empty, disables controller-managed config
entirely. That is the default, so upgrading a router never changes where its configuration comes
from. Configs of a type the router has not allowed are ignored.
Once a config is allowed, the router receives it through the Router Data Model and hands it to the
subsystem that owns its type. It is applied in place: the subsystem reconciles its running state
against the new version rather than the router restarting.
Local YAML wins. Where a router's own config file already configures a subsystem, that subsystem
ignores the controller's config for as long as the local one is present, so moving a subsystem to
controller management is a deliberate act of removing its local configuration. A router with no
local configuration for a subsystem uses whatever the controller sends.
An apply that fails is rolled back to the previously-applied config, and a failure that cannot be
rolled back raises an alert to the controller rather than leaving the router in an unknown state.
### First Config Type: `router.link.v1`
`router.link.v1` is the first controller-managed config type, covering a router's link listeners and
dialers. It is built in, so no config type has to be created before use.
A config looks like this:
```json
{
"listeners": [
{
"binding": "transport",
"bind": "tls:0.0.0.0:6004",
"advertise": "tls:router-east.example.com:6004",
"groups": ["east"]
}
],
"dialers": [
{ "binding": "transport", "groups": ["east", "west"] }
],
"heartbeats": { "sendInterval": "10s", "checkInterval": "1s", "closeUnresponsiveTimeout": "60s" },
"gcMode": "orphaned"
}
```
To put one on a router, create the config, attach it, and allow the type in the router's own config
file (see above):
```
ziti edge create config east-links router.link.v1 "$(cat east-links.json)"
ziti edge update edge-router router-east --config east-links
```
Applying one reconciles the router's link surface in place: listeners whose definitions changed are
rebuilt, listeners that are unchanged are left running, and dialers are replaced. A change confined
to fields that do not describe a listener or dialer does not disturb either. Every successful apply
is logged at info level as `applied link configuration`, with the generation and what changed, so
you can confirm a router received a config even when the change asked nothing of its links.
For links, local config means the router's config file has a `link` section at all. Anything in
it, even only heartbeat or queue settings, takes precedence over the controller's `router.link`
config for the whole type. A `link` section with no listeners or dialers therefore leaves the
router with no links, and the router warns at startup when that happens. To hand a router's links to
the controller, remove its `link` section.
Removing the config from a router removes its link surface entirely.
### Stale Link Collection
A link can outlive the configuration that created it. Change a listener's advertise address or a
dialer's groups and the established link keeps carrying traffic, even though nothing in the current
configuration would re-establish it. These links are "stale": they work until they break, and then
they are gone for good.
`router.link.v1` takes an optional `gcMode` controlling what a router does about them after a config
change:
* `preserve` (default) never closes a link on the basis of the router's own configuration.
* `orphaned` closes links whose supporting listener or dialer is entirely absent.
* `changed` closes links whose supporting listener or dialer still exists but has moved, such as a
renamed advertise address.
`gcMode` covers local staleness only. A peer withdrawing or moving a listener already closes the
affected link under every mode, because the peer decides what it will accept, so `preserve` is not a
guarantee that links survive every change.
**Removing a router's link config closes its links**, whatever `gcMode` said. A removal leaves no
listeners and no dialers, so every remaining link is one the router can never re-establish.
For operators, `ziti ops verify stale-links` reports every link in the network against both
endpoints' current configuration, and `--gc` closes the ones both endpoints confirm as stale. It
judges a router against whatever configuration that router is actually running, so it is useful on
routers configured from local YAML as well as controller-managed ones.
A link is only closed when both of its endpoints agree it is stale and both judged the same
incarnation of it. An endpoint that is offline, too old to answer, or outside `--filter` leaves the
link reported as partial and never closes it.
### Link Heartbeat and Queue Settings
`router.link.v1` also carries the link heartbeat timing (`sendInterval`, `checkInterval`,
`closeUnresponsiveTimeout`) and the `payloadSenderQueueSize` / `ackSenderQueueSize` queue sizes.
Heartbeat timing is applied to links that are already established, retuning them in place, rather
than only reaching links formed after the change. It is applied as a single generation, so a link
never pairs one version's interval with another version's timeout.
**Queue sizes apply to new links only.** `payloadSenderQueueSize` and `ackSenderQueueSize` are read
when a link is built, so changing them has no effect on links that are already established. Those
keep the queues they were built with until they are re-established for some other reason. A change
confined to queue sizes does not trigger stale-link collection, since nothing about an existing link
has become stale.
The timing is validated on both the controller and the router, and some combinations that were
accepted before are now rejected. `closeUnresponsiveTimeout` must be greater than `sendInterval +
checkInterval`, because a responsive link's own heartbeat gap already reaches that sum, and it must
be at least 10s.
**A `closeUnresponsiveTimeout` under 30s now takes effect.** The close check previously sat behind a
fixed 30s threshold, so any shorter timeout silently behaved as 30s. A router configured with a
short timeout will start closing unresponsive links sooner than it did before.
The 10s minimum is there because a healthy connection can legitimately go quiet for several seconds
while TCP recovers from ordinary packet loss, and a timeout inside that window turns routine loss
into a closed link and the cost of re-establishing it. A controller config below the minimum is
rejected. A value below it in a router's own YAML is raised to the minimum with a warning instead,
so a router that started yesterday still starts today.
**Link `sendInterval` and `checkInterval` in a router's own YAML now take effect.** Links
previously ran a fixed 10s send and 1s check interval whatever the config said, reading only
`closeUnresponsiveTimeout`. Because those values were never in force, local timing that fails the
validation above is not allowed to stop a router from starting: the router logs a warning and uses
the defaults instead. That includes an explicit `0s`, which was previously treated as if the field
were absent.
## Wildcard OIDC Issuers
Controllers can serve OIDC for hostnames covered by a wildcard server-certificate SAN. Prior to 2.1,
wildcard SANs were excluded from the set of valid OIDC issuers, so `/oidc/*` requests to a
wildcard-covered hostname returned `404`.
Wildcard SANs cannot be used as a literal OIDC issuer (`https://*.example.com/oidc` is not a usable URL).
Instead, the `edge-oidc` API binding now accepts an `allowedHostnames` option listing the exact hostnames
(covered by the wildcard) that may be served as issuers. If omitted, the wildcard contributes no issuers
and the controller logs a warning at startup; a malformed entry (a non-string value, or one containing a
wildcard character) is a startup error:
```yaml
web:
- name: client-management
apis:
- binding: edge-oidc
options:
allowedHostnames:
- ctrl.example.com
```
Each entry must be an exact hostname (no patterns) that an active server-certificate SAN actually covers;
entries are matched against wildcard SANs using standard X.509 hostname rules. The resulting OIDC issuers
are therefore concrete, fixed hostnames, so the set of valid `iss` values stays closed. Concrete
(non-wildcard) SANs continue to be served as issuers automatically and do not need to be listed.
## Multiple LAN Interfaces for tproxy
The `lanIf` option in `xgress_edge_tunnel` tproxy configs now accepts either a
single string (as before) or a YAML list of interface names. For each intercepted
service address the router inserts one iptables `ACCEPT` rule per configured
interface. Existing single-interface configs are unchanged.
```yaml
# single interface (unchanged)
- binding: tunnel
options:
mode: tproxy
lanIf: enp0s5
# multiple interfaces
- binding: tunnel
options:
mode: tproxy
lanIf:
- enp0s5
- enp0s6
```
The CLI `--lanIf` flag also accepts a comma-separated list or repeated flags:
```bash
ziti tunnel tproxy --lanIf enp0s5,enp0s6
# or
ziti tunnel tproxy --lanIf enp0s5 --lanIf enp0s6
```
## Multiple Resolver Addresses for tproxy
The `resolver` option in `xgress_edge_tunnel` tproxy configs now accepts either a
single string (as before) or a YAML list of `udp://` addresses. A single shared
resolver instance handles all listeners so hostname→IP mappings remain consistent
across interfaces. Existing single-address configs are unchanged.
```yaml
# single address (unchanged)
- binding: tunnel
options:
mode: tproxy
resolver: udp://172.18.102.70:53
# multiple addresses
- binding: tunnel
options:
mode: tproxy
resolver:
- udp://172.18.102.70:53
- udp://192.168.10.1:53
```
## DNS Upstream Query Modes
When more than one DNS upstream is configured for tproxy host-side resolution, the
resolver previously always fanned out: every upstream was queried in parallel and
the first NOERROR response won. For a high-volume router with several upstreams this
multiplies outbound DNS traffic, since every request hits every upstream.
A new `dnsUpstreamMode` option (router config) / `--dnsUpstreamMode` flag
(`ziti tunnel`) controls how multiple upstreams are queried:
* `parallel` (default) - query all upstreams at once; first NOERROR wins. Unchanged
behavior.
* `serial` - query upstreams one at a time in order, starting at the first. Fail
through to the next only on timeout, transport error, or a non-NOERROR response.
* `failover` - like `serial`, but the upstream that last answered is reused as the
starting point, so a healthy upstream isn't re-probed from the top on every query.
* `random` - like `serial`, but each query starts at a randomly chosen upstream.
In all serial modes a healthy upstream costs exactly one outbound query regardless of
how many upstreams are configured. The single-upstream case is unaffected.
In a router's `xgress_edge_tunnel` tproxy config:
```yaml
- binding: tunnel
options:
mode: tproxy
dnsUpstreamMode: serial
dnsUpstream:
- udp://10.96.0.10:53
- tcp://8.8.8.8:53
```
The standalone `ziti tunnel` gains a matching `--dnsUpstreamMode` flag (default
`parallel`):
```
ziti tunnel run --dnsUpstreamMode serial \
--dnsUpstream udp://10.96.0.10:53 \
--dnsUpstream tcp://8.8.8.8:53
```
## Controller Read Throughput Under Load
This release picks up bbolt v1.5.0, which removes a scalability limit that could
stall a controller serving a high rate of reads.
Every controller read - a service list, a policy or posture check, a service
lookup during circuit creation - runs in a bbolt read transaction, and bbolt
guards transaction bookkeeping with a single global mutex that each transaction
takes twice, once on open and once on close. In earlier versions, closing a
transaction held that mutex while linearly scanning the list of all open read
transactions to find itself. The cost of that scan grew with the number of
concurrent readers, so as read concurrency rose the lock was held longer, which
in turn let more readers pile up behind it. Under sustained load the result is a
convoy: thousands of goroutines waiting on one mutex, a machine that looks fully
busy while little work completes, and request timeouts that are not explained by
the amount of actual work being asked of the controller.
bbolt v1.5.0 drops the scan and tracks the open-transaction count directly, so
the critical section no longer grows with concurrency. Controllers most likely to
notice are those with many identities refreshing sessions at once, where the
service-list and policy query rate spikes.
The same release also coalesces contiguous page ranges when a transaction's freed
pages are merged back into the freelist, instead of merging one page at a time.
This mostly benefits transactions that free a large amount of data at once.
## Policy Evaluation Without Store Scans
Every write to a service, identity, edge router, posture check or policy re-evaluates which
policies link to which entities, so the denormalized access tables stay current. Until now that
evaluation scanned. Creating a service, or changing its role attributes, walked every service
policy and every service edge router policy; creating or deleting a service policy walked every
service and every identity, three times over on delete. None of that work found anything for a
policy that names its targets
by `@id`, which is how automation typically builds policies, but it ran inside the single bbolt
write transaction all the same. On a network with a few hundred thousand policies, each of those
writes took seconds, the command queue filled, and the controller answered with 429s and 503s while
mostly idle.
Evaluation now visits only the entities and policies that can be affected by a change: the ones
already linked, the ones named by id, the ones named by a role attribute the entity carries, and
the policies that name `#all`. A policy that names `#all` still visits every target entity, since
every entity is affected. The cost of a write is now proportional to what it touches, not to the
size of the store.
To find policies by what they reference, evaluation uses the role-attribute index each policy
store already keeps for the role-attribute usage queries, which now also records the `#all`
wildcard under the key `all`. The usage queries skip that key: a policy cannot reference an
attribute named `all`, since `#all` is always the wildcard. Policies that name an entity by `@id`
need no index: such a policy is linked to the entity for as long as it exists, so the entity's own
links already lead to it. This is database version 50; the first start of an upgraded controller
adds the wildcard entries to the existing indexes and logs a `policy role-attribute index rebuild
summary` line per index. The same migration rebuilds the posture check to service tables that drive
service-updated events when a posture check changes. They were only maintained when a policy's
posture checks changed, not when its services did, and their service-side entries had been written
into the services' identity tables. The rebuild logs a `rebuilt posture check to service links`
line. No configuration changes.
## Logging Now Uses slog with an Async Handler
The controller, router, and `ziti tunnel` now log through Go's standard
`log/slog` library behind a single asynchronous sink. By default the output is
unchanged - the same human-readable format as before - so no configuration
change is required. Existing `pfxlog`/`logrus` log statements continue to work;
they are bridged into the new sink rather than rewritten, so the migration to
slog is gradual and nothing is lost in the meantime.
The motivation is performance under load. Previously every log call contended on
a single process-wide formatter-plus-writer mutex, which could block many
goroutines at once. Writing is now handed to a background goroutine, off the hot
path of the code doing the logging.
### Asynchronous writes and dropped records
Because writes are buffered through a bounded queue, behavior under saturation
is the one change worth knowing about:
* Records at or above a configurable block threshold (default `warn`) block the
caller until there is room, so warnings, errors, and fatal messages are never
silently dropped.
* Lower-priority records (`info`, `debug`, `trace`) are dropped when the queue is
full instead of blocking. Whenever drops occur, a summary line is emitted
periodically reporting how many records were dropped per level, so the loss is
always visible in the logs.
Under normal load nothing is dropped; this only engages when log volume outruns
the writer.
### New CLI flags
`ziti controller run`, `ziti router run`, and `ziti tunnel` gain three optional
flags to tune the async sink. All have sensible defaults, so leaving them unset
preserves current behavior:
* `--log-queue-size` (default `4096`) - capacity of the async log queue.
* `--log-block-threshold` (default `warn`) - lowest level that blocks rather than
dropping under saturation (`panic|fatal|error|warn|info|debug|trace`).
* `--log-summary-interval` (default `5s`) - how often the dropped-record summary
line is emitted.
The existing `--log-formatter` flag is unchanged: `pfxlog` (default), `json`
(unchanged JSON shape), and `text` (logrus-style `key=value`).
### Per-channel log levels are slog-only
`ziti agent set-channel-log-level <name> <level>` now adjusts only code that has
been migrated to the new slog loggers. Call sites still using `pfxlog.Logger()`
or `pfxlog.ChannelLogger(...)` continue to follow the global level, so because
most call sites are not yet migrated, per-channel overrides have limited reach
today and expand as packages are converted. The global `ziti agent set-log-level
<level>` still affects everything.
## Build Flags
A controller binary can now carry a list of build flags: short names for the
build time choices it was made with. The `/version` endpoint returns them in a
new `buildFlags` field, and `ziti version -v` prints them, so a client or an
operator can tell what a running controller was built to do without asking the
person who built it.
Flags are set with the Go linker at build time. There is one symbol, and one
build owner composes the whole list:
```
go build -ldflags "-X github.com/openziti/ziti/v2/common/build.buildFlags=ALPHA,BRAVO_MODE" ./ziti
```
Names are uppercase letters, digits, and underscores. Anything else in the list
is dropped rather than served. A second `-X` against the same symbol replaces
the first rather than adding to it, and the linker silently ignores an `-X`
whose symbol path it cannot resolve, so a typo in that path yields a binary with
no flags rather than a build error.
Releases from this repository set no flags, so `buildFlags` is empty and `ziti
version -v` prints no build flags line. The names themselves are defined by
whoever produced the build; a client should ignore any it does not recognize.
Build flags are not capabilities. `capabilities` on `/version` describes what
the controller offers over the API, is defined by this repository, and is
enumerated at `/enumerated-capabilities`. Build flags describe how the binary
was built, are open-ended, and are enumerated nowhere.
## Current Beta Features
Beta features are still under development and are subject to change. They should
be usable in their released form. Though unlikely, there is a small chance they will
be removed.
* Basic Permission System
* Alert Events
* Controller-Initiated Control Channel Dials
* Controller Managed Router Configuration
## Deprecated Features
Deprecated features still work, but are no longer recommended and will be removed
in a future release. Plan to migrate off them.
* **Legacy sessions** - The legacy session model, both API sessions and service
sessions, is deprecated and will be removed in OpenZiti 3.0, along with the
supporting infrastructure built around it.
* **Split links** - The split-channel link implementation (separate payload and
ack connections) is a fallback used only when dialing a router that doesn't
support multi-underlay links. The multi-underlay link implementation is a
superset of what split links provide, so nothing is lost by moving off them.
The deprecation proceeds in two steps: in OpenZiti 3.0 the `split` option will
be interpreted as a multi-underlay link with one ack and one payload channel,
retiring the separate split implementation; in OpenZiti 4.0 the `split`
configuration option will be removed entirely.
## Removed Features
* **Link `costTags`** - The `costTags` option on router link listeners has been
removed. It was parsed from router config, advertised to the controller, and
stored on the router model, but was never used for path selection or any other
behavior. Routers ignore the setting if it is still present in config, and the
corresponding protobuf fields are now reserved, so no migration is required
beyond optionally deleting the setting.
* **Edge router API proxy** - The `edge.apiProxy` router option, which served the
controller's Edge API from a router listener, has been removed. Certificate
authentication through the proxy did not work, and the proxy doesn't fit the
planned end-to-end encryption and certificate proof of possession work.
Routers log a warning and ignore the section if it is still present in config.
Clients that used the proxy should connect to the controller directly, or
through a load balancer that passes TLS through.
## Contributors
Thanks to the community members who contributed to this release.
* [@msbusk](https://github.com/msbusk) diagnosed the circuit leak in
[#4184](https://github.com/openziti/ziti/issues/4184) and validated the fix against a
production workload.
## Component Updates and Bug Fixes
* github.com/openziti/channel/v5: [v4.3.11 -> v5.0.39](https://github.com/openziti/channel/compare/v4.3.11...v5.0.39)
* [Issue #315](https://github.com/openziti/channel/issues/315) - Hellos with a value other than 4 bytes under the ReplyFor header key are rejected
* [Issue #311](https://github.com/openziti/channel/issues/311) - A queued message can be discarded unwritten after Send reports success
* [Issue #301](https://github.com/openziti/channel/issues/301) - ReplyFor header with a length other than 4 causes a nil pointer dereference on the rx path
* [Issue #303](https://github.com/openziti/channel/issues/303) - Crafted frame lengths panic the decoder: int truncation on 32-bit, uint32 wrap everywhere
* [Issue #283](https://github.com/openziti/channel/issues/283) - Convert internal logging from pfxlog/logrus to slog
* [Issue #288](https://github.com/openziti/channel/issues/288) - Remove unused reconnecting and memory implementations
* [Issue #285](https://github.com/openziti/channel/issues/285) - Version negotiation responses are never recognized, so the protocol version can never be renegotiated
* [Issue #287](https://github.com/openziti/channel/issues/287) - Dialer retries a hello when the retry cannot change the outcome
* [Issue #280](https://github.com/openziti/channel/issues/280) - MultiListener: no way to decline a grouped channel without stranding the dialer
* [Issue #275](https://github.com/openziti/channel/issues/275) - Multi-listener attaches reconnecting underlays to a closing channel under churn
* [Issue #271](https://github.com/openziti/channel/issues/271) - Treat MinTotalUnderlays as a first-class multi-underlay signal
* [Issue #274](https://github.com/openziti/channel/issues/274) - Add configurable reconnect jitter to BackoffDialPolicy
* [Issue #258](https://github.com/openziti/channel/issues/258) - Add a hook to inject hello headers derived from the peer's certificate
* [Issue #267](https://github.com/openziti/channel/issues/267) - NewSingleChannelWithUnderlay panics on underlays with nil headers (e.g. websocket)
* [Issue #269](https://github.com/openziti/channel/issues/269) - Make channel logging pluggable via injectable slog.Logger
* [Issue #264](https://github.com/openziti/channel/issues/264) - Multi-underlay group reconnect can reject-storm under load
* [Issue #265](https://github.com/openziti/channel/issues/265) - Support reconfiguring heartbeat intervals on a running channel
* [Issue #261](https://github.com/openziti/channel/issues/261) - Add ContentTypeReceiver, a self-describing receive handler
* [Issue #250](https://github.com/openziti/channel/issues/250) - BackoffDialPolicy cannot (re)establish a grouped channel: never sets IsFirstGroupConnection
* [Issue #252](https://github.com/openziti/channel/issues/252) - BackoffDialPolicy misclassifies multi-underlay constraint fill as short-lived/flapping
* [Issue #247](https://github.com/openziti/channel/issues/247) - Config.Binder exposes unexported `*channelImpl` in its public signature
* [Issue #255](https://github.com/openziti/channel/issues/255) - Flaky Test_MultiUnderlayChannels: CloseRandom can close the last required underlay
* [Issue #253](https://github.com/openziti/channel/issues/253) - Multi-underlay channel delays below-Min closure when a dial/backoff is in progress
* [Issue #246](https://github.com/openziti/channel/issues/246) - classic_dialer leaks underlay FD when hello handshake fails
* [Issue #241](https://github.com/openziti/channel/issues/241) - Allow calling LoadOptions on an Options instance
* [Issue #238](https://github.com/openziti/channel/issues/238) - Channelv5
* github.com/openziti/edge-api: [v0.31.0 -> v0.36.1](https://github.com/openziti/edge-api/compare/v0.31.0...v0.36.1)
* [Issue #198](https://github.com/openziti/edge-api/issues/198) - Advertise edge router capabilities in the service edge-router list
* github.com/openziti/foundation/v2: [v2.0.91 -> v2.0.105](https://github.com/openziti/foundation/compare/v2.0.91...v2.0.105)
* [Issue #502](https://github.com/openziti/foundation/issues/502) - logging registry forwards records its root handler would decline
* [Issue #499](https://github.com/openziti/foundation/issues/499) - AtomicBitSet has no set-and-return-previous operation
* [Issue #494](https://github.com/openziti/foundation/issues/494) - Add package-level Panic logging helper
* [Issue #489](https://github.com/openziti/foundation/issues/489) - Add graceful shutdown and idle-wait support to goroutines.Pool
* [Issue #488](https://github.com/openziti/foundation/issues/488) - Add package-level Fatal and SyncEmit helpers to logging
* [Issue #484](https://github.com/openziti/foundation/issues/484) - Add slog logging core (foundation/v2/logging) for upstream libraries
* github.com/openziti/identity: [v1.0.129 -> v1.0.144](https://github.com/openziti/identity/compare/v1.0.129...v1.0.144)
* [Issue #72](https://github.com/openziti/identity/issues/72) - Convert logging from pfxlog/logrus to foundation slog logging
* [Issue #74](https://github.com/openziti/identity/issues/74) - golangci-lint CI fails on go 1.25 module (action installs golangci-lint v1.x)
* github.com/openziti/runzmd: [v1.0.90 -> v1.0.97](https://github.com/openziti/runzmd/compare/v1.0.90...v1.0.97)
* [Issue #58](https://github.com/openziti/runzmd/issues/58) - Variables are not expanded correctly in code blocks
* github.com/openziti/sdk-golang/v2: [v1.7.0 -> v2.0.0-pre5](https://github.com/openziti/sdk-golang/compare/v1.7.0...v2.0.0-pre5)
* [Issue #1048](https://github.com/openziti/sdk-golang/issues/1048) - Write-failed payload is written to the peer as empty data, failing encrypted reads with invalid input
* [Issue #1036](https://github.com/openziti/sdk-golang/issues/1036) - Closing an SDK-hosted xgress conn before its circuit starts blocks for the circuit start timeout
* [Issue #1041](https://github.com/openziti/sdk-golang/issues/1041) - Cost and precedence updates on a hosted service are lost when its listener rebinds
* [Issue #1031](https://github.com/openziti/sdk-golang/issues/1031) - Service refresh retry panics in rand.Int63n when RefreshInterval is 10s or less
* [Issue #1054](https://github.com/openziti/sdk-golang/issues/1054) - Xgress conn closed locally ignores the router's StateClosed and retransmits forever
* [Issue #977](https://github.com/openziti/sdk-golang/issues/977) - Add DirectE2EE header constants and error enum values
* [Issue #1018](https://github.com/openziti/sdk-golang/issues/1018) - Legacy edge conn ignores a StateClosed that arrives after a FIN
* [Issue #1004](https://github.com/openziti/sdk-golang/issues/1004) - UnmarshallPacketPayload indexes the buffer without bounds checks
* [Issue #1006](https://github.com/openziti/sdk-golang/issues/1006) - Chunked payload reassembly trusts the peer's declared size
* [Issue #1011](https://github.com/openziti/sdk-golang/issues/1011) - secretstream: rekey is not implemented, so streams diverge from libsodium peers
* [Issue #1002](https://github.com/openziti/sdk-golang/issues/1002) - Bring the secretstream library into this repo
* [Issue #999](https://github.com/openziti/sdk-golang/issues/999) - Dial for an unknown edge conn id is dropped silently, costing the client its full connect timeout
* [Issue #991](https://github.com/openziti/sdk-golang/issues/991) - Read deadline on ReadAdapter permanently closes the peer's send buffer
* [Issue #987](https://github.com/openziti/sdk-golang/issues/987) - LinkSendBuffer leaks goroutines after send-half close
* [Issue #958](https://github.com/openziti/sdk-golang/issues/958) - ConnectV2 xgress client conn not marked closed on router-initiated teardown
* [Issue #967](https://github.com/openziti/sdk-golang/issues/967) - Move RouterCapabilityConnectV2 constant into edge_client.proto
* [Issue #951](https://github.com/openziti/sdk-golang/issues/951) - Add an SDK acceptance-test framework
* [Issue #952](https://github.com/openziti/sdk-golang/issues/952) - xgress client half-close not delivered to legacy edge hosts
* [Issue #936](https://github.com/openziti/sdk-golang/issues/936) - Implement Connect-V2: sessionless SDK dial
* [Issue #945](https://github.com/openziti/sdk-golang/issues/945) - Migrate to channel/v5
* [Issue #948](https://github.com/openziti/sdk-golang/issues/948) - AddControllerUrlsUpdateListener remover unsubscribes the wrong event
* [Issue #941](https://github.com/openziti/sdk-golang/issues/941) - Prep for channel v5: explicit receive handler registration, drop send priorities
* [Issue #924](https://github.com/openziti/sdk-golang/issues/924) - Make controller http timeout configurable, with a default of 30s
* [Issue #925](https://github.com/openziti/sdk-golang/issues/925) - Switch controllers on a broader set of errors
* [Issue #926](https://github.com/openziti/sdk-golang/issues/926) - Refresh OIDC token using a window to avoid race conditions and herding
* [Issue #927](https://github.com/openziti/sdk-golang/issues/927) - Apply exponential backoff to auth retry attempts
* [Issue #932](https://github.com/openziti/sdk-golang/issues/932) - API Session Certificate chain is not preserved
* github.com/openziti/secretstream: [v0.1.49 -> v0.1.53](https://github.com/openziti/secretstream/compare/v0.1.49...v0.1.53)
* github.com/openziti/transport/v2: [v2.0.215 -> v2.0.228](https://github.com/openziti/transport/compare/v2.0.215...v2.0.228)
* [Issue #176](https://github.com/openziti/transport/issues/176) - tcp.Connection hides CloseWrite, so callers cannot half-close a TCP transport connection
* [Issue #173](https://github.com/openziti/transport/issues/173) - Convert logging from pfxlog/logrus to foundation slog logging
* github.com/openziti/xweb/v3: [v3.0.4 -> v3.0.6](https://github.com/openziti/xweb/compare/v3.0.4...v3.0.6)
* github.com/openziti/ziti/v2: [v2.0.0 -> v2.1.0](https://github.com/openziti/ziti/compare/v2.0.0...v2.1.0)
* [Issue #4705](https://github.com/openziti/ziti/issues/4705) - Version built-in config types independently of the database version
* [Issue #4712](https://github.com/openziti/ziti/issues/4712) - Identity PUT re-enables a disabled identity
* [Issue #4713](https://github.com/openziti/ziti/issues/4713) - Terminator PUT clears hostId, peerData and sourceCtrl
* [Issue #4714](https://github.com/openziti/ziti/issues/4714) - Fabric router PUT crashes the controller
* [Issue #4655](https://github.com/openziti/ziti/issues/4655) - Router doesn't tell SDKs to stop sending on circuits it no longer has
* [Issue #4763](https://github.com/openziti/ziti/issues/4763) - miniziti fails to install Traefik because chart v42.3.1 rejects service.type
* [Issue #4552](https://github.com/openziti/ziti/issues/4552) - transport_udp and proxy_udp listeners can deadlock their own event loop
* [Issue #4592](https://github.com/openziti/ziti/issues/4592) - CLI id-or-name resolution is duplicated, and the fabric version misreads names starting with 'id'
* [Issue #4478](https://github.com/openziti/ziti/issues/4478) - Router never reconnects to a standalone controller after the heartbeat check closes the control channel
* [Issue #4626](https://github.com/openziti/ziti/issues/4626) - Backward compatibility smoke test is stale and doesn't run in Validation (All)
* [Issue #4627](https://github.com/openziti/ziti/issues/4627) - Cancelled CI fablab runs leave AWS instances and VPCs running
* [Issue #4573](https://github.com/openziti/ziti/issues/4573) - Router can drop controllers on a follower's stale controller list and never re-add them
* [Issue #4737](https://github.com/openziti/ziti/issues/4737) - ziti agent writes channel's info logs to stderr, failing the Helm controller readiness probe
* [Issue #4513](https://github.com/openziti/ziti/issues/4513) - Policy evaluation scans every entity or policy on each write
* [Issue #4514](https://github.com/openziti/ziti/issues/4514) - Changing a policy's semantic without changing its roles does not re-evaluate its links
* [Issue #4515](https://github.com/openziti/ziti/issues/4515) - Posture check to service reference counts are only maintained from the posture check side
* [Issue #4726](https://github.com/openziti/ziti/issues/4726) - Changing a service policy's type clears its posture check roles
* [Issue #4509](https://github.com/openziti/ziti/issues/4509) - API request timeouts surface as TLS 'bad record MAC' errors instead of a 503
* [Issue #4580](https://github.com/openziti/ziti/issues/4580) - Terminator cost, precedence and peer data don't converge at the controller after a takeover or update
* [Issue #4596](https://github.com/openziti/ziti/issues/4596) - ER/T terminator re-create reverts health check cost and precedence
* [Issue #4571](https://github.com/openziti/ziti/issues/4571) - Delete existence is decided before dispatch, and a typed router delete reaches the other router type
* [Issue #4579](https://github.com/openziti/ziti/issues/4579) - ziti tunnel host mode: contextProvider.accept dials the upstream and completes the accept inline, so one slow connection blocks every later dial to that service
* [Issue #4559](https://github.com/openziti/ziti/issues/4559) - Identity admin protection is decided from the serving controller's view before dispatch
* [Issue #4560](https://github.com/openziti/ziti/issues/4560) - Granular permissions are granted to a session before MFA completes
* [Issue #4561](https://github.com/openziti/ziti/issues/4561) - The default admin's isAdmin flag can be cleared
* [Issue #4394](https://github.com/openziti/ziti/issues/4394) - Router does not relay an SDK xgress terminator's half-close to tunneler, proxy, or embedded SDK initiators
* [Issue #3588](https://github.com/openziti/ziti/issues/3588) - Sync a controller's raft advertise address with its config on startup
* [Issue #4531](https://github.com/openziti/ziti/issues/4531) - Validate commands on the raft leader before they enter the log, and halt on an entry a member cannot decode
* [Issue #4538](https://github.com/openziti/ziti/issues/4538) - Forwarded API errors lose their structured cause, and duplicate-name or referenced-delete errors return 500 from a follower
* [Issue #4548](https://github.com/openziti/ziti/issues/4548) - Identity creation with authenticators and entity renames swallow their errors
* [Issue #4489](https://github.com/openziti/ziti/issues/4489) - Router panics sending a link state update to a controller it has unregistered
* [Issue #4503](https://github.com/openziti/ziti/issues/4503) - Router pins a CPU core reporting metrics while no controller is registered
* [Issue #4524](https://github.com/openziti/ziti/issues/4524) - Allow case-insensitive terminator addressing via host.v1
* [Issue #4645](https://github.com/openziti/ziti/issues/4645) - Remove the edge router API proxy
* [Issue #4422](https://github.com/openziti/ziti/issues/4422) - Proxied client certificate authentication is dead code and unsafe to repair as is
* [Issue #4406](https://github.com/openziti/ziti/issues/4406) - Router data model root CA public keys carry different intermediates on each controller
* [Issue #4611](https://github.com/openziti/ziti/issues/4611) - Controller deadlocks when posture data changes for an identity with legacy API sessions
* [Issue #3482](https://github.com/openziti/ziti/issues/3482) - Compilation fails on Illumos
* [Issue #4542](https://github.com/openziti/ziti/issues/4542) - Disabling a connected router deadlocks the controller <!-- keep -->
* [Issue #4543](https://github.com/openziti/ziti/issues/4543) - Router deadlocks syncing subscribers when a subscribed identity was deleted or disabled while out of sync
* [Issue #4453](https://github.com/openziti/ziti/issues/4453) - Router gives up announcing its links to a controller after a reconnect, leaving the controller missing links
* [Issue #4454](https://github.com/openziti/ziti/issues/4454) - Checking whether a half-built split link is closed panics
* [Issue #4455](https://github.com/openziti/ziti/issues/4455) - A dialed link the router has no state for is left up and never reported
* [Issue #4456](https://github.com/openziti/ziti/issues/4456) - Controllers can hold stale link listeners for a router using controller-managed link config
* [Issue #4442](https://github.com/openziti/ziti/issues/4442) - Controller panics on tunnel v2 dial to an unknown service and on the default admin check when the db is not open
* [Issue #4402](https://github.com/openziti/ziti/issues/4402) - ziti edge login -f succeeds, every command after it returns 401
* [Issue #4156](https://github.com/openziti/ziti/issues/4156) - SDK terminator removal can enter a self-sustaining retry storm under heavy churn
* [Issue #4155](https://github.com/openziti/ziti/issues/4155) - Router mis-signals congestion on terminator establish/remove timeouts
* [Issue #4378](https://github.com/openziti/ziti/issues/4378) - Api session enforcer delete meter only marks when the batch delete fails
* [Issue #4434](https://github.com/openziti/ziti/issues/4434) - Update to Go 1.27
* [Issue #4413](https://github.com/openziti/ziti/issues/4413) - debian controller & router packages hang on post-install script when installed via ansible
* [Issue #4410](https://github.com/openziti/ziti/issues/4410) - REST error responder logs ApiError.Code as a method value
* [Issue #4349](https://github.com/openziti/ziti/issues/4349) - Router process check reports configured hashes as the expected signer fingerprints
* [Issue #4348](https://github.com/openziti/ziti/issues/4348) - Single-process posture check with no signer fingerprint always fails at the router
* [Issue #4005](https://github.com/openziti/ziti/issues/4005) - Auto-GC stale router links
* [Issue #4246](https://github.com/openziti/ziti/issues/4246) - Router connect scans the entire link table, making a reconnect wave scale with the cube of the router count
* [Issue #4365](https://github.com/openziti/ziti/issues/4365) - REST requests no longer reset the legacy API session timeout
* [Issue #4350](https://github.com/openziti/ziti/issues/4350) - Policy enforcers leak a metrics reference on every run
* [Issue #4369](https://github.com/openziti/ziti/issues/4369) - Router control channel connectivity events can be delivered out of order
* [Issue #4323](https://github.com/openziti/ziti/issues/4323) - Router posture cache grows without bound as clients re-authenticate
* [Issue #4321](https://github.com/openziti/ziti/issues/4321) - Process hashes and signer fingerprints are compared case-sensitively at the router
* [Issue #4314](https://github.com/openziti/ziti/issues/4314) - Router posture cache corrupts process posture state when applying posture responses
* [Issue #4318](https://github.com/openziti/ziti/issues/4318) - Mac and Domain posture checks panic when the client has not reported that data
* [Issue #4319](https://github.com/openziti/ziti/issues/4319) - OS posture check with no versions fails at the router, passes at the controller
* [Issue #4320](https://github.com/openziti/ziti/issues/4320) - Router does not normalize MAC addresses from posture responses
* [Issue #4222](https://github.com/openziti/ziti/issues/4222) - OIDC cert authentication always reports isCertExtendable as true, including for 3rd Party CA certificates
* [Issue #4094](https://github.com/openziti/ziti/issues/4094) - Edge router rejects valid first-party client certificates when the edge signing CA differs from the ctrl-channel CA
* [Issue #4304](https://github.com/openziti/ziti/issues/4304) - Support build time provided capabilities in the /version capabilities list
* [Issue #4296](https://github.com/openziti/ziti/issues/4296) - Bind message with a short cost header panics the router
* [Issue #4199](https://github.com/openziti/ziti/issues/4199) - Add in-place upgrade smoke test
* [Issue #4291](https://github.com/openziti/ziti/issues/4291) - Admin MFA removal doesn't replicate across HA cluster members
* [Issue #4202](https://github.com/openziti/ziti/issues/4202) - wss edge listener rejects all clients on 2.0: certValidatingIdentity forces client-cert verification on a listener that intentionally sets NoClientCert
* [Issue #4184](https://github.com/openziti/ziti/issues/4184) - Router leaks LinkSendBuffer goroutines in `drainDeadlines()` — circuits accumulate until the router OOMs
* [Issue #4278](https://github.com/openziti/ziti/issues/4278) - fabric inspect data-model-index doesn't move for writes outside the router data model
* [Issue #4196](https://github.com/openziti/ziti/issues/4196) - Router control-channel connect/disconnect race can leave a reconnected router de-registered
* [Issue #4247](https://github.com/openziti/ziti/issues/4247) - Router never closes a control channel whose controller stops answering heartbeats, leaving recovery to a ~15 minute TCP timeout
* [Issue #4248](https://github.com/openziti/ziti/issues/4248) - Router startup check samples the controller count at a single instant and kills a router that is momentarily between control channels
* [Issue #4004](https://github.com/openziti/ziti/issues/4004) - Manage router link configuration via the controller
* [Issue #4145](https://github.com/openziti/ziti/issues/4145) - Upgraded controller rejects legacy clients' existing sessions and gives no recovery signal for invalid service tokens
* [Issue #4125](https://github.com/openziti/ziti/issues/4125) - Legacy create-session signs service JWT with a mismatched session id after dedup
* [Issue #4160](https://github.com/openziti/ziti/issues/4160) - Leaderless controller strands terminator operations during cluster membership changes
* [Issue #4204](https://github.com/openziti/ziti/issues/4204) - ziti run/tunnel ignore PFXLOG_NO_JSON and lost the terminal-aware log-format default
* [Issue #4234](https://github.com/openziti/ziti/issues/4234) - Ensure terminator operations are scoped by source router
* [Issue #4240](https://github.com/openziti/ziti/issues/4240) - Legacy v1 create-circuit handler crashes the controller on JWT-prefixed tokens
* [Issue #4232](https://github.com/openziti/ziti/issues/4232) - logging.Install should route stdlib slog.Default() into ziti's sink
* [Issue #4141](https://github.com/openziti/ziti/issues/4141) - Service-policy enforcer deletes valid legacy sessions; type= queries use numeric id against the string-mapped symbol
* [Issue #4118](https://github.com/openziti/ziti/issues/4118) - Overlapping JWKS kids across ext-jwt-signers cause intermittent primary auth failures (GetIssuerByKid collision)
* [Issue #4063](https://github.com/openziti/ziti/issues/4063) - External JWT enrollment fails when a configured role attributes claims selector is absent from the JWT
* [Issue #4124](https://github.com/openziti/ziti/issues/4124) - Managed-auth External JWT Signer created without network-id/managed tags on OpenZiti v2.0.0 controllers
* [Issue #4206](https://github.com/openziti/ziti/issues/4206) - Lock order inversion in ConnectionTracker deadlocks the controller
* [Issue #4193](https://github.com/openziti/ziti/issues/4193) - ER/T terminator create reports "invalid edge router for session" with no router-side error and slow recovery
* [Issue #4079](https://github.com/openziti/ziti/issues/4079) - DisconnectRouter reroutes circuits before marking the router disconnected
* [Issue #4149](https://github.com/openziti/ziti/issues/4149) - upgrading a running 1.x controller/router to 2.x fails to create the service user
* [Issue #3990](https://github.com/openziti/ziti/issues/3990) - Expose service change subscriptions to external SDKs over protobuf
* [Issue #4108](https://github.com/openziti/ziti/issues/4108) - Controller retains bbolt-managed memory past transaction (create-circuit response SIGSEGV)
* [Issue #4067](https://github.com/openziti/ziti/issues/4067) - JWKS peer signer kid is undecodable raw bytes (should be hex) -- %s vs %x on sha1.Sum in oidc_auth/storage.go KeySet()
* [Issue #4071](https://github.com/openziti/ziti/issues/4071) - Unify router capabilities into a single shared bitmask across control and edge channels
* [Issue #3998](https://github.com/openziti/ziti/issues/3998) - Add router-side managed configuration registry
* [Issue #4069](https://github.com/openziti/ziti/issues/4069) - Controller can cache empty apiAddresses forever due to startup race between raft mesh and xweb config load
* [Issue #4066](https://github.com/openziti/ziti/issues/4066) - bootstrap zac for controller with quickstart
* [Issue #4060](https://github.com/openziti/ziti/issues/4060) - support ext-jwt-signer in ziti verify traffic
* [Issue #4036](https://github.com/openziti/ziti/issues/4036) - Own the metrics wire format (MetricsMessage) in ziti
* [Issue #4138](https://github.com/openziti/ziti/issues/4138) - Add l2 service configuration types <!-- keep -->
* [Issue #4137](https://github.com/openziti/ziti/issues/4137) - ziti tunnel ignores --dnsSvcIpRange <!-- keep -->
* [Issue #3988](https://github.com/openziti/ziti/issues/3988) - Support multiple resolver addresses for tproxy mode <!-- keep -->
* [Issue #3972](https://github.com/openziti/ziti/issues/3972) - Support multiple LAN interfaces for tproxy mode <!-- keep -->
* [Issue #4052](https://github.com/openziti/ziti/issues/4052) - ziti cli cached creds don't use refresh token
* [Issue #3881](https://github.com/openziti/ziti/issues/3881) - Add Capability for DNSUPSTREAMS to be used in serial
* [Issue #4035](https://github.com/openziti/ziti/issues/4035) - Controller /metrics endpoint produces duplicate TYPE declarations causing Prometheus to drop samples
* [Issue #4045](https://github.com/openziti/ziti/issues/4045) - Reduce controller link/router management lock contention under high link churn
* [Issue #3884](https://github.com/openziti/ziti/issues/3884) - Implement Connect-V2: sessionless SDK dial
* [Issue #3929](https://github.com/openziti/ziti/issues/3929) - [Backport-2.0] Router does not enforce api-session or identity revocations on live connections
* [Issue #3841](https://github.com/openziti/ziti/issues/3841) - Controller Cluster - new controllers must be able to be dialed by the leader in order to join successfully
* [Issue #3933](https://github.com/openziti/ziti/issues/3933) - edge enrollment: add the list of controllers to successful enrollment response
* [Issue #3992](https://github.com/openziti/ziti/issues/3992) - Overlay edge-oidc listener panics when its redirect_uri is not in the redirectURIs allow-list
* [Issue #4039](https://github.com/openziti/ziti/issues/4039) - cryptic x509 error when cached CA is stale
* [Issue #4010](https://github.com/openziti/ziti/issues/4010) - Non-admin identity with `enrollment` entity permission can create enrollments for any identity including admins, achieving full privilege escalation to admin
* [Issue #4011](https://github.com/openziti/ziti/issues/4011) - Router deletes terminators ~12m after creation when host SDK replies with wrong inspect content type
* [Issue #4002](https://github.com/openziti/ziti/issues/4002) - make quickstart easier - ZAC
* [Issue #4007](https://github.com/openziti/ziti/issues/4007) - add cluster option to ziti run quickstart
* [Issue #3983](https://github.com/openziti/ziti/issues/3983) - Migrate to channel/v5
* [Issue #3916](https://github.com/openziti/ziti/issues/3916) - Convert router/forwarder to slog
* [Issue #3976](https://github.com/openziti/ziti/issues/3976) - Distribute routers and router-target configs through the RDM
* [Issue #3974](https://github.com/openziti/ziti/issues/3974) - Add router.link.v1 config type
* [Issue #3994](https://github.com/openziti/ziti/issues/3994) - support additional addresses in deployments
* [Issue #3934](https://github.com/openziti/ziti/issues/3934) - Consolidate the fabric and edge services data stores
* [Issue #3910](https://github.com/openziti/ziti/issues/3910) - Install slog and route agent log-level callbacks through common/logging
* [Issue #3927](https://github.com/openziti/ziti/issues/3927) - Router does not enforce api-session or identity revocations on live connections; revoked OIDC sessions keep dialing/hosting until access-token expiry
* [Issue #3906](https://github.com/openziti/ziti/issues/3906) - Add named-logger registry, logrus bridge, and pfxlog-shape JSON
* [Issue #3904](https://github.com/openziti/ziti/issues/3904) - Add slog AsyncHandler in preparation for moving to slog for logging
* [Issue #3902](https://github.com/openziti/ziti/issues/3902) - Add agent IPC capability discovery and channel-based log-level commands
* [Issue #3893](https://github.com/openziti/ziti/issues/3893) - Import openziti/agent library into common/agent
* [Issue #3894](https://github.com/openziti/ziti/issues/3894) - Consolidate duplicated agent channel-upgrade code into common/agent
* [Issue #3952](https://github.com/openziti/ziti/issues/3952) - externalIdClaim on CA returns HTTP 500 with empty body for most matcher/parser combinations
* [Issue #3908](https://github.com/openziti/ziti/issues/3908) - Router posture-data updates don't revoke SDK-hosted xgress circuits or hosted terminators
* [Issue #3780](https://github.com/openziti/ziti/issues/3780) - Add configs field to routers
* [Issue #3961](https://github.com/openziti/ziti/issues/3961) - Router panics evaluating an AnyOf process posture check when client reports no process/OS data
* [Issue #1593](https://github.com/openziti/ziti/issues/1593) - Expanded attribute query support in management API; add policy attribute support and usage count
* [Issue #3949](https://github.com/openziti/ziti/issues/3949) - DeleteById swallows errors when firing change events
* [Issue #3867](https://github.com/openziti/ziti/issues/3867) - Tunneler skips iptables rules for services sharing an intercept hostname
* [Issue #3891](https://github.com/openziti/ziti/issues/3891) - oidc auth fails with wildcard server-cert SANs
* [Issue #3914](https://github.com/openziti/ziti/issues/3914) - ziti login fails with oidc + wildcard certs
* [Issue #3938](https://github.com/openziti/ziti/issues/3938) - Carry the link id in a link header instead of only in the channel identity token
* [Issue #3945](https://github.com/openziti/ziti/issues/3945) - Increase certificate serial number namespace to 159 bits
* [Issue #3942](https://github.com/openziti/ziti/issues/3942) - Prep for channel v5: bind handler invocation, send priorities
* [Issue #3920](https://github.com/openziti/ziti/issues/3920) - ziti pki create csr ignores --key-name flag
* [Issue #3744](https://github.com/openziti/ziti/issues/3744) - Add a target field to config type
* [Issue #3684](https://github.com/openziti/ziti/issues/3684) - Keep controller mesh fully connected, as much as possible
* [Issue #3864](https://github.com/openziti/ziti/issues/3864) - e2ee: allow hosting SDK to return e2ee public key in the dial response
* [Issue #3849](https://github.com/openziti/ziti/issues/3849) - Add a recover mechanism for when a controller cluster can't form a quorum