CHANGELOG.md
Rama main · réplica de lectura del repositorio original.
# Changelog
All notable changes to Ojuri will be documented in this file.
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
## [Unreleased]
## [1.8.1] - 2026-09-29
Four instructions that could not be followed and three defects behind them, found
by running every documented script against a live stack rather than reading it.
### Added
- **`ojuri reset-admin`.** Issues a new admin password and prints it once, with
`--password`, `--username` and `--tenant`. It runs inside the RDA container,
which carries bcrypt, knex and the connection details.
### Fixed
- **The synthetic load generator declined everything it sent.** `npm run
seed:load --count N` omitted the trust context the curated demo rows carry
(`is_authenticated`, `device_is_trusted`, `account_age_days`,
`session_to_txn_seconds`, channel and countries). Every absent field defaults
to its least trusted value, so a run reported `accept=0 decline=20` no matter
what `--fraud-ratio` said, and the decision mix it printed meant nothing. The
generated transactions now carry that context, weaker on the fraud-flavoured
ones: at `--fraud-ratio 0.02` a run now accepts, at 1.0 it sends everything
to review.
- **Both load paths sent the timestamp in seconds** where the predict contract
takes milliseconds, so every generated transaction claimed to be from 1970 and
every calendar feature derived from it was wrong. `feature-builder.ts` already
carried a comment about this exact hazard from a previous occurrence.
- **The two install paths could not migrate each other's database.** knex records
whichever migration filename it loaded, and the two paths load different ones:
the container runs compiled JavaScript from `dist`, a checkout runs TypeScript
from `src` through ts-node. So a database created by `ojuri up` refused
`npm run db:migrate` from a checkout, and the reverse, with
`Error: The migration directory is corrupt, the following files are missing:`
followed by all 38 of them. Nothing was corrupt, and nothing in the message
suggested what to do. Both paths now retarget the recorded names to their own
extension before migrating, which is idempotent and a no-op on a database that
has never been migrated.
- **The decision replay CLI could not replay anything.** `npm run replay`
re-posted each audited transaction under its original `transaction_id`, which
the target rejects as a duplicate, so every row came back as a decision
called `ERROR` and the comparison it exists to print was always empty. Rows
now replay under a `replay-<run>-<id>` key, which keeps the original
recoverable in SQL, and a rejection is reported with its code rather than
flattened into `ERROR`. It also sent the timestamp in seconds where the API
takes milliseconds, putting every replayed transaction in 1970 and skewing
every feature derived from its age.
- **The dashboard's Backtest action printed a command that could not work.**
`npm run replay --target <model version> --since 24h` names a script that
needs a checkout, omits the `--` that npm needs to pass flags through at all,
passes a model version where `--target` takes the URL of the deployment to
replay against, and passes `24h` where `--since` goes into a SQL timestamp
comparison. There is no backtest endpoint behind the menu item; it only ever
printed text. It now describes what actually has to happen.
- **`ojuri up` told you to run a command that cannot exist.** Three lines of its
summary, and the dashboard's own login page, pointed at
`npm run reset:admin` — a script in the repository's `package.json`. An
adopter who installed with `npx @ojuri/cli up` has no repository, so the one
instruction printed for a lost password was impossible to follow. They now
name `ojuri reset-admin`, and say that `ADMIN_SEED_PASSWORD` in `.env` is the
password on a database this stack created.
## [1.8.0] - 2026-09-29
What an install gives you when you touch nothing. `npx @ojuri/cli up` used to
bring up the scoring path and stop there: no dashboard, no drift monitoring, and
nothing in the output saying either was missing. It now starts everything except
FIA. Nothing on the `docker compose` path changed, and an `ojuri.yaml` you
already have is read exactly as written.
### Changed
- **Everything but FIA is on by default.** The Sentinel dashboard and MLA now
start with the stack. Sentinel was the bigger omission: decisions were going
to `decisionAuditLog` and the only way to read them was SQL or the API, on a
product whose dashboard is half of what it is. MLA off meant no drift
monitoring and no retraining, so the model an evaluation judged was the one it
started with. FIA stays off, and is now the only service that is: its language
model is a 7.6 GB download and it wants 16 GB of RAM, which is more than an
install should spend uninvited. `services.fia.enabled: true` and another
`ojuri up` turns it on.
- **RDA defaults to one replica under the CLI**, down from three. Three copies
of the same Node process on one laptop buys nothing an evaluation can measure
and costs three times the memory; raise it when a single process stops keeping
up. A bare `docker compose up` still defaults to three, and `RDA_REPLICAS`
still overrides either way.
- **`ojuri up` prints where to go.** Every URL it started, in one table, with
what each is for: Sentinel, the predict endpoint, Grafana, PAA's stats, MLA's.
The admin block now names the dashboard it signs you into, and says plainly
that Grafana has its own credentials, which the shipped pair still being
`admin / admin` made easy to miss. When FIA is off the output says so, and
gives the two steps to turn it on.
- **Switching a service off in the manifest now removes its container.**
Withholding a Compose profile is not enough: `up -d` simply does not mention
the container and it keeps running, and `--remove-orphans` leaves it too,
because a service in an inactive profile is still a defined service. So
turning MLA, FIA or Sentinel off and re-running `ojuri up` left it running,
with the manifest and `ojuri status` both saying otherwise. Switching
observability off removes Prometheus and Grafana the same way. Volumes are
kept, so turning FIA off for an afternoon does not cost the 7.6 GB download.
Two things worth knowing. A bundled datastore you have pointed at your own is
**named, not removed** — the container holds your data, and that is not this
command's to delete. And Compose projects are named after the directory, so
`ojuri up` in `~/work/ojuri` addresses the same containers as a bare
`docker compose` in a checkout of the same basename: a FIA started there by
hand is one the default manifest considers switched off. The output says what
it removed.
- **Dropped the `sentinel-without-fia` warning.** It described the shipped
default once Sentinel came on, and a warning every install earns is a warning
everybody learns to scroll past. The manifest comment and the `up` output both
say what FIA-off costs.
### Fixed
- **Sentinel's drift and retrain controls could not reach MLA.** With the
dashboard served from the stack, NGINX routed `/mla/` to
`host.docker.internal`, which resolves from inside the container but does not
reach a published port back out, so every call the Settings page makes
(`drift-config`, `retrain`, `retrain-runs`, `stats`) timed out as a 502. The
config that fronts Sentinel now names the `mla` service, as the AWS deploy
config already did. The config used when Sentinel is off is unchanged, since
that is the layout where MLA typically runs on the host.
- **System health showed a red failure for a service that was deliberately
off.** Every probe failure was reported as `DOWN` with the message `fetch
failed`, which is what Node calls every network error. With FIA off by
default, the first health page an operator opened had a red card and a
meaningless error on it. A hostname with no DNS record now reads as
`UNKNOWN` — "not running in this deployment" — and a real failure shows the
underlying cause (`connect ECONNREFUSED …`) rather than the wrapper.
- **The compose stack ran MLA's drift check at a threshold nothing else
agreed with.** `docker-compose.yml` defaulted `DRIFT_F1_THRESHOLD` to 0.92
while MLA's own default, `docs/ARCHITECTURE.md` and `CLAUDE.md` all say 0.4,
and the repo's measured F1 on IEEE-CIS is 0.554. Every windowed check would
therefore read as drift (tracked as OJR-05 in `docs/REVIEW_FINDINGS.md`). It
was latent while MLA was off by default; it is not any more, so the compose
default now matches the documented one.
- **`ojuri --version` printed `1.6.0`.** The number came from a constant kept by
hand next to the one in `package.json`, and the two had been apart since
1.7.0, so anyone reporting a bug reported it against the wrong version. It is
read from the package now, which is the only copy that can be wrong.
### Security
- **`ojuri init` generates `MLA_SERVICE_TOKEN`.** RDA accepts it as a bearer
credential for `models:register` and `models:set_status`, and `.env.example`
ships a published development string long enough to clear RDA's 32-character
floor. It was inert while MLA was off by default. With MLA on it would have
been a live credential, identical on every install, for registering and
activating a model. Existing installs keep whatever their `.env` holds:
rotate it by hand if it still starts with `dev-only`. `ojuri validate`, which
`up` runs, now says so on every run while it holds: a warning outside
production and an error in it, matching how the other published defaults are
treated.
- **`ojuri init` generates `GRAFANA_PASSWORD`.** Grafana publishes 3001 on
every interface and its admin can add a datasource, so `admin / admin` was
not a pair to ship either. As with the MLA token, an existing `.env` is never
rewritten; the `up` summary prints the credentials while they are still the
shipped ones and only the variable names once they are not.
## [1.7.2] - 2026-09-28
Friction found by auditing the paths an adopter actually takes, rather than by
waiting for each one to be reported. No migrations, no schema changes. One
behaviour change worth reading before upgrading: a decision can now only be
reviewed once.
### Fixed
- **A retry could be rejected as a changed request.** The idempotency hash was
taken over the request body as your client happened to serialise it, so the
same transaction sent with its fields in a different order looked like a
different request and the retry came back 422 "reused with a different request
body" — sending you looking for a field that changed when none had. Any client
that rebuilds the body from a map or a database row could hit this. Field order
no longer matters; the order of items in a list still does, because a
reordered list really is a different request. Cached responses from before the
upgrade will not match, so a retry in flight across it is scored again rather
than replayed, for one cache lifetime.
- **Errors said what went wrong only in prose.** Every failure the platform
raises now carries a `code` alongside the message. Three conditions return 409
and four return 503, so the status alone never said what to do next, and the
only way to tell them apart was matching the wording of a message that was
written for a human reading a log. `duplicate_transaction` and
`idempotency_in_flight` are both 409 and mean opposite things. The message
wording is not part of the contract; the code is.
- **A decision could be reviewed twice, and the second review won silently.**
Two reviewers acting on the same transaction both succeeded: the later
decision replaced the earlier one with nothing recording that it had happened,
in the table an auditor reads, and each sent its own `decision.overridden`
webhook and its own label to the learning agent, which then trained on
whichever arrived last. A second override is now refused, naming the reviewer
whose decision stands. Reopening a review is not supported.
- **A repeated webhook could not be recognised as a repeat.**
`X-Webhook-Delivery` changed on every attempt, and deliveries are retried, so
a subscriber had nothing stable to deduplicate on and could act twice on one
event. It is now the same for every attempt at the same delivery.
- **The same investigation report had two shapes.** Whether it looked reviewed
depended on which endpoint returned it.
- **`docs/TROUBLESHOOTING.md` listed a port nothing uses** and omitted two that
are published, the same error the README carried.
## [1.7.1] - 2026-09-28
A fix to the one-command install, which failed on exactly the run it exists
for. Nothing else changes; `@ojuri/sdk` is unaffected and stays at 0.0.1.
### Fixed
- **`npx @ojuri/cli up` failed on a first install, on any connection slow
enough to take ten minutes over the images.** Everything the CLI shelled out
to Docker carried a ten-minute ceiling, and starting the stack pulls roughly
three gigabytes on a fresh host, so the install died partway through with
`spawnSync docker ETIMEDOUT`, no containers created, and nothing explaining
why. Found by installing the published package on a machine with no images
cached; CI had never caught it because its runners fetch well inside the
ceiling.
The ceiling was only half of it. Output was buffered rather than shown, so
the longest part of the install printed nothing at all and read as a hang
even when it was working. The pull is now its own step that renders Docker's
own progress as it happens, and the limit on a step like that is an hour,
there to catch a stuck daemon rather than to bound a download. Reading
container state and logs stays as it was, because that output is parsed.
Building from source skips the pull, since there is nothing published to
fetch, and a failure now points at what Docker printed instead of leaving a
blank where the reason belongs.
## [1.7.0] - 2026-09-28
Additive only. Two new publishable packages, a new optional manifest, no
migrations and no schema changes. Every existing install path is unchanged.
### Added
- **A one-command install, as an additional option.** `npx @ojuri/cli@1.7.0 up`
brings the stack up on a machine with Docker and Node 20 and **no clone**:
it writes `ojuri.yaml` and a `.env` with fresh secrets, unpacks the compose
file and the assets it bind-mounts, pulls the images, waits for the
migration and for RDA to answer `/ready`, then prints a runnable `curl` and
the admin password. Neither existing flow changes: `cp .env.example .env`
plus the two-file Compose invocation, and the build-from-source path, work
exactly as documented.
The new `ojuri.yaml` manifest is a layer over `.env`, not a replacement, and
rendering the committed default produces a byte-identical Compose project to
the one you get today, which CI asserts. It adds `ojuri init`, `up`, `down`,
`status`, `doctor` and `render`, a JSON Schema, and validation that refuses
what the architecture cannot do: `paa.replicas` above 1 is an error, because
a second copy sees half the transaction graph and silently stops catching
rings that span both.
`mla`, `fia` and `sentinel` ship disabled, so a default install starts what
it starts today. FIA stays off because it downloads roughly 7.6 GB of model
weights on first start.
- **`@ojuri/sdk`, a typed client for the predict path.** Adopters calling
`POST /v1/predict` no longer hand-roll `fetch`: `packages/sdk` ships a
zero-dependency, dual ESM/CJS client (Node 18+) covering the predict call
and inbound webhook signature verification, published to npm on release.
It deliberately does not wrap the audit reads or the FIA report endpoints —
those are Sentinel's surface, and the dashboard calls them directly.
`test/contracts/sdk-types.contract.test.ts` compares every copied type and
enum against the server declarations and fails this repo's CI on drift in
either direction.
### Fixed
- **`ReasonCodeDto` omitted `basis`.** The wire DTO had drifted from
`ReasonCode`, which carries the field and is what the response factory
actually passes through, so the documented response shape understated what
`/v1/predict` returns.
- **Deploys served a stale dashboard until the CDN cache lapsed.** Sentinel is
a hashed-bundle SPA: a new image writes a new `index.html`, but CloudFront
kept handing visitors the cached one, pointing at the previous bundle. The
stack was current and every artefact check passed — only the browser was
behind. `ojuri-up` now invalidates the distribution, keyed on the image the
sentinel container is actually running rather than on the floating `:v1` tag,
so a boot that changed nothing costs no invalidation. `VERSIONING.md` records
why `curl -H 'Cache-Control: no-cache'` is the wrong way to check for it.
## [1.6.0] - 2026-08-15
Additive only. One new unauthenticated endpoint, two new optional `.env`
variables that default to off, no migrations, no schema changes.
### Added
- **The sign-in page adapts to a public sandbox.** New unauthenticated
`GET /v1/auth/sign-in-options` reports whether this deployment seeded the
shared `demo` account. Where it did, the form prefills `demo` and explains
that the account is shared and its password published; everywhere else the
page is unchanged, keeping the `admin` prefill and the first-run
`npm run db:migrate` instructions. The answer comes from the `users` table
rather than from `SEED_DEMO_USER`, so a migration that ran before the
variable was set cannot point visitors at an account that does not exist.
The password is never returned. Optional `DEMO_CREDENTIALS_URL` adds a link
to wherever it is published.
## [1.5.1] - 2026-08-11
Bug fixes only. No API surface changes an integrator relies on, no migrations,
no new `.env` variables to set.
### Fixed
- **API keys were never persisted.** Sentinel posted the expiry select's label
(`"30 days"`) where a timestamp was expected. `new Date()` produced an
Invalid Date, Postgres rejected the insert, and because the token is minted
*before* the insert the operator was shown a plausible `fdk_…` key that had
never been stored — every request with it then returned 401. The dashboard now
sends an ISO timestamp, and `expiresAt` is validated: a malformed value
returns 400 naming the field instead of a 500 with the failing SQL in the logs.
- **nginx stopped reaching RDA after any restart or slow boot.** The upstream
resolved `rda:3000` once at startup, so nginx booting before RDA cached an
address that no longer existed. `/v1/*` then 502'd while `/` kept serving the
dashboard — the API looked broken while the UI looked fine. Now resolved per
request through the Docker resolver, as `/fia/` and `/mla/` already were.
- **The login form blamed `AUTH_JWT_SECRET` for unrelated outages.** Any 503 was
reported as a missing secret. RDA does return 503 when the secret is absent,
but so does anything in front of it that is down, so a routing failure sent
operators to fix a correctly-set variable. It now distinguishes the two.
- **The sandbox's auto-stop timers were never enabled.** They were armed after
the stack under `set -e`, so the first boot where the stack failed aborted the
bootstrap before reaching them, leaving an instance with no idle stop and no
nightly stop. Timers are now armed first, and a stack that will not start is
reported rather than fatal.
- **A stack half-started when Kafka lost its health race.** Kafka not being
healthy when compose evaluated `depends_on` aborted RDA, PAA and nginx; Kafka
became healthy moments later but the oneshot unit never retried. Now retries
on failure, with a health-check grace period so the first attempt usually wins.
## [1.5.0] - 2026-08-11
Ships a one-instance AWS deployment and fixes three defects found while
standing it up — two of which made services unrunnable rather than
merely awkward. The MLA and FIA fixes change the published images, so
adopters on the floating `:v1` tag receive them automatically.
### Added
- **AWS test environment (`deploy/aws/`)** — Terraform for a single EC2
instance running the whole stack behind CloudFront with AWS-managed
TLS, plus a runbook aimed at operators who are not DevOps engineers.
Ingress is limited to CloudFront's origin-facing prefix list, shell
access is SSM Session Manager (no port 22, no key pair), data stores
bind to loopback, IMDSv2 is required with a hop limit of 1 so no
container can reach the instance role, and every secret is generated
into SSM Parameter Store rather than living in the repo. The instance
stops itself when idle and can be woken from a public button, which
keeps a continuously-available environment at roughly a third of the
cost of leaving it running.
- **`sentinel` service in `docker-compose.yml`**, behind a `sentinel`
profile. The image was already published but had no compose service
and no ingress route, so the dashboard could not be served from the
shipped stack. A plain `docker compose up` is unchanged.
- **`ADMIN_SEED_PASSWORD` passed through to `db-migrate`**, so the seed
migration's existing support for a supplied password can actually be
used. Unset still generates a random one and prints it once.
- **Optional public-demo account (`SEED_DEMO_USER`, default off)** — seeds a
shared `demo` user on a read-mostly `DEMO` role so a throwaway sandbox can be
signed into by anyone, without handing out the super-admin credential. The
role carries reads plus `api_keys:issue` / `api_keys:revoke`, and
deliberately excludes everything that changes decisioning (`rules:update`,
`models:set_status`, `models:set_threshold`, `settings:write`,
`review_queue:override`) or identity (`users:*`, `roles:*`). Requires
`DEMO_USER_PASSWORD` to be set explicitly and refuses to run without it.
`mustChangePassword` is false by design: the credential is shared, and
forcing a rotation would let the first visitor lock out everyone after them.
**Never enable this on a real deployment** — a known-credential account on a
self-hosted fraud platform is a vulnerability, which is why it is opt-in
rather than seeded-then-disabled.
- **`extra_cors_origins` for the AWS deployment** — additional browser origins
permitted to call the API, for a site hosted somewhere other than the front
door. The front door is always included; anything beyond it is a listed,
reviewable decision rather than a wildcard.
### Fixed
- **RDA could not boot with `NODE_ENV=production` from the shipped
compose file** ([#121](https://github.com/ojuri-io/ojuri/issues/121)).
`SENTINEL_CORS_ORIGINS` was never passed into the container, and the
production guard treats its absence as fatal — so the default
configuration crash-looped on *"Refusing to boot with unsafe
defaults"*. Setting it in `.env` did not help: `.env` populates
Compose's interpolation scope, and a variable only reaches a container
if the service's `environment:` block names it.
- **MLA crash-looped on modern kernels**
([#122](https://github.com/ojuri-io/ojuri/issues/122)).
`onnxruntime==1.16.3` ships a native extension marked as requiring an
executable stack, which current loaders refuse — the import failed
before the service started. Now `1.19.2`. The runtime is independent
of the `onnx` / `onnxmltools` / `onnxconverter-common` pins that exist
for XGBoost conversion; those are unchanged and
`tests/test_onnx_conversion.py` passes against the new runtime.
- **FIA chat answers bled into invented follow-up turns.** The follow-up
prompt was a plain `USER:/ASSISTANT:` transcript rather than Phi-3's chat
template, so the model never emitted `<|end|>` — and `<|end|>` was not in the
stop set anyway, since the tokenizer's configured `eos_token` is
`<|endoftext|>`. The model finished its answer, was not stopped, and spent
the remaining `max_new_tokens` writing `QUESTION:`/`ANSWER:` pairs the user
never asked, all returned and persisted as the assistant turn. Measured on a
live deployment: same question, same report, 248,237 ms with four invented
turns before, 66,960 ms and a clean answer after. Report generation shares
the stop-token fix; it never showed the artefact because the parser keeps the
first JSON object and discards what trails it.
- **The wake page is no longer served to callers that wanted JSON.**
CloudFront's error configuration is distribution-wide, so a sleeping sandbox
answered an API call with an HTML page. It now content-negotiates on
`Accept`, and carries CORS headers so a cross-origin caller can read the
error at all rather than seeing an opaque network failure.
- **A stopped instance no longer takes 30 seconds to report itself.**
CloudFront's default three attempts at ten seconds were spent in full,
because a stopped instance swallows connections rather than refusing them.
One attempt at three seconds.
- **`/fia/` and `/mla/` were never stripped before proxying**
([#123](https://github.com/ojuri-io/ojuri/issues/123)), so Sentinel's
investigation reports and retrain controls 404'd against the shipped stack.
nginx substitutes the location prefix only when `proxy_pass` names a literal
upstream; both routes use a variable so the resolver is consulted per
request, and in that form the trailing slash is inert.
- **FIA silently degraded to rule-based reports on any CUDA host.**
`phi3_generator` requested `attn_implementation="flash_attention_2"`
whenever CUDA was present, but `flash-attn` is not a dependency.
transformers raised, the fallback caught it, and the only outward sign
was `llmModelVersion` ending in `-fallback` — on a GPU rented
specifically to avoid that. It now falls back to `eager` when
`flash_attn` is unavailable. The bug was invisible because every
measured run had been on Apple Silicon MPS, which takes the `eager`
branch.
## [1.4.0] - 2026-08-09
This release remediates a full line-by-line architecture review — 45
findings across three rounds (OJR-01–45, per-finding evidence in
`docs/REVIEW_FINDINGS.md`) — and ships a flag-gated log-first audit
pipeline. It also makes permission changes apply to live sessions and
the RDA replica count configurable. No API surface changes, but several
decision-path behaviours differ from 1.3.0 (see Changed) and every fix
was re-verified against a live compose stack: 2,000-request load test
all HTTP 200, p99 84.5 ms at 16-way concurrency vs the 295 ms pre-fix
baseline.
### Added
- **Log-first audit pipeline (`AUDIT_PIPELINE=stream`, default
`queue`)** — the decision event carries the full audit payload and is
published with an awaited `acks=all` send before the response;
`AuditStreamConsumer` materialises `decisionAuditLog` from the topic,
and late values (shadow scores, early-PRE feature snapshots) follow
as `audit.enrichments` events applied as idempotent UPDATEs. Audit
durability moves from "row survives unless the process dies before
flush" to "row durable in Kafka before the caller sees the decision".
Measured: p50/p95/RPS parity with the queue, +10–19 ms p99
end-to-end; server-side decision path p50 1 ms / p99 5 ms; enrichment
delivery 100% (queue mode's patch race dropped ~15% of early-PRE
snapshots). Closes OJR-07 when enabled; the checklist for making it
the default (a major-release change — Kafka becomes a hard dependency
of `/v1/predict`) is in `docs/REVIEW_FINDINGS.md`. Full measurements:
`docs/LOG_FIRST_AUDIT_PROTOTYPE.md`. The prod compose stack threads
`AUDIT_PIPELINE` through to the RDA replicas, so the `.env` opt-in
works on `docker compose up` (previously dev-overlay only).
- **Score calibration reaches serving** — RDA loads the isotonic
breakpoints from `meta.json` and applies them after ONNX output.
`ONNX_CALIBRATION_MODE` defaults to `observe`: the calibrated score
is recorded as `decisionAuditLog.calibratedScore` while decisions
still use the raw score, because every shipped threshold was tuned
against the raw distribution. Flip to `enforce` only after
re-deriving thresholds from calibrated audit data.
- **PAA leader lease** — PAA takes a Redis lease
(`ojuri:paa:leader`) before consuming; a second instance waits for
handover then exits instead of joining the consumer group and
splitting the graph. Renewal fails closed on elapsed TTL, and a
fenced-out instance discards its buffers rather than flushing a
partial-graph snapshot over the new leader's writes.
`PAA_REQUIRE_LEADER_LEASE=false` disables the fence.
### Changed
- **The ONNX breaker fallback is REVIEW, not DECLINE.** opossum fires
the fallback on per-call timeouts too, so a timeout below the
measured p95 under concurrency turned ordinary contention into
customer-facing declines. `CB_ONNX_TIMEOUT` now defaults to 750 ms,
the fallback returns `{ degraded: true }`, and `PredictService` maps
it to `CB_ONNX_FALLBACK_DECISION` (default REVIEW) with
`decisionSource = BREAKER_FALLBACK`. Degraded declines are no longer
published to `transactions.blocked` — there is no model signal for
FIA to investigate.
- **Drift F1 threshold anchors to the deployed champion** — the
window alarm fires at (champion validation F1 −
`DRIFT_F1_MARGIN`) instead of the static `DRIFT_F1_THRESHOLD`, which
stays as the fallback when no champion metrics exist.
- **Model hot-reload serves the version artefact directly** — the copy
into the canonical `MODEL_PATH` is best-effort (the compose files
mount `models/` read-only; EROFS is expected), and a champion that
was ACTIVE before boot is applied at startup, so cold restarts no
longer silently fall back to whatever the canonical file holds.
- **Dev containers run the mounted source directly** —
`module-alias` registers only in compiled builds
(`src/register-aliases.ts`); under ts-node, `tsconfig-paths` resolves
aliases to `src/*.ts`, keeping one module universe (two resolvers
duplicated every tsyringe singleton). nodemon uses `legacyWatch` so
host-side edits through the Docker mount trigger restarts on macOS.
No in-container `npx tsc` after edits any more.
- **Configurable RDA replica count (`RDA_REPLICAS`, default 3)** — the
three hand-copied `rda-1/2/3` compose services collapse into a single
`rda` service scaled via `deploy.replicas`. NGINX balances across the
replica IPs Docker DNS returns (restart nginx after changing the
count so it re-resolves), and Prometheus discovers replicas via DNS
service discovery instead of static targets. The GHCR overlay and CI
reference the `rda` service name; per-replica names (`rda-1:3000`
upstreams, static scrape targets) are gone — update any scripts that
addressed replicas by name.
### Fixed
45 findings; the tracker has per-finding detail. Highlights by service:
- **MLA** — the isotonic calibrator previously affected only the
reported Brier score and never a served score; the calibration split
is carved before SMOTE (fitting on oversampled rows targeted a ~50%
synthetic base rate); drift windows are fed from Postgres ground
truth on the label poll (they previously never received a sample, so
F1/PSI drift could not fire); the label-volume watermark anchors to
the last succeeded `retrainRuns` row, so labels arriving while MLA is
down still count after a restart.
- **PAA** — graph cap is a true LRU; velocity windows are retained by
time, not count; replayed events dedupe instead of double-counting;
graph metadata snapshots for both parties on every event.
- **RDA** — audit ids are tenant-scoped; the audit requeue path bounds
poison entries; the LevelDB replay buffer preserves topic and
partition key per entry (`{ v: 2, ... }` envelope).
- **FIA** — poison messages are bounded by a retry counter and
committed past instead of wedging a partition; offsets commit
per-partition only.
- **Dev/compose** — `db-migrate` and dev services carry the env they
need (`MLA_SERVICE_TOKEN`, `AUDIT_PIPELINE` passthrough).
- **Permission changes now apply mid-session** — the JWT authenticates
identity only; `requireAuth` resolves permissions, `isActive`, and
`mustChangePassword` from Postgres per request through a 30 s
in-process grants cache (`AuthService.verifyTokenLive`). Role edits,
role (un)assignment, user deactivation/deletion, and forced password
rotation take effect on live sessions within the cache TTL —
instantly on the replica that handled the admin call — instead of on
the user's next login (previously up to the 8 h token TTL). The
token's permission claim is still minted for compatibility but is
never consulted for authorization.
- **`/readyz` through NGINX no longer 404s** — `location /ready`
prefix-matched `/readyz` and the `proxy_pass` URI rewrite sent
`/readyzz` upstream. The `/health` and `/ready` aliases are exact
matches now, so the app's own `/livez` and `/readyz` pass through
the catch-all unchanged.
- **`scripts/demo-traffic.mjs` defaulted to an unreachable URL** — the
default `http://localhost:3000` is not published by the production
compose stack (NGINX answers on `:80`), so the documented host-side
invocation exited with "RDA is unreachable". It now probes `:80` then
`:3000`, so both the Docker stack and `npm run start:dev` work with no
`RDA_URL` set. An explicit `RDA_URL` is still used verbatim.
### Documentation
- **README restructured around the adopter journey** — quick start
first, then what the platform does, then a path split for
evaluating / integrating / operating / contributing. Install
prerequisites and the build-from-source path moved into collapsed
sections, and symptoms moved to the new
[`docs/TROUBLESHOOTING.md`](docs/TROUBLESHOOTING.md). Prose rewritten
in plain language throughout.
- **Corrected README instructions that failed when executed** — the
quick-start `curl` returns 409 on a second run (fixed
`transaction_id`, now called out); the sample response showed
`threshold` 0.65 on a `TRANSFER` payload where the seeded segment
threshold is 0.30, and omitted the `basis` field on reason codes;
`"model_version": "default"` was described as no model being
registered, when `00_initial_model_version` seeds exactly that
version as ACTIVE on purpose; `frontend/.env.example` already targets
the Docker stack, so the instruction to uncomment that block was
stale; and `python --version` fails where no bare `python` exists, so
the MLA setup now creates the venv with `python3.11`.
- **`docs/ARCHITECTURE.md` performance table** now carries the
post-review contended measurement (p99 84.5 ms, ~516 RPS) alongside
the superseded 295 ms baseline, which the table had been presenting
as current.
### Notes
- Round 2 of the review (OJR-27–40) consisted of defects introduced by
round 1's fixes, found by three independent re-reviews — the tracker
keeps the full chain visible.
- Reference performance in CLAUDE.md was re-measured post-fix; the
honest contended baseline is now p99 84.5 ms (was 295 ms).
## [1.3.0] - 2026-07-07
This release acts on an independent efficacy validation of the shipped
stack (`efficacy-validation/`). It fixes four correctness gaps between
what the platform did and what its own contracts/README claimed, and
adds two detection improvements. Existing deployments are unaffected —
both seed changes skip rows that already exist — but **fresh installs
behave differently on day 1** (see Changed).
### Added
- **Behavioral rule pack (`04_behavioral_rule_pack.ts`)** — two POST-stage
REVIEW rules on PAA-derived features (`velocity_1h`,
`unique_receivers_24h`), guarded to exclude agents and corporates.
Turns PAA's velocity/graph signals into verdicts the ML model misses
on trusted, authenticated traffic. Measured against the efficacy
harness: velocity-anomaly recall 0.00 → 0.80 with zero added false
positives on agent-network, payroll, airtime, and remittance traffic.
Every feature it reads defaults to 0 on a Redis miss, so the pack is
cold-start-safe.
- **Context-field dropout augmentation (MLA)** — training appends
context-zeroed copies of a fraction (`CONTEXT_DROPOUT_FRACTION`,
default 0.4) of the training rows so the model scores on behaviour,
not payload richness, fixing the shipped model's trust-context
degeneracy (a bare payload scored ~1.0, full context ~0.0).
`CONTEXT_DROPOUT_ENABLED` (default on). See
`mla-service/docs/CONTEXT_DROPOUT_RETRAIN.md` for the runbook and the
measured result: the mechanism closes the context gap (0.9998 →
0.0009) but does not by itself add behavioural typology recall — that
requires training data containing those patterns as fraud.
- **Context-sensitivity probe (RDA)** — alongside the calibration probe,
every model load and hot-swap scores the same transaction with and
without context fields and warns (exposing the gap via
`getModelInfo()`) when the model keys on integration context rather
than behaviour.
### Changed
- **The demo rule pack now seeds inactive by default.** Its amount
thresholds are demo-dataset props that flag large slices of real
traffic (declining every txn ≥ ₦100k, reviewing every ₦500–10k
PAYMENT) and shadow the FATF pack. Set `SEED_DEMO_RULES_ACTIVE=true`
before the first seed run to restore the old behaviour for the demo
dataset. Existing rules keep their operator-set `isActive` state.
- **Fresh installs now register the shipped model and per-type
thresholds.** The `db-migrate` container previously lacked the
`models/` mount, so the initial-model seed silently skipped — every
decision fell back to a flat 0.65 threshold with an empty registry.
Fresh installs now score with the registered `default` model and its
segment thresholds (CASH_OUT 0.70, TRANSFER 0.30, …), so day-1
verdicts differ from 1.2.0's. `v1.x` stays reserved for adopter-trained
models.
- **Calendar features are computed correctly for the first time.** Both
RDA serving and MLA training mis-handled the millisecond event
timestamp (RDA multiplied by 1000; MLA parsed as nanoseconds and
trained on ingestion time), so `hour_of_day` / `day_of_week` /
`is_weekend` / `is_payday_window` / `is_off_hours` were noise.
Verified zero decision change on the deployed model (it barely
weighted these), but retrain before relying on time-of-day signals.
### Notes
- Deployments upgrading from 1.2.0 will see a one-time
context-sensitivity warning in the RDA logs against the shipped model
— that is the new probe working, not a regression.
## [1.2.0] - 2026-07-02
This release closes the label feedback loop — the core mechanism that
lets a self-hosted deployment improve on its own outcomes the way
closed fraud vendors do — and fixes four correctness gaps where the
platform silently did less than its own contracts claimed. Models
trained before 1.2.0 keep working, but see the "Changed" notes on
score distributions before relying on pre-1.2.0 thresholds.
### Added
- **Ground-truth label ingestion API** (#81). `POST /v1/admin/labels`
(new `labels:write` permission) accepts up to 1,000 verified
outcomes per batch — chargebacks, disputes, customer reports — and
upserts `transactions.groundTruthFraud` with provenance. Duplicates
in a batch collapse last-wins; the response splits `applied` vs
`unmatched` so callers can retry rows PAA hasn't flushed yet.
`GroundTruthSource` is now a shared enum across the label paths.
`docs/ADOPTER_TRAINING.md` §4 documents the flow.
- **MLA label-volume retrain trigger** (#81). MLA counts labels
recorded since its watermark every `LABEL_CHECK_INTERVAL_SECONDS`
(default 900) and retrains once `LABEL_RETRAIN_THRESHOLD` (default
500; `0` disables) accumulate — fresh verified labels reach the
model on their own schedule instead of waiting for F1 drift, which
is 3–7 days label-delayed by nature. Runs with or without Kafka,
respects `autoRetrainEnabled`, and reports `label_volume_checks` /
`labels_pending_retrain` in `/stats`. The registry treats the
`label_volume` trigger like `drift` (auto-activate on gate pass).
- **PAA fraud-proximity graph features** (#82). PAA polls
confirmed-fraud transactions (every `FRAUD_LABEL_POLL_INTERVAL_MS`,
default 5 min) and marks both parties as fraud nodes, making two
previously always-default catalogue features real:
`graph_shortest_path_to_fraud` (undirected BFS, depth-capped at 4,
visit-capped at 1000) and `graph_neighborhood_fraud_rate` (fraction
of flagged 1-hop out-neighbors). `recipient_dispute_rate` is now
the only contracted feature without a writer, enforced by test.
- **Real shadow-model scoring** (#83). The SHADOW registry status now
does what it says: OnnxService keeps a small second session pool for
the shadow version (same feature-schema-version refusal as the
champion), scores it on every ML-scored predict overlapped with the
reason-code/POST-rule stages, and writes `shadowScore` onto the same
`decisionAuditLog` row. Strictly observational — failures record
null and never touch the decision or `/readyz`. New
`shadow_inference` stage metric.
- **PAA delivers the full `paa:redis` feature contract** (#78).
Previously only 11 of the ~20 contracted features were written; the
model scored on catalogue defaults for the rest, permanently. Now
implemented: `velocity_1m/5m/15m`, `amount_mean_24h`,
`amount_max_24h`, `unique_receivers_24h/7d` (fan-out signal),
`hour_dev_from_sender_norm`, `graph_is_hub` (top 0.1% out-degree
with an absolute floor), `recipient_lifetime_tx_count` (kept fresh
on pure receivers via partial updates), and a full pair block on a
new `features:pair:{sender}:{receiver}` hash (30-day TTL):
`pair_is_first_send`, `pair_prior_send_count`,
`pair_time_since_last_send`, `pair_round_trip_count_30d`,
`pair_amount_mean_30d`. RDA fetches sender + pair + receiver hashes
in one pipelined round trip. A contract test fails the build if a
contracted feature loses its writer.
- **REVIEW band on the ML score** (#87). A new `review_margin` runtime
setting (seeded 0 = off) routes scores in
`[threshold − margin, threshold)` to a REVIEW decision and into the
analyst queue instead of a silent ACCEPT — turning model uncertainty
into ground-truth labels. Tracks the per-segment threshold
resolution; REVIEW is observational on the wire (`fraud: false`,
never published to the blocked topic). Settings gains a Review band
card with sizing guidance.
- **MLA temporal train/test splits** (#88). Training rows are
time-ordered and split positionally — train on the past, evaluate on
the future. The previous stratified random split leaked future rows
into training and inflated every metric the deployment gate reads.
Falls back to the stratified split for degenerate datasets.
- **MLA absolute deploy floor** (#88). `MIN_DEPLOY_F1` (default 0.3):
the validator refuses candidates below the floor even when they beat
the incumbent, and cold-start models below it register as CANDIDATE
for operator review instead of auto-activating.
- **Sentinel Labels page + label-feedback card** (#86). Manual entry
for verified outcomes (paste ids, per-line verdicts, source picker)
posting to the labels API, and a Settings card showing "labels until
next retrain" from MLA `/stats`.
- **Fraud simulation benchmark** (#90). `scripts/fraud-sim.mjs` +
`scripts/fraud-sim-score.py` + `docs/FRAUD_SIMULATION.md`: a
deterministic persona population with six embedded fraud typologies
(half deliberately rule-evading) driven through `/v1/predict` over a
simulated multi-week window, scored per typology against ground
truth. Reference run: 128k transactions, 34.2% of fraud caught cold
→ 98.8% at 1.1% FPR after one label-driven retrain, on fraud
identities the model never saw.
- **Live ground-truth metrics in champion-vs-shadow** (#91). The
comparison endpoint computes per-model precision/recall/F1 and
McNemar's p from labelled decisions in the window instead of
hardcoding null; the Models page prefers live values over training
metrics, √-scales the score histogram, and fixes the off-plot
tooltip.
- **Demo traffic seeder** (#84). `docker compose --profile demo run
--rm demo-seed` (or `node scripts/demo-traffic.mjs`) posts ~500
realistic NGN transactions — 50 recurring senders plus an embedded
money ring, a 30-receiver mule fan-out, VPN sessions, and a
structuring sequence — so a fresh install has a populated dashboard
and firing rules within a minute. Zero npm dependencies.
### Fixed
- **Reason codes were misattributed** (#77). The explainer used
pre-catalogue vector indices, so every code read a different feature
than its label claimed (`AMOUNT_HIGH` was computed from
`unique_receivers_24h`, `PAGERANK` from `velocity_7d`, …). Specs now
resolve positions from the feature catalogue by name; a regression
test pins the attribution. `PAGERANK` baselines moved to realistic
per-node magnitudes so it no longer adds a large constant to every
explanation.
- **`pair_time_since_last_send` semantics** (#78). Was the sender's
time since their last transaction to *anyone*, mislabeled to the
model as pair recency. Now genuinely pair-scoped.
- **MLA deployment gate had an escape hatch** (#79). A candidate
deployed on F1 improvement alone, bypassing both the McNemar
significance requirement and the >5% precision/recall regression
guards. All three gates are now binding.
- **MLA PSI data-drift detection was inert** (#80). The consumer fed
the detector fields the Kafka event doesn't carry (fabricated as
constant 0) under names that didn't match the baseline's catalogue
keys. PSI now monitors only fields actually present on the event
(`amount`, `account_age_days`, `session_to_txn_seconds`) under exact
catalogue names, omitting absent fields instead of defaulting them.
- **PAA first Redis flush no longer silently dropped** (#89). The
worker now waits for Redis readiness before consuming Kafka — the
lazy client raced the first batch flush and every write in it was
counted as an error and discarded. Also quiets the fraud-sync log
re-reporting the same users every poll.
- **Isotonic calibrator was dropped on automated retrains** (#88). Only
the cold-start script persisted it, so the first drift/label-volume
retrain silently shipped uncalibrated scores. `upload_model` now
writes `calibrator.npz` into every version directory.
### Changed
- **Score distributions will shift after deploying 1.2.0.** Models
trained before this release learned on default values for the ~9
newly-delivered PAA features (#78) and the two fraud-proximity
features (#82); once real values flow, scores move. Plan a retrain
shortly after deploying, and re-validate any hand-tuned per-segment
thresholds against post-upgrade audit data.
- Shadow scoring adds one extra ONNX inference per ML-scored request
*while a SHADOW model is staged* (#83). No cost when no shadow is
registered. If throughput matters on small hardware, retire the
shadow when the comparison window is done.
- `POST /v1/admin/labels` requires the new `labels:write` permission —
grant it to the role your ops integration uses before pointing a
chargeback feed at it (#81).
- Post-1.2.0 training metrics will read **lower** than pre-1.2.0
numbers for the same data (#88) — the temporal split removes the
future-leak inflation. That drop is honesty, not regression; compare
models within the same split methodology only.
- The review queue now includes REVIEW-decision rows alongside
DECLINEs (#87). Queue volume is unchanged until an operator sets a
non-zero `review_margin`.
## [1.1.0] - 2026-06-22
First tagged release. v1.0.0 (2026-06-07) was the soft launch; v1.1.0 is the
first version with a formal git tag, signed container images on GHCR
(`ghcr.io/ojuri-io/{rda,paa,mla,fia,sentinel}`), and a GitHub Release.
Adopters should pin to `:v1` (floating, receives future minors and patches)
or `:v1.1.0` (strict). See [`VERSIONING.md`](./VERSIONING.md).
This release adds the post-v1.0.0 graph-coherence work, adopter training-data
ingest, FATF rule pack, MLA score calibration, per-segment threshold defaults,
and several PAA scale/correctness fixes.
### Added
- **PAA durable edge + node state in Postgres.** New `transactionEdges`
table (Knex migration `20260608000001`) holds the directed-edge
list; `graphMetadata` finally gets the `firstSeen` / `lastSeen` /
`transactionCount` / `totalAmount` / `inDegree` / `outDegree`
columns populated alongside the existing pagerank / community /
clustering outputs. On boot, PAA hydrates the in-memory
`graphology` instance from these tables first and only tails
transactions newer than the latest persisted edge timestamp. Rings
whose seed edges fall outside the 30 d / 100 k row replay window
stop being forgotten across restarts.
- **PAA singleton runtime guard.** New `paa_group_members` Prometheus
gauge that must stay at 1; PAA logs ERROR if a second consumer
joins the `pattern-analysis` group. Backstops the compose change
for ops scenarios (rolling restart overlap, accidental scale-up,
helm-chart drift) where the in-memory graph would otherwise be
fragmented across replicas.
- **Triangle-close recompute trigger.** When a new edge closes a
directed 3-cycle (`A → B`, `B → C` already exists, `C → A` already
exists), PAA fires `computeNetworkMetrics()` on the next event
instead of waiting up to 5 minutes for the scheduled tick. Burst
rings whose edges land in a tight window stop being invisible
during the gap. Throttled by `TRIANGLE_RECOMPUTE_MIN_INTERVAL_MS`
(default `10000`) so a flurry of ring-forming events doesn't
thrash the recompute path.
- **PAA sizing guidance** in `paa-service/README.md`. Covers what a
node represents, what actually happens at the `MAX_GRAPH_NODES`
cap (silent crash loop when no nodes are stale enough to evict),
and the two binding ceilings (1–2k TPS sustained typical, dropping
toward 200–500 with high-degree receiver hubs or graphs past ~500k
nodes; ~1M nodes memory) with a profile-by-profile sizing table
from demo through Tier-1.
- **Adopter training-data ingest.** Operators can now upload labelled
transaction CSVs through the Sentinel "Training imports" page or via
the file-based API. Upload protocol is chunked (5 MB chunks, SHA-256
verify on assemble) so multi-hundred-thousand-row files survive
network blips: `POST /v1/admin/training/upload/init` →
`PUT /v1/admin/training/upload/:id/chunk` × N →
`POST /v1/admin/training/upload/:id/complete` →
`POST /v1/admin/training/import/:jobId/promote` →
`POST /mla/v1/admin/retrain`. Column renames and default values are
applied in the UI (no need to re-export the CSV); a downloadable
template surfaces the canonical headers. New tables: `trainingJobs`,
`trainingUploads`, `transactionsStaging`. See `docs/ADOPTER_TRAINING.md`.
- **Per-segment fraud-threshold defaults** seeded for every
PaySim transaction type in
`src/database/seeds/02_segment_thresholds.ts`: CASH_OUT=0.70,
TRANSFER=0.30, PAYMENT=0.50, DEBIT=0.50, CASH_IN=0.50. Resolution
uses `request.segment ?? request.transaction_type` so adopters get
segment-aware thresholds with no UI work.
- **FATF default rule pack** under `src/database/seeds/03_fatf_rule_pack.ts`:
five rules covering structuring (CASH_OUT under the cash-reporting
threshold), VPN+significant-amount, outbound TRANSFER to FATF
high-risk corridors, account-takeover signature
(rushed session + cross-country IP + high amount), and untrusted
device + significant amount. Defaults are NGN-tuned; edit per-market.
- **MLA isotonic calibration** wraps the post-XGBoost score with
`sklearn.IsotonicRegression`, fitted on a held-out 10% split. The
Brier score (calibrated and uncalibrated) is recorded in
`meta.json` and persisted to the `modelVersions.brierScore` column
so adopters can spot saturation regressions across versions.
- **User-controlled training mode.** New `mlaSettings.trainingMode`
(`FRESH` | `CONTINUED`) and `mlaSettings.continuedTreesPerRound`
let operators choose between full retraining (current behaviour)
and incremental boosting on top of the current production model
(XGBoost `xgb_model=`). Surfaced as a dropdown on the Settings
page; `PUT /mla/v1/admin/drift-config` accepts the two new fields.
- **Rule visibility in audit and predict response.** Every
`/v1/predict` response now carries a `rule` object when a PRE or
POST rule fired (`{ id, name, stage, action, expression }`), and
the decision-audit row stores the rule expression so the Sentinel
detail page can render exactly which rule caused a DECLINE.
- **Curated demo dataset and one-shot loader.** A 20-row
`data/demo/sample-transactions.json` hand-built to exercise all three
decision buckets, plus `npm run demo:load` (extends `scripts/seed-load.ts`
with a `--file` flag that re-mints `transaction_id` / `timestamp` per
send so the dataset can be replayed without idempotency collisions).
- **Demo rules seed** under `src/database/seeds/01_demo_rules.ts`,
installed by `npm run db:seed`. Four PRE-stage rules keyed on
`amount` / `transaction_type` / `segment` so the demo dataset produces
a visible ~9 ACCEPT / ~4 REVIEW / ~7 DECLINE split out of the box even
on a fresh deploy with no PAA cache.
- **Regression guard** for the demo dataset shape: seven Jest assertions
(`test/demo/demo-dataset.spec.ts`) that fail CI if the JSON drifts
from the predict-API contract.
- README quickstart now mentions `npm run reset:admin` for the
"I lost the seeded password" recovery path.
- `.env.example` now documents 11 previously-undocumented env vars
(BRAND_NAME, FEATURE_CATALOG_BASE_PATH / ADOPTER_PATH /
FEATURE_LOOKUP_ROOT, the four `*_HEALTH_URL` knobs,
HEALTH_PROBE_TIMEOUT_MS, MODEL_VERSION_LABEL, MODEL_INPUT_DIMENSION).
### Changed
- **PAA is now a singleton by design.** `paa-2` removed from
`docker-compose.yml`; `deploy.replicas: 1` pins the remaining
`paa-1` service; Prometheus scrape config and CI log-dump list
trimmed accordingly. Reason: PAA holds the transaction graph and
velocity windows in process memory, and Kafka partition assignment
splits the event stream across replicas — running two means each
computes PageRank / Louvain on a partial graph and any ring whose
members hash to different partitions becomes invisible. Adopters
running past the singleton's sustained-throughput ceiling
(typically >2k TPS, sooner for graphs >500k nodes or when high-
degree receiver hubs dominate) will eventually want the
externalised-graph deployment profile (separate work) instead of
scaling PAA.
- **PAA `graphMetadata` populated on every event for both sender and
receiver.** Previously the trigger was `processedCount % 100 === 0`
and only the sender was snapshotted, which dropped 99 % of writes
and left pure-receiver nodes (mules, drain points) missing from
the table entirely. The flush path also switched from a per-row
upsert loop to a single bulk `INSERT … ON CONFLICT … MERGE` per
batch.
- **PAA `pruneOldNodes` switched to event-time clock and an hourly
schedule.** Was wall-clock (mis-classifying historical replays as
recent) and only cap-driven (so the prune effectively never fired
in practice). Now tracks the max observed event timestamp and runs
on a 1 h interval in addition to the existing cap path; the cap
path still drops the oldest 10 % as a relief valve, but the
scheduled path drops everything past the 30 d cutoff.
- **NGINX `/mla/` proxy hardened for Linux hosts.** The new `location
/mla/` block defers `host.docker.internal` resolution to request time
(variable in `proxy_pass` + embedded resolver), and the `nginx`
service in `docker-compose.yml` now declares
`extra_hosts: host.docker.internal:host-gateway`. Without this, nginx
hard-fails at startup on Linux CI runners and the Sentinel
"Retrain now" / drift-config calls return 502.
- **Predict service refactored** into named stages
(`resolveModel`, `loadFeatures`, `evaluatePreRules`, `runInference`,
`evaluatePostRules`, `finalize`) backed by factories
(`DecisionAuditFactory`, `PredictDecisionContextFactory`,
`TransactionEventFactory`, `PredictResponseFactory`). No behavioural
change; the request hot path is now legible from a single 40-line
method.
- Tighter TypeScript strict-family coverage. RDA gained
`noImplicitAny`, `strictPropertyInitialization`,
`noUncheckedIndexedAccess`, and `noImplicitOverride`. PAA (already on
`strict: true`) gained `noUncheckedIndexedAccess` and
`noImplicitOverride`. Net: one real bug fixed in
`velocity.service.ts` (un-guarded `sorted[0]` access), and the
compiler now refuses several whole classes of regression.
### Removed
- Unused `appConfig.onnx.modelRegistryUrl` field. The HTTP-polling
model registry it once fed was retired in favour of the in-process
`ModelRegistryService` subscription flow.
## [1.0.0] - 2026-06-07
Initial public release. The platform is a multi-agent fraud-detection stack
with four backend services (RDA, PAA, MLA, FIA) and an operator dashboard
(Sentinel), all self-hosted and decoupled by Kafka.
### Added
- **RDA (Real-Time Detection Agent).** Fastify HTTP API with `POST /v1/predict`
(sub-5 ms p99 on a developer workstation, ONNX inference, per-segment
thresholds, idempotency keys, inline reason codes), `/livez` / `/readyz` /
`/v1/metrics`, and a full `/v1/admin/*` surface. Three-replica deploy
behind NGINX in `docker-compose.yml`.
- **PAA (Pattern Analysis Agent).** TypeScript Kafka consumer that maintains
a transaction graph (`graphology`) and velocity windows, batching writes
to Redis (features) and Postgres (graph metadata every 100 events).
- **MLA (Model Learning Agent).** Python 3.11 drift monitor (F1 + PSI),
SMOTE-balanced XGBoost retraining, McNemar's test before promotion,
ONNX export, filesystem-based model registry under
`mla-service/models/versions/`.
- **FIA (Fraud Investigation Agent).** Python 3.11 LLM service that consumes
`transactions.blocked`, runs a fine-tuned Phi-3-mini-4k-instruct model
(fallback to a deterministic rule-based report on LLM failure), writes
structured investigation reports to Postgres, and exposes
`POST /v1/reports` and conversational follow-ups via
`POST /v1/reports/:id/messages`.
- **Sentinel dashboard.** Vite + React 18 SPA covering live decisions,
transactions, audit log, rules editor, model registry, FIA investigations,
webhooks, API keys, and user/role administration. Reads use a
`safe(live, empty-fallback)` pattern; the dashboard never displays
synthetic data.
- **Authentication and authorization.** API-key auth (`fdk_<prefix>_<secret>`,
SHA-256-hashed at rest), JWT user auth with bcrypt-hashed users and
role-based permissions, `requireAuth(...perms)` middleware on every
`/v1/admin/*` route. SUPER_ADMIN seeded with a one-time random password
printed at migration time.
- **Rules engine.** JSON-Logic predicate evaluator with PRE / POST stages,
hot-reloaded from Postgres every 30 s.
- **Model registry.** CANDIDATE → SHADOW → ACTIVE → RETIRED lifecycle,
per-segment thresholds, hot-swap on activation.
- **Decision audit log.** Every `/v1/predict` writes a row with champion +
shadow scores, threshold, rule hit, reason codes, feature snapshot.
Reviewer overrides write back to `groundTruthFraud` so the model does not
learn from its own past decisions.
- **HMAC-signed webhooks.** SHA-256 over `t.body`, exponential-backoff
delivery worker, delivery ledger in `webhookDeliveries`. SSRF guards
reject private IP ranges and non-HTTPS schemes.
- **Idempotency.** `Idempotency-Key` header on `/v1/predict`, keyed against
`(tenantId, apiKeyId, key, requestHash)`. Replay returns the cached
response with `Idempotency-Replay: true`. Body divergence returns 422.
- **Resilience.** `opossum` circuit breakers around Redis feature lookup
and ONNX inference. On breaker open, predictions still succeed against
default features.
- **Observability.** Prometheus metrics under `/v1/metrics`,
Grafana dashboard, structured pino logs with sensitive-field redaction.
- **Reason codes.** Lightweight feature-deviation explainer on every
prediction (e.g. `AMOUNT_HIGH`, `VELOCITY_24H`, `PAGERANK`). For deep
reasoning use the FIA report endpoints.
- **Feature catalogue.** Declarative `models/feature-catalog.v1.json` (64
base features across 9 categories) loaded by both RDA and MLA. Adopters
add their own features via `models/feature-catalog.adopter.json`.
- **Replay CLI.** `scripts/replay.ts` runs candidate models against the
live audit log for offline evaluation before promotion.
### Security
- Pre-launch hardening pass: CORS allowlist via `SENTINEL_CORS_ORIGINS`,
per-IP rate limit on `/v1/auth/login`, JWT algorithm / issuer / audience
pinning, dummy bcrypt for unknown users (constant-time login),
webhook SSRF guards, idempotency race protection, FIA HTTP API
authentication, prompt-injection guards on FIA inputs, MLA model loader
switched from `pickle` to XGBoost JSON format, removal of `useragent`
package (replaced with `ua-parser-js`).