docs/benchmarks/competitive-eval-audit-2026-08.md
Rama main · réplica de lectura del repositorio original.
<!-- authority:non-specification -->
# Competitive evaluation audit — 2026-08
Status: **authoritative Phase-0 audit** for the competitive benchmark programme
(OpenSpec change `add-competitive-benchmark-programme`). Every claim below was
verified in code or artifacts during the 2026-08-09 audit, not inherited from
ledgers or summaries. Where a ledger belief was found stale, the correction is
stated explicitly.
Scope: all benchmark code and artifacts in this repository, plus read-only
inspection of three sibling checkouts — upstream Basic Memory
(`../basic-memory`), the Basic Memory fork carrying an Exomem benchmark
provider (`../basic-memory-exomem-provider`), and the Supermemory monorepo
(`../supermemory`).
---
## 1. Asset map
### In this repository (all landed on `main` via PR #390, squash `7a140a2`)
| Asset | Role | State |
|---|---|---|
| `benchmarks/membench/` (~20.9k LOC, 450 tests) | Track B internal instrument: bitemporal oracle, 24 template families (t00–t23), deterministic gates, compile plan, judge backends/blinding/handshake, environment capture, agreement (κ) machinery, Track C hook-activation, Track D journeys | Live, governed by the internal-instrument contract of `rescope-benchmark-instrument-and-public-suite` |
| `benchmarks/lme/` (1.6k LOC, 27 tests) | Exomem-only LongMemEval-S lane (`public-suite-eval`): pinned cleaned dataset, per-question isolated vaults, stub/API readers with metered + full-run founder gates, ceiling/floor bounds, per-ability no-aggregate report, official `evaluate_qa.py` command emission | Live. **Never run on real data** — no LongMemEval file exists on disk; all 27 tests are offline/stub |
| `benchmarks/corpus/` | 10 committed JSON Schemas + the single pinned release `v0.1-seed1` | Frozen under the internal-instrument contract |
| `benchmarks/runs/` (gitignored) | 57 membench run dirs, 2026-07-31 → 2026-08-08 | **All historical-untrusted** (§5) |
| `benchmarks/judge-agreement/` | Blind-sample sheets, judge-vs-gates probes | Partially stale (ledger 4b.45: probe artifact pins a pre-`f15f65f` corpus) |
| `scripts/graph_value_benchmark.py` + `docs/comparison-basic-memory-graph.md` | Graph-value comparison | **WITHDRAWN** both directions (2026-08-09); survives only as an exomem-only regression instrument per rescope task 3b.6 |
| `openspec/changes/add-memory-server-comparison-benchmark/` | Planned CLI-vs-server comparison | **Obsolete** — 0/8 tasks, purpose forbidden by the rescope change |
| `tests/test_membench_*.py`, `tests/test_lme_*.py` | Lean bench suites | Green, offline |
| CI (`.github/workflows/ci.yml`) | `retrieval-eval` job: golden retrieval gate, latency ceiling gate, semantic write-latency gate | **No membench or lme job exists** despite README claims — recorded as a gap, addressed by the programme's new additive `benchmark-protocol` job |
### Sibling checkouts (read-only)
| Asset | Role |
|---|---|
| `../basic-memory/benchmarks/` (`bm-bench`, package `basic_memory_benchmarks`) | Basic Memory's own harness: stages retrieval → qa → rejudge → diagnose → review.html; providers `bm-local`, `bm-cloud`, `mem0-local`, `zep-reference` (stub), `baseline-grep`, `baseline-fullcontext`, and a **Basic-Memory-authored `supermemory-local`**; LongMemEval/LoCoMo(+Penfield corrections)/ConvoMem converters; fixed in-repo answer/judge prompts; `claude -p` (plan-billed) and OpenAI-compatible runners; `--strict-providers` |
| `../basic-memory-exomem-provider` (branch `exomem-provider`) | Fork adding a working Exomem guest provider to bm-bench (warm MCP stdio, +390 LOC provider, +259 LOC tests) with a fair-play bugfix trail. **6 commits ahead, 156 behind upstream** — regenerate, do not rebase |
| `../supermemory` | Supermemory monorepo — **periphery only**: web console, MCP worker (hosted), docs, SDK shims. The engine is closed source; local = precompiled `supermemory-server` binary (port 6767, BYO extraction model); hosted runs proprietary extraction models. MemoryBench is a separate repo (`supermemoryai/memorybench`, MIT, TypeScript), documented here but not vendored |
---
## 2. Reusable / suspect / obsolete / missing
**Reusable, high confidence** (imported as libraries; never modified):
`membench/adapters/base.py` (capability contract; `AdapterUnsupported` vs
`AdapterEnvironmentError` — the invalid-vs-loss line; `NativeAnswer` closed
citations), `membench/environment.py` (two-tier BLOCKING/REPORTED capture),
`membench/judge/{backends,blinding,handshake}.py`, `membench/agreement.py`
(built, tested, **never executed**), `membench/scoring/gates.py::states_value`
and `GateStatus`, `membench/scoring/health.py` (three-tier honesty contract),
`membench/clock.py`, `membench/trackd/journeys.py` shapes,
`membench/adapters/track_a_bridge.py` (bridge pattern), the pydantic →
JSON-Schema export + drift-gate mechanism, the immutable-run-dir +
failures-in-denominator discipline, `benchmarks/lme/` wholesale, bm-bench's
LongMemEval loader with SHA-256 provenance, the fork's provider as a
**checklist**, `scripts/codex_task.sh`, and the privacy gate
(`src/exomem/public_artifact_privacy.py`).
**Stale ledger beliefs corrected by this audit** (verified in code):
- **4b.14 (bare-substring gate matching) is FIXED** — `scoring/gates.py`
routes numeric values through the Decimal path against standalone number
tokens and everything else through word-boundary literal patterns.
- **4b.8 (citations recall-only) is FIXED** — `gate_citations` measures
precision and recall and returns `UNSUPPORTED` (never `PASS`) when
precision is unverifiable.
- **Harness-authored provenance/abstention is half-fixed** — the
`NativeAnswer` path exists and measured 4b.42's effect; the extractive
answerer remains the default and the bounds adapters still author answers.
- **Still open and load-bearing: 4b.18** — judge blinding is defeated by
frontmatter *structure* (key shape identifies the vendor with zero token
hits). In any multi-product comparison this is fatal; the programme
hard-gates every judged number behind a structural-blinding fix with a
structure-swap test.
**Suspect (do not trust without re-derivation):** every historical
cross-contender figure in any document, in either direction; all 57 membench
run dirs (none has a verified environment; the newest was produced from a
dirty tree); the committed judge-probe artifact (4b.45); any claim about
judge quality (4b.18, 4b.23); `ExpectedRecord.gates` fields in published
expected records (4b.10/13/15/16 open).
**Obsolete:** `add-memory-server-comparison-benchmark` (unimplemented,
purpose retired); the comparative half of the graph-value benchmark;
membench's competitor adapters/renderers
(`adapters/basic_memory_local.py`, `native/basic_memory.py`, `graybox*`) —
retired by policy and left retired; sibling worktrees duplicating merged
work.
**Missing entirely:** any Supermemory integration on the Exomem side (none
has ever existed here); a MemoryBench checkout; any LongMemEval dataset on
disk; any executed LME run; LongMemEval-V2 anything; STALE/MemOps anything;
CI coverage for the bench suites.
---
## 3. Validity-failure catalogue
The authoritative long-form record is
`openspec/changes/expand-memory-proof-benchmark/tasks.md` §4b (45 items) and
`docs/memory-proof-benchmark-v01-findings.md`. The classes, with their
sharpest instances:
1. **Index/embedding readiness lies.** The harness never built Basic
Memory's vector index (4b.41 — voided the first headline result);
`EXOMEM_DISABLE_CLIP=1` zeroed *text* retrieval (4b.24); a 3.12→3.14
interpreter change took retrieval 452 hits → 0 and was investigated as a
product regression; Exomem soft-degrades to keyword when
`sentence_transformers` is missing (4b.30 — correct product behavior,
catastrophic benchmark behavior).
2. **Author-configured competitors.** The renderer fed Basic Memory 376
oracle-normalised fact lines phrased in query vocabulary (voided the
second headline result, in the opposite direction); the profile contract
de-tuned Exomem specifically. This class recurred across two full repair
rounds and flipped direction each time — the structural reason the
2026-08-08 independent audit returned REJECT and the rescope retired
self-authored comparison.
3. **Ingestion-lifecycle mismatch.** Track B ingested at raw-source altitude
while scoring provenance chains that could not exist (4b.35: 205 sources,
zero compiled notes, `ingested_into: []` on 204/204); the compile plan
emitted conclusions in claim-id order so 46/88 supersession edges could
not resolve (4b.44); knowledge time was never transmitted to providers.
4. **Corpus integrity.** Entity-name collisions produced byte-identical
prompts with mutually exclusive expected answers (4b.32); four queries
named the forbidden answer in the prompt (4b.21); the committed judge
probe pins a superseded corpus (4b.45).
5. **Scoring-gate defects.** Substring firing (4b.14, now fixed), recall-only
citations (4b.8, now fixed), gates scoring records they were never asked
to judge (4b.36), declared-but-fictional gate names in published records
(4b.10/13/15/16), provenance/abstention authored by the harness rather
than the product.
6. **Judge drift/blinding.** Structural fingerprint defeats blinding (4b.18,
open); 19/19 discrimination is an upper bound from clean one-sentence
candidates (4b.23); judge disposition reversed three times before being
scoped to gate-UNSUPPORTED rows only.
7. **Environment/reproducibility.** "A run directory cannot reproduce its own
result" (resolved into `environment.py`); a Pillow patch bump flagged as
corpus drift took down CI (4b.7, fixed by splitting corpus identity from
environment provenance); GPU unusable under WSL2 → no valid latency claim
from the dev machine (4b.40 withholds cross-contender latency
permanently); Track D journeys changed verdict at midnight (4b.34).
### Prior bm-bench comparisons (fork, 2026-07-31 → 08-01) — all invalid
Eight runs, all on a **3-document / 4-query synthetic corpus** (never
LongMemEval), QA/judge stage never executed. Forensics from leaked run homes
in the fork: semantic search off in 3 of 5 Basic Memory homes (its default is
an implicit function of installed packages); the embedding model failed to
download in a 4th (43 s/query of retry sleeps, scored 0.000 with state
"ok"); `BASIC_MEMORY_CLOUD_MODE` merely unset while unknown projects
fall back to cloud routing; and Basic Memory's MCP returns error-guidance
*strings* with `isError=false`, which the adapter parsed as zero hits. Basic
Memory's honest range on that toy corpus was 0.500 (FTS-only) to 0.750
(hybrid); every published 0.000 was a misconfiguration artifact. Exomem was
never measured against a correctly configured Basic Memory.
### New findings from this audit (not in any ledger)
- **N1 — The direct LongMemEval lane leaks gold labels.**
`benchmarks/lme/dataset.py` (`render_session`) ingests the raw session ID
into provider-visible text, and on real LongMemEval-S evidence sessions
carry an `answer_`-prefixed ID — Basic Memory's converter
(`converters/longmemeval_to_corpus.py`) documents and neutralizes exactly
this leak; ours does not. `benchmarks/lme/adapter.py` additionally writes
`question_id` into note titles and `tags=[question_type]` (the category
label) into frontmatter. No committed fixture can detect it because
fixtures use neutral IDs. The lane has never run on real data, so no
published number is tainted — but the first build item of the programme is
a red test (`fixtures/leaky.json`) that fails against today's code.
- **N2 — Dataset identity diverges across harnesses.** The direct lane pins
`longmemeval-cleaned`; bm-bench pins the *original* `longmemeval`;
MemoryBench uses cleaned. Any cross-harness comparison without a variant
repoint fails dataset-identity equivalence on day one.
- **N3 — bm-bench's Supermemory row is not a memory row.** Its provider
posts to `/v3/search` (document/SuperRAG chunk search) and never passes
`dreaming`, so it measures document-chunk RAG under default dynamic
dreaming whose memory extraction is explicitly unfinished at
`status:"done"`. It is a legitimate but distinct variant — named
`supermemory-local-documents-v3` in this programme — and its defects are
filed upstream as an issue with evidence, never fixed by us
(fixing a competitor's configuration of another competitor repeats the
retired defect class in reverse).
---
## 4. Data flow (per case)
```mermaid
flowchart LR
DS[dataset row\npinned sha256] --> NORM[normalize + NEUTRALIZE\nsession_ordinal, content_sha256,\ntimestamp_semantics]
NORM -->|ProtocolEvent stream\n(gold structurally quarantined)| SCAN{leakage scan\ningest scope: strict}
SCAN -->|clean| ING[provider ingest\nisolated namespace + canaries]
SCAN -->|hit| INVALID1[case INVALID]
ING --> RDY{readiness\npositive verification\nfail closed}
RDY -->|verified| PROBE[known-answer probes\nlexical / semantic / update]
RDY -->|unproven| INVALID2[run INVALID]
PROBE --> Q[exact search query\ntop-k pinned]
Q --> HITS[raw response +\nnormalized hits]
HITS --> PACK[packed answer context\nsize recorded]
PACK --> ANS[frozen reader]
ANS --> JUDGE[official / pinned judge]
JUDGE --> TRACE[(case trace + manifest\nregenerable report)]
```
Every arrow persists an artifact; reports regenerate from artifacts with a
socket guard proving zero provider calls.
---
## 5. Migration of historical artifacts
- All 57 `benchmarks/runs/` membench dirs and all 8 fork bm-bench runs:
**`historical-untrusted`** — none reproduces under a verified environment;
the programme's report renderer refuses to read them.
- `docs/memory-proof-benchmark*.md` cross-contender figures and
`docs/comparison-basic-memory-graph.md`: already marked WITHDRAWN on
`main`; they stay as history with their withdrawal banners.
- Single-product findings that STAND (used as regression baselines, not
comparisons): factual_qa 0 → 99 → 157/180 across the retrieval fix
(PR #378); native provenance 0/272 → 50/238 (4b.42); compiled
supersession 0 → 37 with a one-row score move (4b.43 — the proof that
text-reading gates are blind to lifecycle state, and the design driver
for the new state-snapshot assertions).
- KB research notes (`Benchmark direction locked…`, `Memory-proof benchmark
v0.1 ran…`, `Retrieval fix measured…`, `…foundation locked…`, July's
Basic Memory comparisons) remain accurate as *history*; the 2026-08-08
REJECT audit and this programme's supersession are filed as a new KB note
so the KB thread does not end on the direction-lock.
---
## 6. Fairness questions the implementation must expose (not hide)
| Provider | Open question | How the programme exposes it |
|---|---|---|
| Exomem | Does compiled/native mode earn its cost over source-only? | `exomem-source-only` vs `exomem-native` are separate rows; write-agent budget envelope published |
| Exomem | Does the soft-degrade honesty problem (4b.30) bite users? | Lane D failure-transparency injects the fault and reports what the user sees |
| Basic Memory | Is semantic search actually on? | Fail-closed readiness: `search_vector_chunks` count > 0 + config + log line; `reindex` exit codes explicitly distrusted |
| Basic Memory | Is git history a fair substitute for a revision store? | Two rows (`native-git` / `native-nogit`); split assertions (retention vs linkage), disclosure not penalty |
| Supermemory | When are memories actually ready? | `status:"done"` never treated as extraction-complete; memories-canary polling; `readiness-unverifiable` as a first-class disclosed status for dynamic dreaming |
| Supermemory | Hosted vs local — same product? | Never collapsed; hosted runs proprietary extraction and is always a separate row |
| Supermemory | What org-level config shaped the run? | `GET /v3/settings` captured verbatim into every manifest |
| All | Who authored each configuration value? | Config-provenance table per provider (file:line or docs URL); a knob without provenance refuses to run |
| All | How much glue did Exomem author? | Fairness matrix publishes LOC + endpoint counts per projector/driver; gross asymmetry is itself a finding |
---
## 7. Architecture decision
The minimal change that makes comparison trustworthy is **not a rewrite** —
it is a provider-neutral protocol substrate (events, manifests, traces,
leakage scanning, isolation canaries, readiness, budget) plus
**inversion of configuration authorship**: competitor rows come from
competitor-authored harnesses and providers (bm-bench, MemoryBench), Exomem
enters those harnesses as a guest, and residual Exomem-authored glue is
provenance-cited, disclosed, and adversarially reviewed. Full rationale and
the supersession of the rescope change's retirement clause:
`docs/adr/0002-competitive-eval-architecture.md` and OpenSpec change
`add-competitive-benchmark-programme`.