JIRO_040_ANALYSIS_REPORT.txt
Rama main · réplica de lectura del repositorio original.
================================================================================
JIRO v0.4.0 — VERSION ANALYSIS AND 0.3.3 COMPARISON
================================================================================
Analyzed: 2026-10-22
Method: source inspection, test execution, live CLI invocation
Baseline: jirosearch 0.3.3 (committed, main @ e60207a)
Candidate: 0.4.0 work-in-progress (uncommitted, branch fix/stress-test-findings)
--------------------------------------------------------------------------------
HEADLINE
--------------------------------------------------------------------------------
A large, genuinely engineered body of work exists — 21,504 lines of new code
across 8 new modules, 488 passing tests, 13 ADRs, 16 client adapters. The
architecture is real and the engineering is disciplined.
It is NOT ready to ship as 0.4.0, and nothing is committed:
1. All four headline performance claims have NO benchmark artifact.
The project's own CLAIMS.md rule says such numbers may not be published.
2. The flagship feature (graph run) fails on a trivial query.
3. The version is still 0.3.3 in every file. Nothing is bumped.
4. The entire body of work is uncommitted — 0 commits on the branch.
Verdict: strong Phase 1-4 engineering, presented with pre-launch ambition that
the evidence does not yet support. The gap is verification and tuning, not
architecture.
================================================================================
1. SCOPE — WHAT ACTUALLY EXISTS
================================================================================
New modules (untracked):
Module Files Lines
---------------------- ----- -------
jiro/memory 15 6,228
jiro/runtime 15 4,618
jiro/research 10 3,418
jiro/orchestration 6 2,533
jiro/verification 6 2,300
jiro/graph 6 1,375
jiro/parametric 5 651
jiro/params 3 381
---------------------- ----- -------
TOTAL 66 21,504
Supporting artifacts:
- 20 new test files
- 3 new API routers: graph, params, tasks
- 2 new CLI groups: `jiro tasks`, `jiro graph`
- 13 ADRs (docs/adr/0001-0011 plus 2 phase reports)
- 5 planning docs (docs/v0.4.0/)
- New user docs: memory.md, runtime.md, mcp.md, compliance.md
- Benchmark harness: benchmarks/{claims.yaml, run.py, report.py, workloads/}
- 2 phase handoff docs (Phase 3, Phase 4)
================================================================================
2. CAPABILITY COMPARISON: 0.3.3 -> 0.4.0
================================================================================
Dimension | 0.3.3 (shipped) | 0.4.0 (in tree) | Status
--------------------|----------------------|------------------------|------------
Unit of work | 1 search / 1 scrape | Budgeted query graph | BUILT, FAILS
Search + scrape | Separate endpoints | Unified graph ops | BUILT
Query planner | None | Cost-based planner | BUILT, WORKS
Output | Raw JSON rows | Cited evidence packet | PARTIAL
Claim verification | None | Claim ledger/contradiction | BUILT
Parameter layer | None | Versioned params, TTL | BUILT
Durable tasks | None | Submit/status/cancel | BUILT, WORKS
Agent orchestration | None | 6 agent roles, DAG | BUILT
Memory system | None | 15-file subsystem | BUILT
MCP | 19 tools | +resources/prompts/progress/cancel | BUILT
Client adapters | Manual config | 16 auto-detect adapters | BUILT
Version string | 0.3.3 | 0.3.3 (NOT BUMPED) | MISSING
--------------------------------------------------------------------------------
ENGINE / INTEGRATION FLOOR (unchanged)
--------------------------------------------------------------------------------
Still 9 built-in engines: google, bing, duckduckgo, brave, yandex, baidu,
amazon, ebay, youtube. Plus 6 plugin engines (arxiv, github, google_scholar,
hackernews, reddit, wikipedia). 12 social platforms. No new engines in 0.4.0 —
0.4.0 is about orchestration and evidence, not more sources.
================================================================================
3. WHAT WORKS — VERIFIED LIVE
================================================================================
These were executed, not just read:
[x] `jiro graph plan` — WORKS
Detected "comparison" intent from natural language, built a correct 4-node
DAG (SEARCH, SEARCH, MERGE, COMPRESS) across 3 waves, produced a cost
estimate (~555ms, 2 requests), and even surfaced a counter-argument
("single_search: ~277ms — faster, but no cross-source corroboration").
This is real planning logic with honest tradeoff reporting.
[x] `jiro tasks types` — WORKS
Lists 3 durable task types with schemas: ai_search, ai_agent, batch_scrape.
[x] 488 new-module tests — ALL PASS
graph + memory: 290 passed
runtime + orchestration + params + research: 198 passed
[x] 16 client adapters — CONFIRMED IN CODE
claude, claude_code, cline, codex, continue, cursor, gemini, goose, hermes,
kilo, openclaw, opencode, roo, vscode, windsurf, zed.
The "15+ agents" claim is genuinely met at the code level.
[x] MCP v2 features — PRESENT
resources, prompts, progress tracking, cancellation all implemented.
[x] 13 ADRs — solid design documentation
Notably ADR 0011 (parametric-context-boundary) documents why "writing
weights" is out of scope — intellectual honesty about the boundary.
================================================================================
4. WHAT IS BROKEN OR MISSING
================================================================================
--------------------------------------------------------------------------------
4.1 CRITICAL — All four headline claims have no artifact
--------------------------------------------------------------------------------
Claim | Registry says | Reality
---------------|----------------------------------------|----------
speed-2x | artifact: docs/v0.4.0/benchmarks/ | MISSING
| speed.json |
efficiency-3x | artifact: .../efficiency.json | MISSING
citation- | artifact: .../citations.json | MISSING
complete | |
setup-15 | artifact: .../setup.json | MISSING
benchmarks/claims.yaml exists. benchmarks/run.py exists. The artifact
directory docs/v0.4.0/benchmarks/ does not exist at all.
The project's own stated rule (CLAIMS.md line 3):
"If the artifact does not exist, the number is not published."
By the plan's own standard, NONE of these may be claimed today:
2x faster / 3x more efficient / citation-complete / 15+ agents
The harness is built. It has not been run to produce committed results.
--------------------------------------------------------------------------------
4.2 CRITICAL — Flagship feature fails on a trivial query
--------------------------------------------------------------------------------
Command: jiro graph run "what is the capital of France"
Result: HTTP 502, graph_execution_failed, rc=1
Both search nodes timed out. Root cause is a budget/latency mismatch:
Graph total deadline (model.py:41) : 2,500 ms
Effective per-node deadline (observed) : 1,400 ms
Real search latency (measured live) : 4,630 - 12,905 ms
The node budget is roughly 3x to 9x BELOW the actual search latency floor.
On this machine and network, the graph cannot succeed for any web search.
To the project's credit, the failure is honest — it reports 502, node_timeout,
explicit gaps, empty results, and packet:null. Nothing is faked. The principles
in MASTER_PLAN section "Fail visible" are upheld. But the default budget needs
calibration against real latency before this feature is usable.
--------------------------------------------------------------------------------
4.3 HIGH — Version not bumped anywhere
--------------------------------------------------------------------------------
pyproject.toml version = "0.3.3"
jiro/__init__.py __version__ = "0.3.3"
npm/package.json "version": "0.3.3"
sdk/python/setup.py version="0.3.3"
21,504 lines of new functionality carries no version. Building or publishing
now would ship 0.4.0 features under the 0.3.3 label.
--------------------------------------------------------------------------------
4.4 HIGH — Entire workstream is uncommitted
--------------------------------------------------------------------------------
Branch fix/stress-test-findings: 0 commits beyond main.
86 uncommitted files in the working tree.
There is no point in history at which this work is recoverable. A single
mistaken git clean, checkout, or reset loses 21,504 lines.
--------------------------------------------------------------------------------
4.5 MEDIUM — Claimed feature with no implementation
--------------------------------------------------------------------------------
MASTER_PLAN promises: "handoff protocol" for shared agent sessions.
grep for "handoff" in jiro/mcp.py: 0 occurrences.
Also claimed but unverified: "median time to first successful tool call under
two minutes on a clean machine" (setup-15) — no measurement exists.
--------------------------------------------------------------------------------
4.6 LOW — Live config warning on new runtime code
--------------------------------------------------------------------------------
Every `jiro tasks` invocation prints:
"runtime.lease_ttl_seconds is not greater than
runtime.max_attempt_timeout_seconds; a long attempt that blocks the event
loop could be reclaimed and re-executed"
The runtime's own validator (jiro/runtime/config.py:228) detects a genuine
safety hazard in its default configuration and warns. Defaults are
mis-configured relative to each other. Relevant because it directly describes
a duplicate-execution risk.
================================================================================
5. TEST EVIDENCE
================================================================================
New-module tests : 488 passed, 0 failed
Full suite (my earlier run): 1501 passed, 2 failed, 9 skipped
The 2 full-suite failures are both unrelated to 0.4.0 work:
- test_recipe_api: pre-existing; verified failing on clean main
- test_get_engines_for_category: my own regression, since fixed
Test coverage of new modules is genuinely strong. The gap is not unit testing —
it is integration/environment validation (4.1, 4.2) and shipping hygiene
(4.3, 4.4).
================================================================================
6. ARCHITECTURE ASSESSMENT
================================================================================
The 6-layer design in MASTER_PLAN is coherent:
L6 Agent Fabric — 16 adapters, sessions, one-command setup [built]
L5 Parameter Layer — params, TTL, decay, diff [built]
L4 Evidence — claims, support, packets, ledger [built]
L3 Graph Runtime — planner, cost model, waves, deadline [built, mistuned]
L2 Sense — engines, plugins, social, proxies, SSRF [unchanged]
L1 ... — storage [unchanged]
Strengths worth naming:
- "Additive only" is respected: no existing engine, endpoint or SDK shape
was broken by any of the 8 new modules.
- The capability ladder (R0-R4) is a far more honest framing than an "AGI"
claim, and the plan explicitly refuses the AGI label.
- ADR 0011 draws a clear line around what "internet as model parameter"
does NOT mean, preventing the most likely overclaim.
- Failure reporting is genuinely honest: node_timeout, gaps, engine
attribution, budget accounting — all surfaced.
Weakness:
- The plan's ambition (2x/3x, 15+ agents, R3) is written as though shipped,
while the evidence tier is empty. The writing runs ahead of the artifacts.
================================================================================
7. RECOMMENDED SEQUENCE BEFORE 0.4.0 CAN SHIP
================================================================================
Priority 1 — Protect the work (minutes)
1. Commit the current tree to the branch. 21,504 lines with no commit is the
single largest risk in this analysis.
Priority 2 — Make the flagship work (days)
2. Calibrate the graph node budget against measured latency. Current default
(1400ms/node) cannot serve web search (4,600-12,900ms). Either raise the
budget, make it adaptive, or have the planner estimate from observed
engine latency before committing to a plan.
3. Re-run: jiro graph run "what is the capital of France" must return a cited
packet, not a 502.
Priority 3 — Earn the claims (days)
4. Run benchmarks/run.py to generate the four artifacts, or delete the
claims from the public docs until they can be produced. Publishing
"2x faster" with no artifact contradicts the project's own CLAIMS.md.
5. Resolve the runtime lease_ttl / max_attempt_timeout default mismatch —
it warns about duplicate-execution, which matters for a durable runtime.
Priority 4 — Ship hygiene (hours)
6. Bump version to 0.4.0 across pyproject.toml, jiro/__init__.py,
npm/package.json, sdk/python/setup.py once features are frozen.
7. Implement or remove the "handoff protocol" claim from MASTER_PLAN.
================================================================================
8. BOTTOM LINE
================================================================================
0.4.0 is a real, substantial, well-architected expansion — not a plan document
dressed up as code. 21,504 lines, 488 passing tests, 16 working client
adapters, and 13 ADRs show disciplined engineering. The planner demonstrably
works and reasons about tradeoffs.
The gap between 0.3.3 and 0.4.0 is genuine in capability: from a search-and-
scrape API to a query-graph runtime with evidence compilation, durable tasks,
agent orchestration, and a memory subsystem.
But 0.4.0 is not shippable today for three reasons, in order of severity:
1. Zero of four headline claims have their required artifact. The project
wrote a rule forbidding this and then did it.
2. The flagship graph feature fails on "what is the capital of France"
because its budget is 3-9x below real search latency.
3. Nothing is committed and nothing is versioned.
None of these are architectural problems. All three are finish-line problems —
protect the work, tune the budget, run the benchmark. The foundation is sound;
the last mile is not walked.
================================================================================
END OF REPORT
================================================================================