M1-REPORT.md
Rama main · réplica de lectura del repositorio original.
# M1 — Free adoption unblock report
Date: 2026-07-13
Worktree: `/home/myles/CueCrux/Crux-tierpkg-m1-wt`
Branch supplied by operator: `feat/ingest-loader-m1`
Git commands run: none
## Outcome
Implemented `corecruxctl ingest <path>` as a top-level command. It accepts one
file or recursively walks a directory, turns supported content into local prose
documents, optionally embeds each chunk, batches safe requests, posts them to
`POST /v1/local/ingest`, reports sealed segment/receipt identifiers, and prints
the exact follow-up text-search command.
The preserved default-on work was audited and made consistent: local ingest and
HTTP text search now share one parser where only trimmed `0` or case-insensitive
`false` disables a route. Graph-expand and time-range remain opt-in. Health and
version feature reporting use the same text-search decision.
The existing bootstrap seeder and daemon startup wiring were not changed. Two
new embedded bootstrap patterns document one-command ingest and default-on text
search.
## CLI behavior built
- Supported inputs: `.md`, `.markdown`, `.txt`, and `.json`.
- JSON objects/arrays are recursively flattened to their non-empty string values;
numeric, boolean, and null values are not indexed as prose.
- Hidden directories, `.git`, `node_modules`, and `target` are pruned. Symlinks,
unsupported extensions, NUL-bearing content, non-UTF-8 content, and empty
supported files are skipped and counted.
- Markdown is split at ATX headings before long sections are windowed. Plain
text uses paragraph/whitespace-aware windows targeting 1,800 characters with
a 180-character overlap.
- Every request document carries its relative source path in `title`/`url`, its
RFC3339 mtime in `source_timestamp`, and a stable document ID containing the
BLAKE3 file hash. Every chunk carries `source_path`, `file_hash`, `mtime`,
`chunk_index`, and `chunk_count` metadata.
- Request batching enforces the public 4,096-document and 65,536-chunk limits.
It also fragments document appends at the storage layer's 1,024-event limit
and conservatively keeps serialized requests below 12 MiB so they remain
under the route's ordinary 16 MiB body ceiling.
- `--tenant` defaults to `local`, consistent with other corecruxctl portability
commands; `--corpus` defaults to `docs`; `--daemon-url` defaults to
`http://127.0.0.1:14800`.
- Daemon authentication follows the existing memory CLI convention by sending
`CRUX_AGENT_TOKEN` as a bearer token when present. The ingest route requires
`admin:write` when daemon authentication is enabled.
- `--dry-run` performs no network calls, including no embedding calls. It walks,
parses, chunks, constructs batches, and prints counts only.
- `--embed` requires `CORECRUXD_EMBEDDING_URL`, uses
`CORECRUXD_EMBEDDING_MODEL` (default `nomic-embed-text`), and POSTs each chunk
to the OpenAI-compatible `/v1/embeddings` shape. `OPENAI_API_KEY` is forwarded
when present. Empty, non-finite, or inconsistent-dimension vectors fail closed.
- BM25-only ingest is the default and needs no embedding endpoint.
Observed fixture dry-run output:
```text
files walked: 3
files ingested: 2
skipped: 1 files, 0 directories
chunks: 3
dry run: 2 documents prepared in 1 batches; nothing sealed
query next:
curl -sS -X POST 'http://127.0.0.1:14800/v1/query/text-search' -H 'Content-Type: application/json' -H "Authorization: Bearer $CRUX_AGENT_TOKEN" --data '{"tenant_id":"local","query":"YOUR QUERY","limit":10}'
```
The authorization header is printed only when `CRUX_AGENT_TOKEN` is present.
## Default changes
- `CORECRUXD_LOCAL_INGEST`: ON when unset; only `0` or `false` (trimmed,
case-insensitive for `false`) disables and makes the route return 404.
- `CORECRUXD_QUERY_TEXT_SEARCH`: the same default-on/explicit-off parsing; both
text-search and text-search-expand return 404 when disabled.
- `CORECRUXD_QUERY_GRAPH_EXPAND` and `CORECRUXD_QUERY_TIME_RANGE`: unchanged,
still opt-in.
- `/v1/version` and admin health feature reporting call the same text-search
parser, so reported state matches route state.
## Files authored or updated
CLI and tests:
- `crates/corecruxctl/src/ingest.rs`
- `crates/corecruxctl/src/lib.rs`
- `crates/corecruxctl/src/main.rs`
- `crates/corecruxctl/tests/ingest_dry_run.rs`
- `crates/corecruxctl/tests/fixtures_ingest/guide.md`
- `crates/corecruxctl/tests/fixtures_ingest/records.json`
- `crates/corecruxctl/tests/fixtures_ingest/image.bin`
Default-on audit and coverage:
- `crates/corecruxd/src/config.rs`
- `crates/corecruxd/src/http/mod.rs`
- `crates/corecruxd/src/http/query.rs`
- `crates/corecruxd/src/http/local_ingest.rs`
- `crates/corecruxd/src/http/tests.rs`
- `config.example.env`
Bootstrap and generated documentation aggregate:
- `crates/crux-observe/bootstrap_data/patterns.json`
- `llms-full.txt` (regenerated because a preserved linked-doc edit had made the
aggregate stale)
Preserved M1 files reviewed but not rewritten during this completion pass:
- `docs/agent-guide.md`
- `examples/rust/append_and_query.rs`
Explicitly untouched per the correction:
- `crates/crux-observe/src/bootstrap.rs`
- `crates/corecruxd/src/main.rs` bootstrap seeding/startup logic
Both untouched files are byte-identical by SHA-256 to the independent sibling
`Crux-tierpkg-m2-wt` worktree.
## Tests added
- Markdown heading-aware split.
- Plain-text window size and overlap bounds.
- JSON string-value flattening.
- Hidden/build directory pruning and unsupported-file counting.
- Supported-extension binary-byte skipping.
- Local-ingest request shape and provenance fields.
- Document/chunk/storage/request-size batching behavior.
- OpenAI-compatible embedding endpoint and request shape.
- Clap defaults for top-level `ingest`.
- Fixture-directory dry run with `--embed` proving that dry-run performs no
embedding or daemon network call.
- Shared default-on parser values, stock text-search/local-ingest routes,
explicit-off 404 behavior, and version feature reporting.
- Bootstrap pattern JSON parsing through the existing seeder tests.
## Quality gates and exact summaries
### Formatting
`cargo fmt --all -- --check` — PASS (exit 0, no output).
### Clippy
`cargo clippy --workspace --all-targets -- -D warnings` — BLOCKED by existing,
out-of-scope all-target warnings. The first failures are in
`crates/crux-session/tests/{invocation_chain,end_to_end,ce_full_parity}.rs` and
`crates/crux-session/examples/generate_fixtures.rs` for existing
`unwrap`/`expect`/`println`, plus an existing deprecated-call warning in
`crux-session/src/generator.rs`.
These files were verified against an independent unmodified state without git:
their SHA-256 hashes are byte-for-byte identical to the sibling
`Crux-tierpkg-m2-wt` worktree. The earlier corecruxctl all-target check likewise
found two existing `needless_raw_string_hashes` warnings in `code_chain.rs`,
which is also byte-identical to that sibling worktree.
Changed production targets and the new integration test are clean:
```text
cargo clippy -p corecruxctl --lib --bin corecruxctl -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 6.07s
cargo clippy -p corecruxd --bin corecruxd -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 16.78s
cargo clippy -p corecruxctl --test ingest_dry_run -- -D warnings
Finished `dev` profile [unoptimized + debuginfo] target(s) in 4.48s
```
### Requested package tests
`cargo test -p corecruxctl -p corecruxd -p crux-observe` compiled all requested
packages, then Cargo stopped after the first package failed because this managed
sandbox denies every loopback `TcpListener::bind`. Exact first-package summary:
```text
test result: FAILED. 764 passed; 44 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.73s
```
All 44 failures terminate at mock-server bind with
`Os { code: 1, kind: PermissionDenied, message: "Operation not permitted" }`.
No ingest test failed. Representative failing mock files (`benchmark.rs` and
`extensions.rs`) are byte-identical to the sibling worktree.
The packages Cargo did not reach were run separately:
```text
cargo test -p corecruxd
test result: FAILED. 1814 passed; 20 failed; 2 ignored; 0 measured; 0 filtered out; finished in 63.56s
cargo test -p crux-observe
test result: ok. 61 passed; 0 failed; 1 ignored; 0 measured; 0 filtered out; finished in 0.04s
test result: ok. 5 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.03s
test result: ok. 0 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
```
All 20 daemon failures are also loopback-bind denials in existing mock/proxy,
server-shutdown, MCP bridge, and witness-submit tests. Representative files
`corecruxd/src/{main,mcp_stdio,witness_submit}.rs` and
`corecruxd/src/http/engine_console.rs` are byte-identical to the sibling
worktree. The new default and health tests passed in this run.
Focused M1 tests are green:
```text
cargo test -p corecruxctl ingest::tests --lib
test result: ok. 15 passed; 0 failed; 0 ignored; 0 measured; 793 filtered out; finished in 0.63s
cargo test -p corecruxctl parse_ingest_defaults --bin corecruxctl
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 184 filtered out; finished in 0.00s
cargo test -p corecruxctl dry_run_chunks_fixture_directory_without_network --test ingest_dry_run
test result: ok. 1 passed; 0 failed; 0 ignored; 0 measured; 0 filtered out; finished in 0.00s
```
The `ingest::tests` filter also selects five pre-existing `observe_ingest` tests;
all ten new ingest unit tests are present in the passing set.
### Rustdoc
`cargo doc --no-deps -p corecruxctl -p corecruxd` — PASS.
```text
Finished `dev` profile [unoptimized + debuginfo] target(s) in 6.55s
Generated /home/myles/CueCrux/Crux-tierpkg-m1-wt/target/doc/corecruxctl/index.html and 1 other file
```
### Repository documentation integrity
`scripts/check-agent-docs.sh` — PASS after regenerating `llms-full.txt`.
```text
PASS: every agent-doc reference resolves in the tree.
```
## Deferred or constrained
- No requested CLI behavior is deferred.
- A live non-dry ingest was not attempted: the requested tests explicitly avoid
a live daemon, and this managed sandbox prohibits loopback sockets. Request
serialization, limits, auth header behavior, response parsing, and the
no-network dry run are covered without a daemon.
- The daemon's existing local-ingest handler deserializes but discards optional
document/chunk metadata when mapping to its raw-text seal model. M1 sends all
required provenance in the wire request, but daemon-side persistence of that
optional metadata remains a pre-existing follow-up. Persisting it would change
the sealed prose payload/metadata model and was not broadened into this CLI
deliverable.
- The two mandated workspace-wide gates cannot be made green inside this turn
without either granting loopback-bind permission or expanding scope into
unrelated repository-wide test/example lint cleanup. Their exact failures and
independent-file verification are recorded above.