vendor/rars/VENDORING.md
Rama main · réplica de lectura del repositorio original.
# Vendoring `rars`
`vendor/rars/` is a curated copy of the `rars` crate from our perf fork of
[bitplane/rars](https://github.com/bitplane/rars). It is **not** a git subtree
or submodule — it is a mirrored subset, produced by
[`sync-from-fork.sh`](sync-from-fork.sh).
Upstream-tracking policy is FREEZE + cherry-pick: watch the fork for decoder
fixes, port selectively, re-run the gates, then re-vendor.
## What is vendored
| Path | Source in fork | How |
| --- | --- | --- |
| `src/**` | `crates/rars/src/**` | exact mirror (`rsync --delete`) |
| `tests/fixtures/**` | `crates/rars/tests/fixtures/**` | **exact mirror — all of it** |
| `COPYING` | **not synced - ours** | states this fork's MIT OR Apache-2.0 and why; see below |
| `LICENSE-MIT`, `LICENSE-APACHE` | **not synced - ours** | the two texts upstream does not ship |
| `Cargo.toml` | `crates/rars/Cargo.toml` | **hand-maintained, see below** |
Not vendored: `tests/*.rs` (fork integration tests — we run only the inline
`--lib` tests), `benches/`, `fuzz/`, `python/`, `scripts/`, `target/`.
## Why `tests/fixtures/**` must always come along
The rars unit tests live **inline** in `src/lib.rs` and load inputs via
`env!("CARGO_MANIFEST_DIR")/tests/fixtures/rar15_40/...`. If any referenced
fixture is missing, `cargo test -p rars --lib` fails with `NotFound` on a clean
checkout.
This exact bug shipped once: an earlier sync copied `src/` but dropped
`tests/fixtures/`, and it was hotfixed by hand-vendoring only the two fixtures
the then-current tests happened to read. Hand-picking fixtures re-introduces the
same fragility — the next inline test that references a third fixture breaks
again. So the sync mirrors the **whole** fixtures tree; do not trim it to a
subset.
## Re-vendoring to a newer fork rev
```sh
# 1. Sync src/ + tests/fixtures/ + COPYING from your fork checkout.
RARS_FORK=~/path/to/rars ./vendor/rars/sync-from-fork.sh
# 2. Reconcile Cargo.toml BY HAND *only if* the fork's [dependencies] changed.
# The vendored manifest deliberately differs from the fork's:
# - de-workspaced deps (concrete versions, not `.workspace = true`)
# - version = "0.4.6+nzbfast"
# - [lints.rust] unsafe_code = "deny" (was forbid until the NEON
# blake2sp entry, 22 Aug 2026), unused_must_use = "deny"
# - no [dev-dependencies], no [[bench]] (not vendored)
# The script never overwrites Cargo.toml, so it survives a re-sync.
# 3. Gate: must stay green.
cargo test -p rars --lib
# 4. Commit.
git add -A vendor/rars
git commit -m "vendor/rars: sync to fork rev <rev>"
```
## Local divergences (re-apply after every sync)
`sync-from-fork.sh` mirrors `src/**` with `rsync --delete`, so anything we
change under `src/` is GONE the moment someone re-vendors. Keep the ledger
below current, and walk it after every sync. Each entry names a marker
comment carrying the same date, so `git grep 'nzbfast-local change'` under
`vendor/rars/src` finds what should be here.
**A new entry goes in THIS table, newest first - not in "What is
vendored" above, which lists PATHS and how they are mirrored.** Between
`40d85a1a88` (7 Sep 2026) and 8 Sep 2026, 64 divergence rows were
appended to that first table instead, because it is the first one a
reader scrolling to append meets; they are back here now. A row filed
there is not merely untidy - the re-vendor walk this section exists for
reads THIS table, so those 64 changes would have been silently dropped
on the next `sync-from-fork.sh`. The table below must also stay ONE
table: a blank line between rows ends it in Markdown, and one had crept
in five rows from the bottom.
| Date | What | Where |
| --- | --- | --- |
| 2026-11-09 | **The buffered RAR 5 decoder budgets filter WORK: each record charges its length (at least 4 KiB) against 4 x the member's output plus 256 KiB.** 9 Nov 2026 sweep, section C: `decode_member_from_reader_with_dictionary` pushed every filter record, overlapping or not, and `apply_filters` ran each over its whole range, so a few bytes per record bought records x output of filter passes (the streaming outputs are bounded by `STREAM_MAX_PENDING_FILTERS` and the hold limit; the buffered one by nothing). Conforming filters are disjoint, so their total is at most the output; the budget admits four full overlapping layers plus 64 tiny filters. Checked on the filter symbol only, never in the literal loop. Test `codec::rar50::tests::buffered_decoder_budgets_overlapping_filter_work` (100 identical full-range E8 records decode without the change, 3 still decode with it). NOT YET IN THE FORK. | `src/codec/rar50.rs` (`FILTER_MIN_COST`, `FILTER_WORK_PER_OUTPUT_BYTE`, `FILTER_WORK_SLACK`) |
| 2026-11-09 | **A RAR 5 member whose PACKED size passes the buffered decode limit takes the streaming decoder, whatever its unpacked size.** 9 Nov 2026 sweep, section C: `should_stream_decode` routed on the declared unpacked size alone, and the buffered branch reads the whole packed range into one `Vec` (`packed_data_with_password`), so a header pairing a small unpacked size with a multi-GB packed size read all of it into memory (real archive bytes, but unbounded). A conforming compressed member's packed size is within a few blocks of its unpacked size, so only members already past the whole buffered limit (512 MiB) reroute; the buffered fallback after a streaming filter bail is held to the same bound. Test `rar50::extract::tests::a_member_whose_packed_size_passes_the_buffered_limit_streams` (a routing test; fails without the change). NOT YET IN THE FORK. | `src/rar50/extract.rs` (`should_stream_decode`, `write_file_to`) |
| 2026-11-09 | **RAR 2.9 VM: generic program code is budgeted, a generic filter may not grow its block, and the AUDIO filter's channel loop is clamped to the block.** 9 Nov 2026 sweep, section C. (1) The program table held up to `MAX_VM_PROGRAMS` x `MAX_VM_CODE_SIZE` (512 MiB) of code and a parsed `rarvm::Program` costs up to ~43x its code; GENERIC code (anything `identify_standard_filter` does not recognise, which no known writer emits) is now held to `MAX_VM_GENERIC_CODE_TOTAL` (16 x 64 KiB), returned on a table reset. (2) A generic program sets its own output length (up to RARVM's 256 KiB) and that output replaced the block, so 1-byte blocks could each emit 256 KiB and one span's 8192 filters build a 2 GiB buffer in `filtered_range`; output longer than the block is refused (standard filters never change the length). (3) `audio_decode` looped `regs[0]` channels, a raw u32, and every channel past `data.len()` is an empty pass; clamping is output-identical (measured: `u32::MAX` channels cost ~4 s per filter before). Tests `codec::rar29::tests::vm_generic_program_code_is_budgeted`, `generic_vm_filter_may_not_grow_its_block` (both fail without the change), `audio_filter_channel_count_is_clamped_to_the_block` (passes either way, ~4 s slower without). NOT YET IN THE FORK. | `src/codec/rar29.rs` |
| 2026-11-09 | **`Rar20Decoder::decode_member_from_reader` decodes and flushes in 4 MiB steps instead of materialising the whole member.** 9 Nov 2026 sweep, section C: it decoded the entire member into `output` before writing a byte, so the declared unpacked size (a header field, reachable from a few bytes of matches) was the decoder's resident size. A step boundary is a solid member boundary to the decoder (`copy_match` parks the rest of a match in `pending_match`, `decode_until` drains it), and no RAR 2.0 match reaches past `MAX_HISTORY` (the last offset slot ends at 2^20), which `trim_history` keeps. The PACKED input is still read whole (real archive bytes, not an amplification; a refilling bit reader is the redesign RAR 2.9 got on 16 Sep 2026). Test step is 64 KiB. Test `codec::rar20::tests::decode_member_from_reader_flushes_in_steps` (one 800 KB write without the change). NOT YET IN THE FORK. | `src/codec/rar20.rs` (`RAR20_FLUSH_STEP`, `decode_member_from_reader`) |
| 2026-11-09 | **The pooled RAR 5 worker refuses a digest-less member that decodes short of its declared size, as the serial branch has since 21 Sep 2026.** 9 Nov 2026 sweep A12: `decode_pooled_member` claimed to mirror `write_file_to`'s buffered branch but lacked its declared-length check, so a member with no FILE CRC32 and no BLAKE2sp whose packed stream ended early (`map_truncated_unverified_payload` turns that into `Ok(Vec::new())`) was written out EMPTY and reported extracted whenever the pool took it. Both sites now call one helper, `refuse_undigested_length_mismatch`, which also mends the serial error string (a lost line continuation had embedded a run of spaces in it). Test `rar50::extract::tests::a_digestless_member_that_decodes_short_on_the_pool_is_refused` (fails without the pooled call). NOT YET IN THE FORK. | `src/rar50/extract.rs` |
| 2026-11-09 | **A RAR 5 block header's size field is capped at the format's three-byte limit (2^21 - 1) before anything is allocated, and the writer refuses to exceed it.** 9 Nov 2026 sweep A8: the six header readers checked the field only against the bytes remaining, and on a stream source parsed with an unbounded `expected_len` (nzbfast TODO 118.2 (b), the chase's open-length volume) that is no bound, so `ArchiveSource::read_range` zero-filled an attacker-chosen buffer before the CRC could reject it; an allocation failure aborts the process. Measured: a 1 GiB size field on an open stream allocated 1 GiB and failed `TooShort`; it is now refused `InvalidHeader` from a read of at most 14 bytes. The format's own field width is the limit, so no conforming archive reaches it. Test `rar50::tests::oversized_header_size_on_open_stream_is_refused_before_reading`. NOT YET IN THE FORK: port it before the next `sync-from-fork.sh`. | `src/rar50.rs` (`RAR5_MAX_HEADER_BODY`, `header_body_len`), `src/rar50/write/mod.rs` (`block_header_image`) |
| 2026-09-28 | **Only a multi-block RAR 2.9 member's LAST block ends the file, and the PPMd planner's matches stay inside the archive's dictionary.** Two RAR 4 writer defects that reference unrar 7.23 `t` reports as "checksum error" while this crate's decoder reads both back. (1) The 27 Sep end-of-member row below made EVERY LZ block end the file, and the writer encodes a member over `RAR29_LZ_BLOCK_SIZE` (1 MiB) as several blocks (`encode_member_blocks`), so unrar stopped after the first megabyte of every such member; found by rarkit's oracle pin (40 MiB level-3 text). `encode_member_inner` takes `ends_file`; earlier blocks write the lone `1` ("same file, a new table follows"). (2) PRE-EXISTING: PPMd LZ escapes were capped at 0x1000001, never at `max_match_distance` as the LZ planner's are, so level-5 text (PPMd) drawing a match past unrar's window failed; unrar 7.23 measured as a black box still reads ~200 KB back under a declared 128 KiB and fails 300 KB back, first failure at 282,868 bytes of random-word text. The writer passes its dictionary through new `encode_rar29_ppmd_within` / `encode_rar29_ppmd_with_filter_within`. postfast's RAR 4 compressed containers go through this writer and were affected by both; the in-repo RAR 4 fixture generator writes stored volumes only and was not. ALREADY IN THE FORK as fdb7030 (on `mine/perf`) and hand-applied here, `src/rar15_40/write.rs` byte-identical to it and `src/codec/rar29.rs` differing from it only by the fork's pre-existing 66 lines, so there is no marker comment to keep and the next sync carries it. Claim `rars-rar29-writer-broken-by-end-of-member-fix-28sep`; host repo `research/RARS-RAR29-MULTIBLOCK-AND-PPMD-WINDOW-2026-09-28.md`. | `src/codec/rar29.rs` (`encode_member_inner`/`encode_member_blocks`, the PPMd `*_within` entry points and `best_ppmd_match`), `src/rar15_40/write.rs` (the five PPMd call sites, one test) |
| 2026-09-27 | **A RAR 2.9 LZ member now ENDS THE FILE after its end-of-block symbol instead of announcing a table.** The encoder wrote symbol 256 and then a lone `1` bit, which `read_end_of_block` itself reads as "same file, a new table follows"; it now writes `0` then `1` (new file, the next member reads its own tables). This crate's reader forgave the old ending (`finish_lz_member` accepts a missing table when the trailing bits are zero), so every round trip here was green, but a reader that follows the format parses a phantom table out of whatever its input buffer holds past the member. unrar 4.20, 5.50 and 7.23 all do; when the stale bytes open a PPMd block header with flag 0x40 the phantom replaces the PPMd escape char, and the next PPMd member of a SOLID archive then decodes its escapes as literals and fails its CRC. Found by rarkit's compression rivals round (27 Sep 2026): a 9,681-member solid RAR 4 archive written under `FilterPolicy::Auto` (whose members alternate between PPMd and LZ) failed 5,090 members under all three unrar versions; minimal reproduction PPMd, LZ, PPMd; rebuilt with the fix it tests clean on all three. The line is shared by every RAR 2.9-4.x compressed write, so postfast's non-solid RAR 4 compressed layouts carried the same ending (benign under unrar there, since a non-solid member starts from a fresh decoder) and change by one bit per member. Pinned by `lz_member_ends_the_file_rather_than_announcing_a_table`, which fails on the old line. The reader is NOT tightened: archives written before this still decode here. Ported to the fork as perf `65cf0ff` (27 Sep, claim `rars-fork-rar29-end-of-member-port-and-oracle-pin-28sep`). Claim `nzbfast-rars-writer-filters-4g-and-unrar-hang-27sep`; host repo `research/RARS-RAR29-END-OF-MEMBER-AND-AUTOSIZE-2026-09-27.md`. | `src/codec/rar29.rs` (`encode_member_inner`'s end marker, one test) |
| 2026-09-27 | **A RAR 5 member's FINAL match may run past the member's declared size.** It was refused with `RAR 5 match exceeds output limit`; reference unrar 7.23 writes only the declared bytes and keeps the whole match in the solid window, so the next solid member's distances count the overrun. Found by rarkit's fuzz lane (the fixture is its reproducer, byte for byte). Buffered decoder: the crossing match lands whole, output is truncated, the overrun joins `history`. Ring: `copy_match_across_limit` materializes the overrun behind `emit_cap`, which no flush passes. Flat: the twin, growing the buffer once and never emitting at or past `limit_pos`; `written()` stops at the declared end. Solid chain: a block that ends a member carries `ends_member`, and the apply stage fails the chain (so the extractor's serial retry takes the group) when the output does not sit on that member's end - an interior overrun would otherwise shift every later member. Still refused: a match starting AT the limit, a bad distance on the crossing match, a filter reaching into the overrun, and a wrong declared checksum (unrar exits 3 on that file too). NOT changed, and measured against unrar: SYMBOLS after a member's declared size, which unrar also keeps as window; rars still stops there, so the next member fails its own checksum. ALREADY IN THE FORK as 6a81e4d (on `mine/perf`) and hand-applied here byte-identical to it, so there is no marker comment to keep and the next sync carries it; the fork's `tests/rar50_fixtures.rs` holds the integration tests (not vendored). Claim `nzbfast-rars-solid-match-overruns-member-27sep`. | `src/codec/rar50.rs` (`copy_match`, `decode_member_from_reader_with_dictionary`, `StreamingOutput::{copy_match_across_limit, flush, finish, check_member_end}`, `FlatOutput::{copy_match_across_limit, emit_ready, written, check_member_end}`, `TapeJob`/`BlockTape::ends_member`, four tests), `tests/fixtures/rar50/solid_match_overruns_member*.rar`, `tests/fixtures/rar50/solid_interior_match_overrun.rar` |
| 2026-09-24 | **A split member's continuation is no longer refused for carrying a different stored NAME** (RAR 1.5-4.x and RAR 5, whole-set walk and incremental path). nzbfast GH #92: posted RAR4 sets failed native unpack with `RAR 1.5 split entry name changed`, and the Docker release image ships no unrar to fall back to. Measured: a continuation renamed by case, or to a different name of a different length, with its header CRC re-sealed, tests clean under unrar 7.23 (RAR4 and RAR5) and extracts byte-identical under the FIRST fragment's name, so the four `file.name != pending.name` refusals went and the member keeps the Start fragment's name as before. A continuation from a different member is still refused by the checks that follow (the two broken-chain pins now name `unpacked size changed` / `compression info changed`) and by the member's CRC. Pinned by `rar15_40_split_continuation_with_a_different_name_extracts_under_the_first` and its `rar50_` twin, which rewrite the fixture bytes in-test rather than adding fixtures; both fail on the old code with the #92 error. RAR 1.3's two twins in `rar13.rs` are untouched. Not yet in the fork. | `src/rar15_40/extract.rs`, `src/rar50/extract.rs`, `src/lib.rs` (tests) |
| 2026-09-21 | **A stream source's length is read LIVE, narrowed to the source's own `total_len()` once it reports one.** nzbfast TODO 118.2 (b): a chased RAR5 volume whose `=ybegin size=` is not yet trusted (the random-`size=` poster) now parses with an unbounded `expected_len` and its `FrontierBuffer` learns the length later. Every bound that read the captured `len` - the four `Stream` arms in `source.rs`, `len()`/`bytes()`, and in `rar50.rs` the `read_block_header_from_source` prefix sizing, `parse_block_header_image`'s data-area bound and the walk's `while pos < archive_len` - goes through `stream_len_now` / `ArchiveSource::stream_len` at each use, and `read_block_header_from_source` reads its header prefix BYTE-EXACT on the size vint (CRC plus one vint byte, then one more while the continuation bit is set) instead of a speculative 14-byte read clamped at `len` - over an 8-byte END block that read asked the source for six bytes past the volume, which on an open frontier parks until the download ends (`chase source sealed ... bytes never arrived`) and on one that closes meanwhile answers `Ok(0)` (`input is too short`). `parse_archive_blocks` and `walk_archive_blocks` take `archive_len` as `&dyn Fn() -> usize`; the memory and file parses pass `&\|\| archive_len`. A source that reports no length, or one past `len`, changes nothing. `rar15_40.rs` is untouched: nzbkit waits for a RAR4 volume's bound to close before it parses one (its end IS its length). Marker `nzbfast-local change, 21 Sep 2026`. | `src/source.rs` (`stream_len_now`, `ArchiveSource::stream_len`, the five `Stream` arms), `src/rar50.rs` (`parse_stream_impl`, `resume_walk`, `parse_archive_blocks`, `walk_archive_blocks`) |
| 2026-09-18 | **COMMENT ONLY, and the comment is the point: the three `unsupported_encryption()` guards in `rar15_40.rs` are NOT the reason encrypted compressed RAR 2.9-4.x members failed, and one of them now says so at length.** `unpacked_data_with_rar29`, `unpacked_data_with_rar15` and `write_rar29_to` each refuse an encrypted member, and an 18 Sep 2026 review note read that as the defect behind `rarfast t` on the fork's own `encrypted_newnaming_rar300` fixture printing a bare `Total errors: 1`. They are unreachable for an encrypted member: their ONLY caller is `DecoderSession`'s `CodecState`, which tests `is_encrypted()` first and drives the cipher itself through `packed_reader_for_decode`, and the vendored lib suite already pins that path (`extracts_encrypted_rar300_old_numbered_volume_set` and its new-numbering twin decode the member and match CRC `0xddc95682`). The real defect was in the rarfast CLI, which is NOT vendored: its unrar dialect derived `Opened::is_volume` from the RAR 5 variant alone, so no RAR4 volume chain was ever followed. Fixed in the fork, with two CLI regression tests that drive the chain from the FIRST volume alone; nothing in `vendor/rars` changes behaviour and no nzbfast path moves. Kept here so the next lane reading these three sites does not re-diagnose the same non-defect. Hand-applied byte-identical to the fork hunk, so the next sync carries it. | `src/rar15_40.rs` (three comments) |
| 2026-09-18 | **A genuine DOS RAR 1.55 directory is a directory.** `FileHeader::is_directory` tested only the 0x00e0 window bits; RAR 1.55 writes directory headers with flags 0x8000 only, unpack version 15, host OS 0 and attribute 0x10, so `SUB` extracted as an empty file and nothing beneath it could be created. A pre-2.0 entry from a DOS-attribute host (MS-DOS, OS/2, Win32) with 0x10 set now reads as a directory. ALREADY IN THE FORK as 7cb704f and hand-applied here after the 18 Sep clean-room sync, byte-identical to it, so there is no marker comment to keep and the next sync carries it. The new 668-byte fixture is the writer's own bytes (README beside it); nzbkit-base's own RAR4 header parser had the same flags-only test and is fixed beside this in the host repo. | `src/rar15_40.rs` (`is_directory`, `HOST_*`/`DOS_ATTR_DIRECTORY`, two tests), `tests/fixtures/rar15_40/rar155_dos/` |
| 2026-09-18 | **CLEAN-ROOM SYNC: every row below is now in the fork, and `src/` is an exact mirror of fork `11f2fc9` with nothing re-applied by hand.** This is the walk the section asks for, done in the FORK first because `sync-from-fork.sh` deletes what it does not find there. The rows from the 15 Sep sync to this one that the fork lacked (sweep 16 Sep MEDIUM 8/9/11/13 incl. the RAR 3 VM filter-hold cap, LOW 28/31/65/67, the 24 LOW items, LOW 69/70, `next_volume_follows`, the QO CRC test, the `fold-team-probe` feature, the IO_BUF and SCAN_WINDOW comments) were ported as fork `615dc6b`; every other row was already there. Proven two ways with renames applied (every added line of the 42 vendor commits since `879002e19` found in fork src; every name defined in vendor src resolved there) - residue itemised in the host repo's `research/rars-originality-2026-09-18/chip1-subset-proof/`. The sync also brought the clean-room rewrites of specs A-D, which replace the bodies several older rows below describe (`src/fast.rs` is gone, folded into `codec/address_filters.rs` by fork `994bcd0`; `codec/rar13.rs` is now `codec/rar13/`), and the deferred MAP E renames (`Unpack15/20/29` -> `Rar15/20/29Decoder` and friends; pairs in `research/cleanroom/MAP-E-renames-comments.md`). So a row below that names an old identifier or file describes history, not a hunk to re-apply. `Cargo.toml` by hand: `bench-internals = []` (the fork's src gates the address-filter harness on it) and the unsafe-file comment now says three files. | `src/**`, `tests/fixtures/**`, `Cargo.toml` |
| 2026-09-18 | **`shard_group_crcs`'s per-shard CRC buffer is RIGHT-SIZED to the largest group slice, in BOTH arms, through one `shard_crc_buf_len` helper.** The `parallel` arm allocated a fresh zeroed 256 KiB `IO_BUF` **per shard** - 200 a call, 52 MB of zeroing a call - inside its `into_par_iter().map(..)`, and the serial arm one per call. Both read `min(group.len, buf.len())`, and `rar5::recovery_groups` derives every `group.len` as `min(RAR5_RECOVERY_PARITY_PER_RECORD_MAX, group_count - offset)` where that constant is 64 KiB, so **at least three quarters of every buffer was dead tail on EVERY archive, not on a corpus** - and much more on a small one (a 4 MiB volume has one group of 20,972 bytes, 8% of `IO_BUF`). The size is hoisted out of the `map` so it is computed once rather than 200 times; no signature change, same bytes read, same CRC64 computed. STRICTLY NO WEAKER ON HOSTILE INPUT than what it replaced - `IO_BUF` stays the ceiling through the `.min`, so a crafted `group_count` buys no more than the flat spelling allocated unconditionally - and the `.max(1)` keeps the read loop off a zero stride. **THIS REVERSES A LANDED DECISION.** `research/RARS-SHARD-CRCS-PER-SHARD-BUFFER-2026-09-16.md` (`c813cbd29`) declined exactly this change, on evidence its own limit 4 names as **macOS libmalloc only**: that the untouched tail never becomes resident, and that alloc+free costs the same at 256 KiB and 64 KiB. Both hold on macOS and neither holds on musl. MEASURED ON TWO PARTS, one binary carrying both arms behind a measurement-only environment switch so codegen cannot differ, warm with `mincore` residency proved per run, 32 runs a cell, A/A either side and every headline twice: **Xeon D-1531 static musl, quiet - 0.5848 and 0.5831 min/min on a 16-volume set of 16 MiB volumes** against A/A floors of 1.0014 and 1.0003 (1,378.6 ms to 804.6 ms), and **0.1069 and 0.1067 on a set of 4 MiB volumes** against 0.9967 and 1.0012 (1,165.3 ms to 124.4 ms), with **`minflt` 1,668,592 -> 439,885 a run (ratio 0.264, reproduced to 4 faults) and 1,666,317 -> 3,566 (ratio 0.002)** - the two shapes bracketing musl's recycling threshold between 20,972 and 65,536 bytes; **quiet M1 Ultra - a NO-OP**, 1.0089 and 0.9926 min/min inside an A/A band of 1.0047 and 0.9986, `minflt` flat at 610-621. So it is a musl win and an Apple no-op rather than a trade, which is what makes it safe to land everywhere at once. The base arm's absolute fault count matches the eager-faulting arithmetic to 1.8% (12,800 pages a call x 16 volumes x 8 reps), which is what makes the clock readable. Correctness gate: the driver's `count` mode ran the real `damaged_shards_by_group` against a replica at the other buffer size on 16 of 16 volumes of both fixtures on all three boxes, every line `GATE=agree`; and `detection_finds_the_same_shards_at_any_buffer_size` pins it at 1, 7, 4096 and `IO_BUF` bytes against a fixture asserted to report real damage. `shard_crc_buf_len_is_the_largest_group_and_never_a_quarter_of_io_buf` pins the format ceiling, which is the correction to that note's limit 2 - the case it reserved (a group larger than `IO_BUF`) cannot occur. `IO_BUF`'s own doc block is re-tensed rather than edited away: its 17 Sep census said all six non-`read_chunk_at` sites use the buffer AS the stride, and **the true split is three and three** - `copy_range`, `copy_file_verified` and `crc32_of` do, while both arms of `shard_group_crcs` and `damaged_shards` read `min(group.len, buf.len())` and leave the rest dead. All three of those now size through the helper. `damaged_shards` is included for CONSISTENCY and moves no production timing - every caller of it in the crate is a test, as handoff item 4 priced it - so that a future promotion of it into production cannot silently re-acquire the tail; it is not counted as a measured win anywhere. `map_init` is still not the lever and that note's section 5 stands untouched: right-sizing makes each buffer cheap rather than making them fewer, so it forbids no work stealing. Vendored lib suite 1032 passed, 0 failed. **In the fork** as `e7da8f5` plus the census correction `8c9537b` on `perf` (claim `rcbuf-per-shard-minflt-musl-18sep`), the same edit applied to both trees. The fork's merge `b3d58ed` resolves one conflict against `11f2fc9`, a parallel lane's port of the SUPERSEDED four-and-two wording of this same paragraph, in favour of the corrected text. Findings: `research/RCBUF-MINFLT-MUSL-2026-09-18.md`. | `src/recovery/stream.rs` |
| 2026-09-18 | **Members can extract CONCURRENTLY, each into its own writer, through a NEW entry that nothing in nzbfast calls.** `Archive::extract_to_concurrent` / `extract_volumes_to_concurrent` (crate level and `rar50`) take a worker-side opener, `ConcurrentOpen`, beside the usual `open`: pooled members are decoded and verified then written by the worker that decoded them (stored members up to 8 MiB join the pool for such a caller), and members too big to buffer stream through the MT pipeline K at a time, K from the core count shed until the estimated windows fit the policy or one default window. The walk still calls `open` once per member in archive order. `extract_to`, `extract_volumes_to*` and the existing pool plan are UNCHANGED when no opener is passed (the plan's only new input is a `concurrent: bool` the old callers pass `false`), so the engine's own writers see no contract change. Mirrored from the fork's `perf` commits `d2a5597`, `b82f8db` (a pooled member that fails to decode is re-run serially) and `8b57bc6`, where rarfast's unrar dialect uses it (`research/UNRARFAST-CONCURRENT-MEMBERS-2026-09-18.md`). Not a local divergence: the fork carries it, so a re-sync keeps it. | `src/lib.rs`, `src/rar50.rs`, `src/rar50/extract.rs` (`ConcurrentOpen`, `member_pool_plan`, `extract_volumes_pooled`, `concurrent_big_shape`) |
| 2026-09-18 | **`#[inline(always)]` on `Unpack29::copy_match` and `Unpack29::read_offset`, the two calls the RAR 2.9 match arm makes once per match.** Checked on an unstripped fork build at `5465359`: rustc already inlined `read_offset` and kept `copy_match` out of line, so what the pair changes today is `copy_match` inlined into `decode_lz`; the attribute on `read_offset` holds that inline against a later edit that grows its body. Measured on an M5 Max with the fork's `rarfast`, interleaved arms, same tree either side: rarbench's RAR4 -m3 test leg 0.652 -> 0.646 s over 15 reps and 0.678 -> 0.670 over 31 (about -1%), a held-out RAR4 archive of binaries 0.214 -> 0.211 s for ten tests a sample (-1.4%). An earlier sitting read 3 to 3.7% and this one does not reproduce it; the comment at the site gives the figure measured here. Inlining `Huffman::decode`, `peek_bits` and `read_bits` as well bought nothing, so it is not done. Not re-measured through nzbfast's own RAR4 path: the vendored `rar29.rs` predates the fork's clean-room address-filter rewrite, but both function bodies are byte-identical between the trees (compared at fork `038be1a`) and each tree calls them from the same eight sites. Vendored lib suite 1022 passed, 0 failed. **In the fork** as `038be1a` on `perf` (claim `rarfast-rar4-inline-and-pgo-recipe-18sep`), ported here by hand rather than by `sync-from-fork.sh`, for the reason the 16 Sep rows give. The same fork commit adds `scripts/pgo-build.sh`, which is fork-only and not vendored. Findings: `research/RARFAST-RAR4-INLINE-AND-PGO-2026-09-18.md`. | `src/codec/rar29.rs` |
| 2026-09-17 | **`read_chunk_at`'s CRC64 buffer is RIGHT-SIZED to the record, not a flat `IO_BUF`.** It is allocated once per candidate `{RB}` RECORD inside the scan loop - 1,280 times over a 16-volume set - and the loop it feeds never reads more than `total_size - 0x0c` bytes, so a full 256 KiB was dead tail on every ordinary record (43,604 B on a 16 MiB volume, 17% of the buffer). `total_size` is parsed nineteen lines above, so the buffer is simply sized where it is created: no signature change, no hoist, same bytes read and same CRC computed. STRICTLY SAFER on hostile input than what it replaced - `total_size` is attacker-controlled to a `u32`, so `IO_BUF` stays the ceiling through the `.min` and no crafted header buys more than the old spelling allocated unconditionally, while the bottom end cannot underflow because the `header_size` check above rejects any `total_size` below `CHUNK_FIXED_HEADER` (0x48). The loop bound is now the buffer's own length rather than the constant, which puts the slice in range by construction. `IO_BUF`'s floor rule is respected and is why the `.min` stays: it must not fall below one recovery chunk or this CRC read splits and multiplies `open()` on a per-read-open source - right-sizing to the record IS one chunk, and the count gate holds `crcreads` at 80 a volume in both arms. MEASURED ON TWO PARTS, one binary carrying both arms behind a measurement-only environment switch so codegen cannot differ, warm with `mincore` residency proved per run, 32 runs a cell, A/A either side and every headline twice: **Xeon D-1531 static musl, quiet - record arm 0.3679 median / 0.3778 min-min and 0.3759 / 0.3806 on the repeat** against A/A floors 1.0016 and 1.0002 (2,242 ms to 847 ms), raw fallback 0.6539/0.6458 and 0.6273/0.6460 against 0.9999 and 0.9985, with **`minflt` 753,588 -> 104,038 a run, ratio 0.138, reproduced to ONE fault** and the identical 0.138 in the other caller; **M1 Ultra - 0.9891/0.9892 and 0.9889/0.9887** against floors 0.9997/1.0003 and 0.9996/0.9996, with `minflt` FLAT at 376-381 and the whole saving in `stime` (93.45 -> 88.81 ms, `utime` flat to 0.1%). So it is TWO bills: the fault term is musl's alone and a smaller size term is on both, because the kernel-side cost of an allocation scales with its size even where none of its pages is ever touched. Count gate: 32 of 32 COUNT lines byte-identical between the arms on both callers on both parts. The seven `vec![0u8; IO_BUF]` sites in this file were censused by frequency and only this one is per-record; the `IO_BUF` doc comment now says so, so the next reader does not sprinkle it where it buys nothing. **In the fork** as `7ef0267` on `perf` (claim `read-chunk-at-per-record-iobuf-alloc`), the same edit applied to both trees and the `read_chunk_at` block byte-compared between them. The `IO_BUF` DOC BLOCK differs between the trees by pre-existing drift (vendor carries the 17 Sep in-situ ladder table the fork never took), so the census paragraph is worded to its own surroundings in each. Findings: `research/READ-CHUNK-AT-CRC-BUFFER-2026-09-17.md`. | `src/recovery/stream.rs` |
| 2026-09-16 | **`Unpack29::decode_member_from_reader` refills the packed stream mid-member instead of buffering the whole of it.** Found by the 16 Sep 2026 adversarial review (its item 10): the RAR 2.9-4.x entry point drained the reader to EOF into one `Vec` before decoding a byte (`append_from_reader` -> `read_to_end`), so an 8 GB -m3 RAR4 member cost its packed size in heap whatever any budget said, on the chase path AND on the ordinary disk path. It now holds a `STREAM_INPUT_WINDOW` (1 MiB) window, `compact()` drops what is consumed, and the loop tops the window up each turn. The refill is PROACTIVE, not error-driven: the bit reader has no mid-symbol rollback, so a `NeedMoreInput` raised part-way through a symbol has already consumed bits and moved decoder state, and resuming from there would be a corrupt-output bug - worse than the allocation. So the decoder pauses at a clean symbol boundary (the top of `decode_until`'s loop, and in `decode_lz` immediately after the literal burst and before the generic symbol) whenever fewer than `STREAM_INPUT_MARGIN` (128 KiB) bytes are held and more can still arrive. The margin is sized by the widest excursion past a pause point and the constant's comment carries the arithmetic: `read_vm_code`'s length is a bare 16-bit count read BEFORE `MAX_VM_CODE_SIZE` is checked, so one arm can spend 65538 bytes; `read_tables` is under 1.2 KiB; every other arm is under 16. PPMd blocks are deliberately NOT streamed - a PPMd symbol's input cost is bounded only by the escape chain and `read_vm_code_ppmd` can spend 65539 symbols, so no provable margin exists this side of tens of megabytes - and `decode_until` absorbs the rest of the reader the moment a block selects PPMd, which is the old peak on a shape that was never better than it. The reader is still drained to EOF before `finish_member`, so the chase's consumption watermark and `remaining_bits_are_zero`'s whole-tail scan see exactly what `read_to_end` used to leave them. Measured on a 146,135,476-byte single-member -m3 RAR4 set (`gen_rar4` over 150 MB of near-incompressible data), `bench_extract` to a sink under `/usr/bin/time -l`, five alternating rounds an arm: peak RSS **288.6 MB -> 21.95 MB** (medians; 13.1x), and the new figure is FLAT in member size - it is the 1 MiB window plus `MAX_HISTORY` and its 8 MiB drain hysteresis on the output side. The old figure is about TWICE the packed size, which is `read_to_end`'s geometric growth plus its realloc copy. Wall, 50 timings an arm (ten alternating processes of five iterations, warm cache, dev Mac at load ~52): **0.947x on the median, 0.976x on the minimum** - no regression, and a small win, the 1 MiB buffer staying in cache where a 146 MB one does not. Output is byte-identical across 37 archive groups (every `tests/fixtures` set, every `crates/nzbkit-base/testdata/rar4*` set, and a purpose-built RAR4 corpus of text, x86-opcode, mixed and PPMd members) plus the 146 MB member, with 16 more groups refused identically by both arms. Eight new lib tests hold it, and byte-identical output is explicitly NOT the whole check - `state_digest` compares the carried state a solid successor reads (Huffman levels, offset and length memory, pending match, filter and program tables, block cursor, the retained history window and the absolute count of bits consumed) against the buffered path after every member, with its own negative control. **In the fork** as `47cb5a7` on `perf` (claim `bugsweep16-item10-rar29-refill`), vendor-first and cherry-picked there by hand rather than by `sync-from-fork.sh`, which would have carried the fork's unvendored clean-room RAR 2.9 work with it; one conflict, the test import block, and three hunks of `cargo fmt` collateral reverted out, because the fork tip's own `rar29.rs` is not fmt-clean under this toolchain and `fmt -p rars` rewrote 26 other files besides. Fork suite 1128 passed, 0 failed on the merged tree. NOTE the landed nzbfast commit message names `83764f4` for this port: that was the same change on a local fork branch 166 commits BEHIND `mine/perf`, which was discovered at push time and never pushed; `47cb5a7` on the current tip is the port. | `src/codec/rar29.rs` |
| 2026-09-16 | **Six wall-clock assertions in the recovery tests now assert WORK DONE, and `src/recovery/workgauge.rs` is the instrument.** Four were `elapsed().as_millis() < 500` and two `elapsed().as_secs() < 5`; each was reaching for a claim about complexity or ordering ("this scan is not quadratic in the markers", "this refusal did not size the grid first") that a clock can only proxy - it reds on a busy box and passes on a quiet one whatever the code did. Not hypothetical: nightly's `one-process-loaded` runs `rars:lib` under deliberate load, and a mutation census on 16 Sep 2026 was told a mutation was "caught by 1 test" where the test was `rar5_inline_recovery_scan_is_not_quadratic_on_dense_markers` and it had caught nothing. `workgauge` is two THREAD-LOCAL counters (not global - this crate's `parallel` feature puts rayon workers in the same process, and a shared counter is the defect the one-process jobs exist to find), charged by the production paths and compiled to EMPTY BODIES under `#[cfg(not(test))]`, which is every build nzbfast ships. The two scans now assert the hashing budget itself (`<= 5 * len`: the `4 * len` budget plus the one-candidate overshoot the loop allows, against ~110 GB unbudgeted) AND a floor proving they validated a candidate at all; the three size-from-a-declaration refusals assert ZERO charged cells; `rev_parse_rejects_wire_scale_volume_counts` has no wire-scaled allocation to count - its header buffer is capped at 1 MiB before any count is read - so it names the exact refusal each forged header earns instead, which is strictly stronger than what it had: with the field check deleted the old `is_err()` was GREEN. Each of the six shown red against its own break, checking the test that fired was the one written for it. **Found on the way**: the STREAMING fixture left its two version bytes as filler, and `read_chunk_at` checks them BEFORE it hashes where the in-memory twin hashes first - so every candidate died there, the scan never CRC64'd a byte, and that test's whole subject was never reached. A clock could not tell; a linear walk over 52,000 refused markers is fast, and it passed. **In the fork** as `5d553f4` on `perf` (claim `rars-wallclock-assertions`), taken there from HERE by patch rather than the other way round, and `sync-from-fork.sh` still must not run while the fork's `perf` carries the clean-room `codec/rar13/` oracles. The per-test reasoning, the break table and the load measurements are in that commit's message and at each site. | `src/recovery/workgauge.rs` (new), `src/recovery/mod.rs`, `src/recovery/rar5.rs`, `src/recovery/stream.rs`, `src/rar50.rs` |
| 2026-09-16 | **Every SERVICE block the GENERAL RAR 5 writer emits carries the WRITER's host-OS byte, not a hardcoded `0`.** `write/mod.rs` passed `host_os = 0` - the WINDOWS value - to `stored_file_specific` at all five of its service sites: `CMT`, `QO`, an arbitrary service beside a member, `RR`, and the two encrypted spellings. On a Mac the reference writes `1` there, so this is the mirror of the `write/reference.rs` bug fixed the same day (a hardcoded `1`, invisible from a Mac because `1` IS the Unix answer), with the platforms swapped. Nothing here could see it: the windows conformance leg's `a` rows go through the reference-layout writer, and no test pinned the byte on either side. `SERVICE_HOST_OS` is `pub(super)` now and BOTH writers read it - the two service writers disagreeing is how the last one arrived, and cost nineteen `[files]` divergences. REACHABILITY, which was the open question rather than the value: nothing in `crates/` sets `quick_open` or `archive_comment` (the only site reading either is postfast's `streamed_stored_opts`, container.rs:1532, which REFUSES the streamed path when they are on), so the `CMT` and `QO` arms are unreached from this repo today - but postfast's C10 row sets `recovery_record_pct`, so `RR` IS reached. `RR` was measured rather than inferred, the Windows sitting having covered only `CMT`/`QO`: on the dev Mac against the same rar 7.23, `rar a -m0 -zc.txt`, `-qo+` and `-rr10` each write `host=1`. One test, taking the member's `host_os` as an INPUT set to the opposite of the platform's and pinning the value to the reference writer's own `QO` block rather than to a literal, so it checks both answers from either box; three control arms run, failing on `QO`, `CMT` and `RR` at `left: 0, right: 1`. Stated limit at the site: the two encrypted spellings follow the constant but the test cannot reach them, because a salted key derivation leaves no stable byte to search for by name. **In the fork** as `14e9647` on `perf` (claim `rars-direct-writer-service-host-os-16sep`), taken here by hand off that commit rather than by `sync-from-fork.sh`, for the same reason the FILETIME row below gives - the fork's unvendored `recovery_fold_threads` work would ride along. | `src/rar50/write/mod.rs`, `src/rar50/write/reference.rs` |
| 2026-09-16 | **`IO_BUF` split: the `{RB}` scan's marker-search WINDOW is now `SCAN_WINDOW` at 64 KiB; every CRC and copy site keeps `IO_BUF` at 256 KiB.** One constant served two jobs that want opposite things. The scan resumes at each accepted record's `parity.end`, which is where the next record's marker sits, so `find_marker` hits in the first 4 bytes of every window and the rest is read for nothing: the stride is one recovery chunk (43,604 B on a 16 MiB volume) against a 256 KiB window, so the scan traverses the record region about six times. Measured IN PLACE on the real `scan_inline_recovery_chunks_in`, 17 x 20,266,596 B volumes, one handle: bytes read **23,865,952 -> 8,819,464** (-63.0%), amplification 6.84x -> 2.53x, opens unchanged at 321, and identical chunk counts, parity ranges, shard indices and protected sizes on every volume. Quiet box (80-88% idle), two binaries alternating cross-process, medians of 12-14 rounds: **0.842 at 16 MiB volumes and 0.806 at 6 MiB ones**. 64 KiB rather than smaller because the same loop serves the raw fallback, which slides a whole file finding no marker (4 KiB costs 1.20x there, 64 KiB is free); the CRC sites stay at 256 KiB because they stream with the buffer AS the stride and `read_chunk_at` must not fall below one chunk or its read splits and multiplies `open()`. BOTH window tests are repointed at `SCAN_WINDOW` and that is load-bearing: the boundary test pads by the window size to put a marker AT the edge, and left at `IO_BUF` it still PASSES while testing nothing - verified by running it that way. **In the fork** as `e352fba` on `perf` (claim `rars-io-buf-window-split-16sep`), taken here with `git apply` off that commit. Findings: `research/RARS-IO-BUF-WINDOW-SPLIT-2026-09-16.md`. | `src/recovery/stream.rs` |
| 2026-09-16 | **The RAR 5 REWRITE path takes the vint's own width for its size fields; only CREATE pads to two bytes.** `rar a` streams a member it is still reading, so it reserves two bytes for a packed size it cannot know yet; a rewrite (`rn`, `d`, `ch`, `k`, and every command that re-emits a member whose header it has already parsed) has the size in hand and writes the natural width. rarfast padded on both, so every rewritten member under 128 bytes was two bytes longer than the reference's and the whole archive after it moved. Measured against rar 7.23 on the dev Mac at five sizes: the rewrite header is 29 bytes at 6 and at 127 and 31 at 128, so the crossover is the vint's own and not a minimum; at 20,000 both paths take three. The create header is 27 bytes at 6, 127 AND 128. Two tests, the first from the reference's own header images either side of the crossover and the second holding the create path's padding so a later edit cannot quietly take both; the rewrite golden fails without the change. NO conformance row covers this and both macOS legs are unchanged, which is the same fact: every rewrite row's member clears 128 bytes, where the widths agree. Found on the way to the Windows FILETIME row above and deliberately not folded into it, since it is not a Windows fact. **In the fork** as `a9651ce` on `perf` (claim `rarfast-rewrite-size-width-16sep`), taken here with `git apply` off that commit. | `src/rar50/write/reference.rs` |
| 2026-09-16 | **A member written on WINDOWS carries its time as a Windows FILETIME in a TIME extra record, not as Unix seconds in the header's own field.** The 16 Sep windows `rar` conformance leg reported 26 rows whose written archive had a different sha256 from the reference's, on every writing command, while the same rows matched byte for byte on macOS. Bisected against rar 7.23 on an x86-64 Windows 11 box by diffing two archives of one 256-byte member: 325 bytes against 317, and the host-OS byte, the DOS attribute bits and the stored name separator - the three suspects on record - were all already right. The reference clears `FILE_HAS_UNIX_MTIME` (that field is Unix seconds by definition) and writes an eleven-byte record: size 10, type 3, flags 0x0002 with the unix-format bit CLEAR, then eight bytes of FILETIME. Measured the same over a whole second, a sub-second time, a directory member, a `-htb` member and `rn`/`d`/`ch`/`k`, so the Create/Rewrite split the Unix side has does not exist on this one. The discriminator is the MEMBER's `host_os` and NOT `cfg!(windows)`: a Mac-built archive rewritten on Windows keeps every copied member at `host=1` with the Unix record and gives only the freshly added member the Windows form, measured. `htime_record` and `header_image` in `write/reference.rs` take the host byte; `file_specific` in `write/mod.rs` now RETURNS the record beside the type-specific bytes so every caller must say what it does with it (dropping the field without adding the record would lose a member's time silently at whichever site was missed), and `write/stream.rs`'s three fragment headers set `BLOCK_HAS_EXTRA_AREA` when it is non-empty. Four tests, each with a control arm run, all on macOS because the rule keys on the archive rather than the host. **In the fork** as `38027ab`, `bc773d1` (the `-htb` record-order golden) and `e558553` + `741ddb9` (the machine wording this manifest's leak gate requires) on `perf` (claim `rarfast-windows-created-bytes-16sep`), taken here with `git apply` off those commits rather than by `sync-from-fork.sh`, which would have carried the fork's unvendored `recovery_fold_threads` work with it. Evidence: host repo `research/RARFAST-WINDOWS-CREATED-BYTES-2026-09-16.md`. | `src/rar50/write/reference.rs`, `src/rar50/write/mod.rs`, `src/rar50/write/stream.rs` |
| 2026-09-16 | **Both recovery repairs open the volume ONCE for the pass, not once per read.** `ArchiveSource::read_range_into` is the only read entry point on an archive's backing store whose `File` arm opens the path per READ rather than per CALL, and a census of the crate found its three call sites are all REPAIR: the RAR5 recovery-record repair's `ArchiveRangeSource` (`rar50.rs`), and the legacy RAR 1.5-4.0 protect repair's `ProtectRecoveryBytes::read_into` and `read_sector_window` (`rar15_40.rs`). Each traverses a whole volume one window at a time, twice over, so each pays an `open`+`close` per window, the RAR5 detection's share issued from a rayon team. New `source::RepairRangeSource` opens a file-backed archive once through the `stream::FileSource` the sibling arm `repair_inline_recovery_path` already uses, and falls back to the existing by-range read for the `Memory` and `Stream` shapes, which open nothing per read and are unchanged; it reports the archive's DECLARED length, not the handle's. `ArchiveRangeSource` is gone, `ProtectRecoveryBytes` loses its archive borrow, and `read_into` / `read_tag` / `read_sector_window` / `repair_protect_sectors` take the source. Counted with a libc-level `open`/`openat` interposer, per volume, 20% record, every rebuilt volume byte-compared against a pristine set: RAR5 366.4 / 743.1 / 725.2 / 2259.4 -> **4.0-4.4** at 6.4 / 19.0 / 20.3 / 80.9 MB, legacy 45.1 / 133.2 / 517.4 -> **4.1-4.4** at 6.3 / 20.2 / 80.8 MB (the harness's own byte-compare adds two opens a volume on top and is excluded here; both columns are in the note's `opens.csv`). Both were LINEAR in volume size and both are flat now, which is the signature the per-read open is gone - extraction, which reads through the per-call and cached entry points, has always been flat at three. Cross-process alternating A/B, medians of 20 an arm: RAR5 0.65-0.79 of base (0.54-0.60 on minima), legacy 0.75-0.90 (0.81-0.88), on a box at 5-10x oversubscription, so those ratios are indicative and the counts are not. This does NOT reopen the descriptor-exhaustion question `RangeReaderCache` settles for extraction: that cache exists because a handle per PARSED VOLUME across a large set would exhaust the limit, and this holds one descriptor for one repair of one volume. The whole-volume `clone_prefill` both arms do is untouched and still an APFS clone. **In the fork** as `cdbb587` off `perf` `411201a` (claim `rars-repair-open-fix-narrow-16sep`), ported here by `git apply` of that whole commit - all three files were byte-identical between the trees at `411201a`, checked before the branch was cut. **MERGED TO `perf` as `78c5150`** (fork `cargo test -p rars --lib --features parallel` 1092/1092 on the merged tree), so the next sync base carries it as history rather than depending on a reader spotting this row; the `last-synced-rev` line still names the hunks because the sync has not been re-run. Evidence: host repo `research/RARS-REPAIR-OPEN-FIX-2026-09-16.md`, gated by `research/RARS-ARCHIVESOURCE-OPEN-CENSUS-2026-09-16.md` and `research/NZBFAST-RR-REPAIR-PROFILE-2026-09-16.md`. | `src/source.rs`, `src/rar50.rs`, `src/rar15_40.rs` |
| 2026-09-16 | **A RAR 3 x86 filter block is clipped at the 2 GiB round-trip boundary.** SPEC C Part 7 item 1, the last open point of the clean-room address-filter spec. The RAR 3 x86 transform forms `o = F + p + 1` with no reduction, and encode is the inverse of decode only while `o <= 2^31`; past that the pair is not invertible for every operand value, and the transform must NOT be repaired, because its decode side has to agree with every other RAR 3 reader. The writer reached the region: 128 KiB chunking divides 2 GiB exactly, so a whole-member x86 filter on a member over 2 GiB (the writer caps a member at `u32::MAX`, so 2-4 GiB is writable) put a block start on `0x8000_0000` itself, which is the `F` of the spec vector Z1. `split_large_filter` now clips an E8/E8E9 range at `MAX_RAR3_X86_FILTER_END` = 2^31 + 4 - a block `[s, e)` reaches at most `o = e - 4`, since a trigger byte needs four operand bytes behind it - and emits no record at all for a range left with under five bytes; `filtered_member` refuses a range past the boundary as a writer bug. The spec offered refuse or split and did not pick: this is split, as a clip, because `FilterPolicy::Auto` offers a whole-member x86 filter for EVERY member it compresses, so refusing would turn every RAR 2.9 write of a member over 2 GiB into a hard error. The cost is ratio on the tail past 2 GiB, never correctness. Four new lib tests: the planner block that does not round trip, the clip and its boundary, a range wholly past it emitting nothing, and small members plus the offset-free kinds untouched. Two of them failed before the change. **In the fork** as `6009b50` on `perf` (claim `specc-part7-rar3-x86-past-2gib-16sep`), ported here by hand rather than by `sync-from-fork.sh`: the vendored tree predates the SPEC C transform rewrite entirely (it still carries `e8e9_encode`/`e8e9_decode` in `src/codec/filters.rs` and has no `src/codec/address_filters.rs`), so a mirror would have dragged the whole unvendored rewrite in. The guard is independent of that rewrite - same function, same `o = F + p + 1` rule - so only the writer-policy hunks and their tests came across. Evidence: host repo `research/cleanroom/SPEC-C-address-filters.md` Part 3.2 and Part 7. | `src/codec/rar29.rs` |
| 2026-09-16 | **`flat_slack_only_ever_plans_less_than_the_old_floor` asserted on member sizes that do not exist on a 32-bit target, and so did not fail there - it stopped the `rars` lib test COMPILING.** The cell `slack(4096 * MIB, 64 * MIB)` is exactly `u32::MAX + 1`; `arithmetic_overflow` is deny-by-default and const-folds it, so nightly `armv7-cross` died at `cargo test --no-run` with one error and ran none of the ~5,000 tests in the sweep (run 35078185142 on `f5cdd402`). The grid loop's `8192` member column is the same defect one step later: a RUNTIME overflow panic, since that job sets `overflow-checks = true` precisely to catch arch-width bugs. Both cells are now `#[cfg(target_pointer_width = "64")]`, the idiom this file already uses three times (`rar50.rs` 10691, 11928, 16585); nothing is loosened and no assertion is dropped, because the 1024 MiB cell and the rest of the grid cover the ceiling on every target and the 32-bit cell set is a strict subset of the 64-bit one. `flat_slack_with` itself is width-correct (f64 root, `saturating_add`, both clamps) and is NOT changed. **PORTED to the fork 16 Sep 2026** as `04b6ea4` on `perf` (claim `rars-fork-armv7-flatslack-port-16sep`), so the two trees agree here and a re-vendor no longer reinstates the break - which is what the paragraph this replaces warned was still open, the fork having carried the same test unfixed at `b20261a`, the row below. **This row STAYS after the port**, and re-tensing it rather than deleting it is deliberate: `sync-from-fork.sh` mirrors `src/**` with `rsync --delete`, the re-vendor walk reads THIS table, and a row dropped because "the fork has it now" is how a lane that later re-vendors from a fork rev PREDATING `04b6ea4` puts the compile break back with every gate green. Verified by reproducing the compile error locally at the same target, then running the built binary on real 32-bit ARM under qemu; re-verified on the fork side at `04b6ea4` (1092 lib tests, clippy, and the test itself passing on real 32-bit ARM). | `src/codec/rar50.rs` |
| 2026-09-16 | **The flat plan's slack past the retained window is a function of the member and the dictionary, not a 64 MiB floor.** `flat_plan_bytes` added `max(history_limit, FLAT_SLACK_MIN)` of slack, so every member planned at least 64 MiB of slack whatever its size, and every page of that plan takes a first-touch zero-fill fault the first time the decoder writes it. New `flat_slack` / `flat_slack_with` give `clamp(sqrt(output * history / 3), max(history, FLAT_SLACK_FLOOR), max(history, FLAT_SLACK_MIN))`, the ratio inside the root being the fitted cost of a fault (54.3 us/MiB of plan) against a `make_room` slide (19.4 us/MiB moved) over a 193-cell grid on an M3 Ultra. It only ever plans LESS - the upper clamp is the old expression exactly - so no memory gate, `flat_admitted` admission estimate, memtrack charge or `-mm<size>` cap can be loosened by it, and the old lower bound (slack at least the dictionary, i.e. at most one byte moved per byte emitted) did not move. Under `cfg(test)` the two clamps meet at every dictionary above 4 KiB, so test builds plan the `2 x dictionary` they always did and the differential suite crosses slides exactly as often - which is why the new unit test `flat_slack_only_ever_plans_less_than_the_old_floor` names the ceiling through `flat_slack_with`. Measured on `rarfast t` at 0.913 to 0.958 of base across the 16-128 MiB member with a dictionary of 8 MiB or less, and inside the noise everywhere else; the grid's largest members want MORE slack than the ceiling allows and that was deliberately declined (0.6-2.0% for 64 MiB of resident memory on every large member). **In the fork** as `b20261a` on `perf` (claim `rars-flat-window-slack-16sep`), landed here as `5a73d0532` (claim `rars-flat-window-vendor-port-16sep`) - taken with `git apply` off that fork commit rather than by `sync-from-fork.sh`, which would have carried ~26 other files with it. Two lanes ported it in parallel and produced byte-identical trees; this ledger row is the half neither port carried, and without it the next sync's walk would drop the divergence silently. Evidence: host repo `research/RARFAST-BENCH-2026-09-14.md` section 19, artefacts in `research/rarbench-2026-09-16/flatwin-*`. | `src/codec/rar50.rs` |
| 2026-09-15 | **CORRECTION to the row below: the repair folds' team gates are work PER THREAD, not a flat byte count.** The flat 12 MiB / 12 MiB / 2 MiB form was derived and verified end to end on ONE 20-thread box. On a 32-thread M3 Ultra every one of those thresholds sat in a LOSS: the striped fold was 1.30-1.34x the serial time at a 16 MiB window (and a 16 MiB-volume `rarfast rc` repair cost 12% END TO END with the flat gate on), the whole-grid fold 1.01-1.23x at 12 MiB of work, and the RAR 3 fold 1.12x cold at 2 MiB. Every dispatch wakes and parks the whole pool, so the work a fold must carry scales with the pool. Now `striped_team_min_window_for` = `max(12 MiB, threads x 2 MiB)`, `whole_grid_team_min_work_for` = `max(12 MiB, threads x 1 MiB)` and `reconstruct_team_min_work_for` = `max(2 MiB, threads x 128 KiB)`, each a measured win on BOTH boxes, the floor being what the 20-thread box measured. `fold_team_threads` caches `std::thread::available_parallelism` - never `rayon::current_num_threads`, which would BUILD the global pool the gate is trying not to start. Re-measured on the 32-thread box: the 16 MiB-volume repair is 1.001x the serial build (was 1.123x) and 0.886x the flat-gated one; the 4 MiB-volume sets are unchanged at 1.000-1.002x. Tests pin the formula at 1, 2, 6, 20, 32 and 64 threads, and a test-only `with_fold_team_threads` forces the width so the both-sides differential test still reaches the team arm on a wide host. Evidence: host repo `research/RARFAST-BENCH-2026-09-14.md` section 14.6. | `src/recovery/rar5.rs`, `src/recovery/rar3.rs` |
| 2026-09-15 | **The three repair folds start a rayon team only from the work that was measured to repay one.** `repair_shards_striped`'s per-window fold (what `rarfast rc` and nzbfast's `.rev` and inline repairs take) was slower in parallel than serial at every window up to 8 MiB, warm pool or cold: 2.0-2.6x at 1 MiB, 1.14-1.23x at 4 MiB (rarfast `rc` of a 64 MiB set in 4 MiB volumes, which the quiet-box rounds measured 35% slower with `parallel`). It now folds on the calling thread below `STRIPED_TEAM_MIN_WINDOW` (12 MiB, the first window that won: 0.91-0.94x; 16 MiB 0.81x, 64 MiB 0.49-0.66x). `recover_damaged_shards` gates each rebuilt row on shard length times live sources at `WHOLE_GRID_TEAM_MIN_WORK` (12 MiB; 8 MiB was still 1.04-1.16x), and `recovery/rar3.rs`'s `reconstruct_data_volumes` at `RECONSTRUCT_TEAM_MIN_WORK` (2 MiB; 1 MiB cold was 1.27-1.61x). Measured on an M1 Ultra with an in-process harness, cold process and warm pool; the gated build is 0.97-1.01x the serial one under every gate and keeps every win above them. Pinned by `repair_folds_start_a_team_only_from_the_measured_crossover`, `repair_folds_rebuild_the_same_bytes_on_both_sides_of_the_team_gate` and `rev3_rebuild_starts_a_team_only_from_the_measured_crossover`. VENDOR-FIRST: ported to the fork on branch `perf` as hunks onto spec B's `rar5.rs` and `rar3.rs`, whose fold sites and gates are the same (only the RAR 3 doc comment differs: the fork folds whole sources four to a pass, this tree one byte at a time). The series sync that replaces these two files with spec B's must keep the gates; they are already in the fork's copies. Evidence: host repo `research/RARFAST-BENCH-2026-09-14.md` section 14. | `src/recovery/rar5.rs`, `src/recovery/rar3.rs` |
| 2026-09-15 | **Two more test allowances are `Some(usize::MAX)`, not `Some(64 << 30)`: `tree_wave_width` in `the_hint_budget_narrows_a_wave_without_moving_its_bytes` and `members_in_flight_for` in `a_policy_charges_member_trees_without_moving_the_archive`.** The row below this one fixed the same test's `1 << 40`, which was a COMPILE error; these two compiled and failed at RUN time, because a shift of 30 is under a 32-bit width, so rustc raises nothing and `64 << 30` silently wraps to an allowance of 0. The nightly `armv7-cross` run on `d1621159d` (run 35010884006, 8,343 of 8,345 passing) failed both: `left: 1, right: 8` for the wave width, and one member in flight where the test wants all three. `usize::MAX` means "unbounded" at both widths, and both functions only divide the allowance (`(bytes / 4).min(TREE_HINT_BUDGET_BYTES)`, `working as u64 / 4 / tree`), so it cannot overflow either; the neighbouring `Some(1 << 30)` assertion is representable on 32-bit and stays. Test-only; no production byte moves. These were the only GiB-scale `Some(..)` literals under `src/`. **Ported to the fork** as `bcf8ea5` on `perf` (claim `rars-fork-armv7-test-ports-16sep`, 16 Sep 2026; the wave test came from `43e478b`, the member-tree test from `ad10bd6`/`90020b6`); see VENDOR-REV. | `src/codec/rar50.rs` (test), `src/rar50/write/mod.rs` (test) |
| 2026-09-15 | **Under a working-memory allowance a large streamed member's window keeps its floor of eight blocks; only small-member admission loses it.** `member_window_blocks` was one number used twice in `rar50/write/stream.rs`: as the small-member admission width, which needed its floor gone under an allowance (row below, "A writer allowance only ever NARROWS"), and as the large member's window, which went down with it. At one pool thread a window was then ONE 4 MiB block behind 32 MiB of history, and every window builds a fresh tree finder over its whole span, so about nine bytes were walked per byte encoded. New `member_window_segment_blocks` (wave width, never under eight, allowance or not) feeds the window; `member_window_blocks` still feeds admission and the `2 * segment_blocks` block bound, unchanged. The two travel as `StreamWindows { members, segment }` through `write_members_streamed` and the four `_with_windows` entry points. Measured on rarfast `-m5 -mm64g` over a 64 MiB text member, same archive in every run: one thread 142 s user CPU and 1,328 MiB peak RSS -> 32 s and 1,050 MiB (30 s and 1,053 MiB uncapped); two threads, interleaved, 28 to 38 s wall -> 14 to 15 s; four threads 1,609 -> 1,211 MiB. The tree is sized by the dictionary, not the span, so the floor costs only the three window buffers: the one cell where the peak rose was 8 MiB at one thread, 495 -> 584 MiB. With no allowance nothing moves. Byte-neutral: `streamed_matches_the_in_memory_writer_in_windows_wider_than_admission_{lazily,optimally}` hold an admission of one block with windows of two to the in-memory writer, and `the_hint_budget_narrows_a_wave_without_moving_its_bytes` asserts a one-byte allowance still gives an eight-block window. Research: nzbfast `research/RARFAST-BENCH-2026-09-14.md` section 11. **In the fork** as `b449705` on `perf` (claim `rarfast-window-floor-under-cap-15sep`), with rarfast's `encoder_threads_for_cap` re-fitted to 350 MiB + 20 x dictionary fixed; landed here VENDOR-FIRST and ported as hunks, since the fork's `codec/rar50.rs` carries unvendored clean-room work. | `src/codec/rar50.rs`, `src/rar50/write/stream.rs` |
| 2026-09-15 | **Under a working-memory allowance a streamed window encodes at the pool's wave shared between the windows in flight.** `member_window_wave_width` (with `STREAMED_WINDOWS_IN_FLIGHT = 2`) replaces the direct `encode_block_wave_width_for_budget` call in `encode_lz_member_window`: the streamed writer encodes two windows of a member at once, each allocating its tree-hint buffer for its whole wave (64 MiB a block at the cost-based parse's stride) while both share one pool. Measured with a heap-tracking allocator, rarfast `-m5 -mm64g` over a 64 MiB text member at 32 MiB: live heap 1,315 / 1,528 / 2,141 -> 1,210 / 1,335 / 1,529 MiB at 2 / 4 / 8 threads, Silesia at four threads 1,857 -> 1,571, sha-identical archives. With no allowance nothing moves. Byte-neutral, pinned by `a_streamed_window_under_an_allowance_shares_its_wave_without_moving_a_byte` (four-thread pool, wave two against four, both equal to the whole-member encode). `rar50/write/stream.rs`'s literal three window buffers are left as they are. Pooling the hint buffers across windows was measured and NOT landed (it raised every peak). Research: nzbfast `research/RARFAST-BENCH-2026-09-14.md` section 9.8. **In the fork** as `457afed` on `perf` (claim `rarfast-cost-parse-block-memory-15sep`), copied here as the one function, its doc and its test, not by `sync-from-fork.sh`. The test's huge allowance is `Some(usize::MAX)` (32-bit safe, as the row below explains), ported to the fork with the other such line as `5af99f6`. | `src/codec/rar50.rs` |
| 2026-09-15 | **`the_hint_budget_narrows_a_wave_without_moving_its_bytes` asserts a huge allowance with `Some(usize::MAX)`, not `Some(1 << 40)`.** `encode_block_wave_width_for_budget` takes its allowance as an `Option<usize>`, so on a 32-bit target the shift is a `deny(arithmetic_overflow)` compile error and the whole rars lib test stops building - which took the nightly `armv7-cross` job red before it ran a single test (nzbfast run 35004821725, on `edce9281c`), hiding every other armv7 verdict behind it. `usize::MAX` means the same thing at both widths: the function only computes `(bytes / 2).min(host)` against a saturated host budget, so any allowance past the host's is the default, which is what the assertion pins. Test-only; no production byte moves. The line came in with the row below, copied verbatim from fork `43e478b`. **Ported to the fork** as `bcf8ea5` on `perf` (claim `rars-fork-armv7-test-ports-16sep`, 16 Sep 2026), alongside the two `64 << 30` lines from the row above. | `src/codec/rar50.rs` (test) |
| 2026-09-15 | **A write policy charges each member's match-finder tree: under one, the members encoding side by side share the policy's tree quarter** (TODO 349 B). Each member behind the tree finder holds a tree of ten bytes per dictionary byte, which the wave, the hints and the member window never counted, so the in-memory, volume and streamed writers ran one tree per pool thread: postfast at a 1 GiB budget and `--dictionary 32m` peaked at 2,029 MiB (3,118 MiB with the cost-based parse) on 18 threads. New `filter_policy::members_in_flight_for` (`max(1, (W/4) / (10 x dictionary))`, unbounded with no policy or no tree) caps `resolve_compressed_members`, `resolve_encrypted_compressed_members`, both member resolvers in `volume.rs` (through the new `parallel::map_collect_bounded`) and the small-member queue in `write_members_streamed`; `TREE_MIN_DICTIONARY` becomes `pub(crate)`. Byte-neutral, pinned by `a_policy_charges_member_trees_without_moving_the_archive`; peaks 801 / 834 MiB at the same bytes (host repo `research/RARFAST-BENCH-2026-09-14.md` section 10). Ported to the fork on `perf`. | `src/parallel.rs`, `src/rar50/write/filter_policy.rs`, `src/rar50/write/mod.rs`, `src/rar50/write/volume.rs`, `src/rar50/write/stream.rs`, `src/codec/rar50.rs` |
| 2026-09-15 | **The RAR VM ran any program carrying static data or a CALL wrongly, so the RAR 3 standard IA-64 program ran as the identity.** Two faults, each enough on its own. (1) `Vm::new` placed static data at `GLOBAL_BASE + global_data.len()`; a program's first run passes no globals, so the table landed at `0x3c000`, where the length and file-offset globals written straight after overwrote it, and the IA-64 program (the one standard program with static data) read an all-zero mask table and rewrote nothing. Static data now starts at `GLOBAL_BASE + max(len, SYSTEM_GLOBAL_SIZE)`. (2) `Vm::run` treated every `RET` that left the stack pointer at `MEMORY_SIZE` as the end of the program, which is also true straight after a SUBROUTINE return pops back to an empty stack, so the IA-64 program stopped at the first return of its `CALL`. A top-level `RET` was already reported by `execute_instruction` before it pops; the extra check is gone. The x86 standard programs use neither, which is why they agreed. Production extraction was not affected, since recognised standard filters run natively and only a generic program reaches the VM - but a generic program with static data or a CALL was mis-run. Tests: `static_data_follows_system_globals_when_no_globals_are_passed` and `ia64_standard_program_rewrites_a_branch_slot` in rarvm.rs; and the RGB and DELTA standard bytecode, which no test had ever run through the VM (only AUDIO had one), now each match the native transform (`delta_filter_bytecode_matches_builtin_transform`, `rgb_filter_bytecode_matches_builtin_transform`). `RAR3_ITANIUM_FILTER_BYTECODE` becomes `pub(crate)` here as it already is in the fork. In the fork the spec C module's O2 differential checks IA-64 against the VM again (not vendored: `codec/address_filters.rs` lands here with the clean-room series' final sync). Fork 8793f51 on `perf`, mirrored by copying `rarvm.rs` (byte-identical to the fork's before the change) and hand-applying the two rar29.rs tests, not by `sync-from-fork.sh` (see VENDOR-REV). | `src/codec/rarvm.rs`, `src/codec/rar29.rs` (tests, one visibility) |
| 2026-09-15 | **A writer allowance only ever NARROWS the encoder: each share of `EncodeOptions::working_memory` is capped by its host default, and the streamed writer's member window loses its floor of eight under one.** (2) `member_window_blocks` returned the wave width but never under eight blocks, and that window is also how many small members `write_members_streamed` encodes at once, each on its own thread OUTSIDE the pool with its own match-finder tree and block scratch: with rarfast's pool narrowed to two threads, `-m5 -mm2g` over Silesia still peaked at 2,205 MiB. Under an allowance the window is now the wave width; with none it keeps the floor, so no default moves. Byte-neutral: a member that crosses from the whole-member route to the windowed one gets the same `samples_read_incompressible` verdict and the same blocks, which the streamed-matches-in-memory tests hold; pinned by an assert that a one-byte allowance gives a one-block window and no allowance still gives eight or more. (1) `encode_block_wave_width_for_budget` takes `min(W/2, host wave budget)` and `tree_wave_width` takes `min(W/4, TREE_HINT_BUDGET_BYTES)`; both used to REPLACE the default with the allowance's share, so a large allowance widened a wave the host would not have run. Found by rarfast's `-mm<size>` cap: `-m5 -mm4g` peaked at 7.2 GiB over Silesia against 6.9 GiB with no allowance, and 5.1 against 2.7 GiB on a 256 MiB text, because 1 GiB of parse hints doubled the hint wave from 8 blocks to 16. Byte-neutral by construction (wave width moves no byte; the existing width tests and a sha-identical archive at 1, 2, 4, 8, 16 and 32 pool threads hold it); two new asserts in `the_hint_budget_narrows_a_wave_without_moving_its_bytes` pin that a 64 GiB allowance IS the default and a 1 GiB one holds four blocks of hints. postfast's `rar_write_policy()` tops out at 4 GiB, so its waves can only narrow or stay put. NOT fixed here and written down instead: the policy still does not bound a PROCESS - it sizes a cost-parsed block in flight at 72 MiB where about 175 MiB is measured, and charges no per-member match-finder tree when several members encode side by side - so rarfast holds its cap by sizing the rayon pool as well (nzbfast `research/RARFAST-BENCH-2026-09-14.md`, the `-mm<size>` section). Mirrored by copying `src/codec/rar50.rs` from fork `43e478b` (not `sync-from-fork.sh`, which would carry `7fc7c65`'s clean-room oracles and the rename lane's unsynced `a92cd27`); `cmp` against the fork's file is clean. | `src/codec/rar50.rs` |
| 2026-09-15 | **Renamed the RAR 5 header constants, RAR 2.x sub-block names and split-fragment functions that shared RARLAB UnRAR's internal naming, and reworded comments naming UnRAR internals**, applying MAP-E's "can land now" items 1-4 from the 14 Sep 2026 provenance audit (host repo `research/cleanroom/MAP-E-renames-comments.md`, `research/RARS-PROVENANCE-AUDIT-2026-09-14.md`). Purely mechanical: `HFL_*`/`MHFL_*`/`FHFL_*`/`FHEXTRA_*`/`MHEXTRA_*`/`HTIME_*`/`FSREDIR_*`/`HEAD_*` become a `BLOCK_TYPE_*`/`BLOCK_*`/`ARCHIVE_*`/`FILE_*`/`MAIN_EXTRA_*`/`FILE_EXTRA_*` scheme read from the RAR 5 technote's own field descriptions; `UO_HEAD`/`SIZEOF_UOWNERHEAD` become `SUBBLOCK_UNIX_OWNER`/`UNIX_OWNER_FIXED_SIZE`; `split_fragment_packed_crc`/`_digests` become `nonfinal_fragment_crc`/`_digests` (doc links and test names followed, including a rar50 test name sharing the same prefix that the map did not list). No renamed constant is public. Left untouched: everything inside a clean-room rewrite lane's scope (`codec/rar13.rs`, `rar20.rs`, `rar29.rs`, `rar50.rs`, `recovery/rar3.rs`, `rar5.rs`, `crypto/rar30.rs`) and the CLI's `dialect/unrar/args.rs`, owned by a different chip. `research/cleanroom/check.py` reports 0 denylisted identifiers on every touched file. Applied to the fork at `~/Claude/rars` as `a92cd27` on `perf`, pushed; the fork side then merged in its own 15 Sep work (the `--` end-of-switches dialect fix, not vendored - CLI only - and the streamed recovery record's CRC64/fold worker team below) before this sync ran. `sync-from-fork.sh` hit the exact trap the row below warns about - its full mirror carried fork `7fc7c65`'s five clean-room-oracle files and their two mod lines back in - so those were deleted from this sync same as the row below excludes them (see VENDOR-REV). | `src/lib.rs`, `src/rar15_40.rs`, `src/rar15_40/extract.rs`, `src/rar50.rs`, `src/rar50/extract.rs`, `src/rar50/write/mod.rs`, `src/rar50/write/reference.rs`, `src/rar50/write/stream.rs`, `src/rar50/write/volume.rs` |
| 2026-09-15 | **The streamed recovery record's CRC64 and GF(2^16) fold on a worker team.** `InlineRecoveryFolder` (behind `parallel`, from a 32 MiB prefix) copies its pushes into 8 MiB batches and hands each to up to eight persistent threads, one job each: the batch's CRC64 cut at slice starts (the first piece continues the previous batch's trailing state, the rest start from zero) and its fold split by parity row, each row owned by one job while out. The writer fills the next batch while the last one folds, so two batches are live at most. Fewer than two spawned threads fall back to the inline fold, and a worker panic is resumed on the writer thread. `SliceLayout::walk` is now the single copy of the slice arithmetic. Byte-identical to the whole-prefix builder (`inline_recovery_folder_team_matches_the_whole_prefix_builder`, which two mutants fail). rarfast `a -m0 -rr5` over 512 MiB, two A/B rounds at load average 86 to 123: 0.489 / 0.464 s against the serial fold's 1.263 / 1.932 s, 3.4 to 3.8x rar 7.23 in the same run, peak RSS 100 MiB (serial 84, rar 100). Two rayon-per-write arms had lost here. nzbfast `research/RARFAST-BENCH-2026-09-14.md` 7.8. Fork `dc9d5f4` and `7c01424` (an unused `mut` without `parallel`), mirrored by COPYING these two files rather than by `sync-from-fork.sh`, because fork `7fc7c65`'s clean-room oracles must not be mirrored here (see VENDOR-REV). | `src/recovery/rar5.rs`, `src/recovery/rar5/fold_team.rs` (new) |
| 2026-09-14 | **CRC32 on several cores in both extract pipelines, no CRC32 for a hash-only member, a RAR4 stride copy, and a writer switch that turns level 5's lower-level re-encode off.** (1) `Crc32::combine` and `Crc32::update_pieces` (behind `parallel`, 2 MiB floor): one `crc32fast` stream measured 9.8 GB/s on the dev Mac and was half the CPU of a stored or text `t`; 4 MiB cut four ways and joined ran 25 GB/s. The stored and the packed pipeline digesters now take up to `DIGEST_BATCH` = 4 waiting 1 MiB buffers at once and checksum them together (pools 4 to 8 buffers), and an UNSPLIT member with no CRC32 field computes none (`pipe_stored_chunks` takes `Option<Crc32>`): a BLAKE2sp-only member spent 14% of its test CPU on a CRC32 nothing read. A split fragment always computes it, because its own header is the first fragment's and the CRC32 to verify is the last one's; the first cut read the handed header and failed `rar50_volume_sequence_resumes_the_finishing_volume_after_a_split` (fixed in fork `719cc0e`). rarfast `t` against unrar 7.23: store 0.79x to 1.24x, m3 0.72x to 1.14x, rr5 0.84x to 1.27x, extract-htb 0.84x to 1.09x. (2) `Unpack29::copy_match`: a match of at most 64 bytes at a distance of at least 16 grows the window by a fixed 80 bytes and copies 16-byte strides instead of an `extend_from_within` per match (rar50's `flat_stride_copy` shape), pinned against the byte loop at every stride edge by `copy_match_short_stride_path_matches_bytewise_oracle`; test-rar4-m3 0.81x to 1.13x and extract 0.82x to 1.13x over two rounds. A 16x history drain threshold was tried first and changed nothing but RSS (84 to 140 MiB); not landed. (3) `WriterOptions::level_five_fallbacks` (default `true`, so the library's default bytes do not move) with `with_level_five_fallbacks`: off, level 5 is one encode and the streamed writers accept it. At a 32 MiB dictionary the re-encode bought 66 bytes for 151 s against 22 s; rarfast `-m5` turns it off. Measurements, and the attempts that lost (a rayon CRC64 over recovery slices, 0.848 against 0.752 s): nzbfast `research/RARFAST-BENCH-2026-09-14.md` section 7. Fork commits `eecd733`, `a712a02`, `52ea038`. | `src/crc32/mod.rs`, `src/rar50/extract.rs`, `src/rar50/write/mod.rs`, `src/rar50/write/stream.rs`, `src/codec/rar29.rs` |
| 2026-09-14 | **Streamed writers for the reference stored layout and for compressed trees, so `rarfast a` no longer holds its input.** `assemble_streamed` / `write_reference_stored_streamed` write the reference's stored byte layout from `Read` sources into a `Write + Seek` sink: a member up to 4 MiB is read whole into one reused buffer, a larger one is laid down behind a placeholder-checksum header, copied through a three-stage read / checksum / write ring of four 4 MiB chunks, and its header patched in place; the locator's quick-open offset is patched at the end. BYTE-IDENTICAL to `assemble` / `write_reference_stored` (a test over CRC32 and BLAKE2sp, all three quick-open modes, a prefix block, a directory, an empty member and chunk edges), and the rar conformance table is unchanged at 175/178 rows with the same seven divergences. `member_header_split` now builds through a shared `header_image(fields, digest, ...)` with the checksum computed by the caller; its bytes are unchanged. `StreamedMember` (File or Directory) with `write_compressed_members_streamed` / `write_compressed_member_volumes_streamed` carry the two shapes the entry-slice writers refused - a directory (a header, no data) and an EMPTY file (stored, as the in-memory writer stores it; the volume sink needed its own arm, since an empty member has no fragment to cut and was first written with no header at all) - over which files alone are the entry writer's own bytes (tests hold both, and read a tree back through the crate's extractor across a volume set). The entry-slice writers keep their empty-payload refusal through an associated const on the new private `StreamedItem` trait. Measured on the rarfast create legs in nzbfast `research/RARFAST-BENCH-2026-09-14.md` section 6. Landed in the fork as 18a7d31 on top of f511c96, which ports the three 11 Sep vendor commits the fork lacked; this sync also brings the fork's 7eeeb30 / dfc2764 lint edits here. | `src/rar50/write/reference.rs`, `src/rar50/write/stream.rs`, `src/rar50.rs` (exports) |
| 2026-09-14 | **`write_vint_padded` pads UP to its width and no longer truncates DOWN to it.** It wrote exactly `width` bytes and dropped the high bits, so a quick-open block whose data ran past 16,383 bytes carried `len & 0x3fff` in both size fields and the reference `unrar` 7.23 refused the archive with "Corrupt header is found" on `t` and `x` (while `l` still listed every member). Reached under `-qo+` at about 190 cached headers, and under the default `-qo` by a few hundred members over 4 KiB - which is how `tools/rarbench.py`'s `create-small-m0` leg found it (4,000 files, 16 dirs, verified by the reference unrar). Fixed at the helper (`width.max(vint_width(value))`), so every padded field grows the way the reference's does; the 200-member `-qo+` archive is now exactly the reference's 31,670 bytes. Two tests pin it: the helper's bytes for 16,612 (`e4 81 01`) and a 400-member `-qo+` build whose decoded QO data size must equal the bytes that follow the header. Applied to the fork at `~/Claude/rars` as `cd218d1` on `perf`, pushed, in the same sitting. | `src/rar50/write/reference.rs` |
| 2026-09-11 | **`Gf16MulTable::new` builds its split byte tables by a SUBSET WALK over the field's xtime chain** rather than by 512 `Gf16::mul` calls. Multiplication by a fixed coefficient is GF(2)-linear in its other operand, so every entry of the 256-entry `lo`/`hi` pair is the XOR of the sixteen `c * x^k` chain elements its index has bits set for: ~16 shifts and 510 XORs over a 1 KiB working set, where the log/antilog form was 512 branchy round trips through the module's 768 KiB `exp`/`log` pair. New `power_chain` / `split_tables` free functions, derived from this module's own `PRIMITIVE_POLYNOMIAL` rather than copied from the PAR2 engine's gf16 (the two fields turn out to be the same, which is a coincidence nobody should have to re-check). `Gf16MulTable::new` loses its now-unused `&Gf16` parameter across all six call sites, and with it three dead `let gf = shared_gf16()` bindings. Measured on the dev Mac, two binaries interleaved with the order flipped every rep: the table build is 616 -> 147 ns, worth **-30.3%** on an end-to-end `repair_shards_striped` at a 20,900-byte stripe and **-19.9%** at 41,700 (within-arm spread 3.5% and 1.4%), and NOTHING on a 1 GiB `-rr3` create, which caches per shard and pays only 1,200 builds for the whole archive. The striped repair is the site that matters because it rebuilds per (source, damaged row) PER STRIPE, and nzbfast's `repair_cap()` floor of 8 MiB divided by heavy damage makes a stripe tens of KiB. Pinned by `split_tables_match_mul`, exhaustive over all 65,536 coefficients against `Gf16::mul` with a case counter, negative-controlled by flipping a bit of the reduction constant (970 -> 971 tests). `fold_stripe_into_parity`'s doc comment is re-tensed, not deleted - it carried the still-correct decision NOT to cache the tables, and only its price sentence had gone stale against `05930d7d5d`'s SIMD kernels. Also applied to the fork at `~/Claude/rars` on branch `gf16-split-tables-subset-walk` off `perf`, committed but not pushed. Measurements, the call-site census and the two things deliberately not done: the host repo's `research/JOINT-STAGE2-DEPTH-GATE-2026-09-11.md` section 7.2. | `src/recovery/rar5.rs` |
| 2026-09-11 | **The two slowest tests in nightly's `armv7-cross` job are one test per table cell** rather than one test walking a table. `member_windows_encode_the_same_blocks_as_the_whole_member_walk` becomes six `member_windows_match_the_whole_walk_*` cells plus `member_windows_make_the_same_horizon_choices_as_the_whole_walk`, and `streamed_compressed_archive_matches_the_in_memory_writer` becomes six `streamed_matches_the_in_memory_writer_*` cells; the fixture and the windowed walk are hoisted to module-level helpers and nothing is deleted, loosened or shrunk (963 tests to 974 on that target, and the horizon pair stays ONE test because its size assertion is ACROSS its two arms). That job kills a test at 900 s under qemu and reddens a run whose first attempt timed out, and these two were 605.6 s and 531.1 s of that budget in run 34583096699 - margins of 1.5x and 1.7x with nothing watching them. Measured per cell on the emulated target (qemu-user 8.2.2, the job's own opt-level-2 profile pins, anchored 5.00x / 5.36x / 5.12x against three of that run's own rows): the cost is not spread across the table but concentrated in the `optimal_parse` column, 29.70 s against 1.27 s for the SAME bytes at the same dictionary, and the cost-based parse is CHEAPER at a one-block dictionary than a sub-block one because `TREE_MIN_DICTIONARY` arms the tree finder at 4 MiB. Worst test on the runner falls to an estimated 228 s, below two census rows that were already there. Costs +19% total CPU on the streamed table (each cell rebuilds `compressed_members()`, ~3.8 s under qemu) and nothing measurable on the codec one, paid in parallel across cells that nextest runs concurrently. Measurements, the rig and what is scaled rather than measured: the host repo's `research/ARMV7-CROSS-TEST-CEILING-2026-09-11.md`. | `src/codec/rar50.rs`, `src/rar50/write/stream.rs` |
| 2026-09-11 | `encode_blocks_pooled`'s drain loop takes an unclaimed block and encodes it on the calling thread instead of parking. That thread owns the `in_place_scope` and is a POOL thread at both writer call sites (`rayon::join` in `rar50/write/volume.rs`, `map_slice_collect` in the multi-group walk), so parking it held a thread the `width` jobs it had just spawned needed in order to run at all: N concurrent member encodes on an N-thread pool all parked waiting for jobs none of them could run, a permanent starvation deadlock at near-zero CPU. The 5 ms poll on the park is why it read as a hang rather than a wedge. It cap-killed CI's `unit-one-process` job repeatedly (45m16s at a 45 min cap, reported as `cancelled` and so carrying NO verdict) for an unknown number of pushes: that job runs each unit binary bare, libtest takes its thread count from the runner, and on the two-CPU runner nzbkit's two over-the-cap chase fixtures compress side by side - both pool threads parked and the process wedged four tests into 1,646 for 31 minutes. The owner now parks only once every block is claimed, and a block is only claimed by a thread already running the closure that finishes it. Adds `concurrent_member_encodes_never_starve_a_pool_of_their_own_size` (two-thread pool, Barrier-aligned, `recv_timeout` as the assertion - a join would hang the suite). Negative control: drop the owner's arm and it fails at the full timeout. | `src/codec/rar50.rs` |
| 2026-09-08 | The inline recovery encoder BORROWS its full data shards instead of copying them. `split_prefix_shards` built an owned, zero-padded `Vec` for every shard, so encoding a recovery record held a second complete copy of the archive prefix beside the first; only the tail shard is ever short, so full-length shards are now `Cow::Borrowed` slices of the prefix and only the tail allocates. Measured on a 4 GiB archive at 1% recovery: peak process RSS 12.98 -> 8.75 GB (the saving IS the archive size) and the encode 10.9% faster for not doing the copy, four paired runs spanning -10.73% to -10.96%, against a 0%-recovery control arm that moved +0.05%. Byte-identical records: the four native `rar` cross-checks in `winrar_recovery.rs` pass, including the 128 MB multi-group record that real `rar r` repairs from. The emptiness check is load-bearing - a shard wholly past the end of the prefix gets an inverted range from `split_prefix_shard_ranges`, which must not reach the slice. Candidate from the review's unmerged RAR5 shortlist, validated in the host repo's `research/RAR5-CODEX-QUEUE-VALIDATION-2026-09-08.md`. | `src/recovery/rar5.rs` |
| 2026-09-08 | The Huffman tree is built from TWO QUEUES rather than a binary heap (van Leeuwen). Once the leaves are sorted the merged sums come out non-decreasing on their own, so the next-smallest node is always at the head of one of the two queues and the heap's sift work disappears; the heap re-sifted about 400 symbols for every table of every block. **Output is byte-identical and that is checked end to end, not just asserted**: a 24 MiB mixed fixture creates the same archive to the byte with and without this (same sha256, same 3,496,529 bytes). Tie-breaking is the thing that could have moved it, so both queues carry `(frequency, order)` and compare as whole tuples - a leaf's `order` is always below a merged node's, so a frequency tie still resolves to the leaf exactly as popping the heap did - and `the_two_queue_builder_matches_the_heap_on_ties_and_saturated_sums` differentials 512 cases against the reference builder with case classes that force heavy ties and force `saturating_add` to actually saturate. Measured 3.2% off whole-archive creation on a mixed 4 GiB corpus and 10.0% on repetitive text; treat 3.2% as representative, because repetitive text rebuilds tables far more often per byte. The 10.0% arm ran interleaved against a same-binary A/A control that moved 0.93%, so it is an order of magnitude outside the noise. Candidate from the review's unmerged RAR5 shortlist. | `src/codec/huffman.rs` |
| 2026-09-08 | The streamed writer PARKS AND REUSES its encoder workers. A set of small members spawned one scoped thread per member; workers now wait on a channel and are handed the next member's bytes, so a thread launch is paid once per worker rather than once per member. Measured: 4 GiB / 4,096 mixed members 3.5% faster, 1 GiB repetitive 24.8%, a 32-fixture diversity screen 2.8% - and 0.4%, ie nothing, at 64 large members. The win is thread launches, so it scales with member COUNT and not archive size; the headline number hides that. **Carries a fix the original candidate did not have.** A parked worker is never joined, and `std::thread::scope` panics at its end whenever a thread panicked and nothing consumed the panic (`join` consumes it by taking the result out of the packet; a channel recv does not), so a panicking encoder would have panicked PAST the `InvalidHeader` error `drain_one` builds and that error would never have reached a caller. The worker body is wrapped in `catch_unwind` and converts at the site. `the_streamed_writer_reports_a_panicking_encoder_as_an_error` covers it through the real writer using a test-only member marker rather than a process-global flag another concurrent test could consume, and it was verified as a real test: with the `catch_unwind` removed it fails with "the writer PANICKED instead of returning an error". Candidate from the review's unmerged RAR5 shortlist; the defect and its std-semantics probe are in the host repo's `research/rar5-codex-validation-2026-09-08/`. | `src/rar50/write/stream.rs` |
| 2026-09-08 | Research-only INDEX GEOMETRY knobs (`ratio-lab`): `RARS_INDEX_BUCKETS` and `RARS_INDEX_DEPTH` force their axis of `MatchIndex::shape` exactly, so the bucket count and the ring depth can be swept INDEPENDENTLY - `shape` derives depth from the candidate budget and leaves the buckets at their cap, so an earlier screen that moved both at once could not say which axis did what. With neither set, `index_geometry` IS `index_geometry_default`, which is what makes the research control reproduce ordinary bytes; a malformed value ABORTS rather than falling back to the default (it aborts and does not panic because `shape` runs on an encoder worker, where a panic unwinds one rayon thread and leaves the process alive with no output, which reads as a slow arm rather than as a refusal). Production is untouched and that is proved, not asserted: `--features parallel` of this source and of the baseline have a BYTE-IDENTICAL `__TEXT,__text` section, 712,624 bytes of machine code, `cmp`-equal. Also adds the `opt_*` stage counters for `match_candidates_at`, the OPTIMAL parse's own walk, which the 8 Sep census never reached: it spends 38.2 ring candidates and 17.3 scattered history loads per input byte against the lazy walk's 7.8 and 2.2, so the `smallest` profile is 5x and 8x the walk the earlier census measured. **The geometry is not a lever and no default changed.** Separated over 25 members in both parsers, every byte-changing archive externally extracted and compared: quadrupling the buckets at constant depth removes 141 M tag rejects (one ring-slot load each, no history touched), leaves scattered history loads at 2.24 per input byte unchanged to three decimals, and changes user CPU by nothing resolvable (-5.0% to +11.9% over sixteen one- and eight-worker rows) for a 0.02-0.24% ratio gain and 81-119% more peak RSS per encoder worker; halving the ring depth removes the scattered loads by discarding 42% of the walk's true candidates, which is -26% to -67% of user CPU bought with up to +7.3% of bytes on text. The memory-neutral pairing of the two is the depth cut alone, to within a fifth of a point on every member. Rejected as a default and as an explicit setting; the private research record for 8 Sep 2026 has the per-member screen, the reserved holdout and the balanced timing. | `src/codec/rar50.rs`, `src/codec/rar50/probe_stats.rs` |
| 2026-09-08 | Research-only ring-probe STAGE COUNTERS (`ratio-lab`): how many of `best_match_probe`'s candidates each stage rejects, which break ends each walk, how long the walks are, and how many bytes the length loop compares. `rar5cli` prints them under `RARS_PROBE_STATS=1`. `probe_stat!` has a `#[cfg(not(feature = "ratio-lab"))]` definition that expands to nothing and the module is behind the feature, so a production build is untouched: `--features parallel` of this source and of the baseline have a BYTE-IDENTICAL `__TEXT,__text` section (712,624 bytes of machine code, `cmp`-equal). Measured at the shipped 2 MiB dictionary on 128 MiB of the mixed corpus: 2.23 scattered history loads per input byte, 69.6% of ring candidates rejected by the tag as foreign four-byte words, mean matched length 7.89 bytes, the nice-length stop reached on 283 of 29 M probes and the 256-candidate budget reached on NONE (`MATCH_INDEX_MAX_DEPTH` is 64, so it is unreachable by construction). Four byte-preserving attempts to make the walk cheaper were all measured as REGRESSIONS and are not in the tree; the private research record for 8 Sep 2026 has them and the index-geometry screen that replaced them. | `src/codec/rar50/probe_stats.rs`, `src/codec/rar50.rs`, `examples/rar5cli.rs` |
| 2026-09-08 | Research-only `RARS_MATCH_CANDIDATES` (`ratio-lab`), which varies the ring probe's per-position candidate budget ALONE. A ladder built out of `level` cannot price that budget, because `encode_options_for_level` also flips `lazy_matching` when the level moves. Added to answer where the time goes at the dictionary this repo actually SHIPS: the tree match finder arms at `TREE_MIN_DICTIONARY` = 4 MiB and the default is 2 MiB, so every finder result of the 7-8 Sep campaign describes a setting production does not use. Profiled on the 1 GiB mixed corpus at `-md2097152`, which is postfast's own configuration (`-m3` leaves `compression_level` at `None`): `best_match_probe` plus `consider_match_candidate` and `codec::fast::match_length` are 86.0% of non-idle samples, entropy coding 5.8%, index insertion 4.9%. The budget itself is NOT the lever - paired and externally verified, 64/96/128/256 candidates give the SHA256-IDENTICAL archive (three quarters of the shipped 256 is bit-for-bit dead) for about 3% of wall, because the probe's walk exits early on nearly every position rather than reaching its cap; 48 costs +0.103% of bytes and 32 costs +0.274% ON THAT ONE PAYLOAD. A later qualification over 25 members (the full Silesia corpus plus representative and reserved-holdout fixtures, both parsers, one-large-member and many-small-member layouts, every byte-changing archive externally extracted and compared) settled both halves: the 64-candidate identity is GENERAL, holding on every member and layout tested, while those two ratio costs are NOT - the worst per-file cost at 32 is +3.369% and every Silesia member exceeds +0.3%. A reduced budget buys CPU only where it costs bytes, and on already-compressed input the whole encode is under a tenth of a user second per 48 MiB with nothing to win, so no fast preset was shipped at any value. No default changed. The profile, the paired ladder, the qualification and the rejected lever are in the private research record for 8 Sep 2026. | `src/rar50/write/filter_policy.rs` |
| 2026-09-08 | Sampled filter probes reuse one encoder scratch pool within each region. Every probe still starts with empty dictionary history and the pool is released before the full member encode; plain probes also avoid retaining unused history. Full-byte and error differentials cover all sampled filter kinds, shrinking/empty/regrown inputs and retained scratch. Matched archive and allocation evidence is in `research/rar5-probe-scratch-2026-09-08` in the host repository. | `src/rar50/write/filter_policy.rs` |
| 2026-09-08 | **A memory allowance for the WRITER, `Rar50WritePolicy`, because nothing admitted its allocations.** The 8 Sep cost round measured a single compressed member holding 1.4 to 2.1 GiB of live heap beyond the caller's input at EVERY dictionary from 128 KiB to 32 MiB, out of four pools that took no budget argument: the match-finder tree (ten bytes per dictionary byte, 320 MiB at `-md32m`), the parse hints (a flat `TREE_HINT_BUDGET_BYTES` of 512 MiB), the block wave's scratch (`ENCODE_WAVE_MEMORY_BUDGET_PER_THREAD`, 128 MiB a thread, FLOORED at a gibibyte) and the output archive. A 32-bit target's whole `MemBudget::max_total()` is 1 GiB, so the wave floor ALONE outspent everything the phone posting path could budget, and asking for a wider dictionary silently bought a tree ten times its size. `WriterOptions::write_policy` (`None` by default, so no host-sized default moves) now carries an allowance into `EncodeOptions::working_memory`: half to the block wave, a quarter to the hints, a quarter to the tree - which fixes the widest dictionary admitted, `admitted_dictionary_size` REDUCING the request on the format's own quantum before the payload fit and never raising it, never below the 128 KiB floor. Both wave decisions are byte-neutral by construction and `a_write_policy_narrows_the_wave_without_moving_the_archive` pins that on a four-block member, asserting the two arms really did choose different widths first so the equality cannot pass vacuously; `a_write_policy_admits_only_the_dictionary_it_can_index` pins the reduce-only direction, the floor and the legality of every reduced size. `MemBudget::rar_write_policy()` is the caller-side counterpart of `rar_execution_policy()` (a quarter of the budget, 64 MiB to 4 GiB) and `crates/postfast/src/container.rs` passes it on both the single-archive and volume arms. NOT done here: a bounded input/output path - the writer still reads whole members and returns a whole archive - and no phone-target measurement, only the host relationship that `rar_write_policy_fits_inside_the_budget_it_came_from` pins. | `src/lib.rs`, `src/codec/rar50.rs`, `src/rar50/write/mod.rs`, `src/rar50/write/filter_policy.rs`, `src/rar50/write/stream.rs`, `src/rar50/write/volume.rs`, `src/rar50/extract.rs` |
| 2026-09-08 | Reserve exact space for a plain END header, preventing geometric vector growth from retaining twice the finished archive size when the payload filled the buffer. Archive bytes are unchanged; a large stored-archive capacity regression test covers the allocation behavior. | `src/rar50/write/mod.rs` |
| 2026-09-08 | Research-only large-dictionary cost controls under `ratio-lab`: content-sampled tree insertion, bounded hash chains, direct eight-byte hash hints with the short-match ring retained, and two- or three-prefix hash frontiers. Explicit ladder rungs also combine the cheap indexes with the existing tokenizer horizon choice. `bench_rar5_create --writer-memory` measures requested writer heap separately from preloaded inputs and reports returned output capacity. The ordinary build keeps its existing finder and defaults. | `src/codec/rar50/tree.rs`, `src/codec/rar50.rs`, `examples/rar5cli.rs`, `examples/bench_rar5_create.rs`, `examples/bench_rar5_create/memory.rs` |
| 2026-09-07 | The tree match finder records a per-position CANDIDATE LIST (`TREE_CANDIDATE_SLOTS` distances, nearest first, each reaching strictly farther) instead of one distance, and the cost-based parse takes that list as its candidate source in place of the 64-slot ring walk at tree dictionaries. The ring walk is unchanged below the tree's dictionary threshold and for the lazy parser, which asks for a stride of one and so is byte-identical and pays no memory. A hint budget narrows the wave so the answers held at once do not grow with the box's core count. Measured on the 256 MiB mixed slice at `-md32m`: 112,882,007 bytes at 236.0 user seconds becomes 112,865,011 at 133.9, peak RSS 2.27 GB to 2.02 GB, `rar t` ok. | `src/codec/rar50/tree.rs`, `src/codec/rar50.rs` |
| 2026-09-07 | ADAPTIVE ENTROPY-BLOCK BOUNDARIES on the RAR 5 writer, ON by default (`codec::rar50::boundaries`, new; `EncodeOptions::adaptive_entropy_blocks` and `WriterOptions::adaptive_entropy_blocks`, both `true`, `rar5cli --fixed-entropy-blocks` for the other arm). A tokenizer block's token stream was cut into entropy blocks every `ENTROPY_BLOCK_BYTES` (256 KiB) of INPUT, which is a guess: the right place for a table boundary is where the symbol distribution changes. Token decisions and the rep-distance state are fixed before the emit, so a block's encoded size depends only on its own tokens - which makes the partition a shortest path. Prefix histograms of the emitted symbols at every candidate boundary (a 64 KiB grid plus the fixed layout's own cuts, spans capped at 512 KiB) price any block exactly from two snapshots, a shortest path picks the partition, and each boundary is then moved within +/-32 KiB on a 4 KiB grid by coordinate descent over its two adjacent blocks' exact costs, twice. The cost model counts what `encode_token_block` writes and nothing else - the table description at the member's `algorithm_version` (so RAR 7's 80 distance slots are priced as 80), the leading filter records the first block carries (their symbol 256 folded into the main histogram BEFORE the code lengths are built, their raw payload counted by writing it rather than by a second copy of the field widths), and the block framing - and `statistical_cost_matches_actual_emission` pins it to the emitter over both distance tables and three filter sets. The fixed layout is PRICED by the same model rather than encoded, so only the winner is ever emitted; it is kept as the fallback and a release-mode guard re-emits it if the emitted length ever exceeded the fixed price, so the output can never grow. The prefix snapshots (a 4 KiB grid over 4 MiB is ~1,000 of ~3.5 KB) live in `EncoderScratch` and `LiveSpanEncoder` for the reason the match index does. `ENTROPY_BLOCK_MIN_TOKENS` stays on `entropy_block_token_ranges`, which cuts by input bytes and cannot know what a table costs, and is deliberately NOT a constraint on the search, which is measured rather than assumed: imposing it cost -0.27% instead of -0.76% on the mixed corpus at 32 MiB and -0.58% instead of -1.34% at 128 KiB, and the repeated-payload corpus - the case the floor was added for - came out 919 bytes SMALLER without it. Measured in the POOLED writer, 1 GiB corpora, `rar5cli a -m3` built `--features parallel`, against the same binary with the switch off, every archive `rar t` ok: mixed at `-md33554432` 475,936,821 -> 472,344,571 (-0.755%), mixed at the 128 KiB default 626,897,754 -> 618,500,542 (-1.339%), 400 members non-solid 506,104,176 -> 499,447,258 (-1.315%), the same solid 438,765,468 -> 434,053,901 (-1.074%), repeated 1,087,107 -> 1,086,188 (-0.085%); encode +0.9% to +9.1% of user CPU, decode 0% to +7% (`bench_extract`, five iterations, best of three); blocks on the 1 GiB mixed archive 3,101 -> 6,026 where rar 7.23 writes about 20,600. The feature-gated `codec::rar50::ratio` lab that found this is untouched - only the emit moved. NOT synced: hand-made here, for the reason the rows above give. Research record: `research/CODEX-RAR-RATIO-LAB-REVIEW-2026-09-07.md`. | `src/codec/rar50/boundaries.rs`, `src/codec/rar50.rs`, `src/rar50/write/mod.rs`, `src/rar50/write/filter_policy.rs`, `src/rar50/write/stream.rs`, `src/rar50/write/volume.rs`, `src/rar50/extract.rs`, `examples/rar5cli.rs` |
| 2026-09-07 | `FilterPolicy::Sampled` on the RAR 5 writer (`filter_policy::select_sampled_filters`; `rar5cli -mf`): sample-guided REGIONAL filters, the ratio lab's `regions::select` (review) in production shape. Every 262,143-byte region (the codec's filter chunk, so a record never straddles two LZ blocks) votes on three cheap 16 KiB probes across ten kinds (delta 1-4/8/16/24/32, E8E9, ARM) with a 2% margin over the unfiltered probe, text-reading probes are not probed at all, and one encode carries the winners as ranged specs; a member no region votes for is resolved exactly as `None` (the pooled encoder, the store sampling, the same bytes). Measured 7 Sep 2026 at 32 MiB with the optimal parse: mr 3,027,183 -> 2,686,457 and x-ray 5,161,783 -> 4,093,882, both from +11%/+24% OVER rar 7.23 `-m5 -mcx` to 1.8%/1.4% UNDER it; the mixed corpus votes nothing and is byte-identical. Its price was the path - a member with any filter took the serial 262,143-byte-block filtered walk, no tree finder and no pool - until the row below moved filtered members onto the pooled encoder the same day. NOT the default; no product crate names a filter policy (measurements in the private research record beside `research/rar5-ratio-lab/`). |
| 2026-09-07 | FILTERED MEMBERS THROUGH THE POOLED ENCODER (`Unpack50Encoder::encode_member_with_filters_pooled`, `filtered_lz_member_records`, `filters_in_block`, `filter_token_indices`): a member carrying filter records is transformed whole (the same 262,143-byte chunks, one record per chunk and filter, executable filters keyed by absolute offset) and then walks the same pooled, tree-fed, cost-cut path an unfiltered member does; each 4 MiB block takes the records in its range and the emitter writes each record just before the first token whose span reaches it, as an offset from that token's position (legal RAR 5: symbol 256 anywhere in the stream, the offset relative to the read position), with `boundaries` charging every record to the entropy block that carries it. It used to drop to a serial walk of one-chunk blocks with no tree and no pool, and records could only lead a block. Measured 7 Sep 2026 at 32 MiB with the optimal parse: mr 2,686,457 -> 2,642,508 and x-ray 4,093,882 -> 4,055,259 (3.4% and 2.3% UNDER rar 7.23 `-m5 -mcx`, past the ratio lab's joint tree/filter arms at 2,675,002 / 4,068,072), the filtered encode now as fast as the unfiltered encode of the same transformed bytes (x-ray: 2.05 s against 2.09 s, the raw member 0.76 s - the delta planes are simply more matchable). The old chunked path is kept under `test`/`ratio-lab` as `encode_member_with_filters_chunked`, the ratio lab's exact control. One byte-pinned fixture moved by a byte (`encoder_output_matches_pre_optimization_mixed_stream`, 33464 -> 33463). | `src/codec/rar50.rs`, `src/codec/rar50/boundaries.rs`, `src/codec/rar50/ratio/filtered.rs` |
| 2026-09-07 | `FilterPolicy::Sampled` on the VOLUME writers too (`Rar50VolumeWriter::filter_policy`; `None` or `Sampled`, compressed sets, plain and encrypted, never solid - the other policies are refused there), and the selector's probe kinds STAGED: five kinds first (delta 1/2/4, E8E9, ARM), the wider and odd delta strides only where a delta already won. A member that votes nothing pays six probes per window instead of eleven: on the 256 MiB mixed slice the probes went from +11% user to within noise, and on 100 files of the `small/` set +1%, both byte-identical to `None`. Votes on mr and x-ray are unchanged (delta-2 on every region). `postfast` selects it with the optimal parse (`Packing::optimal_parse`, `--optimal-parse`): one lever for the smallest archive the writer knows how to make. | `src/rar50/write/mod.rs`, `src/rar50/write/volume.rs`, `src/rar50/write/filter_policy.rs`, `examples/rar5cli.rs` |
| 2026-09-07 | DEFAULT RAR 5 DICTIONARY 128 KiB -> 32 MiB (`DEFAULT_RAR50_DICTIONARY_SIZE`; the format's 128 KiB floor and quantum is now `RAR50_DICTIONARY_QUANTUM`), rar's own default, decided on the 7 Sep 2026 dictionary study: on the non-solid archives the poster writes the dictionary reaches inside one member only, 32 MiB captures every gain a member can offer (-7.3% on a large compressible member, -6.7% on a small-file set), and it is the window every RAR reader in the field has budgeted for since 2013; 128 MiB was measured and refused (4x extractor memory for +5.3% on one shape). Landed WITH rar's payload-fit reduction, which the writer never had (`filter_policy::fitted_dictionary_size`, `dictionary_size_for_payload`): the declared dictionary halves while it is strictly greater than twice the largest member (a solid set: its total), never below 128 KiB, archive-wide, reproducing rar 7.23 row for row on the study's six member sizes; every writer applies it (single archive, volume sets plain and encrypted, streamed archive and volumes) and the encoder's match distance follows the declared size, so a post of small files declares a small window and indexes for a small window. Without the fit a 32 MiB default would have asked a phone for 32 MiB to unpack a 200 KiB member. `postfast` gains `Packing::dictionary` and `--dictionary <bytes>` for the one shape where more pays. | `src/rar50/write/mod.rs`, `src/rar50/write/filter_policy.rs`, `src/rar50/write/volume.rs`, `src/rar50/write/stream.rs` |
| 2026-09-07 | VOLUME END HEADERS CARRY THE NEXT-VOLUME FLAG (`ENDARC_NEXT_VOLUME`, 0x0001) on every volume but the last, in every volume writer (in-memory stored/compressed/encrypted with and without a recovery record; streamed stored, stored-with-recovery, compressed and encrypted). Every volume wrote END flags 0 ("last volume") before: harmless while a member straddles the boundary (the file header's split flags carry unrar across) and fatal the moment a member ENDS exactly on a volume's end, where native unrar 7.23 read "last", stopped, and never extracted the members in the later volumes - `unrar t` still exit 0, the second member exit 10 with 0 bytes (the review's native probe, 7 Sep 2026: two 64-byte members in two 119-byte volumes). A full volume is now HELD without its END and sealed when the next volume opens (flag on) or at finish (flag off), since only then is it known. Verified with native `rar` on this writer's output of that exact shape: both members extract byte-exact. | `src/rar50/write/volume.rs`, `src/rar50/write/stream.rs` |
| 2026-09-07 | **A per-region tokenizer horizon choice in the RAR 5 writer**, behind `WriterOptions::tokenizer_horizon_choice` / `EncodeOptions::tokenizer_horizon_choice` (`rar5cli -mh`), OFF by default. A member's 4 MiB regions are the block pool's work items and each was tokenized as ONE block; `encode_member_region` now encodes a region a SECOND time as 1 MiB tokenizer blocks and keeps whichever encoded smaller, so the switch cannot cost a byte on any shape - the arm it is measured against is always one of its candidates. The arms are comparable because a block's tokenizer sees the `max_match_distance` bytes of member INPUT before it and never a previous block's output (`member_block_history`), and because every tokenizer block already starts its parse and its emission from a default `EncoderMatchState`, so no repeat state crosses the choice; only the member's real last block carries the final flag. It composes with the tree match finder, the cost-based parse and the adaptive entropy splitter, which is what prices each arm. Measured on the first 256 MiB of the mixed corpus at `-md32m -mo`: 112,882,007 -> 111,639,169 bytes (-1.10%), which is the review's lab arm `opt-horizon-balanced` to within its two-byte archive framing, and 112,865,011 -> 111,617,844 (-1.11%) once the tree finder's per-position candidate list landed beside it, for 33% more encode CPU and 22% more wall on an idle 20-core box. The idle-box pairing, the poster shapes and the shapes it does NOT reach are in the private research record for 7 Sep 2026. NOT reached: a SOLID member of 4 MiB or less, which `LiveSpanEncoder` encodes on an index and a finder carried across the group - both arms would have to run over that live state and only one may leave its marks on it. `member_windows_make_the_same_horizon_choices_as_the_whole_walk` (part of `member_windows_encode_the_same_blocks_as_the_whole_member_walk` until the 11 Sep 2026 split below) and `the_region_horizon_choice_reaches_the_streamed_writer` hold the windowed writer to the same choices, on material where the choice actually bites. | `src/codec/rar50.rs`, `src/rar50/write/mod.rs`, `src/rar50/write/filter_policy.rs`, `src/rar50/write/stream.rs`, `src/rar50/write/volume.rs`, `examples/rar5cli.rs` |
| 2026-09-07 | **The tree match finder's comparison cap `TREE_NICE_LENGTH` raised from 64 to 256, paid for by comparing a word at a time rather than a byte** (`codec::rar50::tree::common_prefix`). The cap decides which position the finder hands back when several agree that far, and depth is worth bytes on content with long near-identical records: Silesia's twelve files total 52,781,726 at 64 and 52,584,521 at 256 through the production writer (-0.374%, no file larger, and 100.41 against 93.29 user seconds), with `nci` at -8.99% and `xml` at -1.60%; the synthetic corpora and the poster's own 400-file set are SHA256 identical at any cap, solid and not. The depth cannot be split from the wholesale-replacement exit it shares - ending the descent IS the ranking decision, measured at exactly the 64-byte figure when the two are split - so the cost was answered in the comparison instead. Paired on an idle box over a 256 MiB slice of the repeated corpus, every arm writing 1,058,474 bytes: 28.50 s at cap 64 byte at a time, 57.39 s at cap 256 byte at a time, 26.41 s at cap 256 a word at a time. Ratio and CPU record: research/CODEX-RAR-REVIEW-ROUND2-2026-09-07.md section 1 and research/TREE-NICE-LENGTH-2026-09-07.md. |
| 2026-09-07 | **RECONCILED WITH THE FORK, so `sync-from-fork.sh` works again.** The vendored copy had taken 141 commits and 115 rows of this table directly since the 23 Aug sync while the fork sat still, so the documented one-line re-vendor would have `rsync --delete`d working engine code: 28 divergent paths and ~16,400 lines, five of them (`src/memtrack.rs`, `src/rar50/blake2sp/simd.rs`, `src/rar50/write/stream.rs`, `src/recovery/gf16_fold.rs`, the whole `src/codec/rar50/` directory) present ONLY here. `VENDOR-REV` also named `119c2df`, a rev `git cat-file -t` cannot find in the fork at all, so the record and the tree had disagreed about where the fork was for two weeks. Reconciled by the 22 Aug shape rather than commit by commit, because a three-week gap has no tractable commit-by-commit route: ONE commit on the fork's `perf`, `949349e`, mirroring `src/` from vendor at nzbfast `ec623c71e`. That shape is only safe if vendor is a strict SUPERSET, and it was CHECKED rather than assumed: every symbol defined in the fork's `src` resolves in the vendored tree (the three that did not are renames - `check_both` to `check_kernels`, `DefaultLeaves` to a per-arch `Hasher` alias, and one delta test's name), and every fork commit since `1b84503` touching `crates/rars/src` had already been hand-carried across (the tolerant tail, the `.rev` recovery volumes, the RAR4 volume sets and their naming bit, the caller-supplied entropy source, the split STORED plural, the file-time extra record). `sync-from-fork.sh` then re-mirrored from `949349e` and `git diff vendor/rars/src` is EMPTY, which is the whole acceptance test. Fork side, not vendored: its manifest gains `cbc` and the `ratio-lab` feature, and six integration-test sites move to contracts this tree changed under them - five opt into `HashRecord::Blake2sp` (the BLAKE2sp record became a writer option that is off by default on 5 Sep), and `stream_source_rejects_rar4_family_cleanly`, which was RED at the fork's own base, asserted a RAR4 family refusal the one-pass RAR4 work removed, so it now pins that a truncated RAR4 archive fails on its own header. `cargo +1.98.0 test -p rars --tests` in the fork: 1244 passed, 0 failed. The drift guard's `--since` is bumped to this sync, and `.upstream-COPYING.seen` is now committed so the upstream-licence watch reports a real change instead of its own first run. CORRECTION, same day: the fork DOES have a private durable home, whose `perf` is `119c2df` exactly, so the 23 Aug note's "pushed" was accurate and the missing rev was an UNWIRED REMOTE rather than a lost commit: `the upstream rars fork` had only the PUBLIC upstream configured. The private remote is wired up as `mine` now. The two lines had diverged at `c586ae2` so `949349e` could not fast-forward `mine/perf`; RECONCILED the same day with an ours-strategy merge that keeps the mirror's tree byte for byte, on the evidence that `119c2df`'s src is a strict subset of it (no file and no symbol exists only there). Nothing was force-pushed and the redo trio stays reachable through the merge's second parent. The fork tip is `258a508`, held by the private remote and both Macs, so the fork has a durable home rather than one disk; `258a508` also adds the repo-root `rust-toolchain.toml` the fork never had, which is outside the mirror's scope and moves no vendored byte. Never push either line to the public upstream (TODO 11). | `src/**` (28 paths), `VENDOR-REV`, `sync-from-fork.sh` |
| 2026-09-07 | Lab-only joint filter/tree encoder preserves filter block boundaries, records and transformed solid history while supplying tree hints to the existing optimal block encoder. Disabled-tree byte controls, all filter kinds, partial ranges, reset/error state and external archives are checked. Production filtered paths are unchanged. | `src/codec/rar50/ratio/filtered.rs`, `src/codec/rar50/ratio.rs`, `examples/ratio_lab.rs` |
| 2026-09-07 | Feature-gated tree comparison-cap laboratory: 32/64/128/256-byte stopping points, with a fixed 64-byte production path. Brute-force candidate checks, schedule independence, solid/reset round trips and default-output identity cover the new controls. Ratio measurements are in research/rar5-ratio-lab/tree-depth. | `src/codec/rar50/tree.rs`, `src/codec/rar50/ratio.rs`, `examples/ratio_lab.rs` |
| 2026-09-07 | Connect precomputed tree hints to the optimal parser and merge them into its nearest-distance candidate list; an empty hint span preserves the ring path. Ratio lab shares immutable hints across each member's horizon trials and retains explicit ring controls. Adds candidate-validity, parser-connection, round-trip, solid/reset and hint-release checks. The library suite passes 948 tests, four ignored; corpus ratio evaluation is separate. | `src/codec/rar50.rs`, `src/codec/rar50/ratio.rs`, `examples/ratio_lab.rs` |
| 2026-09-07 | **A binary-tree match finder (7-Zip's `bt4` class) in front of the RAR 5 block tokenizer**, for dictionaries at or past 4 MiB. `codec::rar50::tree::TreeMatchFinder` keeps a binary search tree per four-byte hash bucket, ordered by the suffix at each position, over a cyclic window of the dictionary; every position is inserted and searched by one descent, so the longest match in a bucket is found in O(log n) however dense the bucket is - which the ring index, keeping the newest 64 positions of a bucket, cannot be. `encode_blocks_with_tree` walks the member's positions in order ahead of the block pool, one range per 4 MiB block, and hands each block the best distance found at each of its positions (`TreeMatches`, probed in `best_match_probe` beside the rep probes, the ring and the long table); `LiveSpanEncoder` carries one finder across a solid group's members. THE ANSWERS ARE A FUNCTION OF THE POSITION ALONE - positions enter their bucket's tree in increasing order and a node's descendants are always older than it is, so a window that starts a full window before a block holds the same live tree the whole-member walk holds there, and the walkers that split a range BY BUCKET produce the serial walk's answers at any thread count (barriered every 256 KiB, with the window held that far below the cyclic buffer so a walker running ahead cannot overwrite a slot another still counts as live). Measured on the first 256 MiB of the mixed corpus at `-md32m`: 118,772,416 -> 116,498,033 bytes with the far-distance surcharge below, against rar 7.23 `-m3`'s 116,751,028, and matches eight or more MiB back go from 60 K to 2.73 M against rar's 2.21 M. Costs 2.9x the ring's CPU. `member_windows_encode_the_same_blocks_as_the_whole_member_walk` (six `member_windows_match_the_whole_walk_*` cells plus a horizon test since the 11 Sep 2026 split below) and the streamed-writer identity tests hold unchanged. The round's measurements, including the arms not kept, are in the private research record for 7 Sep 2026. | `src/codec/rar50/tree.rs`, `src/codec/rar50.rs`, `src/rar50/write/filter_policy.rs` |
| 2026-09-07 | A far-distance surcharge in the parser's cost estimate (`far_distance_surcharge`, six bits above a megabyte) and a margin on the solid walk's reset rule (`SOLID_RESET_MARGIN`, 64 bytes). Both are documented at their sites with the sweep behind them. The first stops the tree finder's far matches from being underpriced against near ones (117,291,644 -> 116,498,033 on the slice); the second stops a member whose two arms are within noise of each other from resetting a solid stream's history (10,240 members of 10 KiB: 103 resets and 55,921,162 bytes -> 0 resets and 45,213,892, against rar's 46,114,798). | `src/codec/rar50.rs`, `src/rar50/write/filter_policy.rs` |
| 2026-09-07 | Ratio lab combines optimal parsing with all 32 delta widths, regional filters and production auto filters. Extended ultimate retains the previous 37 choices first; optional candidate export supports independent oracle checks. Production codec APIs and defaults unchanged here. | `examples/ratio_lab.rs` |
| 2026-09-07 | Ratio lab `ultimate` portfolio unites earlier candidate lists once and retains new horizon choices; 80 archives externally verified including 27 byte-identical old-mode controls. Separately corrected the progress test to allow workers completing before coordinator scheduling; production behavior unchanged. | `examples/ratio_lab.rs`, test in `src/codec/rar50.rs` |
| 2026-09-07 | Ratio lab: opt-in seven-way power-of-two tokenizer horizon search from 64 KiB to 4 MiB, retaining all four prior choices. Evidence in `research/rar5-ratio-lab/horizon-screen/README.md`; full corpus measurement remains separate from the initial screen. | `src/codec/rar50/ratio.rs`, `examples/ratio_lab.rs` |
| 2026-09-07 | Ratio lab: opt-in balanced 4 MiB/1 MiB tokenizer choice and duplicate single-tail trial elimination. Old two/four-way archive bytes preserved by external controls; evidence in `research/rar5-ratio-lab/balanced/README.md`. | `src/codec/rar50/ratio.rs`, `examples/ratio_lab.rs` |
| 2026-09-07 | Opt-in ratio-lab four-way tokenizer selection adds 1 MiB and 512 KiB candidates per 4 MiB region, retaining the existing two-way mode unchanged. Measured output bounds and external RAR verification in `research/rar5-ratio-lab/wide/README.md`. | `src/codec/rar50/ratio.rs`, `examples/ratio_lab.rs` |
| 2026-09-07 | Ratio lab combines the landed optimal token walker with refined entropy boundaries and per-region 4 MiB/256 KiB tokenizer choice. Opt-in only: 114,435,014 bytes versus RAR 7.23 -m5 -mcx at 114,514,620 on the same 256 MiB synthetic mixed input and 32 MiB dictionary; external extraction/hash checked. Full details in the ratio-lab tokenizer report. Production parser and defaults unchanged here. | `src/codec/rar50/ratio.rs`, `examples/ratio_lab.rs` |
| 2026-09-07 | Ratio-lab tokenizer horizons and exact per-region long/short choice, with complete-archive fallback portfolio. Default horizon remains 4 MiB; production parser and emitter untouched. Evidence: `research/rar5-ratio-lab/tokenizer/README.md`. | `src/codec/rar50/ratio.rs`, `examples/ratio_lab.rs` |
| 2026-09-07 | A cost-based OPTIMAL PARSE, `EncodeOptions::optimal_parse` (`WriterOptions::optimal_parse`, `rar5cli -mo`), one level above the lazy parser and off by default: `walk_tokens_optimal` runs a dynamic program over a 16 Ki-position window, every node carrying its own `EncoderMatchState` so the four rotating repeat distances and the two-bit length-repeat token are inside the search (LZMA's `GetOptimum` shape). `PriceModel` turns each entropy region's token frequencies into the next region's prices, in sixteenths of a bit from integer logarithms (`log2_sixteenths`, never a platform `log2`, so bytes do not move across platforms); `TokenPrices::constant` is `estimated_match_cost` exactly, for the first region. `DistanceArm` prices a length in two array reads once the distance's arm is decided. `match_candidates_at` is the parse's ONE candidate source (for each length, the nearest distance) and is the body a tree match finder replaces. Measured on the 256 MiB mixed slice at a 32 MiB dictionary (with the adaptive block splitter and the length-limited codes, which is what ships): 117,645,910 bytes to 114,407,187, UNDER rar 7.23's own exhaustive `-m5 -mcx` at 114,514,620 (-2.75%, and under rar `-m3`'s 116,751,028) for 3.7x the CPU on an idle box, of which 81% is the ring walk the parse probes every position with; the repeated and 400-file shapes are not larger; the 400-file set goes 499,447,258 to 488,399,398, and the repeated payload is 121 bytes (0.011%) larger, which is stated in the research record. The FIRST price region is a quarter length (`OPTIMAL_FIRST_REGION_BYTES`), which is worth -0.90% on a member set for no CPU because a short member otherwise never leaves the constant model. Nothing is on by default and the lazy parser and its tests are untouched. | `src/codec/rar50.rs`, `src/rar50/write/{mod,filter_policy,stream,volume}.rs`, `examples/rar5cli.rs` |
| 2026-09-07 | Opt-in ratio lab: sampled regional delta/code filters, full-archive ultra portfolio, and measured solid-order search. Existing codec APIs only; production defaults unchanged. Results and reproduction: `research/rar5-ratio-lab/regions/README.md`. | `examples/ratio_lab.rs`, `examples/ratio_lab/regions.rs` |
| 2026-09-07 | LENGTH-LIMITED HUFFMAN CODES: `codec::huffman::lengths_for_frequencies` built an unconstrained tree and, whenever a symbol needed more than `max_bits` bits (15 for every RAR 5 table), threw the frequencies away and returned `uniform_lengths_for_frequencies` - a FLAT code with every used symbol at 8-9 bits. It now builds the OPTIMAL code of that depth by package-merge (Larmore and Hirschberg, run as the coin collector: `package_merge_lengths_for_frequencies`, at most 306 x 15 coins per table, against tokenizing the 256 KiB block the table describes), and the flat code is kept only for the single-symbol case and for an alphabet no code of `max_bits` bits can cover (`used_count > 2^max_bits`), which is what the 1,024-symbol / 1-bit test pins. The unconstrained builder is split out as `unconstrained_lengths_for_frequencies`, answering `None` where it overflows, so the symbol-vector reference test still covers the flat parent-link tree exactly and the limiter is tested on its own: optimality against brute-force enumeration of every complete code under the cap on small alphabets, and Kraft equality plus a cost under the flat code's on Fibonacci-shaped frequency vectors over 306 symbols, the shape that overflows 15 bits. On the first 256 MiB of corpusgen's mixed.bin the fallback fired 79 times over ~770 main tables at `-md32m` and 131 times at the 128 KiB default (main tables of 306 symbols, and 2-4 length tables of 64); measured, every archive `rar t` ok: 32 MiB 118,772,416 -> 118,507,983 (-0.22%), 128 MiB 77,201,547 -> 77,016,740 (-0.24%), 128 KiB default 157,067,269 -> 156,244,335 (-0.52%); user CPU flat over three rounds at 32 MiB (29.14 -> 29.11 s mean). No archive golden moved, and `cargo test --release -p rars --features parallel` is green. | `src/codec/huffman.rs` |
| 2026-09-07 | Ratio lab follow-up: separate adaptive module, finer candidates with coarse fallback and monotonic local boundary refinement; opt-in wide delta portfolio using existing codec filters. No production defaults change. Results in `research/rar5-ratio-lab/refinement/README.md`. | `src/codec/rar50/ratio`, `examples/ratio_lab.rs` |
| 2026-09-07 | Feature-gated ratio laboratory: independent short-repeat, adaptive entropy partition, rep-state carry and exhaustive-probe experiments, plus a whole-archive portfolio and deterministic similarity-order benchmark. Production defaults are unchanged. See `research/rar5-ratio-lab/README.md` for measured wins, regressions and reproduction. | `src/codec/rar50/ratio.rs`, `examples/ratio_lab.rs`, feature/module registration |
| 2026-09-06 | A match index carried LIVE across a solid group's members: `codec::rar50::LiveSpanEncoder` seeds once per group from the dictionary before it (`MatchIndex::seed_history_at`) and walks the members one after another on the same index, which the walks keep current - the tokenizer's walk split from its seeding (`walk_tokens`, called by `encode_tokens_indexed_in_span` after seeding and by the live encoder without it), the tag geometry fixed from the group span's length, a member longer than a block falling back to the block path and the next member re-seeding. `resolve_solid_members_with_rule` runs a group's continued arms through it in order, beside the fresh arms on the pool. Every continued arm used to seed a fresh index from up to 32 MiB of history (~80 ms): 10,240 members of 10 KiB took 368 s of wall and 1,667 CPU-seconds against rar's 24 s. Bytes identical: `pooled_solid_resolve_matches_the_serial_walk` now covers sixty tiny members under a dictionary spanning dozens with forced resets, and `rar5cli -s` matched the previous binary's sha256 on the 200-member set plain and encrypted. The index is shaped as the per-member walk shapes it (`MatchIndex::new_shaped`: history plus member for the shape, the group span for the position packing) - shaped for the whole span it had more buckets and different bytes; groups run as many at once as the pool has threads (`SOLID_GROUP_BYTES` 16 MiB); and the decision walk's re-encodes after a reset run on a live index over a span grown a member at a time (`with_capacity`, the last three positions of a member inserted at the next call), where re-seeding each of them was 24 s of the 10,240-member set's 26 s. Measured on the 8-vCPU guest: 10,240 members of 10 KiB solid 368 -> 18.7-19.5 s wall (1,667 -> 40-44 CPU-seconds; rar 7.23 18.0-18.6 s), 200 members of 2.7 MB 31.0-32.4 -> 27.4-27.5 s (rar 44-47), the same encrypted 32.3-34.7 -> 27.4-30.0 s, 400 members 77.7 -> 59.5 s (rar 82.6); on the 32-core Mac the 10,240-member set 24.9 -> 6.8 s. Every sha256 identical to the previous binary's. | `src/codec/rar50.rs`, `src/rar50/write/filter_policy.rs` |
| 2026-09-06 | STREAMED RECOVERY RECORDS: `recovery::rar5::InlineRecoveryFolder` computes the inline record from the archive's bytes as they pass (every shard a contiguous range of the prefix, so a pushed chunk folds into every parity row at its offset; per-group CRC states slice by slice; a half symbol carried across pushes; the record assembled by `assemble_inline_recovery_record`, factored out of the whole-prefix builder and shared with it), and `write_stored_archive_streamed_with_recovery` / `write_stored_volumes_streamed_with_recovery` (`rar5cli --stream -m0 -rr`) write through a `FoldingSink`, solving the locator offset on header lengths before the first byte (`solve_recovery_offset`, the in-memory writers' rule; the volume set plans its cuts from the sizes first). Byte-identical to `Rar50Writer::recovery_percent` and `Rar50VolumeWriter::recovery_percent`: `inline_recovery_folder_matches_the_whole_prefix_builder` over odd chunking, an odd length and several groups; two writer identity tests; sha256 on the 1 GiB shape single and in sixteen volumes; rar 7.23 `t` on all. The in-memory path built the archive whole and copied it into 200 shard buffers: 3.2 GB working set, most of the wall on a KVM guest. The rows fold in parallel, and `recovery/gf16_fold.rs` adds the SIMD multiply-by-constant (nibble-shuffle NEON / SSSE3 / AVX2 and GFNI bit-matrices, runtime-picked, the PAR2 engine's form over the same field), pinned to the scalar tables by `simd_fold_matches_the_scalar_tables`; `Gf16MulTable::fold_into` runs it and folds the tail scalar, so every recovery fold in the crate (records, REV, repair) takes it. The crate's second `#[allow(unsafe_code)]`, with the argument in the module header and Cargo.toml; the pin test counts two files. Measured on the 8-vCPU KVM guest (GFNI), 1 GiB stored, in-memory / streamed / rar 7.23: `-rr3` 6.2-7.5 / 2.4 / 2.5-3.8 s, RSS 3,179 / 66 / 53 MB; `-rr10` 7.4-9.1 / 4.1-4.4 / 3.3-4.5 s, RSS 3,252 / 217 / 126 MB; `-rr3` in sixteen 64 MiB volumes 6.8-8.3 / 2.1 / 3.1-4.7 s, RSS 2,262 / 6.5 / 31 MB. The fold's parallel shape matters on a guest: rows per piece or rows per chunk cost 7-8 s of system time in pool wakeups and lost to the serial fold; the pieces of a chunk in parallel (`STREAMED_FOLD_PARALLEL`) cost none and carry the many-core Mac's `-rr10` from 7.7 to 2.0 s. | `src/recovery/rar5.rs`, `src/recovery/gf16_fold.rs`, `src/recovery/mod.rs`, `src/rar50/write/stream.rs`, `src/rar50/write/mod.rs`, `src/rar50/write/volume.rs`, `src/rar50.rs`, `src/rar50/blake2sp/mod.rs`, `Cargo.toml`, `examples/rar5cli.rs` |
| 2026-09-06 | `token_statistics_of_an_archive`, an ignored in-crate DIAGNOSTIC (`RARS_STATS_ARCHIVE=<archive> cargo test --release -p rars --features parallel --lib -- --ignored token_statistics --nocapture`): walks a RAR 5 member's compressed blocks with the crate's own table reader and symbol decoders and prints where the bits went - blocks and table sets, literals, filters, repeats, matches by length bucket and by log2 distance, and the far (8 MiB and beyond) matches by length - so two writers' archives over the same input compare class by class. Used to find that the last 1.7% against rar at 32 MiB is 6 MB of copy rar gets from 2.2 M matches 8-32 MiB back (mostly 32-63 bytes long where we match 16-31 nearer), that is, candidate-walk depth on dense text buckets, not table reach: eight long-table variants (denser anchors, shorter keys, wider tables, four ways, the gate opened) measured -0.01% to -0.51% for +4% to +240% wall and none landed; the structural answer is a tree match finder. The research record carries the table. | `src/codec/rar50.rs` Extended 6 Sep to aggregate across a solid stream's members (tables and rep state carried as the decoder carries them), which showed the 10,240-member solid set's 21% gap to rar is match REACH (808 K far matches copying 18.4 MB where ours finds none), not the 0.31 MB table gap; table carry across solid members measured 0.07% and was not kept. |
| 2026-09-06 | Solid arms from a ROLLING WINDOW (`resolve_solid_members_with_rule`, `SOLID_GROUP_BYTES`): the first cut assembled up to a dictionary of history per continued arm and the single-block tokenizer copied history and input into one span again, two 32 MiB copies per member over a set of small members. Members are resolved a group (64 MiB) at a time from one window - the dictionary before the group, then the group back to back - every arm borrowing its history from it, through the new `codec::rar50::encode_lz_member_span_pooled`, which hands the single-block tokenizer a contiguous span to index in place (`encode_lz_block_in_span`'s `indexed_span`, the wide-dictionary blocks' path). The swapped-rule test runs with `SOLID_GROUP_BYTES` of 5,000 under `cfg(test)` so the roll is exercised; `rar5cli -s` sha256 matched the previous binary's. The ENCRYPTED solid paths (single archive, volume set) use the same resolver and the encrypted volume set's non-solid members are resolved on the pool; encryption stays in member order (salts and IVs are drawn in that order) and the seeded-entropy shapes matched the previous binary's sha256. `encode_with_solid_reset_policy_and_progress` is now the tests' reference only. Measured on the 8-vCPU guest, 200 members of 2.7 MB at `-md32m`, two rounds: solid wall 34.0-35.1 -> 34.6-36.5 s (user-bound; system 30-32 -> 15-17 s, RSS 1.73-1.80 -> 1.35 GB), 400 members 77.7 -> 66.1 s (rar 97.4); encrypted solid 213-219 -> 33.0-33.2 s; encrypted 64 MiB volumes 63.6-70.3 -> 9.4-11.1 s; every sha256 identical to the previous binary's, every archive accepted by rar 7.23. | `src/rar50/write/filter_policy.rs`, `src/rar50/write/mod.rs`, `src/rar50/write/volume.rs`, `src/codec/rar50.rs` |
| 2026-09-06 | SOLID SETS resolve both arms of every member on the pool (`filter_policy::resolve_solid_members`, used by the single-archive and volume-set solid paths): the walk encoded every member twice on one thread - continued against the history, fresh without it, the smaller kept - and cloned the encoder per member; 200 members of 2.7 MB at `-md32m` took 203-206 s on the 8-vCPU guest against 9.6 s non-solid and 44-47 s for rar. A member's history is the raw bytes of the members before it since the last reset, never an earlier encode's output, so every fresh arm and every continued arm (speculatively against the no-reset history) is encoded at once via `parallel::map_collect`, and the walk only decides, re-encoding a continued arm against the true history only when a reset lies within a dictionary of it. Rule and histories unchanged, so bytes and flags are the serial walk's: `pooled_solid_resolve_matches_the_serial_walk` holds it under the real rule and under a swapped rule that forces resets, and `rar5cli -s` matched the old binary's sha256 on the 200-member set (30.2 / 31.1 s after). The serial policy (`encode_with_solid_reset_policy_and_progress`, still used by the encrypted solid paths) no longer clones the encoder: the only outcome needing the pre-member state replaces the encoder wholesale. Per-set `EncoderScratchPool` for every member resolve (single archive, volume set, solid arms, streamed jobs): each member used to fault in 40 MiB of fresh index and token buffers, 8 of the 200-member set's 9.6 s wall in system time on the guest; system time 7.9 -> 3.7 s and wall 9.5 -> 8.7-9.0 s on that set, 9.9 -> 8.8 s in volumes, 10.5-12.4 -> 9.2 s streamed, sha256 identical on all; the solid set keeps most of its system time in the per-arm history copies (two 32 MiB copies per member), the next step there. Volume-set members carry their digests from the resolve (`rayon::join` beside the encode, as the single archive does) instead of computing them at emission. review queue item 15 (progress-counter contention) rejected on reading: the tokenizer reports once per MiB. | `src/rar50/write/filter_policy.rs`, `src/rar50/write/mod.rs`, `src/rar50/write/volume.rs`, `src/rar50/write/stream.rs`, `src/codec/rar50.rs` |
| 2026-09-06 | The in-memory compressed VOLUME writer (`write_compressed_volume_set_impl`) resolves its non-solid members on the pool (`parallel::map_collect`, each member the exact resolve the serial loop did: `resolve_compressed_volume_member`) instead of one at a time; the solid path keeps its serial encoder. Found by the streamed writer's pairing: 150 members of 2.7 MB into 64 MiB volumes took 45 s on the 8-vCPU guest against 7.3 s as a single archive of the same members, which resolves on the pool since 5 Sep. Bytes identical - the volume goldens and the streamed identity tests hold it; the same set on the guest went 45.0 -> 7.0 / 7.2 s (rar 13.9 / 19.0). | `src/rar50/write/volume.rs` |
| 2026-09-06 | STREAMED COMPRESSED WRITERS (`write_compressed_archive_streamed`, `write_compressed_volumes_streamed`; `rar5cli --stream -m3`): a compressed member is encoded a SEGMENT at a time from a `Read + Seek` source through the block pool, via the codec's new `encode_lz_member_window` (the blocks from `first_block` on of a span whose earlier blocks are the member bytes before them; `MemberWindow` threads the first block and the final-segment flag through `encode_lz_member_blocks_in_waves` / `encode_blocks_pooled`, so only the member's true last block carries `is_last`). Windows are whole blocks - the previous segment's tail, at least the dictionary, plus a segment of the pool's width in blocks (`member_window_blocks`, never under eight) - so every block sees the history the whole-member walk hands it and the anchors of the history blocks are computed from the span as that walk computes them; the concatenation is the whole-member output byte for byte (a codec test at three dictionaries; two writer tests against `Rar50Writer` / `Rar50VolumeWriter` at windows of one and eight blocks; sha256 identity on six 1 GiB CLI shapes here and eight on the VPS, all accepted by rar 7.23). Two windows are encoded on threads of their own with a third being read, so the pool never idles at a window's tail. `EncoderScratchPool` hands the workers' 72 MiB scratch buffers on from window to window and member to member - without it every window faulted in fresh ones and the KVM guest's system time went 8 -> 18 s over a gigabyte, the streamed writer 30% slower than in-memory. Members up to one window are read whole and resolved as concurrent jobs (`resolve_member`: the sampler, the encode, the compare against size), emitted in order through a `MemberSink` (`ArchiveSink`, `VolumeSetSink`), at most the pool's width of them holding two windows of member data - one at a time, 150 members of 2.7 MB were 15x slower than in memory; a member longer than a window drains the jobs first. The sampler split into `incompressible_sample_windows` + `samples_read_incompressible` so the windowed member's verdict is read from its source. Stated limits: the single archive holds one windowed member's packed bytes (the header carries their length); the volume set holds a volume's worth and cannot store a windowed member the encode failed to shrink after the sampler passed it (compressed there, stored by `Rar50VolumeWriter`; needs three 256 KiB+ windows each shrinking over 0.5% while the whole does not - no corpus shape); level 5 refused by name. Measured on the 8-vCPU KVM guest (idle, three warm rounds), 1 GiB `-m3 -md32m`, in-memory / streamed / rar 7.23: mixed 34.0-36.5 / 29.7-31.4 / 83.7-92.4 s, RSS 1,981 / 1,004 / 305 MB; mixed in eight 64 MiB volumes 32.0 / 28.6 / 87.1 s, RSS 1,981 / 653 / 305 MB; mixed at the 128 KiB default 14.8 / 12.8 s; rep 6.5-6.8 / 6.0-6.3 / 15.0-15.2 s, RSS 1,302 / 505 / 305 MB; 150 members of 2.7 MB 7.3-7.5 / 7.6-9.5 / 15.9-16.6 s, RSS 812 / 272 / 91 MB. | `src/rar50/write/stream.rs`, `src/codec/rar50.rs`, `src/rar50/write/filter_policy.rs`, `src/rar50.rs`, `examples/rar5cli.rs` |
| 2026-09-06 | The streaming stored writers take ENCRYPTED members and encrypted headers (`write_encrypted_stored_archive_streamed`, `write_encrypted_stored_volumes_streamed`): a member is enciphered a megabyte at a time through `StreamCipher` (chained AES-CBC, the partial block at a chunk's end carried to the next, the last zero-padded), its cipher bytes cut into volume fragments at the same offsets `Rar50VolumeWriter` cuts them, headers built by the in-memory writers' functions; the salt and IV of every member and the header keys are drawn in the ORDER each in-memory writer draws them (the archive: header keys first; the volume set: members first), so the bytes match under seeded entropy - four tests hold it, and `rar5cli --stream -p/-hp` matched the in-memory CLI's sha256 on four shapes, every archive accepted by rar 7.23. Peak RSS for a 1 GiB encrypted member 2,050 -> 5 MB. Measured on the 8-vCPU KVM guest (idle), 1 GiB encrypted stored, three alternations: single archive in-memory 6.5-7.2 s / streamed 2.71-2.90 s / rar 7.23 2.48-3.36 s; header-encrypted sixteen 64 MiB volumes 6.4-6.8 / 2.75-2.92 / 2.90-3.15 s; system time 4.6-5.4 -> 1.1-1.3 s, what remains being the AES itself (1.6 s of user); peak RSS 2,048 / 2 / 17 MB. | `src/rar50/write/stream.rs`, `src/rar50.rs` |
| 2026-09-06 | STREAMING STORED WRITERS (`rar50::write::stream`, re-exported as `write_stored_archive_streamed`, `write_stored_volumes_streamed`, `StreamedStoredEntry`, `crc32_of_reader`): stored members copied from a `Read` source into `Write` sinks a megabyte at a time, headers built with the functions the in-memory writers use, nothing held - where every writer in the crate takes slices and returns a `Vec` per archive, so a 1 GiB member was two gigabytes of fresh pages before a byte reached the disk. The file header precedes its data and carries the member's CRC32, so the caller supplies it (`crc32_of_reader` is the pre-pass; the page cache makes the second read cheap where the fresh pages were not). Byte-identical to `Rar50Writer::stored_entries` and `Rar50VolumeWriter::stored_entries` over the same members - two tests hold it, including a member cut mid-volume and a partial last volume. First cut, deliberately narrow: encryption, the BLAKE2sp record, recovery records, solid sets, comments and quick-open are refused by name. review queue item 17's first step. `rar5cli --stream` drives it. Measured on an 8-vCPU KVM guest (EPYC, Ubuntu 24.04, idle), 1 GiB stored, three alternations against the in-memory writer and rar 7.23: single archive in-memory 3.7-6.4 s / streamed 1.05-1.16 s / rar 1.33-1.68 s; sixteen 64 MiB volumes 3.5-4.3 / 1.09-1.21 / 1.61-1.76 s; peak RSS 2,048 MB / 1 MB / 17 MB. On the M3 0.48 -> 0.38 s and 0.38 -> 0.22 s, RSS 2,050 -> 4 MB. The writer `rar` has, on the shape a Usenet post is. | `src/rar50/write/stream.rs`, `src/rar50/write/mod.rs`, `src/rar50/write/volume.rs`, `src/rar50.rs` |
| 2026-09-06 | Encrypted members are encrypted IN PLACE. A stored member's plaintext was copied into a padded buffer, encrypted there, and copied again into the archive; a compressed member's packed bytes were cloned before encryption. `EncryptedStoredPayload` now holds the key material and digests, and for a stored member no ciphertext at all: the fragment writers encrypt the caller's plaintext straight into the archive through a `CipherStream` (AES-CBC chained across every fragment of the member; a block that straddles a volume boundary is encrypted once and paid out across the two fragments), and a compressed member's packed buffer is encrypted where the encoder produced it (`encrypted_payload` takes it by value). `BlockData` carries a block's data area as bytes in hand or plaintext to encrypt, through `write_block_with` and `append_encrypted_header_block_with`. Output BYTE-IDENTICAL under seeded entropy on eleven shapes (stored and compressed, single and in volumes of odd sizes, header-encrypted, with a recovery record, with BLAKE2sp), every one accepted by rar 7.23. Peak RSS for a 1 GiB encrypted stored member 3,074 -> 2,050 MB. Measured where it shows, an 8-vCPU KVM guest (EPYC, Ubuntu 24.04, idle), 1 GiB encrypted stored member, three alternations: single archive 6.0-8.4 -> 4.9-5.5 s wall (system time 4.0-6.4 -> 3.1-3.7 s), sixteen 64 MiB volumes 6.5-6.8 -> 4.7-4.9 s, header-encrypted volumes of 100,000,001 bytes 6.0-6.4 -> 4.6-5.2 s, peak RSS 3,072 -> 2,048 MB on every shape; on the M3 the copies were within noise. review queue item 6 and its stored half (`research/rar5-priority-queue-2026-09-06`). | `src/rar50/write/mod.rs`, `src/rar50/write/volume.rs` |
| 2026-09-06 | A volume WITHOUT a recovery record is built in place: `VolumeSetWriter::start_volume` writes the volume head (`write_volume_head`: signature, head-crypt block, main header) into the buffer the member fragments then append to, and `finish_current_volume` closes it with the end header (`write_volume_end`) - where every volume was assembled as a body and copied whole into a second buffer behind its head (`write_volume_from_body`), a gigabyte copied and faulted in twice over a 16-volume set. The head's length depends on nothing the body decides, so the bytes are IDENTICAL (sha256 on 1 GiB stored and compressed sets); a recovery-record volume keeps the body-then-wrap path, whose head carries the record's offset. One volume body less in peak RSS (2115 -> 2051 MB on a 64 MiB-volume set). Measured where it shows, an 8-vCPU KVM guest (EPYC, Ubuntu 24.04, idle) where every fresh page is an EPT fault: 1 GiB stored in sixteen 64 MiB volumes 5.3-7.6 s -> 3.6-3.9 s wall, system time 4.8-7.1 -> 3.4-3.7 s, three alternations - the volume set now costs what the single archive costs there (3.5 s); on the M3 the copy was already noise. review queue item 1 (`research/rar5-priority-queue-2026-09-06`), measured here. | `src/rar50/write/volume.rs` |
| 2026-09-06 | A recovery-record archive is emitted ONCE. The record's offset sits in the main header's locator as a varint, so the record's position depends on the header's length, which depends on the offset; the writer resolved that by emitting the whole archive at offset 0, reading where the record landed and emitting again - up to four passes, each copying every payload and generating the Reed-Solomon parity over the whole archive (`resolve_writer_plan_offset`, `write_volume_from_body`). Everything after the main header has the same length whatever the offset, so `emit_resolved_writer_plan_recovery_direct` runs one MEASURING pass (offset 0, a zero-filled record of the right size in place of the parity: `recovery_record_data(.., placeholder)`), solves the offset on head lengths alone (`resolved_archive_head_len`) and emits once with the real parity; a volume's body is known before its head, so `write_volume_from_body` solves without measuring (`volume_head_len`). The result is the fixed point the loop converged to, so the bytes are IDENTICAL (sha256 checked on 32 MiB and 1 GiB, single and in volumes, stored and compressed); the loop stays as the fallback for a fixed point that fails to reproduce. Measured on a 1 GiB stored member (this Mac, load 5): `-rr3` 2.66 -> 1.17 s wall and 9.4 -> 3.2 CPU-seconds, in 16 volumes 3.54 -> 1.39 s and 8.2 -> 2.8, `-rr10` 3.49 -> 1.51 s and 26.8 -> 9.0; the compressed 32 MiB-dictionary archive with `-rr3` 5.91 -> 5.29 s (the encode dominates there). review queue item 2 (`research/rar5-priority-queue-2026-09-06`), measured here. | `src/rar50/write/mod.rs`, `src/rar50/write/volume.rs` |
| 2026-09-06 | FIX: the RAR5 inline recovery record was CORRUPT TO RARLAB for every archive over about 13 MB. rar 7.23 `t` said "Recovery record is corrupt" on our 32 MiB, 64, 128, 256, 512 MiB and 1 GiB archives and OK on 8 MiB; our own reader read all of them. Two fields in the `{RB}` record header were single-group-only: the u64 at 0x16 is the group's OFFSET within its shard (rar writes 0, 0x10000, 0x20000; we wrote 0) and the u64 trailing the 200 CRC states must be ONE value across every record of the archive (rar refused its own archive with two records disagreeing and accepted it with all set to zero or to an arbitrary constant, CRCs recomputed; we wrote a per-GROUP value). The u32 at 0x1e is the group's share of the last data shard's real bytes (unchecked by rar, now written as rar writes it). Diagnosed by dumping rar's records beside ours (`examples/dump_rr`) and mutating single fields of rar's archive with the record CRC64 recomputed. Proof: rar 7.23 tests the fixed 32 MiB / 1 GiB stored / 1 GiB compressed records OK, and `rar r` REPAIRED our archive with 300 KB of damage in two places to bytes identical to the input. Single-group archives are byte-identical to before. Found by the 6 Sep creation bench while timing recovery records; the `#[ignore]` interop test covers only the read direction (RARLab archives, our repair), which is why nothing caught the write direction. | `src/recovery/rar5.rs` |
| 2026-09-06 | SHARED ANCHOR LISTS for the long-match table (`MemberAnchors`, `LongSeed`): each block's tokenizer used to rescan its whole history for anchor positions - up to 32 MiB per 4 MiB block, eight passes over every byte of a member, about 9% of a large-dictionary encode - where a block's anchors are a property of its bytes alone. The member now holds one list of anchor offsets per 4 MiB block, computed by the first worker that needs it (`block()`, under a per-block mutex) and read by the others; `seed_long_from_anchors` inserts the lists covering the block's history newest-first with keep-first semantics, which is the order and result of the scan, so the table and the output are IDENTICAL (checked by sha256 on four shapes: mixed at 32 MiB / 16 MiB / 128 KiB and the 400-file set); solid history from an earlier member is still scanned - and scanned LAST (corrected later the same day): it is the OLDEST part of the window, and the first cut scanned it first, which handed an older occurrence the slot a newer one should have had whenever both anchored the same way - invisible on every shape the identity checks used, since none carried incoming history, and it would have broken the streamed compressed writer's window identity below. The pool's drain loop releases lists no in-flight block can need (`release_below`), so the lists in memory are the dictionary's blocks plus one (about 5 MB at 32 MiB), not the member's. Mixed 1 GiB at 32 MiB, paired on a loaded box: 117 / 143 -> 108 / 132 CPU-seconds (-7%); 16 MiB dictionary -3%; the 128 KiB default unchanged (its history is seeded forward, list-free). | `src/codec/rar50.rs` |
| 2026-09-06 | RING SLOT TAGS: a `MatchIndex` slot holds the position in its low `pos_bits` (enough for the indexed span) and, above them, up to eight bits of the position's four-byte hash that the bucket does not use (`tag_bits`, `MATCH_TAG_BITS_MAX`); `candidates_tagged` yields each candidate with whether its tag matches the probe word's, and `best_match_probe` rejects a mismatch before the one-byte filter's history load, spending the candidate exactly as a prefix mismatch did so the walk stops where it stopped. At a 32 MiB dictionary every bucket is full and each of a probe's 64 candidates cost a scattered load into 36 MiB of history - the mixed 1 GiB corpus went 202-204 -> 150-156 CPU-seconds (-25%), the 400-file set 76-90 -> 64-71, the 128 KiB default unchanged (sparse buckets), output BYTE-IDENTICAL on all three (the same sha256). The split is recomputed when a worker's scratch index is reused for a span of another length (`set_span`; block 0 has no history, block 8 has 32 MiB, and a stale split decoded every tag against the wrong bits - caught because the first build's output hash moved). Descends from the review's `rar5-fingerprint` pass, whose 24-bit variant was limited to 8-16 MiB spans; this one takes whatever bits the span leaves, so it reaches the 32 MiB case that needed it. | `src/codec/rar50.rs` |
| 2026-09-06 | (review, `research/rar5-crclast-2026-09-05`) compute a split member's whole-file CRC32 only for the LAST fragment: `bool::then_some(crc32(entry.data))` evaluated the checksum eagerly for every fragment, `then(\|\| crc32(entry.data))` defers it - four call sites (single stored entries, stored sets, compressed sets that store, genuinely compressed fragments); the CRC still covers the whole file in the same header, volumes byte-identical (202 volumes over eight sets, RARLab 7.23 `rar t` ok). A 1 GiB member in 16 volumes no longer hashes 15 GiB it throws away: measured here on that shape, three pairs, 1.04/0.94/0.99 s -> 0.44/0.42/0.43 s wall and 0.62 -> 0.08 s of user CPU, the sixteen volumes byte-identical between the two builds and accepted by `rar t`. | `src/rar50/write/mod.rs`, `src/rar50/write/volume.rs` |
| 2026-09-06 | ENTROPY BLOCKS: the RAR5 encoder writes a fresh Huffman table set per ~256 KiB of input (`ENTROPY_BLOCK_BYTES`), cutting each 4 MiB tokenizer block's token stream at token boundaries into RAR 5 compressed blocks of its own (`encode_token_blocks` / `encode_token_block`), the rep-distance state running on through them as the reader's does across every block of a file, the initial filters on the first, `is_last` on the last, and a floor of 1,024 tokens per table (`ENTROPY_BLOCK_MIN_TOKENS`: a stream of 4 KiB matches has 64 tokens per 256 KiB, and tables alone grew the repeated corpus 1.3% before the floor). (Pricing literals per 256 KiB segment was measured too: -0.4% of size for +13% of CPU, the sharper prices sending the parser after more matches; not kept - `LiteralPrices` stays one table per tokenizer block.) One table set per 4 MiB block was the larger half of the size gap to `rar` at equal dictionary: on the 1 GiB mixed corpus (thirds text, records, compressed bytes) at a 32 MiB dictionary 585.6 -> 531.1 MB, at 1 MiB 663.6 -> 593.4 MB (rar 590.1), at the 128 KiB default 722.8 -> 628.9 MB (rar 7.23 at 128k: 634.3). Wall unchanged (the tokenizer, its history seed and the pool task are untouched); the reader's table decodes cost nothing measurable at 256 KiB (0.39 -> 0.38-0.40 s for the 1 GiB extraction) and 25% at 64 KiB, which is why the cut is not finer. Output CHANGES for members over 256 KiB; every archive verified by RARLab `rar t`. | `src/codec/rar50.rs` |
| 2026-09-06 | LONG-MATCH TABLE: `MatchIndex::long`, one position per entry keyed by a hash of the 32 bytes at an ANCHOR position (a position whose four-byte hash has its low five bits clear - one in 32, content-defined so the same bytes anchor the same way wherever they recur), sized one entry per 32 bytes of span (2^10..2^20 entries, 4 MiB at a 32 MiB dictionary). Seeded newest-first over the WHOLE history in one pass (`seed_long_history` - the ring's seed stops when its buckets are full, this one must not, reach being its point), fed on every forward insertion, probed after the ring walk when the best so far is under 32 bytes; a repeated span is found at its first anchor and the rep-distance probe carries it on. The ring keeps the newest 64 positions per bucket, which at a 32 MiB dictionary is the newest ~4 MiB of any prefix's positions, so a repeat farther back was invisible: `rar -m1` matched our size at half our wall. Mixed 1 GiB at 32 MiB 531.1 -> 476.9 MB (rar 7.23: 468.9 - from +24.9% to +1.7%); the 400-file set 588.5 -> 507.9 MB (rar 505.4, +0.5%); the repeated corpus 1,087,107 bytes as before; wall 7.0 -> 7.2 s and CPU unchanged (186 s paired); 1 MiB and 128 KiB dictionaries unchanged by this table (the ring already covers them). Record: the lane's research handoff of 6 Sep 2026. | `src/codec/rar50.rs` |
| 2026-09-05 | RAR5 creation uses symbol-indexed canonical encoder codes, byte-chunk bit output, direct inverse match slots, four-byte candidate filtering with unchanged search limits, and avoids duplicate token/input buffers. Header-encrypted members and services append directly to the archive instead of copying their payload through a temporary Vec. Reference/golden tests preserve codes, bits, match choices and output; creation/extraction experiments and rejected variants are in `research/rar5-create-2026-09-05/README.md`. | `src/codec/rar50.rs`, `src/rar50/write/mod.rs` |
| 2026-09-05 | In parallel builds, buffered and streaming RAR5 decode reuse the distance slot's already-validated extra-bit count when constructing the distance, matching the tape worker instead of repeating the public conversion's checks. Tape workers return immediate length slots 0–7 before loading the length table or entering the bit reader. Exhaustive slot/bit-offset tests cover the short-length boundary; existing distance-LUT and differential archive tests cover validation. Local ARM64 experiments, rejected alternatives and counter-based confirmation are recorded in `research/rar5-fresh-2026-09-05/README.md`. | `src/codec/rar50.rs` |
| 2026-09-05 | The block pool's memory budget scales with the box - 128 MiB per pool thread, never under 1 GiB - and the per-block estimate is what a worker actually holds (72 MiB: ring index, literal price prefix, tokens, output and input copy, plus the dictionary when the history is copied rather than borrowed). The flat 1 GiB with the previous 212 MiB estimate held a 32 MiB-dictionary encode to FOUR workers on a 32-core desktop: the mixed 1 GiB corpus at `-md32m` took 26.4 s wall for 103 CPU-seconds where rar 7.23 took 14.5 s; with the budget it takes 7.6 s (189 CPU-seconds, on a loaded box), bytes identical. On an 8-thread box the 1 GiB floor holds. Found by the 5 Sep public-position bench. | `src/codec/rar50.rs` |
| 2026-09-05 | The RAR5 encoder PRICES ITS LITERALS (`LiteralPrices`): once per 4 MiB block, the order-0 Huffman code lengths over the block's bytes (four interleaved histograms, every fourth byte above 64 KiB), prefix-summed. A match is accepted only when its estimated bits are below the literals it replaces and its score is the bits it saves (`consider_match_candidate` takes the range's literal bits; the old `16 * length - cost` accepted every 4-byte match); the lazy parser charges the deferred byte's real price; acceleration is keyed to `saw_prefix_match` from `best_match_probe`, not to whether a match was taken. Output CHANGES and is never larger on the corpus: mixed 8 MiB archive -2.3%, 256 MiB -0.9%, 1 MiB text -10.9%, Rust source -0.4%; time +5..+18% on compressible input (more probes at the positions poor matches used to cover), ~1.5 ms per block for the table. The incompressibility samples shrank to 256 KiB (three of them) for the same verdict at a quarter of the cost. Three test fixtures with five-symbol alphabets or a 10-byte lookahead were re-cut; measurements and the constant-price probe that motivated the model are in the round's research record. | `src/codec/rar50.rs`, `src/rar50/write/filter_policy.rs` |
| 2026-09-05 | batch four adjacent match-index insertions from one u64 input load, preserving every position, hash, ring slot and insertion order. Normal-worker 32 MiB random / 64 MiB repetitive archive creation improves 8% / 12%, with 14–15% fewer whole-process instructions; mixed archives are roughly neutral in time. Parallel/serial tests, armv7 build, byte comparisons and RARLab checks pass. See `research/rar5-large-2026-09-05/README.md`. | `src/codec/rar50.rs` |
| 2026-09-05 | account for the bounded match index when estimating wide-dictionary block-wave memory. Two blocks with a 128 MiB dictionary fit the existing 1 GiB wave budget; smaller dictionaries keep their prior width. Mixed 256 MiB archive creation takes 42–50% less time, with essentially unchanged instructions and 24–38% higher measured process RSS. Full 1 GiB / 128 MiB dictionary archive validation, parallel/serial tests and armv7 build pass. See `research/rar5-gig-2026-09-05/README.md`. | `src/codec/rar50.rs` |
| 2026-09-05 | seed sufficiently large match histories newest-first, storing only retained entries and stopping when every bucket is full; select scalar or four-prefix scanning from a small diversity sample. Candidate order and subsequent insertions match the forward ring exactly; combined spans above u32 retain the original counter-wrap behavior. Random 128 MiB creation / 128 MiB dictionary uses 46% less wall time and 50% fewer process instructions; a 1 GiB mixed case uses 32.5% fewer instructions with unchanged wall time. Default writer dictionary is neutral. See `research/rar5-seed-2026-09-05/README.md` for all variants, limitations and validation. | `src/codec/rar50.rs` |
| 2026-09-05 | on our persistent worker pool, blocks with dictionaries >=32 MiB borrow a contiguous history/input span. Only an incoming solid-history seam allocates a combined buffer; the tokenizer borrows it instead of copying it again. Price the same 1 GiB worker budget from the remaining owned history, allowing four large-dictionary workers when incoming history is empty. The final guarded 1 GiB / 128 MiB dictionary check takes 43.46 -> 26.11 seconds (-39.9%) and 32.1% less peak process RSS; 256 MiB / 128 MiB dictionary improves 43.4% across two pairs. Default-dictionary 1 GiB wall time is neutral; smaller source/raw controls are 2-4% slower. Packed bytes and match policy are unchanged. Full selection and synchronization record: `research/rar5-span-2026-09-05/README.md`. | `src/codec/rar50.rs` |
| 2026-09-05 | The BLAKE2sp hash record is an OPTION, off by default: `WriterOptions::hash_record`, `HashRecord::Crc32Only` (what `rar` writes unless asked for `-htb`) or `HashRecord::Blake2sp` (what this writer wrote unconditionally until now). Every file header still carries its CRC32; with the record off the extra area is empty and `HFL_EXTRA` follows it, and an encrypted member's MAC covers the CRC32 alone (`EncryptedStoredPayload::blake2sp_mac` is `None`). Threaded through the plan (`ResolvedRar50WritePlan::hash_record`), both member resolvers, `PayloadDigests`, the fragment writers and the service writers. Reason: the digest was 85% of storing a member (one thread at 2.4 GB/s, and it cannot be split across threads - the eight leaf chains are already the SIMD lanes) and every reader pays the same hash over every extracted byte, for a guarantee nothing in the Usenet pipeline relies on. Measured on a 32-core Apple desktop: 256 MiB stored member 107 -> 14 ms (1.77 -> 0.27 CPU-seconds), encrypted stored 8 MiB -27%, compressed unchanged. Readers verify CRC32 when no record is present, as they always did for `rar`'s own archives. | `src/rar50/write/mod.rs`, `src/rar50/write/volume.rs` |
| 2026-09-05 | Large-member creation, round 4. (1) A member of 16 MiB or more is SAMPLED before the full encode: three 1 MiB windows (front, middle, back) raw-encoded on their own, and if none shrinks by more than half a percent the member is stored without encoding it (`filter_policy::sampled_incompressible`, on the single-volume, encrypted and both volume-set compressed paths; solid and filtered members are exempt). The writer used to compress the whole member and then compare lengths: a 256 MiB random archive went 0.35 s -> 0.12 s (store speed) and 10.4 -> 1.4 CPU-seconds. A compressible region all three samples miss is stored; that is the accepted trade. (2) The payload's CRC32 and BLAKE2sp (`PayloadDigests`) are computed beside the sampling and encode with `rayon::join` in the single-volume compressed resolver and carried to the emitter (`StoredCompressedWithDigests`, `Compressed { digests }`), instead of serially after the encode: mixed 256 MiB archive -15%, sampled-store 256 MiB -10%. Other paths still digest at emission. | `src/rar50/write/mod.rs`, `src/rar50/write/volume.rs`, `src/rar50/write/filter_policy.rs` |
| 2026-09-05 | The block scheduler is a POOL, not waves: `encode_blocks_pooled` runs `width` worker tasks on rayon that take the next block off a shared counter until the member is exhausted, so a slow block never idles the others behind a barrier; the calling thread drains finished blocks into the output IN ORDER as they complete (in-flight output stays near `width` blocks), reports progress on each completion or every 5 ms (condvar, not a sleep), and raises the cancel flag the tokenizers poll. Each worker keeps an `EncoderScratch` - the match index (reset by its bucket counts alone; slots past a count are never read) and the token vector - across its blocks instead of faulting 16 MiB of fresh pages per block. Mixed 256 MiB archive -12%, 128 MiB lz -17%, scratch a further -3% and -90 MiB peak; bytes identical. Replaces both the wave loop and the no-callback `par_iter` branch. | `src/codec/rar50.rs` |
| 2026-09-05 | Two encoder-walk savings. (1) Candidates come newest first, so a later one is farther and can only beat the best by being STRICTLY longer: one byte compare at `best.length - 1` skips it before the prefix compare, the length loop and the cost estimate (a byte of length is 16 score points; the most a farther distance saves on the length ladder is two bits). The nice-length stop is applied only after a candidate is actually considered, so behind a 64+ byte repeat match the walk now passes shorter candidates cheaply and stops at the first that could tie it, where it used to stop on the first candidate whatever it was (Rust source: a 160-byte match at distance 5,008 behind a 75-byte repeat; -27% time, 9 bytes smaller). (2) The lazy parser's probe at `pos + 1` is KEPT when it defers, since the next iteration would run exactly that search; the deferred position is inserted before the probe so the probe sees what the fresh search would have. Archives -5..-6%, a few bytes smaller. | `src/codec/rar50.rs` |
| 2026-09-05 | On the current u64 writer, wide writes use inline <=32-bit chunks and spills leave dead high accumulator bits alone. The wide usize shift passes through u64, fixing the imported armv7 build overflow. Parallel block waves without a callback use an indexed parallel map, preserving bounded concurrency and block-order errors; archive writers propagate an absent reporter instead of supplying no-op callbacks. Reporter/cancellation behavior and packed bytes are preserved. Current-main integration details, candidate races, reference tests and full armv7 build evidence are in `research/rar5-current-2026-09-05/README.md`. | `src/codec/rar50.rs`, `src/rar50/write/mod.rs` |
| 2026-09-05 | DELTA reconstruction uses independent per-channel accumulators and contiguous row stores for 2/3/4 channels, preserving the generic path for other counts and handling partial final rows. Single-channel DELTA encoding transforms the input in place through a separate helper, avoiding temporary allocation/copying without slowing other channels. Scalar differential tests cover all legal counts, row/vector boundaries, buffer reuse and invalid counts. Selection, rejected variants and full RAR5 encode/decode controls are recorded in `research/rar5-delta-2026-09-05/README.md`. | `src/codec/filters.rs` |
| 2026-09-05 | Shared creation-time Huffman construction uses flat parent links and bulk heap construction instead of per-leaf symbol vectors and repeated symbol-list merges. Node indexes preserve the original creation-order tie breaker and saturating frequency sums; a reverse traversal converts parent links to depths in place. Depth limits and packed output are unchanged. Differential reference tests and RAR5 creation measurements are recorded in `research/rar5-tree-2026-09-05/README.md`. | `src/codec/huffman.rs` |
| 2026-09-05 | The RAR5 encoder's match index is a FLAT RING TABLE under a four-byte multiplicative hash (`MatchIndex`): buckets sized to the indexed span (2^10..2^16), each a fixed ring of `depth` u32/usize slots where `depth` follows the candidate budget (4..64), insertion overwrites the oldest slot and a probe reads the newest `depth` back; a match of 64+ bytes ends the walk (`MATCH_NICE_LENGTH`). It replaces 4,096 growable vectors under a three-byte hash, in which most of the default 256 candidates a probe walked were other trigrams' positions, and with it the 5 Sep bucket-reserve row below (no per-bucket allocation remains) - the u32-position row still applies, the ring stores `P`. Output CHANGES: a budget of 256 now means the 64 newest same-hash positions, and the candidates are real. Measured on a 32-core Apple desktop against the rows above: mixed 8 MiB archive 212 -> 126 ms and 2.4% smaller, Rust source 460 -> 228 ms and 0.6% larger, 8 MiB repeated dictionary 37 -> 18 ms and 8x smaller (the repeats are found now), 1 MiB text 13% smaller, random -32%; depth 128 was 29% slower for 0.1-0.6% of size, depth 32 gave the size win back. Two tests pin the ring's order and depth, and the review's candidate-limit/repeat-choice reference now walks the same index. | `src/codec/rar50.rs` |
| 2026-09-05 | The blocks of one RAR5 member are encoded in PARALLEL. A member over 4 MiB is cut into 4 MiB blocks whose tokenizers depend on raw input only (block k's history is the `max_match_distance` bytes before it - input, or the incoming solid history, never a previous block's output), so under `parallel` a wave of blocks runs on rayon (`encode_lz_member_blocks`, `encode_block_wave`) and the results are concatenated in order; the wave is capped by cores and by a 1 GiB working-memory estimate (`ENCODE_WAVE_MEMORY_BUDGET`). The calling thread keeps the single-threaded progress callback, reporting the wave's tokenized bytes every 5 ms and raising a flag every block's tokenizer polls at its checkpoints, so cancellation lands within a checkpoint as on the serial walk. Output is byte-identical to the serial walk, pinned by `member_blocks_encode_identically_serially_and_in_a_wave`; progress order and cancellation by `member_block_waves_report_monotone_progress_and_honour_cancellation`. Measured on a 32-core Apple desktop: mixed 32 MiB archive 2.37 s -> 0.24 s with the two rows below. Without `parallel` the wave is one block and the walk is the old one. | `src/codec/rar50.rs` |
| 2026-09-05 | Literal-run ACCELERATION in the RAR5 tokenizer: after 32 consecutive literals with no match at all, the probe step grows to 2, 3, ... (LZ4 / zstd fast-level rule; `LITERAL_SKIP_STRENGTH` 5, `LITERAL_SKIP_MAX` 16). Every position is still inserted into the index, so a later match can point back into a skipped run; only the probe at the skipped position is saved. Output CHANGES for inputs with long literal runs (the mixed-stream golden grew 13 bytes, +0.04%; the 8 MiB corpora moved by at most +0.015% and the repeated-dictionary fixture shrank), and random input encodes 7-17x faster (1 MiB lz 47 -> 7.5 ms, 8 MiB archive 862 -> 51 ms). RARLab 7.23 `rar t` accepts the output. | `src/codec/rar50.rs` |
| 2026-09-05 | `BitWriter` accumulates in a u64 and spills 32 bits at a time instead of pushing a byte per loop iteration; `BitWriter::continuing(bytes, bit_pos)` takes a table stream's partial last byte back into the accumulator, and a write wider than 32 bits is split. Byte-identical output (the width/offset oracle test and every golden stream hold); `bit_writer_continues_a_partial_byte` covers the mid-byte start. | `src/codec/rar50.rs` |
| 2026-09-05 | RAR5 creation stores match positions as u32 when the complete current-input plus usable-history span fits, with a usize fallback for larger spans or length-sum overflow. The span is checked once before dispatch, and the common matcher is generic over the private position type. Existing search policy and output are preserved. Differential token/progress tests exercise both widths; arithmetic boundary tests cover u32 limits and overflow without allocating multi-GiB inputs. Variants, CPU/memory results and extraction controls are in `research/rar5-positions-2026-09-05/README.md`. | `src/codec/rar50.rs` |
| 2026-09-05 | RAR5 creation reserves 64 positions per match bucket when the current input plus usable history is over 16 KiB and at most 256 KiB. Tiny and large indexes retain on-demand growth. Search order, candidate limits and output remain unchanged. This trades up-front slot storage for fewer allocations on medium inputs; the later compact-position change reduces that reservation from 2 MiB to 1 MiB on 64-bit targets. Boundary comparisons, rejected history-copy/index variants and the original measurements are in `research/rar5-allocation-2026-09-05/README.md`. | `src/codec/rar50.rs` |
| 2026-09-05 | RAR5 and RAR4 creation compare complete 8-byte words before the byte tail when extending a match. Checked immutable slices support overlapping repeats; little-endian XOR and trailing-zero counting preserve the first mismatch on either host byte order. A byte-oracle test covers alignment, distance, mismatch and length boundaries. Hash-table, index-layout and word-loop experiments, compatibility checks and workload-specific performance results are in `research/rar5-hash-2026-09-05/README.md`. | `src/codec/fast.rs` |
| 2026-09-05 | RAR5 creation coalesces consecutive literals into two-word length/distance tokens and emits filter records separately. Frequency counting and bit emission read literal bytes from the current block input. Disabled match searching emits one literal run without a hash index or history copy, preserving progress/cancellation checkpoints. Golden output, token-layout and callback tests cover compatibility. Further candidates, memory/CPU comparisons and rejected search-index variants are in `research/rar5-next-2026-09-05/README.md`. | `src/codec/rar50.rs` |
| 2026-09-04 | `Rar50VolumeWriter::stored_entries`: a split STORED set holding SEVERAL members. `::stored_entry`, `::encrypted_stored_entry`, `rar15_40::write::write_stored_volumes` and `::write_compressed_volumes` each split ONE member across the volumes, and only `::compressed_entries` took a slice, so the commonest layout on the wire - several files in one split store set - could not be written at all and `postfast`'s container plane refused it by name (gap H0 of `research/POSTFAST-VS-NESTED-CORPUS-2026-09-03.md`, the nested corpus's own baseline leg r1). Adds the `Rar50VolumeEntries::StoredSet` variant and `write_stored_volume_set_impl`, which walks the slice through the same `VolumeSetWriter` the compressed set already uses: members pack end to end, only the one landing on a volume boundary carries the split flags, each member's whole-file CRC rides its own last fragment and the blake2sp hash record rides a member that was not split. An empty member and a set that fits one volume are refused rather than silently dropped. Adds `stored_volume_set_splits_several_members_and_reads_them_back` (lib test: byte-exact readback member for member, per-member CRC placement, corruption refused). **In the fork** the same day, with three more integration tests there including `reference_unrar_extracts_a_multi_file_stored_rar50_volume_set` - unrar 7.23 answered `All OK` on a four-volume three-member set and wrote all three members byte-exact. | `src/rar50/write/mod.rs`, `src/rar50/write/volume.rs`, `src/lib.rs` |
| 2026-09-04 | The volume writers' remaining single-entry gaps, all three of them refused by name in `postfast`'s container plane. (1) `rar15_40::write::write_stored_volume_set` and `::write_compressed_volume_set`: `write_stored_volumes` and `write_compressed_volumes` each split ONE member across the volumes, so a RAR4 set holding several files - the commonest layout on the wire - had no emitter at all. Members now walk one payload cursor through `write_volume_set`, so only the member landing on a volume boundary carries the split flags. (2) The same pair reaches header encryption, which the container plane refused on the reading that `write_header_encrypted_split_volumes` is entered only from the stored writer; `write_compressed_volumes_impl` has always entered it too and nothing tested that path, which is how the wrong reading survived. (3) `Rar50VolumeWriter::encrypted_stored_entries`, the plural that was missing beside `::encrypted_stored_entry` while `::encrypted_compressed_entries` always took a slice - the asymmetry that made an encrypted multi-member split set emitable compressed and not stored. A RAR4 multi-member set needed one thing its RAR5 twin did not: an `ENDARC` block with the next-volume flag on every volume but the last. A single member is split across EVERY volume, so `FHD_SPLIT_AFTER` alone chains the set and the writer wrote no end header; a multi-member set has volumes whose last file is complete, and **unrar 7.23 stopped at the first of those and wrote one member out of three**. The end headers are written after the whole set is packed, because only then is it known which body is last. Adds five lib tests (the RAR4 stored plural with per-member CRC placement, an ENDARC-flag assertion and a corruption arm; the header-encrypted compressed plural and its single-member control; the one-volume and empty-set refusals) and the RAR5 encrypted stored plural with a wrong-password arm. **In the fork** the same day (`rar4-volume-set`, f94c217), with two more integration tests there that hand the RAR4 sets to the local unrar 7.23 - `All OK`, every member byte-exact. Hand-ported rather than run through `sync-from-fork.sh`, for the reason the rows above give: vendor is ahead of the fork on sixteen files and the sync would rsync `--delete` an older engine over the newer one. The three writer files were byte-identical between the trees beforehand, so the port is exact. | `src/rar15_40.rs`, `src/rar15_40/write.rs`, `src/rar50/write/mod.rs`, `src/rar50/write/volume.rs`, `src/lib.rs` |
| 2026-09-04 | Optional caller-supplied entropy on the crypto writers, so an encrypted archive can be generated reproducibly. `Entropy` (`src/write_entropy.rs`, new) is a `Copy` choice on both `WriterOptions` - `Os` by default, so every existing caller keeps drawing from `getrandom::fill` and writes the same bytes it wrote before - and `Entropy::Seeded([u8; 32])` derives each draw from `BLAKE2s(key = seed, message = counter)` instead. The choice reaches the five draw sites (the RAR 5 header salt, the RAR 5 data salt and IV, the per-header IV, and the RAR 3.x 8-byte salt) through a per-thread scope the writer entry points install for one archive, rather than a parameter threaded through the whole plan-and-emit machinery, so a writer arm added later inherits the setting instead of silently drawing from the OS. Documented at the type as reproducible-test-generation only: a predictable salt weakens the encryption it feeds. Unblocks postfast's container plane, which refuses every encrypted shape by name today. Twelve tests (five on the mechanism, seven end to end); `unrar 7.23` opens a seeded archive of both generations with its password. **In the fork** since 4 Sep 2026 (perf 0477bdc, the exact patch, tests included) - HAND-PORTED here rather than synced, because vendor/ is still ahead of the fork on 16 other files and sync-from-fork.sh would rsync `--delete` an older engine over them. | `src/write_entropy.rs`, `src/lib.rs`, `src/rar15_40.rs`, `src/rar15_40/write.rs`, `src/rar50/write/mod.rs`, `src/rar50/extract.rs` |
| 2026-09-04 | `rar50::write::reference` (new module): RAR 5 archives in the byte layout the REFERENCE `rar` 7.23 writes, so `rarfast` can be a drop-in whose stored output a conformance table can compare by sha256. Additive and reached by nothing in nzbfast today: the fork's own writer keeps every layout choice it had (BLAKE2sp hash record, no locator, minimum-width vints), which is what the posting-layout catalog is pinned to. What the new module reproduces, all of it MEASURED against rar 7.23 on the dev Mac and none of it derivable from the format note: the main header's LOCATOR record and the fixed-width quick-open offset it reserves (the width is `vint_width(sum(data + 33 + 3 * name_len) << 12)`, fitted over twenty shapes and then confirmed by predicting two crossings exactly); a two-byte minimum on `PackSize`, `UnpSize` and `CompressionInfo` in a file header, and one byte for a directory's sizes; a whole-second mtime in the header's own field on CREATE and in an HTIME record on a REWRITE; CRC32 in the header by default with `-htb` moving it to a BLAKE2sp record; the skip-if-unknown flag on the main and end headers only; a `QO` quick-open service block caching the header of every member over 4096 bytes, with the three `-qo` spellings; a `CMT` comment block; and volume sets, where each volume stops 82 bytes short of its size before the tail and is zero-padded to it, every fragment carries a checksum with the whole member's on the last, and the locator estimate is over the whole members rather than the fragment. Verified byte-for-byte against the reference on stored, directory, blake2, comment, quick-open and 8k/12k volume shapes, and through the four in-place editing commands (`d`, `rn`, `k`, `ch`) whose conformance rows compare the rewritten archive's sha256. Four lib tests, including a whole-archive golden. **In the fork** since 4 Sep 2026 (branch `perf`, tip 1299bfe, the exact patch) - HAND-PORTED here rather than synced, for the reason the row above gives. | `src/rar50/write/reference.rs`, `src/rar50/write/mod.rs`, `src/rar50.rs` |
| 2026-09-04 | `rar50::write::reference::assemble_with_recovery` (and the `main_header_recovery` / `locator_reserve_width_rewrite` pair behind it): the assembly `rar rr[N]` writes when it adds a data recovery record to an archive that ALREADY EXISTS, which the reference will only verify if three things measured against rar 7.23 on the dev Mac are right - the main header's archive flags gain 0x0008, the locator record gains flag 0x02 and a SECOND fixed-width offset after the quick-open one (written even to an archive created with `-qo-` that carried no locator at all), and the reserve width is taken from the SIZE of the archive being rewritten rather than the create path's member estimate: `vint_width((len + 1) << 12)`, confirmed by prediction at both crossings (510 bytes reserves three and 511 four; 65,534 four and 65,535 five). `assemble` and the new entry point share `assemble_body`, so the locator reserve, the quick-open block and the end header still have one implementation. The record is built AFTER the main header is final, because it covers the archive as it stands at that line. Consumed by `rarfast`'s `rr` command in `crates/rars-cli`, which is NOT vendored; the rar conformance leg went 20 divergences to 15. **In the fork** the same day at `c3a20ab`, HAND-PORTED here with `git diff 1299bfe c3a20ab --src-prefix=a/vendor/rars/src/ --dst-prefix=b/vendor/rars/src/ --relative=crates/rars/src` and `git apply` - `sync-from-fork.sh` was NOT run and must not be until the drift reconciliation item is done, because vendor is still ahead of the fork on other files. | `src/rar50/write/reference.rs`, `src/rar50.rs` |
| 2026-09-04 | `rar50::write::rev` (new module): the `.rev` recovery-volume WRITER, the inverse of the rebuild path this crate already had. The format was read from our own reader, but the two things the reader cannot say were MEASURED: the code word pads the longest volume up to an EVEN length (an odd `-v12289b` set gets a 12,290-byte payload), and the number a `.rev` header carries is `data_count + row` rather than the number in its own file name. Proved rather than argued: building a set over the five WinRAR volumes already in `tests/fixtures/rar50/` reproduces WinRAR's own `multivol_rev.part1.rev` and `part2.rev` byte for byte, and the fork's conformance table compares the `.rev` rarfast writes against rar 7.23's sha256 and passes. It is STRIPED rather than a call to `encode_parity_shards`, which takes every shard as a slice - twenty 500 MB volumes with four recovery volumes would ask for 12 GB before writing a byte, and a volume set is exactly where that is the normal case: recovery rows go in batches so the open file count does not follow `-rv999`, and inside a batch one window of each data volume is folded in turn, so the working set is `(batch + 1) * window` and follows neither count. `recovery::rar5::fold_stripe_into_parity` is `encode_parity_shards` taken apart for it, and `data_volume_matches` is the bounded CRC check `rc` needs. Additive and reached by nothing in nzbfast today; it is what the fork's `rv[N]`, `-rv[N]` and a real `rc` are built on. One doc CORRECTION rides along, at the site: `Rev5RebuildSink`'s first argument indexes the MISSING volumes rather than the slots, and the two coincide whenever slot 0 is the first one gone, which is often enough to hide it in a test. Six lib tests. **In the fork** since 4 Sep 2026 (branch `perf`, tip 3e4f034, the exact patch) - HAND-PORTED here rather than synced, for the reason the rows above give. | `src/rar50/write/rev.rs`, `src/rar50/write/mod.rs`, `src/rar50.rs`, `src/recovery/rar5.rs` |
| 2026-09-04 | An opt-in TOLERANT TAIL on the RAR 5 parse, `ArchiveReadOptions::rar50_tolerant_tail`, default off. A member's header declares its packed length and the walk refused a header whose data area ran past the end of the file - attributing the refusal to that member's own header offset, so nothing before it survived either and a partially downloaded archive read as "not a RAR archive" with nothing listed. That check is what stops a hostile or corrupt length driving a read past the end of a mapping and the fuzz corpus exercises it, so it is NOT relaxed: with the option set the walk CLAMPS the last block's data range to the bytes the file holds, ends there, and marks the archive (`rar50::Archive::has_truncated_tail`, `BlockHeader::data_is_truncated` per block); a header the file no longer holds in full ends the walk the same way, and only a SHORT READ is forgiven - a bad CRC or a malformed field still refuses on either policy. Every range a tolerant parse hands out still lies inside the file, which is the property the strict check exists for, and a lib test walks 40-odd cut lengths asserting it. Nothing in nzbfast passes the option today, so the one-pass pipeline and the fuzz harness are untouched; `rarfast`'s unrar dialect does, which is what closed the `test-truncated` conformance waiver (128 rows, 0 waived, 0 divergences against unrar 7.23). **In the fork** since 4 Sep 2026 (branch `perf`, tip 40913f6, the exact patch) - HAND-PORTED here rather than synced, for the reason the two rows above give; `lib.rs` needed a hand edit because vendor's `ArchiveReadOptions` carries two split-hash fields the fork does not. | `src/rar50.rs`, `src/lib.rs`, `src/error.rs`, `src/rar50/extract.rs` |
| 2026-09-04 | An embedded ("Protect+") recovery record on the RAR 3.x/4.x WRITER, so `postfast`'s C10 plane is expressible on both archive generations instead of RAR 5 only. The READER half already shipped (`repair_protect_head`, `repair_protect_to_path`, the streaming `ProtectPlan` row above); nothing could WRITE one, because all three `rar15_40::write` validators built an `allowed` `FeatureSet` with `recovery_record` off and refused any other, and `crates/postfast/src/container.rs` refused a rar4 C10 profile by name against that. Adds `recovery::rar3::plan_newsub_recovery` / `::build_newsub_recovery_data` (the CRC tag table then the XOR parity sectors, tag = `!crc32(sector) & 0xffff`, parity slot = `index % parity_sectors`, trailing sector zero-padded), `WriterOptions::recovery_percent` with `with_recovery_percent`, and an `append_recovery_record` step on the stored and compressed single-archive paths that writes the `RR` NEWSUB block last and folds `MHD_PROTECT` into the main header at write time (the flag is INSIDE the protected run, so patching it afterwards would leave sector 0 reading as damaged). Refused by name rather than silently dropped: a percent without the feature flag or the flag without a percent (`validate_recovery_request`), a RAR 1.5 or 2.0 target (those want the old-style `PROTECT_HEAD` block), a header-encrypted archive (the record would be written in the clear and `newsub_recovery_data` declines an encrypted one), and a VOLUME set (`rar` writes one record per volume over that volume's own bytes; `validate_volume_writer_inputs` refuses, because the volume writers share the single-archive validators and would otherwise pass and write nothing). The percentage rule is `plan_inline_recovery`'s, floored, so one `recovery_record_pct` means the same on both generations; `rar 7.23` rounds instead, measured on a 391-sector archive at 4/8/12/19/39/78/117/195 against this rule's 3/7/11/19/39/78/117/195. Ten tests, and the one that matters is `rar4_recovery_record_we_wrote_is_repaired_by_winrar` in `tests/winrar_recovery.rs`: RARLab's own `rar` finds the record, names the damaged sector from our tags and rebuilds the file byte-exact. **NOT synced**: hand-made here, for the reason the two rows above give - vendor/ is ahead of the upstream rars fork on many files and `sync-from-fork.sh` would rsync `--delete` an older engine over them. | `src/recovery/rar3.rs`, `src/rar15_40.rs`, `src/rar15_40/write.rs`, `src/lib.rs`, `tests/integration/winrar_recovery.rs` |
| 2026-09-04 | `WriterOptions::volume_numbering` on the RAR 1.5-4 writers: which NAMING the caller will give the volumes, which is what decides `MHD_NEWNUMBERING` in every volume's main header. The writer returns one byte vector per volume and the caller names them, so the convention cannot be inferred there, and left undeclared the four volume entry points always wrote the classic numbering. A set named `set.part01.rar` with that bit clear is one no reference reader can follow: `unrar` 7.23 swaps the EXTENSION of the first volume's name, looks for `set.part01.r00`, prints `Cannot find volume` and then reports a checksum error on the member it had to truncate, exiting 6. `VolumeNumbering::Classic` is the default, so every existing caller writes the bytes it wrote before; `NewStyle` sets the bit on EVERY volume, which is what `rar` itself does - the RAR 3.00 fixtures `multivol_oldnaming_*` and `multivol_newnaming_*` already in `tests/fixtures/` are the same archive written twice two seconds apart and differ by this bit, the header CRCs over it, and nothing else. Measured against `unrar` 7.23: the bit is REQUIRED under `.partNN.rar` naming, and merely tolerated under `set.rar`/`.r00` and under `set.001`/`.002`. A lib test drives all four writers under both settings and asserts the two sets differ at exactly three bytes per volume, so the default stays byte-identical; the fork's reference test extracts a `.partNN` set with `unrar` and then pins the negative, the same bytes without the flag under the same names. **In the fork** since 4 Sep 2026 (branch `rar4-new-numbering`, tip `4695872`, the exact patch) - HAND-PORTED here with `git diff perf 4695872 --src-prefix=a/vendor/rars/src/ --dst-prefix=b/vendor/rars/src/ --relative=crates/rars/src` and `git apply`, for the reason the rows above give: `sync-from-fork.sh` would rsync `--delete` an older engine over the files vendor is still ahead on. | `src/rar15_40.rs`, `src/rar15_40/write.rs`, `src/lib.rs` |
| 2026-09-03 | Tiny RAR 5 member-pool work ranges (at most eight members and 32 KiB decoded in total) remain grouped on the worker result channel. The coordinator keeps one in-order cursor and stores out-of-order results by batch start, reducing channel receives and reorder-tree operations from once per member to once per batch on the measured 8 x 4 KiB shape. Larger work ranges publish allocation-free singleton packets as each member finishes, avoiding head-of-line delay behind the range's remaining decodes. Member results remain in archive order, decoded bytes retain the existing global in-flight byte charge, and checked size addition makes overflow select the singleton path. | `src/rar50/extract.rs` |
| 2026-09-03 | Parallel RAR 5 block tapes return their payload, literal, packed-op, and out-of-band filter vector capacities to the scanner through a bounded, non-blocking recycle channel. The existing flat 8-byte `TapeOp` and its length/distance slot lookup fast paths are unchanged; recycling now includes the separate filter allocation that makes that representation possible. Workers also borrow rather than clone each cached nested decode-table `Arc`. Early output, errors, a full channel, and flat-tape continuation never wait for a returned buffer, and a hostile bundle above 4 MiB is dropped, bounding retained recycle capacity to 32 MiB at the eight-worker ceiling. Earlier rotating common-shape measurements of the recycler reduced retired instructions about 4%; this rebase deliberately does not add the separately tested outer lazy-table `Arc` ownership transfer. | `src/codec/rar50.rs` |
| 2026-09-03 | RAR 5 literal bursts consume a primary-LUT control symbol and hand its already-decoded id directly to buffered, streaming, or packed-tape dispatch. Previously each short-code match/control was looked up by the burst, rejected as non-literal, then immediately peeked and looked up a second time by `HuffmanTable::decode`; the packed tape still routes the handed-off symbol through `decode_tape_op`, preserving its 8-byte op shape and length/distance slot-table fast paths. Zero LUT entries remain untouched for canonical fallback. A boundary regression test pins that the handoff consumes exactly one code. | `src/codec/rar50.rs` |
| 2026-09-03 | The primary RAR 5/7 Huffman LUT uses 9 index bits (2 KiB per table) instead of 12 (16 KiB); wider codes take the already-outlined canonical fallback. A position-balanced M1 sweep of 9/10/11/12-bit release builds, with duplicate 12-bit controls and CRC-gated output over common RAR5 shapes, measured the 9-bit build about 2.3% faster on the 400-member compressed shape and 0.3-0.7% faster on solid, RAR7-dictionary, and large mixed archives. The boundary/fallback test derives its long-code width from this constant. | `src/codec/rar50.rs` |
| 2026-09-03 | The flat RAR 5 match copier spells out its at-most-four 16-byte strides instead of retaining a loop backedge and loop-carried offsets. Common-shape profiling found 88.8-91.7% of matches are at most 16 bytes and more than 99.75% are at most 64 bytes with distance at least 16, so this is the dominant flat-output path; overlapping matches retain `copy_within` semantics. Exhaustive tests cover every length through 64 at distances 16, 17, 31, 64 and 127. In position-balanced M1 product races the isolated change reduced solid, RAR7-dictionary and large mixed extraction by 4.5-5.9%, and its combination with the MATCH dispatch row below reduced them by 7.0-9.6%; the 400-member shape was neutral. | `src/codec/rar50.rs` |
| 2026-09-03 | The flat RAR 5 short-match path performs its first 16-byte stride unconditionally. Decoded match lengths are at least two, so the old zero-length branch ran on every real fast-path match but was never taken. The internal zero-length edge remains sound: the over-copy is outside the unchanged logical end, later output overwrites it before emission, and `fast_end` guarantees the stride's slack; the exhaustive short-match test includes length zero. In nine position-balanced, load-guarded local ARM64 ThinLTO pairs, retired instructions fell in all nine pairs on solid (median 0.296%), RAR7-dictionary (0.310%), and large mixed (0.242%) archives; the 400-member shape was neutral (+0.010%). Two exact copies of the baseline binary differed by -0.004% at the median. Wall and cycle results were noisy and are not claimed. | `src/codec/rar50.rs` |
| 2026-09-03 | Parallel RAR 5 tape apply fast-dispatches a fully resolved `MATCH` before the remaining op-kind switch in both the streaming-ring and flat-buffer executors. Common-shape profiling counted `MATCH` as 86.10% of 217.7 million tape ops; on arm64 ThinLTO this changes each instantiated hot dispatch from a five-instruction decision tree (`cmp`/range branch/zero branch/`cmp`/equality branch) to one `cmp`/`b.ne` pair with `MATCH` as fall-through. The output-limit check still precedes dispatch, and rep-state mutation, copy validation, filter order, and deferred errors retain their original sequence. | `src/codec/rar50.rs` |
| 2026-09-03 | Short-code Huffman LUT entries mark symbols below 256 in otherwise-unused bit 31. The literal burst can therefore classify its overwhelmingly common literal entry with one arm64 bit-test branch instead of subtracting one, shifting, and branching; zero remains the shared decoder's miss sentinel, and masked symbol extraction keeps the marker invisible to every table. ThinLTO codegen removes two instructions from the packed-tape worker's per-symbol loop and shrinks that worker by 12 bytes. In three position-balanced local arm64 hardware-counter pairs, retired instructions fell 2.12% on the 400-member compressed shape, 0.99% on solid, 0.40% on the RAR7-dictionary shape, and 1.64% on the large mixed shape. Cycle results were load-sensitive, but were consistently lower for the small and large shapes (paired medians -1.76% and -4.97%); correctness was CRC-gated. | `src/codec/rar50.rs` |
| 2026-09-03 | RAR 5 bit-operand cache hits are forced inline while refill and error handling remain in one outlined helper. The previous 580-byte `read_bits` stayed a real call in ThinLTO ARM64 builds because it included the byte-tail refill ladder; every match length/distance then paid the call plus the indirect `Result<u32>` ABI even when the 64-bit cache was already full. The split removes that leaf from the common path and shrinks the benchmark's text by about 9 KiB. Nine position-balanced local common-shape rounds reduced retired instructions 4.5% on the many-member compressed shape and 7.2-8.7% on solid, RAR7-dictionary, and large mixed archives; CRC-gated outputs and all focused RAR 5 tests agree. The over-wide regression check also pins that a full cache cannot bypass the 32-bit limit or consume input. | `src/codec/rar50.rs` |
| 2026-09-03 | BLAKE2sp leaves off aarch64 run `blake2s_simd`'s OWN blake2sp (`src/rar50/blake2sp/simd.rs`, `Hasher` aliases it) instead of `portable.rs`'s eight `blake2s_simd::State`s on a four-thread team. The old path spawned four scoped threads per buffered chunk and each GATHERED its two leaves' 64-byte blocks out of the 512-byte groups into a freshly zeroed scratch buffer before a ONE-instance compression - a megabyte of allocation, a megabyte of copying and four spawns per megabyte of input, and no many-way SIMD anywhere. The crate's blake2sp reads the leaf blocks in place at a stride of eight and compresses them eight-wide under AVX2 or four-wide under SSE4.1, runtime-detected (`Implementation::detect`), with its own portable compression as the fallback; same tree, so the digests are equal by construction, and no new `unsafe` appears (`unsafe_is_confined_to_the_neon_entry` still passes). aarch64 is UNTOUCHED and must stay so: `blake2s_simd` has no many-way kernel there (`guts::MAX_DEGREE` is 1) and its blake2sp measured 0.82 GB/s on an M3 Ultra against the NEON kernel's 2.89 - "just use the crate everywhere" would be a 3.5x regression on Apple silicon. Measured on an i5-10600KF, best of 5 over 6 process runs, in the 256 KiB pieces the extract pipelines feed: 1.485 -> 2.905 GB/s (1.96x) on ONE thread instead of four; one-shot 2.475 -> 2.925 (1.18x). `portable.rs` is now compiled only under `cfg(test)` off aarch64 and is kept as the INDEPENDENT implementation the agreement tests cross-check both shipping kernels against - do not delete it to buy the lines. Audit round 25. | `src/rar50/blake2sp/` |
| 2026-09-03 | `Rar50SplitFragmentDigests` on `ArchiveReadOptions` (default `Always` = today's behaviour for every caller; `DeferForStoredMembers` opted into by `nzbkit::mem::rar_read_options`). A STORED split member's packed bytes ARE its unpacked bytes, so the per-fragment packed digest records and the whole-member record digest every byte TWICE - once on the producer thread, in series with the read. The format needs only the member's own digest for a verdict; the fragment records are unrar's damage LOCALIZER (UIERROR_CHECKSUMPACKED), and a member whose final digest proves cannot have had a bad fragment. Under the opt-in the whole-set walks skip them for a stored member and, if the member's digest then fails, `PendingSplitRefs::localize_fragment_damage` re-reads the packed bytes and reports the same volume-naming `SplitFragmentCrc32Mismatch` / `SplitFragmentHashMismatch`. It never applies to a compressed member (whose two records cover different bytes), never to a member carrying no whole-member digest of its own, and never on a walk that releases volumes behind the read. **What it gives up, stated plainly: a stored set whose fragment digest RECORD is damaged while its payload is sound now extracts** - correctly, the member's own digest being what says so - where the eager check refused it. `rar50_deferred_fragment_digests_accept_a_lying_record_over_sound_data` pins that, and `rar50_deferred_fragment_digests_still_name_a_damaged_volume` pins that broken DATA is still named against its volume on all three split shapes. Measured on an M3 Ultra over 1 GiB, paired against an exact-base control, 3 reps each: stored `-htb` 13.962 -> 7.730 G instructions (-44.6%), 1.55 -> 0.80 s user CPU, 0.52 -> 0.45 s wall; stored CRC32 1.410 -> 1.152 G (-18.3%); the compressed `-htb` control 54.857 -> 54.812 G (-0.08%, gated out by `is_stored`). Audit round 25. | `src/lib.rs`, `src/rar50/extract.rs` |
| 2026-09-03 | Parallel RAR 5 tape workers inline the primary-LUT hit only at the distance-code decode used by every fully resolved match. This removes the call and indirect `Result` return from that dominant site while keeping the other Huffman call sites outlined; forcing the whole decoder inline retired fewer instructions but regressed cycles and was rejected. Nine position-balanced M3 rounds were neutral on the 400-member compressed shape and reduced retired instructions by 3.0-3.8% and cycles by 4.9-6.4% on solid, RAR7-dictionary, and large mixed archives; an exact-binary A/A control differed by 0.03% instructions. The helper's LUT-hit and fallback semantics are byte-for-byte the existing `decode` path. | `src/codec/rar50.rs` |
| 2026-09-03 | A distance table's short-code LUT entries cache their already-validated extra-bit count in unused bits 24..=28 and mark an invalid slot in bit 30. The parallel resolved-match path therefore replaces two slot-bound comparisons plus the `DISTANCE_SLOT_BITS` address/load with one invalid-bit branch and one bitfield extract; the shared decoder masks the metadata from its symbol result, while long codes and short input tails still take the existing checked lookup. Tests pin generic decoding, valid metadata, short invalid-slot consumption, and long-code fallback errors. Against the exact combined predecessor, nine position-balanced local arm64 hardware-counter pairs were neutral on the 400-member shape (+0.01% retired instructions), and reduced instructions in all nine pairs on solid (median -1.24%), RAR7-dictionary (-1.29%), and large mixed (-1.07%); the exact-binary RAR7 A/A median was +0.014%. All measured extractions remained CRC-gated. | `src/codec/rar50.rs` |
| 2026-09-03 | Parallel-feature buffered decode and the below-MT streaming serial fallback reuse the distance-specific Huffman LUT hit and its cached extra-bit count. They previously called the generic outlined decoder and then repeated the distance-slot table lookup and validation; non-parallel builds retain that path unchanged. A deterministic `mt_workers_cap(1)`/`flat_limit = 0` test compares streaming output and final rep/last-length state with buffered decode for dense-match algorithm-v0 and algorithm-v1 members. In a seven-pair exact-product M1 race (five CRC-gated extractions per sample), the 400-member compressed shape reduced retired instructions 2.35%, cycles 7.84%, user time 7.95%, and wall time 7.65%, winning all seven pairs; the already-parallel solid, RAR7-dictionary, and large mixed controls remained instruction-neutral (within 0.03%). An exact-binary small-shape A/A differed by 0.005% instructions and 0.009% cycles. Forced inlining grows the arm64 ThinLTO `decode_block_serial` instance by 988 bytes; the measured win is confined to buffered/split extraction, cap-one execution, and streaming work below the MT threshold. | `src/codec/rar50.rs` |
| 2026-09-03 | Streaming and flat RAR 5 DELTA filters swap their two reusable scratch vectors after channel reconstruction instead of copying the entire decoded filter range back into the source vector. Buffered decode retains its slice-based copy because its destination is the member output itself; E8/E8E9/ARM behavior is unchanged. A 20-round position-balanced ARM64 race on a 128 MiB real-RAR5, four-channel DELTA-heavy member reduced median paired wall time 6.1% (19/20 wins), retired instructions 1.53% (20/20), and cycles 4.44%; an identical-binary control was within 1.4% on all three. A realistic 131 MiB x86 application member with only 1.7 MiB of DELTA ranges was neutral, as were the filter-free common compressed shapes, so this is deliberately a zero-cost specialization rather than a claimed general extraction win. Ring and flat DELTA tests pin both swap consumers. | `src/codec/rar50.rs` |
| 2026-09-03 | The reusable RAR 5 DELTA scratch keeps its initialized prefix, growing only when necessary and otherwise truncating to the next filter range, because channel reconstruction overwrites every output byte. This removes a redundant zero-fill after the preceding buffer-swap change. An additional 20-round position-balanced ARM64 race on the 128 MiB four-channel DELTA-heavy member reduced retired instructions 0.34% (19/20 wins); paired wall time was 1.57% lower but noisy (11/20 wins). A realistic x86 application member with little DELTA-filtered output remained neutral. The dirty-buffer regression now covers scratch growth, exact reuse, and truncation. | `src/codec/filters.rs` |
| 2026-09-03 | RAR 5 DELTA reconstruction validates one contiguous encoded slice per channel and writes through a strided destination iterator, replacing per-byte checked source lookup and loop-carried source/destination indexes without unsafe code. On a serial 128 MiB four-channel real-RAR5 DELTA-heavy extraction, a 30-round position-balanced ARM64 race reduced median paired wall time 3.39% and cycles 3.20% (29/30 wins each); retired instructions were flat (+0.007%), as was the identical-binary control. Two parallel runs agreed on wall-time direction (-1.60% and -2.03%) while their cycle results were load-sensitive. Dirty-buffer tests compare against an independent scalar reference across odd lengths and all valid channel counts. | `src/codec/filters.rs` |
| 2026-09-03 | The public archive-family facade borrows the already typed archive for common single-volume RAR 5 extraction and progress instead of cloning its parsed block/name table into a one-element `Vec`; multi-volume and mixed-family validation is unchanged. On a clean claimed M1, 40 position-alternated pairs of four timed extractions of a real 10,000 x 4 KiB STORE archive reduced paired extraction time 3.844%, retired instructions 3.056%, cycles 2.490% (40/40 extraction wins), and peak footprint 36.281%. The 10,000 x 4 KiB compressed control was -1.772% extraction, -0.217% instructions, and +0.711% cycles. An exact-binary 30-pair STORE A/A was -0.154% extraction, -0.034% instructions, and -0.188% cycles. | `src/lib.rs` |
| 2026-09-03 | Plaintext RAR 5 STORE members of at most 8 KiB consume their payload from an existing memory/read-ahead view, avoiding the boxed reader, zeroed scratch, and second copy while retaining identical write-before-digest ordering, integrity/size checks, and short-range errors; encrypted, streaming, empty, and larger members retain the old paths. On a clean M1, exact facade-only to view-without-cursor comparisons over 40 alternated pairs of eight extractions reduced 10,000 x 4 KiB STORE extraction 24.888%, instructions 28.285%, and cycles 20.041% (40/40 wins); 2,048 x 8 KiB fell 24.008%, 27.729%, and 19.725% (40/40). The 10,000 x 4 KiB compressed control was instruction-neutral (-0.017%), and bypass controls were flat at 16 KiB (-0.046% extraction, +0.090% instructions, -0.027% cycles) and 32 KiB (+0.096%, +0.133%, +0.149%). The 10,000-member run retains exactly one archive payload descriptor per extraction session beyond process stdio; views add no descriptors or per-member allocation growth. | `src/source.rs`, `src/rar50.rs`, `src/rar50/extract.rs` |
| 2026-09-03 | The flat RAR 5 apply path appends a SHORT literal run with constant-length copies - a one-byte store, then a power-of-two ladder to fifteen bytes - instead of `copy_from_slice`, whose runtime length is a `memcpy` libcall whatever that length is; longer runs keep the libcall, which is the right shape for the incompressible runs that carry most of the literal bytes. A run ends at the next match and matches are 86% of tape ops, so runs are overwhelmingly tiny: a counting build over 1 GiB of the census -m3 shapes measured 67.8-69.4% of literal runs at ONE byte and 97.1-97.7% at three bytes or fewer, and a `sample` of the merged decoder over a 4 GiB fixture put 9.6% of the apply thread - the pipeline's wall - at the return address of that single call site. On an idle M1 Ultra over five rotating rounds with every output tree digested: bigv 0.850 -> 0.830 s (-2.35%, 4/5 paired wins), solid 0.990 -> 0.940 (-5.05%, 5/5), r7dict 0.870 -> 0.840 (-3.45%, 5/5); the buffered-decoder control (400 members) and the stored, encrypted-stored and tiny shapes are instruction-neutral within 1%. A new test walks every ladder length 0..=20 at three starting offsets. | `src/codec/rar50.rs` |
| 2026-09-03 | The flat RAR 5 op walk keeps the rep shift register and the last match length in LOCALS for the length of a tape, written back to the decoder on every exit including the error paths, instead of reading and writing them through `&mut self` on every match: behind the reference they were two loads and three stores of the decoder struct per match AND a loop-carried dependency through store-to-load forwarding on the wall thread. The REP arm goes with them - `reps[..=index].rotate_right(1)` takes the array's ADDRESS, which pinned the whole register back in memory (four stores and three reloads per match survived the first cut of this change), so the rotation is spelled out per index, exact because only symbols 258..=261 reach it and the index is therefore 0..=3 by construction. The flat/buffered differential suite compares the carried rep state and last length after every shape. Idle M1 Ultra, five rotating rounds: bigv -3.53% wall against the pre-literal-ladder base (5/5 paired wins), solid -5.05% (5/5), r7dict -4.60% (5/5). | `src/codec/rar50.rs` |
| 2026-09-03 | `FlatOutput` maintains `limit_pos`, the physical write position at which the group's declared output is complete, beside the `fast_end` bound it is recomputed with on every slide. The apply walk asks "is this member finished" once per TAPE OP and was answering it from `written()`, which is `origin + pos - base`: three loads, an add and a subtract, seven instructions every op; it is now one load and one compare. The identity holds because both callers build the plan with the same `output_size` they pass the walk, so `output_end == base + output_size`; a debug assertion pins that and the differential suite drives a full member plus three smaller output prefixes through it. Idle M1 Ultra, five rotating rounds: retired instructions -1.75% on bigv, -2.14% on solid, -2.17% on r7dict against the arm above, wall unchanged at the 10 ms clock on those legs. | `src/codec/rar50.rs` |
| 2026-09-03 | The flat RAR 5 op walk moves the output buffer OUT of `FlatOutput` for the length of a tape and keeps the write cursor, the two folded bounds and the emit watermark in locals. Behind `&mut FlatOutput` every op reloaded the buffer pointer, the buffer length, the position, `fast_end`, `fast_distance_max` and the emit watermark and stored the position back, because the walk contains calls taking the same reference; on arm64 that was eight loads and a store per match. Every fallback - a declined stride copy, a literal run past the ladder, a filter, a due emit - puts the buffer back before it calls anything that reads it and takes it again afterwards, on the error path too, so the walk always hands the buffer back. The stride copy and the literal ladder move into `flat_stride_copy` and `flat_literal_ladder`, which take the buffer and the bounds as arguments: that is what lets them stay in registers, and it keeps ONE copy of each rule. `FlatOutput::copy_match` becomes the unhoisted spelling of the same two steps and is now the oracle the exhaustive short-match and slow-path unit tests drive; the walk is covered end to end by the flat/buffered differential suite, which compares output, error values, rep state and last length across shapes, three output prefixes, filters and (with the test build's 4 KiB slack) constant window slides. Idle M1 Ultra, five rotating rounds, digests identical: bigv 0.850 -> 0.800 s (-5.88% against the round's base, 5/5 paired wins), solid 0.990 -> 0.910 (-8.08%, 5/5), r7dict 0.870 -> 0.800 (-8.05%, 5/5), retired instructions -3.3 to -4.1%. | `src/codec/rar50.rs` |
| 2026-09-03 | A flat RAR 5 literal run of at most sixteen bytes that has sixteen bytes of tape BEHIND it is copied as one fixed 16-byte word whatever its true length, which makes the dominant one-to-three-byte run branchless; only the last run of a block falls short of that and takes the ladder. Nothing over-READS - the tape-length term is what guarantees the word is inside the block's own literal vector - and the over-WRITE past the run is the stride copier's argument: `fast_end` guarantees the slack, the scribbled bytes sit at or beyond the new position, every later write lands exactly there and overwrites them before the position moves past, matches only read below it, and the emit gate never releases bytes at or above it. Idle M1 Ultra, five rotating rounds, digests identical: bigv 0.850 -> 0.780 s (-8.24% against the round's base, 5/5 paired wins, -1.15% over the arm above), solid 0.990 -> 0.890 (-10.10%, 5/5), r7dict 0.870 -> 0.790 (-9.20%, 5/5), retired instructions -3.6 to -4.6%. Measured MIXED on a loaded box and reversed on an idle one - see research/RAR-PERF-AUDIT-2026-09-02.md round 28's method note. | `src/codec/rar50.rs` |
| 2026-09-03 | The 22 rows above dated 2026-09-02/03 are the RAR5 optimisation branch, merged onto main by the accretive 3 Sep lane after an interleaved race on M1, Zen 4 and M3 Ultra found it accretive to the 2 Sep rounds (research/RAR-PERF-AUDIT-2026-09-02.md round 24). An independent read-only review of the whole diff found no memory-safety, panic or unbounded-resource defect and confirmed every cap the rows claim. Two behavioural deltas it noted, recorded rather than changed: (1) the per-session range-reader cache is keyed on the volume PATH and never revalidated, so a volume replaced on disk DURING one extraction session is served from the old inode for the rest of that session (the cache lives for one `DecoderSession` or pool worker only; nzbfast never rewrites a volume mid-extraction, and every watermark clears it); (2) a truncated plaintext STORE member of at most 8 KiB now fails with `UnexpectedEof` BEFORE writing its readable prefix, where the boxed-reader path wrote the prefix and then failed - same error, no partial file on disk. | `src/source.rs`, `src/rar50/extract.rs` |
| 2026-09-03 | A live-byte counter for the decode path's own working memory (`src/memtrack.rs`, new), and one charge from it: each compressed member's decode charges the sliding flat plan it is admitted to (`flat_plan_bytes`, the same input the codec allocates from - or the retained dictionary when the plan is over the caller's flat cap) plus the member pipe's four 1 MiB pool buffers, released by RAII when the member ends. Counters only: nothing in this crate reads the value, and the product's memory-floor attribution (`nzbkit::memgauge`, tier `rars_work`) pulls it at snapshot time, because this crate cannot depend on that one. It exists because 41 to 58% of a compressed-RAR chase's peak was unattributed and this crate's working set was one of the candidates (research/RAR-PERF-AUDIT-2026-09-02.md rounds 14 and 35); measured, the whole rars term is 105 MB of a 2,246 MB peak, so the candidate was NOT the answer and now says so in a log line rather than in an argument. The tape workers' buffers are deliberately NOT charged: they are allocated in `src/codec/rar50.rs`, which another lane held for the whole of round 35, and round 3 measured them at 149 to 152 MB for a whole 8-worker member decode - the largest single thing still inside `unattributed` on a compressed shape, and the next re-sync's row to write. | `src/memtrack.rs`, `src/lib.rs`, `src/rar50/extract.rs` |
| 2026-09-02 | RAR 5 extraction retains one positional file handle per decoder session/member-pool worker and reuses it for consecutive member ranges, instead of reopening and seeking the same volume for every member. The cache holds only one volume at a time (the pool is capped at eight workers), so descriptor use remains O(workers), never O(volumes); every range reader tracks an explicit logical cursor. It is explicitly cleared before every progress watermark that promises a volume can be deleted, so an open handle cannot pin reclaimed disk space. Memory and growing-stream sources are unchanged, and a short file range still fails with `UnexpectedEof`. Two 20-pair runs over a real RAR 5 CRC-only STORE archive with 10,000 distinct 4 KiB members measured 0.1155 -> 0.015 s and 0.114 -> 0.017 s medians (6.7-7.7x); an exact-copy A/A control differed by 1 ms (0.1205 vs 0.1195 s), and a one-member 1 GiB STORE control was flat within noise (0.102 vs 0.103 s). | `src/source.rs`, `src/rar50.rs`, `src/rar50/extract.rs` |
| 2026-09-02 | RAR 5/7 Huffman tables retain only `u16` symbol ids in canonical-code order instead of repeating code, length, and a pointer-sized id in every entry (16 -> 2 bytes per populated symbol on 64-bit). Existing bucket metadata supplies decode indexes and reconstructs `(code, length)` for the writer, so encoder output and malformed-code errors stay unchanged. Construction places symbols directly into their canonical buckets instead of allocating sortable records. New tests pin scattered multi-length codes, encoder lookup and missing-symbol errors, long-code fallback, the largest RAR5/RAR7 symbol, and refusal rather than truncation outside `u16`. A directional exact-base M1 race moved the 400-member compressed shape about 2% and left the large-member shape inside timing noise; embedding the 2 KiB primary LUT in the table was separately measured neutral and is not retained. | `src/codec/rar50.rs` |
| 2026-09-02 | The non-solid RAR 5 member pool claims adjacent tiny members in adaptive batches of at most eight, amortising its shared work-channel and byte-budget locks. It preserves at least eight batches per worker and limits each batch to one worker's share of the 64 MiB in-flight budget; a single oversized member still advances alone, and checked addition fails closed on hostile size totals. Tiny compressed-member runs improved about 35-40%, while 64-member and 1 MiB-member controls selected batches of one and remained flat. | `src/rar50/extract.rs` |
| 2026-09-02 | Tiny file-backed payload ranges (at most 8 KiB) share a 192 KiB forward read-ahead window in each bounded extraction-worker cache. Live readers own their window through `Arc`; an idle cache reuses the allocation without re-zeroing it. A failed speculative read is retried over the required prefix, optional-tail failures are not exposed as member failures or cached as EOF, and truncation still reports `UnexpectedEof`. A 51-run rotating exact-base M1 race measured about 17% less wall time on 10,000 distinct 4 KiB STORE members and 9% on an independent 2,048-member 8 KiB shape. The initially tested 16 KiB boundary was 1.6% slower on that host despite winning locally, so it is deliberately bypassed; 16/32 KiB, one-member 1 GiB, memory, and growing-stream paths remain on the direct reader. Retained memory is 192 KiB per active session/worker, at most about 1.5 MiB for the capped pool. | `src/source.rs` |
| 2026-08-27 | `ArchiveSource::range_reader`'s File arm returns the guarded `OwnedRangeReader::File` shape instead of `file.take(len)`, and `copy_range_to`'s File arm routes through `range_reader` - `take` answers `Ok(0)` with bytes still owed on a truncated volume, which the chained extract walks read as fragment EOF. Same defect the 26 Aug row above closed on the owned readers; these are the pre-opened volume walks (`ArchiveSource::File`, how the daemon extracts) that row could not see. `a_range_that_ends_early_fails_rather_than_reading_as_eof` now also drives the File `range_reader` and `copy_range_to`. Found by the 27 Aug 2026 full-tree bug sweep, finding 12. | `src/source.rs` |
| 2026-08-27 | Seven `(end - position) as usize` narrowings (and the `source_len - offset` / `len - position` twins) in the recovery window loops now clamp in u64 FIRST (`(end - position).min(IO_BUF as u64) as usize`, or `.min(buf.len() as u64)` where the bound is a buffer). On `armv7-unknown-linux-musleabihf` a remaining span that is a multiple of 4 GiB cast to `usize` is 0, `read_at` of an empty buffer returns Ok, `position` never moves, and the RR repair or fallback scan of an 8-20 GiB volume wedges forever; identical codegen on 64-bit. Same class as nzbkit's `chunk_len` sweep (`adb3681e9`); `tools/chunk-narrow-gate.py` does not cover `vendor/`. Found by the 27 Aug 2026 full-tree bug sweep, finding 13. | `src/recovery/stream.rs`, `src/rar50.rs` |
| 2026-08-26 | `OwnedRangeReader`'s File and Stream arms, and `BlockingRangeReader`, return `UnexpectedEof` (`short_range()`) when the inner read answers 0 with bytes still owed, instead of `Ok(0)`. `Ok(0)` on a range that ended short is indistinguishable from a clean end, and both sequential extract walks (`rar50/extract.rs`, `rar15_40/extract.rs`) read it as "this fragment is finished" and move to the NEXT fragment - so a volume whose payload was cut short (a chase whose yEnc size fell short of the header's range, a file truncated after parse) still "extracted", with later members decoded from the wrong bytes and a member carrying no CRC of its own written truncated in silence. The exact-range path (`stream_read_exact`) has always failed closed on the same input; this is the sequential path agreeing with it. Adds `a_range_that_ends_early_fails_rather_than_reading_as_eof` (all three readers). Found by the 26 Aug 2026 full-tree bug sweep, finding 5. | `src/source.rs` |
| 2026-08-26 | `repair_prefix_streaming_impl` tests `damaged.is_empty()` BEFORE `states.is_empty() || slots.is_empty()`, and a group that is damaged with no usable record is now `TooManyDamagedShards` rather than a silent `continue`. Skipping it left no trace, so an archive whose every damaged group was skipped returned `Ok(vec![])` - the value the API contract reserves for "the record says the prefix is already intact". The caller cannot tell the two apart: `crates/nzbfast/src/rarfix.rs` renamed its destination over the volume on that answer and reported the volume repaired, over a byte-for-byte copy of the broken one with the member CRCs still wrong. The empty-rows arm below already answered `TooManyDamagedShards` for the same situation whenever slots existed but were unusable, so this is that answer reaching the case it could not see. Adds `a_damaged_group_with_no_usable_record_is_refused_not_reported_intact`. Found by the 26 Aug 2026 full-tree bug sweep, finding 6. | `src/recovery/stream.rs` |
| 2026-08-25 | `growable_buffer_read_blocks_until_bytes_arrive` waits for the reader to BLOCK (spinning on `blocked_waits()` under a 30s loud timeout) instead of sleeping 20 ms and hoping. The sleep was standing in for "the reader has reached the point where it must wait", and on a Windows runner executing six test shards at once, thread start-up plus scheduling outran it: the writer appended first, the reader found all six bytes on its first call, `blocked_waits` was 0 and the assertion failed. Intermittent windows-unit red; took main red on 25 Aug 2026 at baf2be1c (run 32803615314, shard 4/6, both nextest tries). Test only - no production code moves, and the data assertion was never affected. | `src/source.rs` |
| 2026-08-23 | TODO 17c item 2, follow-up: `next_x86_opcode` and its `_impl`/`_scalar` helpers have ONE definition again. The row above converted BOTH byte-identical copies to `memchr` and left the duplication in place, so the same function was maintained twice with a near-duplicate lib test in each file. `src/codec/fast.rs` keeps `match_length` (and the `fast`-gated portable-simd imports it needs) and now carries `pub(crate) use crate::fast::next_x86_opcode;`, so the five `super::fast::next_x86_opcode` call sites in `codec/filters.rs` and `codec/rar50.rs` and the `crate::fast::` one in `src/x86_filter_scan.rs` resolve unchanged. The two test modules were merged into `src/fast.rs`, keeping the lane-boundary fixture, the E8-vs-E8E9 separation and the fallback arm for a mask no caller uses; net -83 lines, 763 rars tests green. Pure refactor: a temporary differential harness ran both copies over a 1 MiB pseudorandom corpus across 6 masks x 7 start offsets x 8 end bounds and both digested to `0xfd50b90bcddaa56e` before AND after the collapse. | `src/fast.rs`, `src/codec/fast.rs` |
| 2026-08-22 | Sweep 8 M10: `clone_prefill` no longer unlinks the caller-claimed `dest` and then `std::fs::copy`s to the same NAME. The caller takes that path with `create_new` precisely so no symlink can be followed through, and the unlink threw the claim away - a link installed in the window received the whole volume, outside the job, and the `symlink_metadata` guard ran after the damage. The copy now lands in a staging directory this call creates exclusively (`create_dir` refuses an existing entry of any kind) and is published with `rename`, which replaces a link at the target rather than following it; the reflink/clonefile win survives because the copy destination still does not exist. Adds `stage_beside` and `open_repair_dest` (`O_NOFOLLOW` on unix) - the three repair-to-path entry points open `dest` right after the prefill, and a prefill that DECLINED leaves whatever is there alone. Three new lib tests. Cargo.toml gains a unix-only `libc` dep, constants only (`unsafe_code = "forbid"` still holds). | `src/recovery/stream.rs`, `src/rar50.rs`, `src/rar15_40.rs`, `Cargo.toml` |
| 2026-08-22 | NEON BLAKE2sp leaves (TODO 11 later list): `src/rar50/blake2sp.rs` became `blake2sp/{mod,portable,many}.rs`. `LeafSet` trait over the eight leaf states; `portable.rs` is the old `blake2s_simd` + 4-thread team (every non-aarch64 target), `many.rs` is a four-lane NEON kernel (aarch64 default) that compresses four leaves per pass. `Hasher` is now an alias for `TreeHasher<DefaultLeaves>`, so the extract.rs call sites did not change. **Cargo.toml `unsafe_code` went `forbid` -> `deny`**: `many.rs` carries the crate's ONE unsafe block, the call that enters the `#[target_feature(enable = "neon")]` kernel (stable Rust offers no safe way in, and the argument is in the module comment); `unsafe_is_confined_to_the_neon_entry` pins it to that single file. Measured on an M5 Max, 1 GiB: 0.87 CPU-s vs 1.88 for the thread team. Tests hold both kernels to `blake2s_simd`'s blake2sp on every length 0..1536 and on random inputs/chunkings. **In the fork** since 22 Aug 2026 (perf branch 1b84503, which ported every unported row above it as well - the fork had sat at c586ae2 of 24 Jul); the fork crate owns its own `[lints]` table to carry the `deny`. | `src/rar50/blake2sp/`, `Cargo.toml` |
| 2026-08-22 | TODO 17c item 1: `pipe_stored_chunks` keeps every pooled buffer at its full `STORED_PIPE_BUF` length for life and carries the fill level beside it, instead of truncating to the read count, clearing on the way back to the pool and growing again with `resize(STORED_PIPE_BUF, 0)`. That memset ran on EVERY round trip - 1 MiB per chunk, 1 GiB per GiB extracted - over bytes `read` was about to overwrite. The data channel now carries `(Vec<u8>, usize)`. Measured on a 1 GiB real-`rar` `-m0` member, 42 paired runs: 30 pairs faster, median paired CPU ratio 0.973-0.978. | `src/rar50/extract.rs` |
| 2026-08-22 | TODO 17c item 2: `next_x86_opcode_scalar` (BOTH copies) searches with `memchr`/`memchr2` instead of a masked byte loop. `cmp_mask` is only ever `0xff` (E8) or `0xfe` (E8E9), and masking with `0xfe` accepts exactly `0xe8`/`0xe9`, so the two are a one- and a two-byte search; any other mask keeps the byte loop, and a new lib test in each file pins that arm (both copies were collapsed to one the next day - see the 2026-08-23 row, and re-apply that row after this one). Scan primitive 2481 -> 7237 MiB/s (E8) and 2282 -> 4009 MiB/s (E8E9) at ~1.5% opcode density; end to end on a 129 MiB x86 payload with 1199 filter blocks, 14 paired runs: median paired wall ratio 0.961. `Cargo.toml` gains `memchr = "2"`, already in the workspace lockfile via aho-corasick. | `src/fast.rs`, `src/codec/fast.rs`, `Cargo.toml` |
| 2026-08-22 | TODO 17c item 1 rider: a lib test drives `pipe_stored_chunks` through a ragged read schedule (1 MiB, 7, 300 KB, 1, ~1 MB) and asserts the consumer sees exactly the content and the CRC matches, so no part of a previous round can reach either past the fill level. Test only; the row above is the change it guards. | `src/rar50/extract.rs` |
| 2026-08-22 | TODO 17b: the PPMd glue pass's step-1 threading is its own `Suballocator::thread_free_blocks`, held to a literal node-by-node front-insertion reference by two free-list fixtures. The O(n) build itself is older (29 Jul 2026, `78feeafbb`, replacing the reference's quadratic prepend); nothing pinned that it produced the SAME list, and the pre-existing glue tests cannot tell the orders apart because they only assert which blocks merged. The order sets which cell the next allocation takes, so it decides model-restart timing and decoder compatibility. Reversing the bucket walk fails both fixtures. | `src/codec/ppmd.rs` |
| 2026-08-22 | Split STORED members get the whole-file CRC on their last fragment. Both stored branches of the RAR 5 volume writer (`write_stored_volumes_impl`, and the stored fallback inside `write_compressed_volume_set_impl` that an incompressible entry takes) passed `None` for `data_crc32`, so such a set carried no checksum anywhere and `extract_volumes_to` returned Ok on corrupt bytes - every nzbfast fixture built by `Rar50VolumeWriter` over an incompressible payload was unverifiable. The compressed branch already did this. Adds `stored_split_volumes_carry_a_final_crc_and_refuse_corruption` (lib test, both writers, corruption refused with `checksum mismatch`). **In the fork** since 23 Aug 2026 (perf f17ff11, the exact patch; a lib test there too). | `src/rar50/write/volume.rs`, `src/lib.rs` |
| 2026-08-22 | TODO 17e: `reconstruct_data_volumes` has no per-byte fallback any more - `erasure_correction_matrix`'s error propagates. The fallback existed because a full-length 255-symbol codeword scanned alpha^0 twice and could not derive coefficients; that root-scan fix is older (29 Jul 2026, `7fc3ecdfd`, the `.max(1)` on the scan range) and **this row depends on it** - re-applying the deletion without it would turn a decodable set into a `DecodeFailed`. The per-byte Forney loop moved into the test module, where it stays as the differential oracle. `no_full_length_erasure_set_needs_a_per_byte_fallback` sweeps 200+55, 250+5 and 128+127 at one erasure and at the whole parity budget. | `src/recovery/rar3.rs` |
| 2026-08-20 | `delta_decode_into`: the RAR 5 filter path reuses one delta buffer per member instead of allocating and zeroing a fresh one per filter block. Adds the `delta_scratch` field on both RAR 5 output structs and a fourth argument to `apply_filter_to_range`. `delta_decode` (the allocating shape) stays for RAR 2.9 and the tests. | `src/codec/filters.rs`, `src/codec/rar50.rs` |
| 2026-08-20 | Streaming legacy (RAR 2/3) recovery repair, C7: `repair_protect_to_file`/`_to_path` on the rar15_40 `Archive` scan protected sectors by bounded range reads and patch a cloned/copied destination in place, instead of `repair_protect_head`'s two whole-volume buffers (2.02 GiB -> 11 MiB peak RSS on a 1 GiB volume). Adds `Error::LegacyRepairTooLarge` (budget refusal, incl. the compressed-NEWSUB decode pre-check), reroutes the `ArchiveReader::repair_recovery_to_file`/`_to_path` Rar15To40 arms, and adds 7 `streaming_protect_*` lib tests plus the `legacy_rar3_streaming_repair_matches_winrar_byte_for_byte` leg in `tests/winrar_recovery.rs`. `repair_protect_head` (buffered) stays as the tests' oracle. | `src/rar15_40.rs`, `src/lib.rs`, `src/error.rs`, `tests/winrar_recovery.rs` |
| 2026-08-20 | C7 rider (sweep 8 M9): `repair_protect_sectors` pass 1 bails at `parity_sectors + 1` mismatches (same `InvalidHeader(exceeds_parity)` value as the post-scan check) instead of collecting every damaged sector index first - the unbounded `Vec<usize>` grew source_len/64 bytes on a widely-corrupt volume before any check ran, breaking the documented peak-memory contract. The `working > budget` check also moved above the `used_slots` allocation it already accounts for. | `src/rar15_40.rs` |
| 2026-09-16 | Probe-only, and OFF in every shipped build: a non-default `fold-team-probe` feature lets `RARS_STRIPED_FOLD_MIN_WINDOW` move the striped repair fold's team gate, read once into a cached atomic. The gate is what stops the team starting, so a bench cannot otherwise ask whether the team would win at a window the gate refuses, and cannot get a one-variable arm pair at all (moving the memory budget moves the window and the cap with it). Reached from `nzbfast-unpack`'s pass-through feature of the same name by its two `#[ignore]`d `revscan_*` drivers. No shipped default changed. Findings: `research/NZBFAST-REPAIR-FOLD-GATE-SMALL-VOLUME-2026-09-16.md`. | `src/recovery/rar5.rs`, `Cargo.toml` |
| 2026-09-16 | The RAR 5 parser now decodes the END-of-archive record's own flags vint into `BlockHeader::end_flags`, and `Archive::next_volume_follows()` reads bit 0x0001 ("another volume of this set follows") back out; `rars::Archive` delegates it and answers `None` for every other family. The flag was WRITTEN on every non-final volume since 7 Sep 2026 and read by nobody: it is the only continuity a volume carries when its boundary falls BETWEEN whole members, which is what an archiver emits whenever the next file header will not fit. Without it nzbfast's obfuscated-RAR partition - which has no usable names to group by - cut such a set in two at that boundary and failed the job with every byte present. A malformed END body leaves the field `None` rather than refusing the archive, so `Some(false)` ("I am the last") is distinguishable from "does not say". New `end_of_archive_records_whether_another_volume_follows` round-trips the writer against the parser over a multi-member stored volume set. Found by the 16 Sep 2026 bug sweep, item 51. | `src/rar50.rs`, `src/lib.rs`, `src/rar50/extract.rs` |
| 2026-10-28 | The striped repair's overlapped arm (`overlapped_stripe_loop`) can no longer park forever on a dead fold worker. The main thread kept a clone of the reply channel's sender alive for the whole scope, so a worker that died mid-fold left `back_rx.recv()` with nothing to end it: no CPU, no error, and `thread::scope` never reached its join. The worker now owns the only sender and catches its own panic, so its death reads as `BadRecoveryChunk` on the main thread rather than a hang, or an unwind out of the scope. Found chasing nzbfast issue 115 (an unrepairable job stuck in Repairing for 14 hours inside the RR pass). Adds `rar5_striped_repair_fails_rather_than_hangs_when_its_fold_worker_dies` (a cfg(test) hook kills the worker at one window length; red against the clone with a 60 s timeout). **Not yet in the fork.** | `src/recovery/rar5.rs` |
## Deep gate before a release (decoder changes especially)
The `--lib` gate above and the fuzzers use small inputs, so they are blind to
size-dependent decode bugs: back-references reaching past the streaming window
(>64 MiB), solid cross-member history, and volume-split members only appear in
large real archives. The fork carries a real-`rar` round-trip rig for exactly
these axes at `crates/rars/tests/real_archive_diff.rs`. It is NOT vendored (a
`tests/*.rs` file) and needs a local `rar` 7.x, so run it in the fork checkout
before cutting a release or dropping the external `unrar` fallback:
```sh
cargo test -p rars --test real_archive_diff -- --ignored --nocapture
```
It builds ~76 MiB archives across RAR5/RAR7 dictionary sizes (128 MiB..1 GiB),
solid multi-file sets, and multivolume splits, and asserts rars decodes each
byte-for-byte. This is what would have caught the 64 MiB streaming-window cap.
The whole export ships in the public repo (`vendor` is in
`packaging/PUBLIC_MANIFEST`; `/vendor/` is leak-scan-exempt as third-party
source), which satisfies build-from-source.