mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
benbrownandClaude Opus 4.8
899d55f4b6
An adversarial audit found the enforcement story ahead of the code: gates that were documented as hard gates were advisory, manual, or absent, and several human-facing claims outran the machine record. This makes the claims true. High findings - Differential fuzz was documentation, not enforcement: verdict hardcoded fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and it is inherently baseline-vs-candidate, so it runs on the accept path and reports Skipped in the standalone `just gates` flow. New hard rule: a machine `accepted` verdict is impossible unless the fuzz gate actually passed — otherwise it caps at needs-human-review. - The agent's system prompt claimed FP/concurrency/unsafe patches were "routed to human review automatically". No such mechanism existed. Added a conservative lexical risk classifier (crates/verdict/src/risk.rs) that forces needs-human-review on any would-be accept carrying those signals; fp-tolerance equivalence always signals. - The local verdict path ran agent-influenced code unsandboxed. It now runs under scripts/no-net.sh (network namespace) and fails closed. Upstream test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when present (scripts/pin-testsuite.sh generates them). Also fixed, found while fixing the above - The pipeline script's `set -e` was inert (brace group as the left operand of `&&`), so a failing candidate build did not abort the run and verdict could bench a stale binary. Now a scoped subshell whose status is branched on at top level. - A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path) stranded RUNNING.lock and left read_verdict polling forever. Failure accounting now lives outside the wrapper. - harnessd's patch allowlist ignored git rename/copy headers and --no-prefix diffs; the rename-escape case is now a committed regression test. - The workspace did not compile on Windows (ungated Unix APIs). cfg-gated; 71 portable tests now run on both platforms. - pin-check regenerated the corpus manifest instead of verifying it; it is now verify-only, with a deliberate `just pin-corpus` to re-pin. - Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics (compare --perf-stat), never part of the accept decision. Docs now match the code, including what is NOT yet enforced: no target ships a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not wired to the fp-tolerance gate — each is stated as mechanized-but-inert rather than done. Corrected the stale "33 ledger rows" (34), the playbook's class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation claim (five accepted rows carry sanitizers_clean=false), the unsupported "40,000 requests" figure, and the blanket "historical rows record fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
74 lines
3.8 KiB
Markdown
74 lines
3.8 KiB
Markdown
# Case study 1 — tokei: compounding algorithmic wins (Rust, class 5)
|
||
|
||
**Target:** [tokei](https://github.com/XAMPPRocky/tokei) v14.0.0 — a
|
||
source-code line counter. **Workload:** 221 MB / 4360 files (8 replicas
|
||
of the Pro Git book + comrak's own tree), single rayon thread, CPU
|
||
pinned. **Result:** three independent, CI-verified class-5 wins on the
|
||
same hot function — each proposed *by the unattended agent* and
|
||
100%-audited. Multiplied together they are **~1.26× over the Phase-0
|
||
baseline** (1.107 × 1.037 × 1.099), but that product is an
|
||
approximation, not a measured figure: each speedup carries its own
|
||
bootstrap CI against its own baseline, and confidence intervals do not
|
||
compose through multiplication. The trustworthy claims are the three
|
||
individual increments below.
|
||
|
||
## The hot path
|
||
|
||
tokei spends the overwhelming majority of its instructions in one
|
||
function — `LanguageType::parse_lines` and the per-byte loop inside
|
||
`SyntaxCounter::perform_multi_line_analysis`. The cache-sim profile put
|
||
it at **55.9% of instructions, 62% of all branches, 44% of branch
|
||
mispredicts** on the pinned corpus: for every byte of every line, the
|
||
counter tested a stack of quote/comment token matchers.
|
||
|
||
## The three wins
|
||
|
||
| run | mechanism | speedup (median) | 95% CI |
|
||
|---|---|---|---|
|
||
| `phase0-tokei-002` (class 3) | read path: `fs::read` pre-sized, BOM-only decode | 1.099 | [1.080, 1.127] |
|
||
| `phase2-tokei-001` (class 5) | 256-entry first-byte gate table for the token matchers | 1.107 | [1.084, 1.131] |
|
||
| `phase2-tokei-003` (class 5) | forward-only whitespace break (drop the `rposition` rescan) | 1.037 | [1.027, 1.052] |
|
||
| `phase2-tokei-008` (class 5) | streaming leftmost `important_syntax` scan (one vectorized pass, not per-line restart) | 1.099 | [1.073, 1.112] |
|
||
|
||
Each is a **redundant-work elimination**: the first-byte gate replaces
|
||
five per-byte matcher loops with one table load; the whitespace-break
|
||
change drops a full backward scan of the file tail per byte; the
|
||
streaming scan replaces a per-line aho-corasick restart (which kept
|
||
memmem's SIMD prefilter in its short-haystack slow path) with a single
|
||
leftmost search advanced by integer comparisons.
|
||
|
||
## Why the numbers are trustworthy
|
||
|
||
- **Byte-identical golden replay** on the pinned corpus for every win —
|
||
a code counter that miscounts one line fails the gate.
|
||
- **10,000-input differential fuzz** (`scripts/diff-fuzz-tokei.py`),
|
||
pristine vs candidate, **0 divergences** — after canonicalizing
|
||
tokei's benign parallel-walker output ordering (a real finding:
|
||
raw-byte differential comparison over-counts when the target is
|
||
legitimately nondeterministic in presentation).
|
||
- **ASan/LSan clean** on the patched tree over the full corpus.
|
||
- **Interleaved A/B** vs a pristine-rebuilt baseline, bootstrap 95% CI
|
||
lower bound ≥ the 2% bar. `phase2-tokei-008` was the first accept
|
||
through the *machine-enforced* sanitizer gate (see case study 3).
|
||
|
||
## What it demonstrates
|
||
|
||
The richest wins on a mature Rust binary were not exotic — they were
|
||
"this loop does work it doesn't need to, on every byte." The agent found
|
||
each by reading the cache-sim profile, checking the ledger to avoid
|
||
re-attempting a class on the same hotspot, and proposing the cheapest
|
||
untried mechanism. Three separate hypotheses, three separate verified
|
||
increments, all on the same 40%-of-runtime function — and one honest
|
||
rejection in between (`phase2-tokei-009`, a real +5.7% median whose
|
||
noisy CI lower bound of +0.2% could not clear the bar, so it does not
|
||
count).
|
||
|
||
---
|
||
|
||
> **Note (2026-07-13).** The 10,000-input differential fuzz cited above
|
||
> ran **out-of-band** (`scripts/diff-fuzz-tokei.py`, per-attempt), not
|
||
> inside the verdict pipeline — so these ledger rows record
|
||
> `fuzz_iters=0` in their machine record. The fuzz gate is now
|
||
> pipeline-integrated; rows written after this date record the
|
||
> iteration count actually executed.
|