Claude
03113312ad
phase3-cjson-002 (class 5+3): stack-buffer number parsing + strtod round-trip replacing sscanf. +8.85% median, 95% CI [+7.5%, +10.0%] on the C target, banked. 100% audit: byte-identical golden replay, independent ASan+UBSan clean, 10k-input differential fuzz 0 divergences, patch bounds/free-gating verified by eye. - verdict: an infrastructure sanitizer-BUILD failure (missing runtime, broken toolchain) now caps the accept at needs-human-review and logs it, instead of crashing the verdict and losing a measured bench. - results/phase3/: four public case studies (tokei class-5 wins; build flags + allocators; the comrak caught false-accept; the cJSON cross-language proof) + README. Compound-speedup claim stated as an approximation, not a measured figure (CIs don't multiply). - .gitignore: cjson build-artifact dirs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
3.1 KiB
Case study 3 — the caught false-accept (comrak, class 3)
This is the most important case study in the project, because it is the one where the machine got it wrong and the process caught it.
What happened
phase2-comrak-010: the unattended agent, reading comrak's cache-sim
profile, saw that AST-arena teardown at process exit —
drop_in_place::<Vec<Node<RefCell<Ast>>>> plus its share of the glibc
free path — cost real instructions and cache misses in a
run-to-completion CLI. Its hypothesis: comrak already ends in
process::exit, so the OS reclaims memory in bulk anyway; wrap the arena
in std::mem::ManuallyDrop to skip the per-node drop walk, and add a
std::hint::black_box(&arena) before the scope closes so the leak stays
reachable and a leak checker won't flag it.
The pipeline ran it:
- Upstream tests: PASS
- Golden replay: byte-identical
- Interleaved A/B: +10.7% median, 95% CI [+7.9%, +12.1%]
- Verdict:
accepted
A clean, large, CI-significant win. And it was wrong.
The audit overturned it
Phase 2 runs a 100% human audit of every accepted win (SPEC §10).
The audit ran the sanitizers the pipeline had not: LeakSanitizer flagged
a 384-byte arena chunk as unreachable at exit. The agent's
black_box(&arena) keeps the arena live only until its scope closes —
but process::exit runs after that scope, so at the moment of exit the
arena is genuinely unreachable. A real leak. The agent's own hypothesis
had even named the risk ("if the sanitizer still flags the leak,
rejected-gates is the honest outcome") — the pipeline simply had no
sanitizer gate to trip, because in Phase 2 ASan/LSan were a per-attempt
manual step.
Called per SPEC §10: a pipeline false accept, caught by audit before anything shipped. The win was never banked. Shipped false accepts across the entire project remain zero.
The fix made the machine enforce the tier
The corrective action was not "be more careful." It was structural:
verdict now builds the patched tree with ASan+LSan (ASan+UBSan for
C/C++) and runs the pinned workload on every would-be accept, capping
any sanitizer-flagged win at needs-human-review. The tier rule SPEC §8
always specified is now enforced by the machine, not the auditor. The
very next Rust accept (phase2-tokei-008) and the first C accept
(phase3-cjson-002) both passed through this automated gate.
The 10k-input differential fuzz on the overturned patch found 0 divergences — the output equivalence was genuinely real. Only the teardown-leak tier was wrong. That is the subtle failure mode the layered gates exist for: a change can be behaviorally equivalent on output and still unacceptable on a dimension the bench and the golden replay can't see.
What it demonstrates
The product is not "an agent that finds speedups." The product is
trust — and trust is demonstrated most by the case where the
optimistic path failed and the structural controls held. An agent
graded only by accept-rate would have banked this win. The ledger is
append-only, so the accepted row still stands, immutable — with this
document as the permanent corrective record beside it.