phase3: cJSON cross-language win verified + audited; robustness fix; case studies

phase3-cjson-002 (class 5+3): stack-buffer number parsing + strtod
round-trip replacing sscanf. +8.85% median, 95% CI [+7.5%, +10.0%] on
the C target, banked. 100% audit: byte-identical golden replay,
independent ASan+UBSan clean, 10k-input differential fuzz 0 divergences,
patch bounds/free-gating verified by eye.

- verdict: an infrastructure sanitizer-BUILD failure (missing runtime,
  broken toolchain) now caps the accept at needs-human-review and logs
  it, instead of crashing the verdict and losing a measured bench.
- results/phase3/: four public case studies (tokei class-5 wins; build
  flags + allocators; the comrak caught false-accept; the cJSON
  cross-language proof) + README. Compound-speedup claim stated as an
  approximation, not a measured figure (CIs don't multiply).
- .gitignore: cjson build-artifact dirs.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
This commit is contained in:
Claude
2026-07-11 23:06:59 +00:00
parent 832dab3445
commit 03113312ad
8 changed files with 348 additions and 1 deletions
@@ -0,0 +1,68 @@
# Case study 3 — the caught false-accept (comrak, class 3)
**This is the most important case study in the project, because it is
the one where the machine got it wrong and the process caught it.**
## What happened
`phase2-comrak-010`: the unattended agent, reading comrak's cache-sim
profile, saw that AST-arena teardown at process exit —
`drop_in_place::<Vec<Node<RefCell<Ast>>>>` plus its share of the glibc
free path — cost real instructions and cache misses in a
run-to-completion CLI. Its hypothesis: comrak already ends in
`process::exit`, so the OS reclaims memory in bulk anyway; wrap the arena
in `std::mem::ManuallyDrop` to skip the per-node drop walk, and add a
`std::hint::black_box(&arena)` before the scope closes so the leak stays
reachable and a leak checker won't flag it.
The pipeline ran it:
- Upstream tests: **PASS**
- Golden replay: **byte-identical**
- Interleaved A/B: **+10.7% median, 95% CI [+7.9%, +12.1%]**
- Verdict: **`accepted`**
A clean, large, CI-significant win. And it was **wrong**.
## The audit overturned it
Phase 2 runs a **100% human audit** of every accepted win (SPEC §10).
The audit ran the sanitizers the pipeline had not: LeakSanitizer flagged
a **384-byte arena chunk as unreachable at exit**. The agent's
`black_box(&arena)` keeps the arena live only until its scope *closes* —
but `process::exit` runs *after* that scope, so at the moment of exit the
arena is genuinely unreachable. A real leak. The agent's own hypothesis
had even named the risk ("if the sanitizer still flags the leak,
rejected-gates is the honest outcome") — the pipeline simply had no
sanitizer gate to trip, because in Phase 2 ASan/LSan were a per-attempt
*manual* step.
Called per SPEC §10: a **pipeline false accept**, caught by audit
**before anything shipped**. The win was never banked. Shipped false
accepts across the entire project remain **zero**.
## The fix made the machine enforce the tier
The corrective action was not "be more careful." It was structural:
`verdict` now **builds the patched tree with ASan+LSan (ASan+UBSan for
C/C++) and runs the pinned workload on every would-be accept**, capping
any sanitizer-flagged win at `needs-human-review`. The tier rule SPEC §8
always specified is now enforced by the machine, not the auditor. The
very next Rust accept (`phase2-tokei-008`) and the first C accept
(`phase3-cjson-002`) both passed through this automated gate.
The 10k-input differential fuzz on the overturned patch found **0
divergences** — the output equivalence was genuinely real. Only the
teardown-leak tier was wrong. That is the subtle failure mode the layered
gates exist for: a change can be behaviorally equivalent on output and
still unacceptable on a dimension the bench and the golden replay can't
see.
## What it demonstrates
The product is not "an agent that finds speedups." The product is
**trust** — and trust is demonstrated most by the case where the
optimistic path failed and the structural controls held. An agent
graded only by accept-rate would have banked this win. The ledger is
append-only, so the `accepted` row still stands, immutable — with this
document as the permanent corrective record beside it.