mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
Claude
03113312ad
phase3-cjson-002 (class 5+3): stack-buffer number parsing + strtod round-trip replacing sscanf. +8.85% median, 95% CI [+7.5%, +10.0%] on the C target, banked. 100% audit: byte-identical golden replay, independent ASan+UBSan clean, 10k-input differential fuzz 0 divergences, patch bounds/free-gating verified by eye. - verdict: an infrastructure sanitizer-BUILD failure (missing runtime, broken toolchain) now caps the accept at needs-human-review and logs it, instead of crashing the verdict and losing a measured bench. - results/phase3/: four public case studies (tokei class-5 wins; build flags + allocators; the comrak caught false-accept; the cJSON cross-language proof) + README. Compound-speedup claim stated as an approximation, not a measured figure (CIs don't multiply). - .gitignore: cjson build-artifact dirs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
69 lines
3.1 KiB
Markdown
69 lines
3.1 KiB
Markdown
# Case study 3 — the caught false-accept (comrak, class 3)
|
|
|
|
**This is the most important case study in the project, because it is
|
|
the one where the machine got it wrong and the process caught it.**
|
|
|
|
## What happened
|
|
|
|
`phase2-comrak-010`: the unattended agent, reading comrak's cache-sim
|
|
profile, saw that AST-arena teardown at process exit —
|
|
`drop_in_place::<Vec<Node<RefCell<Ast>>>>` plus its share of the glibc
|
|
free path — cost real instructions and cache misses in a
|
|
run-to-completion CLI. Its hypothesis: comrak already ends in
|
|
`process::exit`, so the OS reclaims memory in bulk anyway; wrap the arena
|
|
in `std::mem::ManuallyDrop` to skip the per-node drop walk, and add a
|
|
`std::hint::black_box(&arena)` before the scope closes so the leak stays
|
|
reachable and a leak checker won't flag it.
|
|
|
|
The pipeline ran it:
|
|
|
|
- Upstream tests: **PASS**
|
|
- Golden replay: **byte-identical**
|
|
- Interleaved A/B: **+10.7% median, 95% CI [+7.9%, +12.1%]**
|
|
- Verdict: **`accepted`**
|
|
|
|
A clean, large, CI-significant win. And it was **wrong**.
|
|
|
|
## The audit overturned it
|
|
|
|
Phase 2 runs a **100% human audit** of every accepted win (SPEC §10).
|
|
The audit ran the sanitizers the pipeline had not: LeakSanitizer flagged
|
|
a **384-byte arena chunk as unreachable at exit**. The agent's
|
|
`black_box(&arena)` keeps the arena live only until its scope *closes* —
|
|
but `process::exit` runs *after* that scope, so at the moment of exit the
|
|
arena is genuinely unreachable. A real leak. The agent's own hypothesis
|
|
had even named the risk ("if the sanitizer still flags the leak,
|
|
rejected-gates is the honest outcome") — the pipeline simply had no
|
|
sanitizer gate to trip, because in Phase 2 ASan/LSan were a per-attempt
|
|
*manual* step.
|
|
|
|
Called per SPEC §10: a **pipeline false accept**, caught by audit
|
|
**before anything shipped**. The win was never banked. Shipped false
|
|
accepts across the entire project remain **zero**.
|
|
|
|
## The fix made the machine enforce the tier
|
|
|
|
The corrective action was not "be more careful." It was structural:
|
|
`verdict` now **builds the patched tree with ASan+LSan (ASan+UBSan for
|
|
C/C++) and runs the pinned workload on every would-be accept**, capping
|
|
any sanitizer-flagged win at `needs-human-review`. The tier rule SPEC §8
|
|
always specified is now enforced by the machine, not the auditor. The
|
|
very next Rust accept (`phase2-tokei-008`) and the first C accept
|
|
(`phase3-cjson-002`) both passed through this automated gate.
|
|
|
|
The 10k-input differential fuzz on the overturned patch found **0
|
|
divergences** — the output equivalence was genuinely real. Only the
|
|
teardown-leak tier was wrong. That is the subtle failure mode the layered
|
|
gates exist for: a change can be behaviorally equivalent on output and
|
|
still unacceptable on a dimension the bench and the golden replay can't
|
|
see.
|
|
|
|
## What it demonstrates
|
|
|
|
The product is not "an agent that finds speedups." The product is
|
|
**trust** — and trust is demonstrated most by the case where the
|
|
optimistic path failed and the structural controls held. An agent
|
|
graded only by accept-rate would have banked this win. The ledger is
|
|
append-only, so the `accepted` row still stands, immutable — with this
|
|
document as the permanent corrective record beside it.
|