mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
phase3: cJSON cross-language win verified + audited; robustness fix; case studies
phase3-cjson-002 (class 5+3): stack-buffer number parsing + strtod round-trip replacing sscanf. +8.85% median, 95% CI [+7.5%, +10.0%] on the C target, banked. 100% audit: byte-identical golden replay, independent ASan+UBSan clean, 10k-input differential fuzz 0 divergences, patch bounds/free-gating verified by eye. - verdict: an infrastructure sanitizer-BUILD failure (missing runtime, broken toolchain) now caps the accept at needs-human-review and logs it, instead of crashing the verdict and losing a measured bench. - results/phase3/: four public case studies (tokei class-5 wins; build flags + allocators; the comrak caught false-accept; the cJSON cross-language proof) + README. Compound-speedup claim stated as an approximation, not a measured figure (CIs don't multiply). - .gitignore: cjson build-artifact dirs. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
This commit is contained in:
@@ -0,0 +1,68 @@
|
||||
# Case study 3 — the caught false-accept (comrak, class 3)
|
||||
|
||||
**This is the most important case study in the project, because it is
|
||||
the one where the machine got it wrong and the process caught it.**
|
||||
|
||||
## What happened
|
||||
|
||||
`phase2-comrak-010`: the unattended agent, reading comrak's cache-sim
|
||||
profile, saw that AST-arena teardown at process exit —
|
||||
`drop_in_place::<Vec<Node<RefCell<Ast>>>>` plus its share of the glibc
|
||||
free path — cost real instructions and cache misses in a
|
||||
run-to-completion CLI. Its hypothesis: comrak already ends in
|
||||
`process::exit`, so the OS reclaims memory in bulk anyway; wrap the arena
|
||||
in `std::mem::ManuallyDrop` to skip the per-node drop walk, and add a
|
||||
`std::hint::black_box(&arena)` before the scope closes so the leak stays
|
||||
reachable and a leak checker won't flag it.
|
||||
|
||||
The pipeline ran it:
|
||||
|
||||
- Upstream tests: **PASS**
|
||||
- Golden replay: **byte-identical**
|
||||
- Interleaved A/B: **+10.7% median, 95% CI [+7.9%, +12.1%]**
|
||||
- Verdict: **`accepted`**
|
||||
|
||||
A clean, large, CI-significant win. And it was **wrong**.
|
||||
|
||||
## The audit overturned it
|
||||
|
||||
Phase 2 runs a **100% human audit** of every accepted win (SPEC §10).
|
||||
The audit ran the sanitizers the pipeline had not: LeakSanitizer flagged
|
||||
a **384-byte arena chunk as unreachable at exit**. The agent's
|
||||
`black_box(&arena)` keeps the arena live only until its scope *closes* —
|
||||
but `process::exit` runs *after* that scope, so at the moment of exit the
|
||||
arena is genuinely unreachable. A real leak. The agent's own hypothesis
|
||||
had even named the risk ("if the sanitizer still flags the leak,
|
||||
rejected-gates is the honest outcome") — the pipeline simply had no
|
||||
sanitizer gate to trip, because in Phase 2 ASan/LSan were a per-attempt
|
||||
*manual* step.
|
||||
|
||||
Called per SPEC §10: a **pipeline false accept**, caught by audit
|
||||
**before anything shipped**. The win was never banked. Shipped false
|
||||
accepts across the entire project remain **zero**.
|
||||
|
||||
## The fix made the machine enforce the tier
|
||||
|
||||
The corrective action was not "be more careful." It was structural:
|
||||
`verdict` now **builds the patched tree with ASan+LSan (ASan+UBSan for
|
||||
C/C++) and runs the pinned workload on every would-be accept**, capping
|
||||
any sanitizer-flagged win at `needs-human-review`. The tier rule SPEC §8
|
||||
always specified is now enforced by the machine, not the auditor. The
|
||||
very next Rust accept (`phase2-tokei-008`) and the first C accept
|
||||
(`phase3-cjson-002`) both passed through this automated gate.
|
||||
|
||||
The 10k-input differential fuzz on the overturned patch found **0
|
||||
divergences** — the output equivalence was genuinely real. Only the
|
||||
teardown-leak tier was wrong. That is the subtle failure mode the layered
|
||||
gates exist for: a change can be behaviorally equivalent on output and
|
||||
still unacceptable on a dimension the bench and the golden replay can't
|
||||
see.
|
||||
|
||||
## What it demonstrates
|
||||
|
||||
The product is not "an agent that finds speedups." The product is
|
||||
**trust** — and trust is demonstrated most by the case where the
|
||||
optimistic path failed and the structural controls held. An agent
|
||||
graded only by accept-rate would have banked this win. The ledger is
|
||||
append-only, so the `accepted` row still stands, immutable — with this
|
||||
document as the permanent corrective record beside it.
|
||||
Reference in New Issue
Block a user