mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 19:08:27 +00:00
benbrownandClaude Opus 4.8
899d55f4b6
An adversarial audit found the enforcement story ahead of the code: gates that were documented as hard gates were advisory, manual, or absent, and several human-facing claims outran the machine record. This makes the claims true. High findings - Differential fuzz was documentation, not enforcement: verdict hardcoded fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and it is inherently baseline-vs-candidate, so it runs on the accept path and reports Skipped in the standalone `just gates` flow. New hard rule: a machine `accepted` verdict is impossible unless the fuzz gate actually passed — otherwise it caps at needs-human-review. - The agent's system prompt claimed FP/concurrency/unsafe patches were "routed to human review automatically". No such mechanism existed. Added a conservative lexical risk classifier (crates/verdict/src/risk.rs) that forces needs-human-review on any would-be accept carrying those signals; fp-tolerance equivalence always signals. - The local verdict path ran agent-influenced code unsandboxed. It now runs under scripts/no-net.sh (network namespace) and fails closed. Upstream test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when present (scripts/pin-testsuite.sh generates them). Also fixed, found while fixing the above - The pipeline script's `set -e` was inert (brace group as the left operand of `&&`), so a failing candidate build did not abort the run and verdict could bench a stale binary. Now a scoped subshell whose status is branched on at top level. - A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path) stranded RUNNING.lock and left read_verdict polling forever. Failure accounting now lives outside the wrapper. - harnessd's patch allowlist ignored git rename/copy headers and --no-prefix diffs; the rename-escape case is now a committed regression test. - The workspace did not compile on Windows (ungated Unix APIs). cfg-gated; 71 portable tests now run on both platforms. - pin-check regenerated the corpus manifest instead of verifying it; it is now verify-only, with a deliberate `just pin-corpus` to re-pin. - Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics (compare --perf-stat), never part of the accept decision. Docs now match the code, including what is NOT yet enforced: no target ships a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not wired to the fp-tolerance gate — each is stated as mechanized-but-inert rather than done. Corrected the stale "33 ledger rows" (34), the playbook's class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation claim (five accepted rows carry sanitizers_clean=false), the unsupported "40,000 requests" figure, and the blanket "historical rows record fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
79 lines
3.9 KiB
Markdown
79 lines
3.9 KiB
Markdown
# Phase 3 — playbook + proof
|
||
|
||
**Exit criteria (SPEC §5): 3–5 public case studies · zero regressions
|
||
shipped · playbook ≥6 classes · C/C++ targets · coz.**
|
||
|
||
Phase 3 turns the trust layer and the agent loop from Phases 1–2 into
|
||
*evidence*: verified optimization wins across the playbook and across
|
||
languages, each one reproducible from a pinned corpus through the same
|
||
gated pipeline, with every attempt — win or loss — in the append-only
|
||
ledger.
|
||
|
||
## What changed for Phase 3
|
||
|
||
- **The trust layer went language-agnostic.** Build isolation moved from
|
||
a cargo-specific `CARGO_TARGET_DIR` injection to an explicit `{out}`
|
||
placeholder plus optional `sanitizer`/`sanitizer_binary` build
|
||
templates (`crates/diff-test/src/target.rs`). A C/C++ target is now
|
||
just a `target.toml` with `clang` commands. Proven zero-regression:
|
||
tokei ran a full verdict through the refactored pipeline to a correct
|
||
null result before any C target was added.
|
||
- **A C target joined the roster** — cJSON (v1.7.19, MIT), with a
|
||
trust-layer harness driver, a hermetic Unity/CTest gate, ASan+UBSan
|
||
sanitizers, a deterministic pinned corpus, and its own A/A calibration
|
||
(0/20 false positives, 20/20 injected-slowdown detections). CI gates
|
||
it on every push.
|
||
- **coz** is wired (`-DHOTPATH_COZ` progress point, `just coz`,
|
||
`scripts/coz-*`); its apt runtime aborts in its own init on this
|
||
container's glibc (documented in `results/cjson/coz/README.md`), so
|
||
callgrind cache-sim is the working profiler.
|
||
|
||
## Playbook classes exercised (≥6 required)
|
||
|
||
| Class | Mechanism | Exercised on | Result |
|
||
|---|---|---|---|
|
||
| 1 | build config (LTO, codegen-units) | tokei | ✅ win (+10.4%) |
|
||
| 2 | allocator swap (mimalloc) | comrak | needs-human-review (+4.6%) |
|
||
| 3 | allocation churn | tokei, comrak, **cjson** | ✅ wins + one overturned |
|
||
| 4 | data layout | comrak | attempted (rejected) |
|
||
| 5 | algorithmic / redundant work | tokei ×3, **cjson** | ✅ wins |
|
||
| 6 | SIMD / autovectorization | comrak, tokei | attempted (rejected) |
|
||
|
||
Six classes exercised (1–6); class 5 and class 3 carry verified wins on
|
||
both a Rust and a C target.
|
||
|
||
## The case studies
|
||
|
||
1. [`01-tokei-algorithmic-rust.md`](01-tokei-algorithmic-rust.md) —
|
||
three independent CI-verified class-5 wins on a Rust code counter
|
||
(~1.26× multiplied, stated as an approximation, not a measured
|
||
figure), each unattended and audited.
|
||
2. [`02-build-flags-and-allocators.md`](02-build-flags-and-allocators.md)
|
||
— the cheap wins first: LTO (class 1) accepted, mimalloc (class 2)
|
||
correctly held at needs-human-review.
|
||
3. [`03-comrak-caught-false-accept.md`](03-comrak-caught-false-accept.md)
|
||
— the pipeline auto-accepted a leaking teardown patch; the audit
|
||
overturned it and hardened the pipeline. The most important study.
|
||
4. [`04-cjson-c-target.md`](04-cjson-c-target.md) — the cross-language
|
||
proof: a real sub-threshold win correctly rejected, then a byte-
|
||
identical number-handling win accepted on the C target.
|
||
|
||
## The through-line
|
||
|
||
Every number here carries its 95% bootstrap CI and its workload. Every
|
||
win survived byte-identical golden replay, a 10k-input differential
|
||
fuzz, and sanitizers. Every rejection is a real mechanism that simply
|
||
didn't clear the 2% CI-lower-bound bar — and it is in the ledger too,
|
||
because a rejection is a complete, valid outcome. Nothing shipped that
|
||
the pipeline couldn't prove.
|
||
|
||
> **Note (2026-07-13).** "Every win survived … a 10k-input differential
|
||
> fuzz, and sanitizers" is true of the **evidence**, not of the machine
|
||
> record at the time: through Phase 5 the fuzz ran **out-of-band**
|
||
> (`scripts/diff-fuzz-<target>.py`, per-attempt) and the Phase 3 rows
|
||
> therefore carry `fuzz_iters=0`. The fuzz gate is now integrated into the
|
||
> accept path, where a passed fuzz gate is a **precondition** for a machine
|
||
> `accepted` verdict; rows written after this date record the iteration
|
||
> count actually executed. See the per-case-study notes in
|
||
> `01-tokei-algorithmic-rust.md` and `04-cjson-c-target.md`.
|