benbrownandClaude Opus 4.8
899d55f4b6
An adversarial audit found the enforcement story ahead of the code: gates that were documented as hard gates were advisory, manual, or absent, and several human-facing claims outran the machine record. This makes the claims true. High findings - Differential fuzz was documentation, not enforcement: verdict hardcoded fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and it is inherently baseline-vs-candidate, so it runs on the accept path and reports Skipped in the standalone `just gates` flow. New hard rule: a machine `accepted` verdict is impossible unless the fuzz gate actually passed — otherwise it caps at needs-human-review. - The agent's system prompt claimed FP/concurrency/unsafe patches were "routed to human review automatically". No such mechanism existed. Added a conservative lexical risk classifier (crates/verdict/src/risk.rs) that forces needs-human-review on any would-be accept carrying those signals; fp-tolerance equivalence always signals. - The local verdict path ran agent-influenced code unsandboxed. It now runs under scripts/no-net.sh (network namespace) and fails closed. Upstream test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when present (scripts/pin-testsuite.sh generates them). Also fixed, found while fixing the above - The pipeline script's `set -e` was inert (brace group as the left operand of `&&`), so a failing candidate build did not abort the run and verdict could bench a stale binary. Now a scoped subshell whose status is branched on at top level. - A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path) stranded RUNNING.lock and left read_verdict polling forever. Failure accounting now lives outside the wrapper. - harnessd's patch allowlist ignored git rename/copy headers and --no-prefix diffs; the rename-escape case is now a committed regression test. - The workspace did not compile on Windows (ungated Unix APIs). cfg-gated; 71 portable tests now run on both platforms. - pin-check regenerated the corpus manifest instead of verifying it; it is now verify-only, with a deliberate `just pin-corpus` to re-pin. - Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics (compare --perf-stat), never part of the accept decision. Docs now match the code, including what is NOT yet enforced: no target ships a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not wired to the fp-tolerance gate — each is stated as mechanized-but-inert rather than done. Corrected the stale "33 ledger rows" (34), the playbook's class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation claim (five accepted rows carry sanitizers_clean=false), the unsupported "40,000 requests" figure, and the blanket "historical rows record fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
4.6 KiB
Case study 4 — cJSON: the cross-language proof (C, classes 3 + 5)
Target: cJSON v1.7.19 — an ultralightweight C JSON parser/printer, MIT. Workload: a 5.6 MB deterministic JSON corpus (20k records: nested objects/arrays, escaped/unicode strings, integers, dense floats), parsed and serialized, single-threaded, CPU-pinned. Result: the first verified win on a C/C++ target — proof the trust layer, the gates, and the ROI story carry across languages, not just across Rust crates.
Getting C through the same pipeline
Nothing in the trust layer assumes cargo anymore. cJSON is "just a
target.toml" pointing at clang build commands, with:
- a hermetic Unity/CTest suite (19 tests:
parse_number,parse_string,print_*,misc_tests, …) as the upstream-test gate; - a trust-layer harness driver (
targets/cjson/harness.c) compiled with the patchedcJSON.c— the agent can patch the parser but not the measurement rig (it lives outside the workspace allowlist); - ASan + UBSan as the sanitizer gate (the C analogue of the Rust nightly ASan path);
- its own A/A calibration: 0/20 false positives, 20/20 injected-slowdown detections — the C workload is a trusted anchor before any verdict runs.
The profile
callgrind cache-sim put ~22% of instructions in glibc float I/O —
__printf_fp (number serialization) and strtod/str_to_mpn (number
parsing) — plus parse_value, parse_string, and heavy malloc/free
churn. A number-dense JSON document spends most of its time turning
numbers into strings and back.
Attempt 1 — the honest rejection (phase3-cjson-001, class 3)
parse_number heap-allocated a temporary NUL-terminated buffer
(malloc + memcpy + free) for every number, just to hand a
clean string to strtod. With ~100k+ numbers in the corpus that is a
large share of the allocation churn. The fix: a 64-byte stack buffer
for the common short-number case, heap only for pathologically long
number strings — byte-identical (strtod sees the same bytes).
- Speedup: 1.012 median, 95% CI [1.004, 1.022] → rejected-bench.
A real mechanism, a real +1.2%, but the CI lower bound (+0.4%) does not clear the 2% bar — because the corpus's wall-clock is dominated by float formatting, not by malloc. The statistical bar rejected a genuine win, exactly as designed. It is in the ledger.
Attempt 2 — the accepted win (phase3-cjson-002, classes 5 + 3)
Adding the class-5 lever: print_number verified every fractional
number's "%1.15g" round-trip by calling sscanf("%lg") — a full
strtod plus format-string interpretation, per number. Replacing it
with a direct strtod is the identical parse with none of the
format-string overhead. Combined with the stack buffer:
- Speedup: 1.089 median, 95% CI [1.075, 1.100] → accepted.
- Upstream Unity suite: PASS. Golden replay: byte-identical.
- ASan + UBSan: clean — through the machine-enforced sanitizer gate.
- 10,000-input differential fuzz (
scripts/diff-fuzz-cjson.py), pristine vs candidate, 0 divergences.
A +8.9% verified win on a C target, banked.
A finding worth keeping
The first cJSON accept nearly wasn't recorded: the ASan+UBSan build
failed to link because the clang sanitizer runtime (libclang_rt.asan)
was not installed, and sanitizer_check treated that as a hard error —
crashing the verdict and losing the measured bench. The fix hardened the
gate: an infrastructure build failure (missing runtime, broken
toolchain) now caps the accept at needs-human-review and logs the
failure for the operator, rather than discarding a real result. "Cannot
verify" is a review tier, not a crash.
What it demonstrates
The progression — a real win rejected for being too small, then a larger win accepted only after golden replay, fuzzing, and sanitizers all held — is the whole method in miniature, now shown on C. Nothing about the trust story is Rust-specific. The cheapest safe lever (skip a per-number allocation) wasn't enough; the algorithmic lever (stop doing a redundant heavyweight parse) was — and the pipeline proved it the same way it proves everything else.
Note (2026-07-13). The 10,000-input differential fuzz cited above ran out-of-band (
scripts/diff-fuzz-cjson.py, per-attempt), not inside the verdict pipeline — so thephase3-cjson-002ledger row recordsfuzz_iters=0in its machine record. The fuzz gate is now pipeline-integrated; rows written after this date record the iteration count actually executed.