mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
benbrownandClaude Opus 4.8
899d55f4b6
An adversarial audit found the enforcement story ahead of the code: gates that were documented as hard gates were advisory, manual, or absent, and several human-facing claims outran the machine record. This makes the claims true. High findings - Differential fuzz was documentation, not enforcement: verdict hardcoded fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and it is inherently baseline-vs-candidate, so it runs on the accept path and reports Skipped in the standalone `just gates` flow. New hard rule: a machine `accepted` verdict is impossible unless the fuzz gate actually passed — otherwise it caps at needs-human-review. - The agent's system prompt claimed FP/concurrency/unsafe patches were "routed to human review automatically". No such mechanism existed. Added a conservative lexical risk classifier (crates/verdict/src/risk.rs) that forces needs-human-review on any would-be accept carrying those signals; fp-tolerance equivalence always signals. - The local verdict path ran agent-influenced code unsandboxed. It now runs under scripts/no-net.sh (network namespace) and fails closed. Upstream test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when present (scripts/pin-testsuite.sh generates them). Also fixed, found while fixing the above - The pipeline script's `set -e` was inert (brace group as the left operand of `&&`), so a failing candidate build did not abort the run and verdict could bench a stale binary. Now a scoped subshell whose status is branched on at top level. - A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path) stranded RUNNING.lock and left read_verdict polling forever. Failure accounting now lives outside the wrapper. - harnessd's patch allowlist ignored git rename/copy headers and --no-prefix diffs; the rename-escape case is now a committed regression test. - The workspace did not compile on Windows (ungated Unix APIs). cfg-gated; 71 portable tests now run on both platforms. - pin-check regenerated the corpus manifest instead of verifying it; it is now verify-only, with a deliberate `just pin-corpus` to re-pin. - Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics (compare --perf-stat), never part of the accept decision. Docs now match the code, including what is NOT yet enforced: no target ships a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not wired to the fp-tolerance gate — each is stated as mechanized-but-inert rather than done. Corrected the stale "33 ledger rows" (34), the playbook's class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation claim (five accepted rows carry sanitizers_clean=false), the unsupported "40,000 requests" figure, and the blanket "historical rows record fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
96 lines
4.6 KiB
Markdown
96 lines
4.6 KiB
Markdown
# Case study 4 — cJSON: the cross-language proof (C, classes 3 + 5)
|
|
|
|
**Target:** [cJSON](https://github.com/DaveGamble/cJSON) v1.7.19 — an
|
|
ultralightweight C JSON parser/printer, MIT. **Workload:** a 5.6 MB
|
|
deterministic JSON corpus (20k records: nested objects/arrays,
|
|
escaped/unicode strings, integers, dense floats), parsed and serialized,
|
|
single-threaded, CPU-pinned. **Result:** the first verified win on a
|
|
C/C++ target — proof the trust layer, the gates, and the ROI story
|
|
carry across languages, not just across Rust crates.
|
|
|
|
## Getting C through the same pipeline
|
|
|
|
Nothing in the trust layer assumes cargo anymore. cJSON is "just a
|
|
`target.toml`" pointing at `clang` build commands, with:
|
|
|
|
- a **hermetic Unity/CTest suite** (19 tests: `parse_number`,
|
|
`parse_string`, `print_*`, `misc_tests`, …) as the upstream-test gate;
|
|
- a **trust-layer harness driver** (`targets/cjson/harness.c`) compiled
|
|
*with* the patched `cJSON.c` — the agent can patch the parser but not
|
|
the measurement rig (it lives outside the workspace allowlist);
|
|
- **ASan + UBSan** as the sanitizer gate (the C analogue of the Rust
|
|
nightly ASan path);
|
|
- its own **A/A calibration**: 0/20 false positives, 20/20
|
|
injected-slowdown detections — the C workload is a trusted anchor
|
|
before any verdict runs.
|
|
|
|
## The profile
|
|
|
|
callgrind cache-sim put **~22% of instructions in glibc float I/O** —
|
|
`__printf_fp` (number serialization) and `strtod`/`str_to_mpn` (number
|
|
parsing) — plus `parse_value`, `parse_string`, and heavy malloc/free
|
|
churn. A number-dense JSON document spends most of its time turning
|
|
numbers into strings and back.
|
|
|
|
## Attempt 1 — the honest rejection (`phase3-cjson-001`, class 3)
|
|
|
|
`parse_number` heap-allocated a temporary NUL-terminated buffer
|
|
(`malloc` + `memcpy` + `free`) for **every** number, just to hand a
|
|
clean string to `strtod`. With ~100k+ numbers in the corpus that is a
|
|
large share of the allocation churn. The fix: a 64-byte **stack buffer**
|
|
for the common short-number case, heap only for pathologically long
|
|
number strings — byte-identical (strtod sees the same bytes).
|
|
|
|
- **Speedup: 1.012 median, 95% CI [1.004, 1.022]** → **rejected-bench.**
|
|
|
|
A real mechanism, a real +1.2%, but the CI lower bound (+0.4%) does not
|
|
clear the 2% bar — because the corpus's wall-clock is dominated by float
|
|
*formatting*, not by malloc. The statistical bar rejected a genuine win,
|
|
exactly as designed. It is in the ledger.
|
|
|
|
## Attempt 2 — the accepted win (`phase3-cjson-002`, classes 5 + 3)
|
|
|
|
Adding the class-5 lever: `print_number` verified every fractional
|
|
number's `"%1.15g"` round-trip by calling **`sscanf("%lg")`** — a full
|
|
`strtod` *plus* format-string interpretation, per number. Replacing it
|
|
with a direct `strtod` is the identical parse with none of the
|
|
format-string overhead. Combined with the stack buffer:
|
|
|
|
- **Speedup: 1.089 median, 95% CI [1.075, 1.100]** → **accepted.**
|
|
- Upstream Unity suite: **PASS**. Golden replay: **byte-identical**.
|
|
- **ASan + UBSan: clean** — through the machine-enforced sanitizer gate.
|
|
- **10,000-input differential fuzz** (`scripts/diff-fuzz-cjson.py`),
|
|
pristine vs candidate, **0 divergences**.
|
|
|
|
A **+8.9%** verified win on a C target, banked.
|
|
|
|
## A finding worth keeping
|
|
|
|
The first cJSON accept nearly wasn't recorded: the ASan+UBSan build
|
|
failed to link because the clang sanitizer runtime (`libclang_rt.asan`)
|
|
was not installed, and `sanitizer_check` treated that as a hard error —
|
|
crashing the verdict and losing the measured bench. The fix hardened the
|
|
gate: an **infrastructure** build failure (missing runtime, broken
|
|
toolchain) now caps the accept at `needs-human-review` and logs the
|
|
failure for the operator, rather than discarding a real result. "Cannot
|
|
verify" is a review tier, not a crash.
|
|
|
|
## What it demonstrates
|
|
|
|
The progression — a real win *rejected* for being too small, then a
|
|
larger win *accepted* only after golden replay, fuzzing, and sanitizers
|
|
all held — is the whole method in miniature, now shown on C. Nothing
|
|
about the trust story is Rust-specific. The cheapest safe lever (skip a
|
|
per-number allocation) wasn't enough; the algorithmic lever (stop doing a
|
|
redundant heavyweight parse) was — and the pipeline proved it the same
|
|
way it proves everything else.
|
|
|
|
---
|
|
|
|
> **Note (2026-07-13).** The 10,000-input differential fuzz cited above
|
|
> ran **out-of-band** (`scripts/diff-fuzz-cjson.py`, per-attempt), not
|
|
> inside the verdict pipeline — so the `phase3-cjson-002` ledger row
|
|
> records `fuzz_iters=0` in its machine record. The fuzz gate is now
|
|
> pipeline-integrated; rows written after this date record the
|
|
> iteration count actually executed.
|