mirror of https://github.com/bdbrown4/peltier.io.git synced 2026-11-07 10:58:28 +00:00

Files

T

benbrownandClaude Opus 4.8 899d55f4b6 gates: close the audit findings — enforce the gates the docs claimed

An adversarial audit found the enforcement story ahead of the code: gates
that were documented as hard gates were advisory, manual, or absent, and
several human-facing claims outran the machine record. This makes the
claims true.

High findings

- Differential fuzz was documentation, not enforcement: verdict hardcoded
  fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case
  studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real
  pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and
  it is inherently baseline-vs-candidate, so it runs on the accept path and
  reports Skipped in the standalone `just gates` flow. New hard rule: a
  machine `accepted` verdict is impossible unless the fuzz gate actually
  passed — otherwise it caps at needs-human-review.
- The agent's system prompt claimed FP/concurrency/unsafe patches were
  "routed to human review automatically". No such mechanism existed. Added
  a conservative lexical risk classifier (crates/verdict/src/risk.rs) that
  forces needs-human-review on any would-be accept carrying those signals;
  fp-tolerance equivalence always signals.
- The local verdict path ran agent-influenced code unsandboxed. It now runs
  under scripts/no-net.sh (network namespace) and fails closed. Upstream
  test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when
  present (scripts/pin-testsuite.sh generates them).

Also fixed, found while fixing the above

- The pipeline script's `set -e` was inert (brace group as the left operand
  of `&&`), so a failing candidate build did not abort the run and verdict
  could bench a stale binary. Now a scoped subshell whose status is branched
  on at top level.
- A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path)
  stranded RUNNING.lock and left read_verdict polling forever. Failure
  accounting now lives outside the wrapper.
- harnessd's patch allowlist ignored git rename/copy headers and --no-prefix
  diffs; the rename-escape case is now a committed regression test.
- The workspace did not compile on Windows (ungated Unix APIs). cfg-gated;
  71 portable tests now run on both platforms.
- pin-check regenerated the corpus manifest instead of verifying it; it is
  now verify-only, with a deliberate `just pin-corpus` to re-pin.
- Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics
  (compare --perf-stat), never part of the accept decision.

Docs now match the code, including what is NOT yet enforced: no target ships
a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not
wired to the fp-tolerance gate — each is stated as mechanized-but-inert
rather than done. Corrected the stale "33 ledger rows" (34), the playbook's
class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation
claim (five accepted rows carry sanitizers_clean=false), the unsupported
"40,000 requests" figure, and the blanket "historical rows record
fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-07-14 13:10:14 -04:00

4.6 KiB

Raw Permalink Blame History

Case study 4 — cJSON: the cross-language proof (C, classes 3 + 5)

Target: cJSON v1.7.19 — an ultralightweight C JSON parser/printer, MIT. Workload: a 5.6 MB deterministic JSON corpus (20k records: nested objects/arrays, escaped/unicode strings, integers, dense floats), parsed and serialized, single-threaded, CPU-pinned. Result: the first verified win on a C/C++ target — proof the trust layer, the gates, and the ROI story carry across languages, not just across Rust crates.

Getting C through the same pipeline

Nothing in the trust layer assumes cargo anymore. cJSON is "just a target.toml" pointing at clang build commands, with:

a hermetic Unity/CTest suite (19 tests: parse_number, parse_string, print_*, misc_tests, …) as the upstream-test gate;
a trust-layer harness driver (targets/cjson/harness.c) compiled with the patched cJSON.c — the agent can patch the parser but not the measurement rig (it lives outside the workspace allowlist);
ASan + UBSan as the sanitizer gate (the C analogue of the Rust nightly ASan path);
its own A/A calibration: 0/20 false positives, 20/20 injected-slowdown detections — the C workload is a trusted anchor before any verdict runs.

The profile

callgrind cache-sim put ~22% of instructions in glibc float I/O — __printf_fp (number serialization) and strtod/str_to_mpn (number parsing) — plus parse_value, parse_string, and heavy malloc/free churn. A number-dense JSON document spends most of its time turning numbers into strings and back.

Attempt 1 — the honest rejection (`phase3-cjson-001`, class 3)

parse_number heap-allocated a temporary NUL-terminated buffer (malloc + memcpy + free) for every number, just to hand a clean string to strtod. With ~100k+ numbers in the corpus that is a large share of the allocation churn. The fix: a 64-byte stack buffer for the common short-number case, heap only for pathologically long number strings — byte-identical (strtod sees the same bytes).

Speedup: 1.012 median, 95% CI [1.004, 1.022] → rejected-bench.

A real mechanism, a real +1.2%, but the CI lower bound (+0.4%) does not clear the 2% bar — because the corpus's wall-clock is dominated by float formatting, not by malloc. The statistical bar rejected a genuine win, exactly as designed. It is in the ledger.

Attempt 2 — the accepted win (`phase3-cjson-002`, classes 5 + 3)

Adding the class-5 lever: print_number verified every fractional number's "%1.15g" round-trip by calling sscanf("%lg") — a full strtod plus format-string interpretation, per number. Replacing it with a direct strtod is the identical parse with none of the format-string overhead. Combined with the stack buffer:

Speedup: 1.089 median, 95% CI [1.075, 1.100] → accepted.
Upstream Unity suite: PASS. Golden replay: byte-identical.
ASan + UBSan: clean — through the machine-enforced sanitizer gate.
10,000-input differential fuzz (scripts/diff-fuzz-cjson.py), pristine vs candidate, 0 divergences.

A +8.9% verified win on a C target, banked.

A finding worth keeping

The first cJSON accept nearly wasn't recorded: the ASan+UBSan build failed to link because the clang sanitizer runtime (libclang_rt.asan) was not installed, and sanitizer_check treated that as a hard error — crashing the verdict and losing the measured bench. The fix hardened the gate: an infrastructure build failure (missing runtime, broken toolchain) now caps the accept at needs-human-review and logs the failure for the operator, rather than discarding a real result. "Cannot verify" is a review tier, not a crash.

What it demonstrates

The progression — a real win rejected for being too small, then a larger win accepted only after golden replay, fuzzing, and sanitizers all held — is the whole method in miniature, now shown on C. Nothing about the trust story is Rust-specific. The cheapest safe lever (skip a per-number allocation) wasn't enough; the algorithmic lever (stop doing a redundant heavyweight parse) was — and the pipeline proved it the same way it proves everything else.

Note (2026-07-13). The 10,000-input differential fuzz cited above ran out-of-band (scripts/diff-fuzz-cjson.py, per-attempt), not inside the verdict pipeline — so the phase3-cjson-002 ledger row records fuzz_iters=0 in its machine record. The fuzz gate is now pipeline-integrated; rows written after this date record the iteration count actually executed.

4.6 KiB Raw Permalink Blame History