gates: close the audit findings — enforce the gates the docs claimed

An adversarial audit found the enforcement story ahead of the code: gates
that were documented as hard gates were advisory, manual, or absent, and
several human-facing claims outran the machine record. This makes the
claims true.

High findings

- Differential fuzz was documentation, not enforcement: verdict hardcoded
  fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case
  studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real
  pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and
  it is inherently baseline-vs-candidate, so it runs on the accept path and
  reports Skipped in the standalone `just gates` flow. New hard rule: a
  machine `accepted` verdict is impossible unless the fuzz gate actually
  passed — otherwise it caps at needs-human-review.
- The agent's system prompt claimed FP/concurrency/unsafe patches were
  "routed to human review automatically". No such mechanism existed. Added
  a conservative lexical risk classifier (crates/verdict/src/risk.rs) that
  forces needs-human-review on any would-be accept carrying those signals;
  fp-tolerance equivalence always signals.
- The local verdict path ran agent-influenced code unsandboxed. It now runs
  under scripts/no-net.sh (network namespace) and fails closed. Upstream
  test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when
  present (scripts/pin-testsuite.sh generates them).

Also fixed, found while fixing the above

- The pipeline script's `set -e` was inert (brace group as the left operand
  of `&&`), so a failing candidate build did not abort the run and verdict
  could bench a stale binary. Now a scoped subshell whose status is branched
  on at top level.
- A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path)
  stranded RUNNING.lock and left read_verdict polling forever. Failure
  accounting now lives outside the wrapper.
- harnessd's patch allowlist ignored git rename/copy headers and --no-prefix
  diffs; the rename-escape case is now a committed regression test.
- The workspace did not compile on Windows (ungated Unix APIs). cfg-gated;
  71 portable tests now run on both platforms.
- pin-check regenerated the corpus manifest instead of verifying it; it is
  now verify-only, with a deliberate `just pin-corpus` to re-pin.
- Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics
  (compare --perf-stat), never part of the accept decision.

Docs now match the code, including what is NOT yet enforced: no target ships
a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not
wired to the fp-tolerance gate — each is stated as mechanized-but-inert
rather than done. Corrected the stale "33 ledger rows" (34), the playbook's
class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation
claim (five accepted rows carry sanitizers_clean=false), the unsupported
"40,000 requests" figure, and the blanket "historical rows record
fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-14 13:10:14 -04:00
co-authored by Claude Opus 4.8
parent 21352efe6c
commit 899d55f4b6
57 changed files with 3308 additions and 692 deletions
+6
View File
@@ -37,6 +37,12 @@ workload = "5.6 MB synthetic JSON (20k records: nested objects/arrays, escaped/u
tests = "cmake -S targets/cjson/workspace -B targets/cjson/testbuild -DENABLE_CJSON_TEST=On -DCMAKE_BUILD_TYPE=Debug >/dev/null 2>&1 && cmake --build targets/cjson/testbuild -j >/dev/null 2>&1 && ctest --test-dir targets/cjson/testbuild --output-on-failure"
# Golden: full re-serialization + checksum of the corpus, byte-hashed.
golden = "{binary} corpora/cjson/input/big.json 1"
# Differential fuzz, pipeline-integrated: diff-test substitutes {iters},
# {baseline} (the pristine binary the accept path rebuilds) and {candidate},
# then parses the script's final FUZZ-RESULT line. The harness owns where the
# pristine build lands, so the script takes the baseline from argv and never
# hardcodes a path; paths may be repo-root-relative or absolute.
fuzz = "python3 scripts/diff-fuzz-cjson.py targets/cjson/fuzz-work {iters} {baseline} {candidate}"
[corpus]
manifest = "corpora/cjson/MANIFEST.sha256"
+6
View File
@@ -33,6 +33,12 @@ workload = "Pro Git book, all translations, 11.0 MB concatenated markdown, singl
tests = "cargo test --release --manifest-path targets/comrak/workspace/Cargo.toml"
# Golden replay: corpus through the binary, stdout must be byte-identical.
golden = "cat corpora/comrak/progit.md | {binary}"
# Differential fuzz, pipeline-integrated: diff-test substitutes {iters},
# {baseline} (the pristine binary the accept path rebuilds) and {candidate},
# then parses the script's final FUZZ-RESULT line. The harness owns where the
# pristine build lands, so the script takes the baseline from argv and never
# hardcodes a path; paths may be repo-root-relative or absolute.
fuzz = "python3 scripts/diff-fuzz-comrak.py targets/comrak/fuzz-work {iters} {baseline} {candidate}"
[corpus]
manifest = "corpora/comrak/MANIFEST.sha256"
+12
View File
@@ -21,4 +21,16 @@ grep -A0 '^"vendor/' "$toml" | while IFS= read -r line; do
git -C "$ws/$sub" fetch -q origin "$scommit" 2>/dev/null || true
git -C "$ws/$sub" checkout -q "$scommit"
done
# Upstream test-suite pin: verify against repo root when present; a
# mismatch means the fetched suite is not the one that was audited.
suite="corpora/$name/TESTSUITE.sha256"
if [ -f "$suite" ]; then
if ! sha256sum -c --quiet "$suite"; then
echo "fetch: upstream test-suite pin MISMATCH for $name ($suite) — refusing" >&2
exit 1
fi
echo "$name test suite verified against $suite"
else
echo "note: no $suite — pin the suite with: scripts/pin-testsuite.sh $name <paths...>"
fi
echo "$name pinned at $commit"
+6
View File
@@ -23,6 +23,12 @@ workload = "8 replicas of pinned progit + comrak/src trees, 221 MB, 4360 files,
[gates]
tests = "cargo test --release --manifest-path targets/tokei/workspace/Cargo.toml"
golden = "RAYON_NUM_THREADS=1 {binary} --sort code corpora/tokei/input"
# Differential fuzz, pipeline-integrated: diff-test substitutes {iters},
# {baseline} (the pristine binary the accept path rebuilds) and {candidate},
# then parses the script's final FUZZ-RESULT line. The harness owns where the
# pristine build lands, so the script takes the baseline from argv and never
# hardcodes a path; paths may be repo-root-relative or absolute.
fuzz = "python3 scripts/diff-fuzz-tokei.py targets/tokei/fuzz-work {iters} {baseline} {candidate}"
[corpus]
manifest = "corpora/tokei/MANIFEST.sha256"