gates: close the audit findings — enforce the gates the docs claimed

An adversarial audit found the enforcement story ahead of the code: gates
that were documented as hard gates were advisory, manual, or absent, and
several human-facing claims outran the machine record. This makes the
claims true.

High findings

- Differential fuzz was documentation, not enforcement: verdict hardcoded
  fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case
  studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real
  pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and
  it is inherently baseline-vs-candidate, so it runs on the accept path and
  reports Skipped in the standalone `just gates` flow. New hard rule: a
  machine `accepted` verdict is impossible unless the fuzz gate actually
  passed — otherwise it caps at needs-human-review.
- The agent's system prompt claimed FP/concurrency/unsafe patches were
  "routed to human review automatically". No such mechanism existed. Added
  a conservative lexical risk classifier (crates/verdict/src/risk.rs) that
  forces needs-human-review on any would-be accept carrying those signals;
  fp-tolerance equivalence always signals.
- The local verdict path ran agent-influenced code unsandboxed. It now runs
  under scripts/no-net.sh (network namespace) and fails closed. Upstream
  test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when
  present (scripts/pin-testsuite.sh generates them).

Also fixed, found while fixing the above

- The pipeline script's `set -e` was inert (brace group as the left operand
  of `&&`), so a failing candidate build did not abort the run and verdict
  could bench a stale binary. Now a scoped subshell whose status is branched
  on at top level.
- A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path)
  stranded RUNNING.lock and left read_verdict polling forever. Failure
  accounting now lives outside the wrapper.
- harnessd's patch allowlist ignored git rename/copy headers and --no-prefix
  diffs; the rename-escape case is now a committed regression test.
- The workspace did not compile on Windows (ungated Unix APIs). cfg-gated;
  71 portable tests now run on both platforms.
- pin-check regenerated the corpus manifest instead of verifying it; it is
  now verify-only, with a deliberate `just pin-corpus` to re-pin.
- Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics
  (compare --perf-stat), never part of the accept decision.

Docs now match the code, including what is NOT yet enforced: no target ships
a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not
wired to the fp-tolerance gate — each is stated as mechanized-but-inert
rather than done. Corrected the stale "33 ledger rows" (34), the playbook's
class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation
claim (five accepted rows carry sanitizers_clean=false), the unsupported
"40,000 requests" figure, and the blanket "historical rows record
fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-14 13:10:14 -04:00
co-authored by Claude Opus 4.8
parent 21352efe6c
commit 899d55f4b6
57 changed files with 3308 additions and 692 deletions
+9 -2
View File
@@ -8,8 +8,15 @@ Every corpus directory carries a `MANIFEST.sha256` (`sha256sum` format:
before every run and refuses to run on any mismatch — a tampered corpus
is a stop-the-line event.
Regenerate a manifest only as a deliberate human action:
`just pin-check <target>` regenerates the corpus and verifies it against
the manifest; it never rewrites the manifest. Regenerate a manifest only
as a deliberate human action:
```sh
cd corpora/<target> && find . -type f ! -name MANIFEST.sha256 -exec sha256sum {} + > MANIFEST.sha256
just pin-corpus <target> # gen-corpus.sh --pin
```
A directory may also carry a `TESTSUITE.sha256` pinning the upstream test
suite (repo-root-relative paths). `diff-test` verifies it before running
upstream tests and `targets/fetch.sh` verifies it after every fetch;
generate it with `scripts/pin-testsuite.sh <target> <path>...`.
+20 -4
View File
@@ -4,8 +4,16 @@
# with escapes and unicode, ints, floats (the strtod/sprintf hot path),
# booleans, nulls. No randomness — byte-identical on every machine, so
# the MANIFEST.sha256 pin is reproducible. Run from repo root:
# sh corpora/cjson/gen-corpus.sh
# sh corpora/cjson/gen-corpus.sh [--check|--pin]
# Default/--check: regenerate and verify against MANIFEST.sha256 (never
# rewrites it). --pin: deliberately rewrite the manifest.
set -eu
mode="check"
case "${1:-}" in
"" | --check) ;;
--pin) mode="pin" ;;
*) echo "usage: gen-corpus.sh [--check|--pin]" >&2; exit 2 ;;
esac
DIR="$(cd "$(dirname "$0")" && pwd)"
OUT="$DIR/input/big.json"
mkdir -p "$DIR/input"
@@ -57,6 +65,14 @@ with open(sys.argv[1], "w") as f:
PY
echo "wrote $OUT ($(wc -c < "$OUT") bytes)"
# Pin the corpus and regenerate the manifest.
( cd "$DIR/input" && find . -type f -print0 | sort -z | xargs -0 sha256sum ) > "$DIR/MANIFEST.sha256"
echo "manifest: $DIR/MANIFEST.sha256"
if [ "$mode" = "pin" ]; then
( cd "$DIR/input" && find . -type f -print0 | sort -z | xargs -0 sha256sum ) > "$DIR/MANIFEST.sha256"
echo "re-pinned $DIR/MANIFEST.sha256 — a deliberate human action (corpora/README.md);"
echo "commit the new manifest with justification."
else
if [ ! -f "$DIR/MANIFEST.sha256" ]; then
echo "no MANIFEST.sha256 — pin the corpus with: sh corpora/cjson/gen-corpus.sh --pin" >&2
exit 1
fi
( cd "$DIR/input" && sha256sum -c ../MANIFEST.sha256 )
fi
+15 -1
View File
@@ -1,7 +1,21 @@
#!/bin/sh
# Regenerate the comrak bench corpus from the pinned progit checkout
# (targets/comrak/target.toml [source.submodules]), then verify the pin.
# Default/--check verifies MANIFEST.sha256 (never rewrites it); --pin
# deliberately rewrites it.
set -eu
mode="check"
case "${1:-}" in
"" | --check) ;;
--pin) mode="pin" ;;
*) echo "usage: gen-corpus.sh [--check|--pin]" >&2; exit 2 ;;
esac
cd "$(dirname "$0")"
cat ../../targets/comrak/workspace/vendor/progit/*/*/*.markdown > progit.md
sha256sum -c MANIFEST.sha256
if [ "$mode" = "pin" ]; then
sha256sum progit.md > MANIFEST.sha256
echo "re-pinned MANIFEST.sha256 — a deliberate human action (corpora/README.md);"
echo "commit the new manifest with justification."
else
sha256sum -c MANIFEST.sha256
fi
+15 -1
View File
@@ -2,12 +2,26 @@
# Regenerate the tokei bench input: 8 replicas of two pinned trees
# (progit @ 61833a5, comrak src/ @ 45c1995), .git removed. Verifies the
# manifest for copy1; copies 2-8 are byte-identical replicas of copy1.
# Default/--check verifies MANIFEST.sha256 (never rewrites it); --pin
# deliberately rewrites it.
set -eu
mode="check"
case "${1:-}" in
"" | --check) ;;
--pin) mode="pin" ;;
*) echo "usage: gen-corpus.sh [--check|--pin]" >&2; exit 2 ;;
esac
cd "$(dirname "$0")"
rm -rf input
mkdir -p input/copy1
cp -r ../../targets/comrak/workspace/vendor/progit input/copy1/progit
cp -r ../../targets/comrak/workspace/src input/copy1/src
rm -rf input/copy1/progit/.git
(cd input/copy1 && sha256sum -c ../../MANIFEST.sha256 --quiet) || { echo "corpus pin FAILED"; exit 1; }
if [ "$mode" = "pin" ]; then
(cd input/copy1 && find . -type f -print0 | sort -z | xargs -0 sha256sum) > MANIFEST.sha256
echo "re-pinned MANIFEST.sha256 — a deliberate human action (corpora/README.md);"
echo "commit the new manifest with justification."
else
(cd input/copy1 && sha256sum -c ../../MANIFEST.sha256 --quiet) || { echo "corpus pin FAILED"; exit 1; }
fi
for i in 2 3 4 5 6 7 8; do cp -r input/copy1 "input/copy$i"; done