mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
gates: close the audit findings — enforce the gates the docs claimed
An adversarial audit found the enforcement story ahead of the code: gates that were documented as hard gates were advisory, manual, or absent, and several human-facing claims outran the machine record. This makes the claims true. High findings - Differential fuzz was documentation, not enforcement: verdict hardcoded fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and it is inherently baseline-vs-candidate, so it runs on the accept path and reports Skipped in the standalone `just gates` flow. New hard rule: a machine `accepted` verdict is impossible unless the fuzz gate actually passed — otherwise it caps at needs-human-review. - The agent's system prompt claimed FP/concurrency/unsafe patches were "routed to human review automatically". No such mechanism existed. Added a conservative lexical risk classifier (crates/verdict/src/risk.rs) that forces needs-human-review on any would-be accept carrying those signals; fp-tolerance equivalence always signals. - The local verdict path ran agent-influenced code unsandboxed. It now runs under scripts/no-net.sh (network namespace) and fails closed. Upstream test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when present (scripts/pin-testsuite.sh generates them). Also fixed, found while fixing the above - The pipeline script's `set -e` was inert (brace group as the left operand of `&&`), so a failing candidate build did not abort the run and verdict could bench a stale binary. Now a scoped subshell whose status is branched on at top level. - A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path) stranded RUNNING.lock and left read_verdict polling forever. Failure accounting now lives outside the wrapper. - harnessd's patch allowlist ignored git rename/copy headers and --no-prefix diffs; the rename-escape case is now a committed regression test. - The workspace did not compile on Windows (ungated Unix APIs). cfg-gated; 71 portable tests now run on both platforms. - pin-check regenerated the corpus manifest instead of verifying it; it is now verify-only, with a deliberate `just pin-corpus` to re-pin. - Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics (compare --perf-stat), never part of the accept decision. Docs now match the code, including what is NOT yet enforced: no target ships a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not wired to the fp-tolerance gate — each is stated as mechanized-but-inert rather than done. Corrected the stale "33 ledger rows" (34), the playbook's class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation claim (five accepted rows carry sanitizers_clean=false), the unsupported "40,000 requests" figure, and the blanket "historical rows record fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
+9
-2
@@ -8,8 +8,15 @@ Every corpus directory carries a `MANIFEST.sha256` (`sha256sum` format:
|
||||
before every run and refuses to run on any mismatch — a tampered corpus
|
||||
is a stop-the-line event.
|
||||
|
||||
Regenerate a manifest only as a deliberate human action:
|
||||
`just pin-check <target>` regenerates the corpus and verifies it against
|
||||
the manifest; it never rewrites the manifest. Regenerate a manifest only
|
||||
as a deliberate human action:
|
||||
|
||||
```sh
|
||||
cd corpora/<target> && find . -type f ! -name MANIFEST.sha256 -exec sha256sum {} + > MANIFEST.sha256
|
||||
just pin-corpus <target> # gen-corpus.sh --pin
|
||||
```
|
||||
|
||||
A directory may also carry a `TESTSUITE.sha256` pinning the upstream test
|
||||
suite (repo-root-relative paths). `diff-test` verifies it before running
|
||||
upstream tests and `targets/fetch.sh` verifies it after every fetch;
|
||||
generate it with `scripts/pin-testsuite.sh <target> <path>...`.
|
||||
|
||||
@@ -4,8 +4,16 @@
|
||||
# with escapes and unicode, ints, floats (the strtod/sprintf hot path),
|
||||
# booleans, nulls. No randomness — byte-identical on every machine, so
|
||||
# the MANIFEST.sha256 pin is reproducible. Run from repo root:
|
||||
# sh corpora/cjson/gen-corpus.sh
|
||||
# sh corpora/cjson/gen-corpus.sh [--check|--pin]
|
||||
# Default/--check: regenerate and verify against MANIFEST.sha256 (never
|
||||
# rewrites it). --pin: deliberately rewrite the manifest.
|
||||
set -eu
|
||||
mode="check"
|
||||
case "${1:-}" in
|
||||
"" | --check) ;;
|
||||
--pin) mode="pin" ;;
|
||||
*) echo "usage: gen-corpus.sh [--check|--pin]" >&2; exit 2 ;;
|
||||
esac
|
||||
DIR="$(cd "$(dirname "$0")" && pwd)"
|
||||
OUT="$DIR/input/big.json"
|
||||
mkdir -p "$DIR/input"
|
||||
@@ -57,6 +65,14 @@ with open(sys.argv[1], "w") as f:
|
||||
PY
|
||||
|
||||
echo "wrote $OUT ($(wc -c < "$OUT") bytes)"
|
||||
# Pin the corpus and regenerate the manifest.
|
||||
( cd "$DIR/input" && find . -type f -print0 | sort -z | xargs -0 sha256sum ) > "$DIR/MANIFEST.sha256"
|
||||
echo "manifest: $DIR/MANIFEST.sha256"
|
||||
if [ "$mode" = "pin" ]; then
|
||||
( cd "$DIR/input" && find . -type f -print0 | sort -z | xargs -0 sha256sum ) > "$DIR/MANIFEST.sha256"
|
||||
echo "re-pinned $DIR/MANIFEST.sha256 — a deliberate human action (corpora/README.md);"
|
||||
echo "commit the new manifest with justification."
|
||||
else
|
||||
if [ ! -f "$DIR/MANIFEST.sha256" ]; then
|
||||
echo "no MANIFEST.sha256 — pin the corpus with: sh corpora/cjson/gen-corpus.sh --pin" >&2
|
||||
exit 1
|
||||
fi
|
||||
( cd "$DIR/input" && sha256sum -c ../MANIFEST.sha256 )
|
||||
fi
|
||||
|
||||
@@ -1,7 +1,21 @@
|
||||
#!/bin/sh
|
||||
# Regenerate the comrak bench corpus from the pinned progit checkout
|
||||
# (targets/comrak/target.toml [source.submodules]), then verify the pin.
|
||||
# Default/--check verifies MANIFEST.sha256 (never rewrites it); --pin
|
||||
# deliberately rewrites it.
|
||||
set -eu
|
||||
mode="check"
|
||||
case "${1:-}" in
|
||||
"" | --check) ;;
|
||||
--pin) mode="pin" ;;
|
||||
*) echo "usage: gen-corpus.sh [--check|--pin]" >&2; exit 2 ;;
|
||||
esac
|
||||
cd "$(dirname "$0")"
|
||||
cat ../../targets/comrak/workspace/vendor/progit/*/*/*.markdown > progit.md
|
||||
sha256sum -c MANIFEST.sha256
|
||||
if [ "$mode" = "pin" ]; then
|
||||
sha256sum progit.md > MANIFEST.sha256
|
||||
echo "re-pinned MANIFEST.sha256 — a deliberate human action (corpora/README.md);"
|
||||
echo "commit the new manifest with justification."
|
||||
else
|
||||
sha256sum -c MANIFEST.sha256
|
||||
fi
|
||||
|
||||
@@ -2,12 +2,26 @@
|
||||
# Regenerate the tokei bench input: 8 replicas of two pinned trees
|
||||
# (progit @ 61833a5, comrak src/ @ 45c1995), .git removed. Verifies the
|
||||
# manifest for copy1; copies 2-8 are byte-identical replicas of copy1.
|
||||
# Default/--check verifies MANIFEST.sha256 (never rewrites it); --pin
|
||||
# deliberately rewrites it.
|
||||
set -eu
|
||||
mode="check"
|
||||
case "${1:-}" in
|
||||
"" | --check) ;;
|
||||
--pin) mode="pin" ;;
|
||||
*) echo "usage: gen-corpus.sh [--check|--pin]" >&2; exit 2 ;;
|
||||
esac
|
||||
cd "$(dirname "$0")"
|
||||
rm -rf input
|
||||
mkdir -p input/copy1
|
||||
cp -r ../../targets/comrak/workspace/vendor/progit input/copy1/progit
|
||||
cp -r ../../targets/comrak/workspace/src input/copy1/src
|
||||
rm -rf input/copy1/progit/.git
|
||||
(cd input/copy1 && sha256sum -c ../../MANIFEST.sha256 --quiet) || { echo "corpus pin FAILED"; exit 1; }
|
||||
if [ "$mode" = "pin" ]; then
|
||||
(cd input/copy1 && find . -type f -print0 | sort -z | xargs -0 sha256sum) > MANIFEST.sha256
|
||||
echo "re-pinned MANIFEST.sha256 — a deliberate human action (corpora/README.md);"
|
||||
echo "commit the new manifest with justification."
|
||||
else
|
||||
(cd input/copy1 && sha256sum -c ../../MANIFEST.sha256 --quiet) || { echo "corpus pin FAILED"; exit 1; }
|
||||
fi
|
||||
for i in 2 3 4 5 6 7 8; do cp -r input/copy1 "input/copy$i"; done
|
||||
|
||||
Reference in New Issue
Block a user