mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
gates: close the audit findings — enforce the gates the docs claimed
An adversarial audit found the enforcement story ahead of the code: gates that were documented as hard gates were advisory, manual, or absent, and several human-facing claims outran the machine record. This makes the claims true. High findings - Differential fuzz was documentation, not enforcement: verdict hardcoded fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and it is inherently baseline-vs-candidate, so it runs on the accept path and reports Skipped in the standalone `just gates` flow. New hard rule: a machine `accepted` verdict is impossible unless the fuzz gate actually passed — otherwise it caps at needs-human-review. - The agent's system prompt claimed FP/concurrency/unsafe patches were "routed to human review automatically". No such mechanism existed. Added a conservative lexical risk classifier (crates/verdict/src/risk.rs) that forces needs-human-review on any would-be accept carrying those signals; fp-tolerance equivalence always signals. - The local verdict path ran agent-influenced code unsandboxed. It now runs under scripts/no-net.sh (network namespace) and fails closed. Upstream test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when present (scripts/pin-testsuite.sh generates them). Also fixed, found while fixing the above - The pipeline script's `set -e` was inert (brace group as the left operand of `&&`), so a failing candidate build did not abort the run and verdict could bench a stale binary. Now a scoped subshell whose status is branched on at top level. - A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path) stranded RUNNING.lock and left read_verdict polling forever. Failure accounting now lives outside the wrapper. - harnessd's patch allowlist ignored git rename/copy headers and --no-prefix diffs; the rename-escape case is now a committed regression test. - The workspace did not compile on Windows (ungated Unix APIs). cfg-gated; 71 portable tests now run on both platforms. - pin-check regenerated the corpus manifest instead of verifying it; it is now verify-only, with a deliberate `just pin-corpus` to re-pin. - Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics (compare --perf-stat), never part of the accept decision. Docs now match the code, including what is NOT yet enforced: no target ships a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not wired to the fp-tolerance gate — each is stated as mechanized-but-inert rather than done. Corrected the stale "33 ledger rows" (34), the playbook's class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation claim (five accepted rows carry sanitizers_clean=false), the unsupported "40,000 requests" figure, and the blanket "historical rows record fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
Executable
+63
@@ -0,0 +1,63 @@
|
||||
#!/bin/sh
|
||||
# Pin an upstream test suite: hash every file under the given repo-root-
|
||||
# relative paths (typically targets/<t>/workspace/tests/...) into
|
||||
# corpora/<target>/TESTSUITE.sha256, sha256sum two-space format. diff-test
|
||||
# verifies this pin before running upstream tests; targets/fetch.sh
|
||||
# verifies it after every fetch. Run from repo root:
|
||||
# scripts/pin-testsuite.sh <target> <path>...
|
||||
set -eu
|
||||
if [ $# -lt 2 ]; then
|
||||
echo "usage: scripts/pin-testsuite.sh <target> <path>..." >&2
|
||||
exit 2
|
||||
fi
|
||||
target="$1"
|
||||
shift
|
||||
out="corpora/$target/TESTSUITE.sha256"
|
||||
if [ ! -d "corpora/$target" ]; then
|
||||
echo "pin-testsuite: corpora/$target does not exist (run from repo root?)" >&2
|
||||
exit 1
|
||||
fi
|
||||
for p in "$@"; do
|
||||
if [ ! -e "$p" ]; then
|
||||
echo "pin-testsuite: $p not found (paths are repo-root-relative)" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
# Build the pin out-of-place and install it only on success: a run that fails
|
||||
# anywhere below must not truncate, replace, or half-write an existing pin —
|
||||
# that pin is what diff-test refuses to run against on mismatch.
|
||||
list="$out.files.$$"
|
||||
tmp="$out.new.$$"
|
||||
trap 'rm -f "$list" "$tmp"' EXIT INT TERM
|
||||
|
||||
# LC_ALL=C: byte-order sort, so the same tree pins identically on every
|
||||
# machine. -print0/-z: filenames with spaces or newlines survive intact.
|
||||
find "$@" -type f -print0 | LC_ALL=C sort -z > "$list"
|
||||
count=$(tr -dc '\0' < "$list" | wc -c | tr -d ' ')
|
||||
if [ "$count" -eq 0 ]; then
|
||||
# Was: `find ... | xargs -0 sha256sum > "$out"` — with no matches, xargs
|
||||
# still ran sha256sum with zero file arguments, so it hashed STDIN (the
|
||||
# empty pipe) and wrote a bogus "<sha-of-empty> -" entry, then reported
|
||||
# SUCCESS. That is a pin of nothing that later "verifies" by reading
|
||||
# stdin. Refuse instead, and leave any existing pin untouched.
|
||||
echo "pin-testsuite: no regular files under: $* — refusing to write $out" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# --text is GNU's default on Linux (a no-op there); it is spelled out because
|
||||
# sha256sum defaults to *binary* mode on some platforms, emitting "<hash> *path"
|
||||
# — a one-space-plus-asterisk form that diff-test's pin verifier (which splits
|
||||
# on the two-space separator) does not parse. Paths stay repo-root-relative:
|
||||
# find was handed repo-root-relative paths and we run from the repo root.
|
||||
xargs -0 sha256sum --text -- < "$list" > "$tmp"
|
||||
|
||||
lines=$(wc -l < "$tmp" | tr -d ' ')
|
||||
if [ "$lines" -ne "$count" ]; then
|
||||
echo "pin-testsuite: hashed $lines of $count files — refusing to write a partial pin" >&2
|
||||
exit 1
|
||||
fi
|
||||
mv "$tmp" "$out"
|
||||
echo "wrote $out ($count files)"
|
||||
echo "pinning a test suite is a deliberate human action (corpora/README.md):"
|
||||
echo "commit the new pin with justification."
|
||||
Reference in New Issue
Block a user