mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 19:08:27 +00:00
benbrownandClaude Opus 4.8
899d55f4b6
An adversarial audit found the enforcement story ahead of the code: gates that were documented as hard gates were advisory, manual, or absent, and several human-facing claims outran the machine record. This makes the claims true. High findings - Differential fuzz was documentation, not enforcement: verdict hardcoded fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and it is inherently baseline-vs-candidate, so it runs on the accept path and reports Skipped in the standalone `just gates` flow. New hard rule: a machine `accepted` verdict is impossible unless the fuzz gate actually passed — otherwise it caps at needs-human-review. - The agent's system prompt claimed FP/concurrency/unsafe patches were "routed to human review automatically". No such mechanism existed. Added a conservative lexical risk classifier (crates/verdict/src/risk.rs) that forces needs-human-review on any would-be accept carrying those signals; fp-tolerance equivalence always signals. - The local verdict path ran agent-influenced code unsandboxed. It now runs under scripts/no-net.sh (network namespace) and fails closed. Upstream test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when present (scripts/pin-testsuite.sh generates them). Also fixed, found while fixing the above - The pipeline script's `set -e` was inert (brace group as the left operand of `&&`), so a failing candidate build did not abort the run and verdict could bench a stale binary. Now a scoped subshell whose status is branched on at top level. - A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path) stranded RUNNING.lock and left read_verdict polling forever. Failure accounting now lives outside the wrapper. - harnessd's patch allowlist ignored git rename/copy headers and --no-prefix diffs; the rename-escape case is now a committed regression test. - The workspace did not compile on Windows (ungated Unix APIs). cfg-gated; 71 portable tests now run on both platforms. - pin-check regenerated the corpus manifest instead of verifying it; it is now verify-only, with a deliberate `just pin-corpus` to re-pin. - Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics (compare --perf-stat), never part of the accept decision. Docs now match the code, including what is NOT yet enforced: no target ships a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not wired to the fp-tolerance gate — each is stated as mechanized-but-inert rather than done. Corrected the stale "33 ledger rows" (34), the playbook's class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation claim (five accepted rows carry sanitizers_clean=false), the unsupported "40,000 requests" figure, and the blanket "historical rows record fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
37 lines
1.5 KiB
Bash
Executable File
37 lines
1.5 KiB
Bash
Executable File
#!/bin/sh
|
|
# Reproduce a target workspace at its pinned commit (CI + fresh clones).
|
|
# Usage: targets/fetch.sh <name> (run from repo root)
|
|
set -eu
|
|
name="$1"
|
|
toml="targets/$name/target.toml"
|
|
repo=$(grep -m1 '^repo = ' "$toml" | cut -d'"' -f2)
|
|
commit=$(grep -m1 '^commit = ' "$toml" | cut -d'"' -f2)
|
|
ws="targets/$name/workspace"
|
|
if [ ! -d "$ws/.git" ]; then
|
|
git clone "$repo" "$ws"
|
|
fi
|
|
git -C "$ws" fetch -q origin "$commit" 2>/dev/null || git -C "$ws" fetch -q origin
|
|
git -C "$ws" checkout -q "$commit"
|
|
# Pinned submodules (comrak's bench corpus lives in one).
|
|
grep -A0 '^"vendor/' "$toml" | while IFS= read -r line; do
|
|
sub=$(printf '%s' "$line" | cut -d'"' -f2)
|
|
surl=$(printf '%s' "$line" | sed 's/.*repo = "\([^"]*\)".*/\1/')
|
|
scommit=$(printf '%s' "$line" | sed 's/.*commit = "\([^"]*\)".*/\1/')
|
|
[ -d "$ws/$sub/.git" ] || git clone "$surl" "$ws/$sub"
|
|
git -C "$ws/$sub" fetch -q origin "$scommit" 2>/dev/null || true
|
|
git -C "$ws/$sub" checkout -q "$scommit"
|
|
done
|
|
# Upstream test-suite pin: verify against repo root when present; a
|
|
# mismatch means the fetched suite is not the one that was audited.
|
|
suite="corpora/$name/TESTSUITE.sha256"
|
|
if [ -f "$suite" ]; then
|
|
if ! sha256sum -c --quiet "$suite"; then
|
|
echo "fetch: upstream test-suite pin MISMATCH for $name ($suite) — refusing" >&2
|
|
exit 1
|
|
fi
|
|
echo "$name test suite verified against $suite"
|
|
else
|
|
echo "note: no $suite — pin the suite with: scripts/pin-testsuite.sh $name <paths...>"
|
|
fi
|
|
echo "$name pinned at $commit"
|