mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
gates: close the audit findings — enforce the gates the docs claimed
An adversarial audit found the enforcement story ahead of the code: gates that were documented as hard gates were advisory, manual, or absent, and several human-facing claims outran the machine record. This makes the claims true. High findings - Differential fuzz was documentation, not enforcement: verdict hardcoded fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and it is inherently baseline-vs-candidate, so it runs on the accept path and reports Skipped in the standalone `just gates` flow. New hard rule: a machine `accepted` verdict is impossible unless the fuzz gate actually passed — otherwise it caps at needs-human-review. - The agent's system prompt claimed FP/concurrency/unsafe patches were "routed to human review automatically". No such mechanism existed. Added a conservative lexical risk classifier (crates/verdict/src/risk.rs) that forces needs-human-review on any would-be accept carrying those signals; fp-tolerance equivalence always signals. - The local verdict path ran agent-influenced code unsandboxed. It now runs under scripts/no-net.sh (network namespace) and fails closed. Upstream test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when present (scripts/pin-testsuite.sh generates them). Also fixed, found while fixing the above - The pipeline script's `set -e` was inert (brace group as the left operand of `&&`), so a failing candidate build did not abort the run and verdict could bench a stale binary. Now a scoped subshell whose status is branched on at top level. - A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path) stranded RUNNING.lock and left read_verdict polling forever. Failure accounting now lives outside the wrapper. - harnessd's patch allowlist ignored git rename/copy headers and --no-prefix diffs; the rename-escape case is now a committed regression test. - The workspace did not compile on Windows (ungated Unix APIs). cfg-gated; 71 portable tests now run on both platforms. - pin-check regenerated the corpus manifest instead of verifying it; it is now verify-only, with a deliberate `just pin-corpus` to re-pin. - Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics (compare --perf-stat), never part of the accept decision. Docs now match the code, including what is NOT yet enforced: no target ships a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not wired to the fp-tolerance gate — each is stated as mechanized-but-inert rather than done. Corrected the stale "33 ledger rows" (34), the playbook's class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation claim (five accepted rows carry sanitizers_clean=false), the unsupported "40,000 requests" figure, and the blanket "historical rows record fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
+114
-24
@@ -1,8 +1,25 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Differential fuzz for phase2-tokei-001: pristine vs candidate tokei
|
||||
"""Differential fuzz for tokei attempts: pristine vs candidate tokei
|
||||
must produce byte-identical output on mutated source inputs targeting
|
||||
the patched code path (quote/comment token scanning).
|
||||
|
||||
python3 scripts/diff-fuzz-tokei.py <workdir> <iters> <baseline-bin> <candidate-bin>
|
||||
|
||||
All four arguments are required and are supplied by the harness: diff-test
|
||||
substitutes {iters}, {baseline} and {candidate} into the target's
|
||||
[gates].fuzz template and runs it from the repo root. Both binary paths may
|
||||
be repo-root-relative OR absolute (the accept path rebuilds the pristine
|
||||
baseline to an absolute path) — relative ones resolve against the repo root.
|
||||
Nothing here may hardcode a baseline location: the harness owns where the
|
||||
pristine build lands, and a hardcoded path is a gate that always hard-fails.
|
||||
|
||||
Final stdout line is the machine contract diff-test parses:
|
||||
FUZZ-RESULT iters=<n> divergences=<m>. Exit 0 iff the run completed with
|
||||
zero divergences, 1 on divergence (the line is still printed, with the real
|
||||
count), 2 on a usage/setup error — which prints no FUZZ-RESULT and is
|
||||
therefore graded a hard gate FAILURE, never a pass. `iters` counts mutated
|
||||
files (tokei scans a directory, so files are batched per invocation).
|
||||
|
||||
Seeds: files from the pinned corpus (many languages). Mutations focus on
|
||||
quote/comment structure: inserting/deleting/duplicating quote chars,
|
||||
comment openers/closers, backslashes, and random bytes; plus boundary
|
||||
@@ -13,13 +30,12 @@ import random
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import NoReturn
|
||||
|
||||
ROOT = Path("/home/user/peltier.io")
|
||||
BASE = ROOT / "targets/tokei/baseline/release/tokei"
|
||||
CAND = ROOT / "targets/tokei/candidate-4e3da7e6bd1c/release/tokei"
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
CORPUS = ROOT / "corpora/tokei/input/copy1"
|
||||
WORK = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("/tmp/difffuzz")
|
||||
ITERS = int(sys.argv[2]) if len(sys.argv) > 2 else 10000
|
||||
|
||||
USAGE = "usage: diff-fuzz-tokei.py <workdir> <iters> <baseline-bin> <candidate-bin>"
|
||||
|
||||
TOKENS = [
|
||||
b'"', b"'", b'`', b'"""', b"'''", b"/*", b"*/", b"//", b"#", b"--",
|
||||
@@ -27,6 +43,60 @@ TOKENS = [
|
||||
b"r#\"", b"\"#", b"=begin", b"=end", b"'''", b'f"""', b"%{", b"}",
|
||||
]
|
||||
|
||||
|
||||
def die(msg: str) -> NoReturn:
|
||||
"""Fatal setup error: exit 2 having printed NO FUZZ-RESULT line. diff-test
|
||||
grades a fuzz command that never reports the line as a hard gate failure,
|
||||
which is the right posture for a gate that could not run at all — it must
|
||||
never be gradeable as a pass."""
|
||||
print(f"diff-fuzz-tokei: {msg}", file=sys.stderr)
|
||||
raise SystemExit(2)
|
||||
|
||||
|
||||
def parse_args(argv: list[str]) -> tuple[Path, int, Path, Path]:
|
||||
"""<workdir> <iters> <baseline-bin> <candidate-bin>, all positional."""
|
||||
if len(argv) != 5:
|
||||
die(f"expected 4 arguments, got {len(argv) - 1}\n{USAGE}")
|
||||
work, raw_iters, baseline, candidate = argv[1:5]
|
||||
if not raw_iters.isdigit():
|
||||
die(f"iters must be a non-negative integer, got {raw_iters!r}\n{USAGE}")
|
||||
iters = int(raw_iters)
|
||||
if iters < 1:
|
||||
# A zero-iteration run would print FUZZ-RESULT iters=0 divergences=0
|
||||
# and exit 0 — a fuzz gate that passes without fuzzing anything.
|
||||
die(f"iters must be >= 1, got {iters}\n{USAGE}")
|
||||
# `ROOT / p` keeps a repo-root-relative path anchored at the repo root and
|
||||
# leaves an absolute path untouched (pathlib drops the left operand when
|
||||
# the right side is absolute). The harness supplies either form.
|
||||
return ROOT / work, iters, ROOT / baseline, ROOT / candidate
|
||||
|
||||
|
||||
def need_binary(path: Path, label: str) -> Path:
|
||||
if not path.is_file():
|
||||
die(
|
||||
f"{label} binary not found: {path}\nthe harness substitutes "
|
||||
"{baseline}/{candidate} in [gates].fuzz — check targets/tokei/target.toml"
|
||||
)
|
||||
return path
|
||||
|
||||
|
||||
def keep_dir(work: Path, tag: str) -> Path:
|
||||
"""First unused divergence-<tag> directory under `work`.
|
||||
|
||||
The RNG is seeded, so re-running in the same work dir re-diverges at the
|
||||
same batch: the kept-divergence directory already exists. The old plain
|
||||
`rename()` raised (the destination is a non-empty directory) and killed
|
||||
the run *before* the FUZZ-RESULT line, so diff-test reported a genuine
|
||||
divergence — the whole point of the gate — as "fuzz command did not
|
||||
report FUZZ-RESULT". This never raises; it picks the next free name.
|
||||
"""
|
||||
path, n = work / f"divergence-{tag}", 2
|
||||
while path.exists():
|
||||
path = work / f"divergence-{tag}-{n}"
|
||||
n += 1
|
||||
return path
|
||||
|
||||
|
||||
def mutate(data: bytes, rng: random.Random) -> bytes:
|
||||
ops = rng.randrange(1, 6)
|
||||
b = bytearray(data)
|
||||
@@ -51,6 +121,7 @@ def mutate(data: bytes, rng: random.Random) -> bytes:
|
||||
b[pos:pos] = b"\n"
|
||||
return bytes(b)
|
||||
|
||||
|
||||
def canonicalize(payload: bytes) -> str:
|
||||
"""Parse tokei JSON and sort every list so benign output-order
|
||||
nondeterminism (parallel walker arrival order) doesn't count as a
|
||||
@@ -78,43 +149,62 @@ def run(binary: Path, d: Path) -> tuple[str, bytes, int]:
|
||||
canon = f"UNPARSEABLE:{e}:{r.stdout[:200]!r}"
|
||||
return canon + f"||rc={r.returncode}", r.stdout, r.returncode
|
||||
|
||||
def main() -> int:
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
work, iters, baseline, candidate = parse_args(argv)
|
||||
need_binary(baseline, "baseline")
|
||||
need_binary(candidate, "candidate")
|
||||
if not CORPUS.is_dir():
|
||||
die(f"corpus not found: {CORPUS} (generate it with corpora/tokei/gen-corpus.sh)")
|
||||
|
||||
rng = random.Random(0xB16B00B5)
|
||||
seeds = sorted(CORPUS.rglob("*"))
|
||||
seeds = [p for p in seeds if p.is_file() and p.stat().st_size < 200_000][:400]
|
||||
if not seeds:
|
||||
print("no seeds found", file=sys.stderr)
|
||||
return 2
|
||||
WORK.mkdir(parents=True, exist_ok=True)
|
||||
fdir = WORK / "case"
|
||||
die(f"no seeds found under {CORPUS}")
|
||||
work.mkdir(parents=True, exist_ok=True)
|
||||
fdir = work / "case"
|
||||
divergences = 0
|
||||
done = 0
|
||||
runs = 0
|
||||
# Batch: mutate BATCH files per tokei invocation (tokei scans a dir),
|
||||
# keeping each seed's extension so language detection stays realistic.
|
||||
BATCH = 50
|
||||
while done < ITERS:
|
||||
while done < iters:
|
||||
if fdir.exists():
|
||||
for f in fdir.iterdir():
|
||||
f.unlink()
|
||||
fdir.mkdir(exist_ok=True)
|
||||
for j in range(min(BATCH, ITERS - done)):
|
||||
batch = min(BATCH, iters - done)
|
||||
for j in range(batch):
|
||||
seed = rng.choice(seeds)
|
||||
name = f"m{done + j}{seed.suffix or '.txt'}"
|
||||
(fdir / name).write_bytes(mutate(seed.read_bytes(), rng))
|
||||
a_canon, a_raw, a_rc = run(BASE, fdir)
|
||||
b_canon, b_raw, b_rc = run(CAND, fdir)
|
||||
a_canon, a_raw, a_rc = run(baseline, fdir)
|
||||
b_canon, b_raw, b_rc = run(candidate, fdir)
|
||||
if a_canon != b_canon:
|
||||
divergences += 1
|
||||
keep = WORK / f"divergence-{done}"
|
||||
fdir.rename(keep)
|
||||
(keep / "_baseline.out").write_bytes(a_raw + f"\nrc={a_rc}".encode())
|
||||
(keep / "_candidate.out").write_bytes(b_raw + f"\nrc={b_rc}".encode())
|
||||
print(f"DIVERGENCE at batch starting {done}; inputs+outputs kept in {keep}")
|
||||
done += BATCH
|
||||
# Keeping the reproducer is best-effort: an I/O failure here must
|
||||
# never cost the FUZZ-RESULT line and turn a real divergence into
|
||||
# an unparseable gate error (see keep_dir). If the rename fails,
|
||||
# fdir survives and the next batch cleans it out.
|
||||
try:
|
||||
keep = keep_dir(work, str(done))
|
||||
fdir.rename(keep)
|
||||
(keep / "_baseline.out").write_bytes(a_raw + f"\nrc={a_rc}".encode())
|
||||
(keep / "_candidate.out").write_bytes(b_raw + f"\nrc={b_rc}".encode())
|
||||
print(f"DIVERGENCE at batch starting {done}; inputs+outputs kept in {keep}")
|
||||
except OSError as e:
|
||||
print(f"DIVERGENCE at batch starting {done}; could not keep artifacts: {e}")
|
||||
done += batch
|
||||
runs += 1
|
||||
if done % 1000 == 0:
|
||||
print(f"{done}/{ITERS} mutated files, {divergences} divergences", flush=True)
|
||||
print(f"done: {done} mutated files across {done // BATCH} runs, {divergences} divergences")
|
||||
print(f"{done}/{iters} mutated files, {divergences} divergences", flush=True)
|
||||
print(f"done: {done} mutated files across {runs} runs, {divergences} divergences")
|
||||
# Machine contract, always the final stdout line on any completing run.
|
||||
print(f"FUZZ-RESULT iters={done} divergences={divergences}", flush=True)
|
||||
return 1 if divergences else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
sys.exit(main(sys.argv))
|
||||
|
||||
Reference in New Issue
Block a user