gates: close the audit findings — enforce the gates the docs claimed

An adversarial audit found the enforcement story ahead of the code: gates
that were documented as hard gates were advisory, manual, or absent, and
several human-facing claims outran the machine record. This makes the
claims true.

High findings

- Differential fuzz was documentation, not enforcement: verdict hardcoded
  fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case
  studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real
  pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and
  it is inherently baseline-vs-candidate, so it runs on the accept path and
  reports Skipped in the standalone `just gates` flow. New hard rule: a
  machine `accepted` verdict is impossible unless the fuzz gate actually
  passed — otherwise it caps at needs-human-review.
- The agent's system prompt claimed FP/concurrency/unsafe patches were
  "routed to human review automatically". No such mechanism existed. Added
  a conservative lexical risk classifier (crates/verdict/src/risk.rs) that
  forces needs-human-review on any would-be accept carrying those signals;
  fp-tolerance equivalence always signals.
- The local verdict path ran agent-influenced code unsandboxed. It now runs
  under scripts/no-net.sh (network namespace) and fails closed. Upstream
  test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when
  present (scripts/pin-testsuite.sh generates them).

Also fixed, found while fixing the above

- The pipeline script's `set -e` was inert (brace group as the left operand
  of `&&`), so a failing candidate build did not abort the run and verdict
  could bench a stale binary. Now a scoped subshell whose status is branched
  on at top level.
- A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path)
  stranded RUNNING.lock and left read_verdict polling forever. Failure
  accounting now lives outside the wrapper.
- harnessd's patch allowlist ignored git rename/copy headers and --no-prefix
  diffs; the rename-escape case is now a committed regression test.
- The workspace did not compile on Windows (ungated Unix APIs). cfg-gated;
  71 portable tests now run on both platforms.
- pin-check regenerated the corpus manifest instead of verifying it; it is
  now verify-only, with a deliberate `just pin-corpus` to re-pin.
- Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics
  (compare --perf-stat), never part of the accept decision.

Docs now match the code, including what is NOT yet enforced: no target ships
a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not
wired to the fp-tolerance gate — each is stated as mechanized-but-inert
rather than done. Corrected the stale "33 ledger rows" (34), the playbook's
class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation
claim (five accepted rows carry sanitizers_clean=false), the unsupported
"40,000 requests" figure, and the blanket "historical rows record
fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-14 13:10:14 -04:00
co-authored by Claude Opus 4.8
parent 21352efe6c
commit 899d55f4b6
57 changed files with 3308 additions and 692 deletions
+107 -23
View File
@@ -4,7 +4,21 @@ produce byte-identical stdout (serialization + checksum) AND identical exit
code on mutated JSON. cJSON is a deterministic single-threaded parser, so
raw byte comparison needs no canonicalization.
python3 scripts/diff-fuzz-cjson.py <workdir> <iters> <candidate-bin>
python3 scripts/diff-fuzz-cjson.py <workdir> <iters> <baseline-bin> <candidate-bin>
All four arguments are required and are supplied by the harness: diff-test
substitutes {iters}, {baseline} and {candidate} into the target's
[gates].fuzz template and runs it from the repo root. Both binary paths may
be repo-root-relative OR absolute (the accept path rebuilds the pristine
baseline to an absolute path) — relative ones resolve against the repo root.
Nothing here may hardcode a baseline location: the harness owns where the
pristine build lands, and a hardcoded path is a gate that always hard-fails.
Final stdout line is the machine contract diff-test parses:
FUZZ-RESULT iters=<n> divergences=<m>. Exit 0 iff the run completed with
zero divergences, 1 on divergence (the line is still printed, with the real
count), 2 on a usage/setup error — which prints no FUZZ-RESULT and is
therefore graded a hard gate FAILURE, never a pass.
Seeds: slices of the pinned corpus plus hand JSON fragments. Mutations
target parser edges: structural tokens, string escapes, number forms
@@ -15,13 +29,12 @@ import random
import subprocess
import sys
from pathlib import Path
from typing import NoReturn
ROOT = Path("/home/user/peltier.io")
BASE = ROOT / "targets/cjson/baseline/cjson-bench"
CAND = ROOT / sys.argv[3] if len(sys.argv) > 3 else ROOT / "targets/cjson/baseline/cjson-bench"
ROOT = Path(__file__).resolve().parents[1]
CORPUS = ROOT / "corpora/cjson/input/big.json"
WORK = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("/tmp/difffuzz-cjson")
ITERS = int(sys.argv[2]) if len(sys.argv) > 2 else 10000
USAGE = "usage: diff-fuzz-cjson.py <workdir> <iters> <baseline-bin> <candidate-bin>"
TOKENS = [
b"{", b"}", b"[", b"]", b'"', b":", b",", b"\\", b"\\n", b"\\u00e9", b"\\uD83D",
@@ -38,6 +51,59 @@ SEEDS_STATIC = [
]
def die(msg: str) -> NoReturn:
"""Fatal setup error: exit 2 having printed NO FUZZ-RESULT line. diff-test
grades a fuzz command that never reports the line as a hard gate failure,
which is the right posture for a gate that could not run at all — it must
never be gradeable as a pass."""
print(f"diff-fuzz-cjson: {msg}", file=sys.stderr)
raise SystemExit(2)
def parse_args(argv: list[str]) -> tuple[Path, int, Path, Path]:
"""<workdir> <iters> <baseline-bin> <candidate-bin>, all positional."""
if len(argv) != 5:
die(f"expected 4 arguments, got {len(argv) - 1}\n{USAGE}")
work, raw_iters, baseline, candidate = argv[1:5]
if not raw_iters.isdigit():
die(f"iters must be a non-negative integer, got {raw_iters!r}\n{USAGE}")
iters = int(raw_iters)
if iters < 1:
# A zero-iteration run would print FUZZ-RESULT iters=0 divergences=0
# and exit 0 — a fuzz gate that passes without fuzzing anything.
die(f"iters must be >= 1, got {iters}\n{USAGE}")
# `ROOT / p` keeps a repo-root-relative path anchored at the repo root and
# leaves an absolute path untouched (pathlib drops the left operand when
# the right side is absolute). The harness supplies either form.
return ROOT / work, iters, ROOT / baseline, ROOT / candidate
def need_binary(path: Path, label: str) -> Path:
if not path.is_file():
die(
f"{label} binary not found: {path}\nthe harness substitutes "
"{baseline}/{candidate} in [gates].fuzz — check targets/cjson/target.toml"
)
return path
def keep_dir(work: Path, tag: str) -> Path:
"""First unused divergence-<tag> directory under `work`.
The RNG is seeded, so re-running in the same work dir re-diverges at the
same iteration: the kept-divergence directory already exists. The old
plain `mkdir()` raised FileExistsError there and killed the run *before*
the FUZZ-RESULT line, so diff-test reported a genuine divergence — the
whole point of the gate — as "fuzz command did not report FUZZ-RESULT".
This never raises; it picks the next free name instead.
"""
path, n = work / f"divergence-{tag}", 2
while path.exists():
path = work / f"divergence-{tag}-{n}"
n += 1
return path
def mutate(data: bytes, rng: random.Random) -> bytes:
b = bytearray(data)
for _ in range(rng.randrange(1, 8)):
@@ -65,36 +131,54 @@ def run(binary: Path, path: Path) -> tuple[bytes, int]:
return r.stdout, r.returncode
def main() -> int:
def main(argv: list[str]) -> int:
work, iters, baseline, candidate = parse_args(argv)
need_binary(baseline, "baseline")
need_binary(candidate, "candidate")
if not CORPUS.is_file():
die(f"corpus not found: {CORPUS} (generate it with corpora/cjson/gen-corpus.sh)")
rng = random.Random(0x1CE_B00C)
corpus = CORPUS.read_bytes()
seeds = list(SEEDS_STATIC)
# windows of the real corpus (often invalid mid-slice — exercises the
# error path, which must diverge identically or not at all)
for o in range(0, len(corpus) - 4096, len(corpus) // 200):
# error path, which must diverge identically or not at all). The max(1, …)
# guards are no-ops on the pinned 5.6 MB corpus (identical seed set, so
# the fuzz stays reproducible) and stop a degenerate tiny corpus from
# raising a zero-step ValueError before FUZZ-RESULT is ever printed.
for o in range(0, max(1, len(corpus) - 4096), max(1, len(corpus) // 200)):
for n in (64, 512, 4096):
seeds.append(corpus[o : o + n])
WORK.mkdir(parents=True, exist_ok=True)
case = WORK / "case.json"
work.mkdir(parents=True, exist_ok=True)
case = work / "case.json"
divergences = 0
for i in range(ITERS):
for i in range(iters):
payload = mutate(rng.choice(seeds), rng)
case.write_bytes(payload)
a_out, a_rc = run(BASE, case)
b_out, b_rc = run(CAND, case)
a_out, a_rc = run(baseline, case)
b_out, b_rc = run(candidate, case)
if a_out != b_out or a_rc != b_rc:
divergences += 1
keep = WORK / f"divergence-{i}"
keep.mkdir(exist_ok=True)
(keep / "input.json").write_bytes(payload)
(keep / "baseline.out").write_bytes(a_out + f"\nrc={a_rc}".encode())
(keep / "candidate.out").write_bytes(b_out + f"\nrc={b_rc}".encode())
print(f"DIVERGENCE at iter {i} (rc {a_rc} vs {b_rc}); kept in {keep}")
# Keeping the reproducer is best-effort: an I/O failure here must
# never cost the FUZZ-RESULT line and turn a real divergence into
# an unparseable gate error (see keep_dir).
try:
keep = keep_dir(work, str(i))
keep.mkdir(parents=True, exist_ok=True)
(keep / "input.json").write_bytes(payload)
(keep / "baseline.out").write_bytes(a_out + f"\nrc={a_rc}".encode())
(keep / "candidate.out").write_bytes(b_out + f"\nrc={b_rc}".encode())
print(f"DIVERGENCE at iter {i} (rc {a_rc} vs {b_rc}); kept in {keep}")
except OSError as e:
print(f"DIVERGENCE at iter {i} (rc {a_rc} vs {b_rc}); "
f"could not keep artifacts: {e}")
if (i + 1) % 1000 == 0:
print(f"{i + 1}/{ITERS}, {divergences} divergences", flush=True)
print(f"done: {ITERS} inputs, {divergences} divergences")
print(f"{i + 1}/{iters}, {divergences} divergences", flush=True)
print(f"done: {iters} inputs, {divergences} divergences")
# Machine contract, always the final stdout line on any completing run.
print(f"FUZZ-RESULT iters={iters} divergences={divergences}", flush=True)
return 1 if divergences else 0
if __name__ == "__main__":
sys.exit(main())
sys.exit(main(sys.argv))
+108 -22
View File
@@ -4,7 +4,21 @@ produce byte-identical HTML (stdout), stderr, and exit code on mutated
markdown. comrak is a single-threaded stdin->stdout filter, so raw byte
comparison needs no canonicalization.
python3 scripts/diff-fuzz-comrak.py <workdir> <iters> <candidate-bin>
python3 scripts/diff-fuzz-comrak.py <workdir> <iters> <baseline-bin> <candidate-bin>
All four arguments are required and are supplied by the harness: diff-test
substitutes {iters}, {baseline} and {candidate} into the target's
[gates].fuzz template and runs it from the repo root. Both binary paths may
be repo-root-relative OR absolute (the accept path rebuilds the pristine
baseline to an absolute path) — relative ones resolve against the repo root.
Nothing here may hardcode a baseline location: the harness owns where the
pristine build lands, and a hardcoded path is a gate that always hard-fails.
Final stdout line is the machine contract diff-test parses:
FUZZ-RESULT iters=<n> divergences=<m>. Exit 0 iff the run completed with
zero divergences, 1 on divergence (the line is still printed, with the real
count), 2 on a usage/setup error — which prints no FUZZ-RESULT and is
therefore graded a hard gate FAILURE, never a pass.
Seeds: windows of the pinned Pro Git corpus. Mutations focus on inline/
block structure the parser and teardown paths exercise: emphasis, code
@@ -15,13 +29,12 @@ import random
import subprocess
import sys
from pathlib import Path
from typing import NoReturn
ROOT = Path("/home/user/peltier.io")
BASE = ROOT / "targets/comrak/baseline/release/comrak"
CAND = ROOT / sys.argv[3]
ROOT = Path(__file__).resolve().parents[1]
CORPUS = ROOT / "corpora/comrak/progit.md"
WORK = Path(sys.argv[1])
ITERS = int(sys.argv[2])
USAGE = "usage: diff-fuzz-comrak.py <workdir> <iters> <baseline-bin> <candidate-bin>"
TOKENS = [
b"**", b"*", b"__", b"_", b"`", b"```", b"~~~", b"~~", b"[", b"](", b")",
@@ -30,6 +43,59 @@ TOKENS = [
]
def die(msg: str) -> NoReturn:
"""Fatal setup error: exit 2 having printed NO FUZZ-RESULT line. diff-test
grades a fuzz command that never reports the line as a hard gate failure,
which is the right posture for a gate that could not run at all — it must
never be gradeable as a pass."""
print(f"diff-fuzz-comrak: {msg}", file=sys.stderr)
raise SystemExit(2)
def parse_args(argv: list[str]) -> tuple[Path, int, Path, Path]:
"""<workdir> <iters> <baseline-bin> <candidate-bin>, all positional."""
if len(argv) != 5:
die(f"expected 4 arguments, got {len(argv) - 1}\n{USAGE}")
work, raw_iters, baseline, candidate = argv[1:5]
if not raw_iters.isdigit():
die(f"iters must be a non-negative integer, got {raw_iters!r}\n{USAGE}")
iters = int(raw_iters)
if iters < 1:
# A zero-iteration run would print FUZZ-RESULT iters=0 divergences=0
# and exit 0 — a fuzz gate that passes without fuzzing anything.
die(f"iters must be >= 1, got {iters}\n{USAGE}")
# `ROOT / p` keeps a repo-root-relative path anchored at the repo root and
# leaves an absolute path untouched (pathlib drops the left operand when
# the right side is absolute). The harness supplies either form.
return ROOT / work, iters, ROOT / baseline, ROOT / candidate
def need_binary(path: Path, label: str) -> Path:
if not path.is_file():
die(
f"{label} binary not found: {path}\nthe harness substitutes "
"{baseline}/{candidate} in [gates].fuzz — check targets/comrak/target.toml"
)
return path
def keep_dir(work: Path, tag: str) -> Path:
"""First unused divergence-<tag> directory under `work`.
The RNG is seeded, so re-running in the same work dir re-diverges at the
same iteration: the kept-divergence directory already exists. The old
plain `mkdir()` raised FileExistsError there and killed the run *before*
the FUZZ-RESULT line, so diff-test reported a genuine divergence — the
whole point of the gate — as "fuzz command did not report FUZZ-RESULT".
This never raises; it picks the next free name instead.
"""
path, n = work / f"divergence-{tag}", 2
while path.exists():
path = work / f"divergence-{tag}-{n}"
n += 1
return path
def mutate(data: bytes, rng: random.Random) -> bytes:
b = bytearray(data)
for _ in range(rng.randrange(1, 8)):
@@ -57,34 +123,54 @@ def run(binary: Path, payload: bytes) -> tuple[bytes, bytes, int]:
return r.stdout, r.stderr, r.returncode
def main() -> int:
def main(argv: list[str]) -> int:
work, iters, baseline, candidate = parse_args(argv)
need_binary(baseline, "baseline")
need_binary(candidate, "candidate")
if not CORPUS.is_file():
die(f"corpus not found: {CORPUS} (generate it with corpora/comrak/gen-corpus.sh)")
rng = random.Random(0xC0FFEE)
corpus = CORPUS.read_bytes()
# Seed windows across the whole book, varied sizes.
# Seed windows across the whole book, varied sizes. The max(1, …) guards
# are no-ops on the pinned 11 MB corpus (identical seed set, so the fuzz
# stays reproducible); they stop a degenerate tiny corpus from raising a
# zero-step ValueError, or yielding no seeds at all, before FUZZ-RESULT
# is ever printed.
seeds = [
corpus[o : o + n]
for o in range(0, len(corpus) - 65536, len(corpus) // 300)
for o in range(0, max(1, len(corpus) - 65536), max(1, len(corpus) // 300))
for n in (512, 4096, 32768)
]
WORK.mkdir(parents=True, exist_ok=True)
if not seeds:
die(f"corpus is empty, no seeds to fuzz from: {CORPUS}")
work.mkdir(parents=True, exist_ok=True)
divergences = 0
for i in range(ITERS):
for i in range(iters):
payload = mutate(rng.choice(seeds), rng)
a = run(BASE, payload)
b = run(CAND, payload)
a = run(baseline, payload)
b = run(candidate, payload)
if a != b:
divergences += 1
keep = WORK / f"divergence-{i}"
keep.mkdir()
(keep / "input.md").write_bytes(payload)
(keep / "baseline.out").write_bytes(a[0] + b"\n--stderr--\n" + a[1] + f"\nrc={a[2]}".encode())
(keep / "candidate.out").write_bytes(b[0] + b"\n--stderr--\n" + b[1] + f"\nrc={b[2]}".encode())
print(f"DIVERGENCE at iter {i}; kept in {keep}")
# Keeping the reproducer is best-effort: an I/O failure here must
# never cost the FUZZ-RESULT line and turn a real divergence into
# an unparseable gate error (see keep_dir).
try:
keep = keep_dir(work, str(i))
keep.mkdir(parents=True, exist_ok=True)
(keep / "input.md").write_bytes(payload)
(keep / "baseline.out").write_bytes(a[0] + b"\n--stderr--\n" + a[1] + f"\nrc={a[2]}".encode())
(keep / "candidate.out").write_bytes(b[0] + b"\n--stderr--\n" + b[1] + f"\nrc={b[2]}".encode())
print(f"DIVERGENCE at iter {i}; kept in {keep}")
except OSError as e:
print(f"DIVERGENCE at iter {i}; could not keep artifacts: {e}")
if (i + 1) % 1000 == 0:
print(f"{i + 1}/{ITERS}, {divergences} divergences", flush=True)
print(f"done: {ITERS} inputs, {divergences} divergences")
print(f"{i + 1}/{iters}, {divergences} divergences", flush=True)
print(f"done: {iters} inputs, {divergences} divergences")
# Machine contract, always the final stdout line on any completing run.
print(f"FUZZ-RESULT iters={iters} divergences={divergences}", flush=True)
return 1 if divergences else 0
if __name__ == "__main__":
sys.exit(main())
sys.exit(main(sys.argv))
+114 -24
View File
@@ -1,8 +1,25 @@
#!/usr/bin/env python3
"""Differential fuzz for phase2-tokei-001: pristine vs candidate tokei
"""Differential fuzz for tokei attempts: pristine vs candidate tokei
must produce byte-identical output on mutated source inputs targeting
the patched code path (quote/comment token scanning).
python3 scripts/diff-fuzz-tokei.py <workdir> <iters> <baseline-bin> <candidate-bin>
All four arguments are required and are supplied by the harness: diff-test
substitutes {iters}, {baseline} and {candidate} into the target's
[gates].fuzz template and runs it from the repo root. Both binary paths may
be repo-root-relative OR absolute (the accept path rebuilds the pristine
baseline to an absolute path) — relative ones resolve against the repo root.
Nothing here may hardcode a baseline location: the harness owns where the
pristine build lands, and a hardcoded path is a gate that always hard-fails.
Final stdout line is the machine contract diff-test parses:
FUZZ-RESULT iters=<n> divergences=<m>. Exit 0 iff the run completed with
zero divergences, 1 on divergence (the line is still printed, with the real
count), 2 on a usage/setup error — which prints no FUZZ-RESULT and is
therefore graded a hard gate FAILURE, never a pass. `iters` counts mutated
files (tokei scans a directory, so files are batched per invocation).
Seeds: files from the pinned corpus (many languages). Mutations focus on
quote/comment structure: inserting/deleting/duplicating quote chars,
comment openers/closers, backslashes, and random bytes; plus boundary
@@ -13,13 +30,12 @@ import random
import subprocess
import sys
from pathlib import Path
from typing import NoReturn
ROOT = Path("/home/user/peltier.io")
BASE = ROOT / "targets/tokei/baseline/release/tokei"
CAND = ROOT / "targets/tokei/candidate-4e3da7e6bd1c/release/tokei"
ROOT = Path(__file__).resolve().parents[1]
CORPUS = ROOT / "corpora/tokei/input/copy1"
WORK = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("/tmp/difffuzz")
ITERS = int(sys.argv[2]) if len(sys.argv) > 2 else 10000
USAGE = "usage: diff-fuzz-tokei.py <workdir> <iters> <baseline-bin> <candidate-bin>"
TOKENS = [
b'"', b"'", b'`', b'"""', b"'''", b"/*", b"*/", b"//", b"#", b"--",
@@ -27,6 +43,60 @@ TOKENS = [
b"r#\"", b"\"#", b"=begin", b"=end", b"'''", b'f"""', b"%{", b"}",
]
def die(msg: str) -> NoReturn:
"""Fatal setup error: exit 2 having printed NO FUZZ-RESULT line. diff-test
grades a fuzz command that never reports the line as a hard gate failure,
which is the right posture for a gate that could not run at all — it must
never be gradeable as a pass."""
print(f"diff-fuzz-tokei: {msg}", file=sys.stderr)
raise SystemExit(2)
def parse_args(argv: list[str]) -> tuple[Path, int, Path, Path]:
"""<workdir> <iters> <baseline-bin> <candidate-bin>, all positional."""
if len(argv) != 5:
die(f"expected 4 arguments, got {len(argv) - 1}\n{USAGE}")
work, raw_iters, baseline, candidate = argv[1:5]
if not raw_iters.isdigit():
die(f"iters must be a non-negative integer, got {raw_iters!r}\n{USAGE}")
iters = int(raw_iters)
if iters < 1:
# A zero-iteration run would print FUZZ-RESULT iters=0 divergences=0
# and exit 0 — a fuzz gate that passes without fuzzing anything.
die(f"iters must be >= 1, got {iters}\n{USAGE}")
# `ROOT / p` keeps a repo-root-relative path anchored at the repo root and
# leaves an absolute path untouched (pathlib drops the left operand when
# the right side is absolute). The harness supplies either form.
return ROOT / work, iters, ROOT / baseline, ROOT / candidate
def need_binary(path: Path, label: str) -> Path:
if not path.is_file():
die(
f"{label} binary not found: {path}\nthe harness substitutes "
"{baseline}/{candidate} in [gates].fuzz — check targets/tokei/target.toml"
)
return path
def keep_dir(work: Path, tag: str) -> Path:
"""First unused divergence-<tag> directory under `work`.
The RNG is seeded, so re-running in the same work dir re-diverges at the
same batch: the kept-divergence directory already exists. The old plain
`rename()` raised (the destination is a non-empty directory) and killed
the run *before* the FUZZ-RESULT line, so diff-test reported a genuine
divergence — the whole point of the gate — as "fuzz command did not
report FUZZ-RESULT". This never raises; it picks the next free name.
"""
path, n = work / f"divergence-{tag}", 2
while path.exists():
path = work / f"divergence-{tag}-{n}"
n += 1
return path
def mutate(data: bytes, rng: random.Random) -> bytes:
ops = rng.randrange(1, 6)
b = bytearray(data)
@@ -51,6 +121,7 @@ def mutate(data: bytes, rng: random.Random) -> bytes:
b[pos:pos] = b"\n"
return bytes(b)
def canonicalize(payload: bytes) -> str:
"""Parse tokei JSON and sort every list so benign output-order
nondeterminism (parallel walker arrival order) doesn't count as a
@@ -78,43 +149,62 @@ def run(binary: Path, d: Path) -> tuple[str, bytes, int]:
canon = f"UNPARSEABLE:{e}:{r.stdout[:200]!r}"
return canon + f"||rc={r.returncode}", r.stdout, r.returncode
def main() -> int:
def main(argv: list[str]) -> int:
work, iters, baseline, candidate = parse_args(argv)
need_binary(baseline, "baseline")
need_binary(candidate, "candidate")
if not CORPUS.is_dir():
die(f"corpus not found: {CORPUS} (generate it with corpora/tokei/gen-corpus.sh)")
rng = random.Random(0xB16B00B5)
seeds = sorted(CORPUS.rglob("*"))
seeds = [p for p in seeds if p.is_file() and p.stat().st_size < 200_000][:400]
if not seeds:
print("no seeds found", file=sys.stderr)
return 2
WORK.mkdir(parents=True, exist_ok=True)
fdir = WORK / "case"
die(f"no seeds found under {CORPUS}")
work.mkdir(parents=True, exist_ok=True)
fdir = work / "case"
divergences = 0
done = 0
runs = 0
# Batch: mutate BATCH files per tokei invocation (tokei scans a dir),
# keeping each seed's extension so language detection stays realistic.
BATCH = 50
while done < ITERS:
while done < iters:
if fdir.exists():
for f in fdir.iterdir():
f.unlink()
fdir.mkdir(exist_ok=True)
for j in range(min(BATCH, ITERS - done)):
batch = min(BATCH, iters - done)
for j in range(batch):
seed = rng.choice(seeds)
name = f"m{done + j}{seed.suffix or '.txt'}"
(fdir / name).write_bytes(mutate(seed.read_bytes(), rng))
a_canon, a_raw, a_rc = run(BASE, fdir)
b_canon, b_raw, b_rc = run(CAND, fdir)
a_canon, a_raw, a_rc = run(baseline, fdir)
b_canon, b_raw, b_rc = run(candidate, fdir)
if a_canon != b_canon:
divergences += 1
keep = WORK / f"divergence-{done}"
fdir.rename(keep)
(keep / "_baseline.out").write_bytes(a_raw + f"\nrc={a_rc}".encode())
(keep / "_candidate.out").write_bytes(b_raw + f"\nrc={b_rc}".encode())
print(f"DIVERGENCE at batch starting {done}; inputs+outputs kept in {keep}")
done += BATCH
# Keeping the reproducer is best-effort: an I/O failure here must
# never cost the FUZZ-RESULT line and turn a real divergence into
# an unparseable gate error (see keep_dir). If the rename fails,
# fdir survives and the next batch cleans it out.
try:
keep = keep_dir(work, str(done))
fdir.rename(keep)
(keep / "_baseline.out").write_bytes(a_raw + f"\nrc={a_rc}".encode())
(keep / "_candidate.out").write_bytes(b_raw + f"\nrc={b_rc}".encode())
print(f"DIVERGENCE at batch starting {done}; inputs+outputs kept in {keep}")
except OSError as e:
print(f"DIVERGENCE at batch starting {done}; could not keep artifacts: {e}")
done += batch
runs += 1
if done % 1000 == 0:
print(f"{done}/{ITERS} mutated files, {divergences} divergences", flush=True)
print(f"done: {done} mutated files across {done // BATCH} runs, {divergences} divergences")
print(f"{done}/{iters} mutated files, {divergences} divergences", flush=True)
print(f"done: {done} mutated files across {runs} runs, {divergences} divergences")
# Machine contract, always the final stdout line on any completing run.
print(f"FUZZ-RESULT iters={done} divergences={divergences}", flush=True)
return 1 if divergences else 0
if __name__ == "__main__":
sys.exit(main())
sys.exit(main(sys.argv))
+25
View File
@@ -0,0 +1,25 @@
#!/bin/sh
# Network-isolation wrapper for the local verdict path (SPEC §10): exec the
# given command inside a fresh network namespace so target code built and
# run by the pipeline cannot reach the network. harnessd invokes it as
# `no-net.sh sh -c <pipeline-script>` (HOTPATH_VERDICT_WRAPPER overrides).
# CI gets the same guarantee from `docker --network=none` instead.
set -eu
if [ $# -eq 0 ]; then
echo "usage: scripts/no-net.sh <command> [args...]" >&2
exit 2
fi
if unshare --net --map-current-user true 2>/dev/null; then
exec unshare --net --map-current-user -- "$@"
fi
if [ "${HOTPATH_ALLOW_UNISOLATED:-}" = "1" ]; then
echo "no-net.sh: WARNING: unshare --net unavailable and HOTPATH_ALLOW_UNISOLATED=1 —" >&2
echo "no-net.sh: WARNING: running WITHOUT network isolation. Target code can reach the network." >&2
exec "$@"
fi
echo "no-net.sh: cannot create a network namespace (unshare --net --map-current-user failed)." >&2
echo "no-net.sh: refusing to run target code unisolated. Options:" >&2
echo "no-net.sh: - run on a host with unprivileged user namespaces enabled" >&2
echo "no-net.sh: - in CI, run the whole pipeline under docker --network=none" >&2
echo "no-net.sh: - set HOTPATH_ALLOW_UNISOLATED=1 to override (loud warning, not for accepts)" >&2
exit 97
+63
View File
@@ -0,0 +1,63 @@
#!/bin/sh
# Pin an upstream test suite: hash every file under the given repo-root-
# relative paths (typically targets/<t>/workspace/tests/...) into
# corpora/<target>/TESTSUITE.sha256, sha256sum two-space format. diff-test
# verifies this pin before running upstream tests; targets/fetch.sh
# verifies it after every fetch. Run from repo root:
# scripts/pin-testsuite.sh <target> <path>...
set -eu
if [ $# -lt 2 ]; then
echo "usage: scripts/pin-testsuite.sh <target> <path>..." >&2
exit 2
fi
target="$1"
shift
out="corpora/$target/TESTSUITE.sha256"
if [ ! -d "corpora/$target" ]; then
echo "pin-testsuite: corpora/$target does not exist (run from repo root?)" >&2
exit 1
fi
for p in "$@"; do
if [ ! -e "$p" ]; then
echo "pin-testsuite: $p not found (paths are repo-root-relative)" >&2
exit 1
fi
done
# Build the pin out-of-place and install it only on success: a run that fails
# anywhere below must not truncate, replace, or half-write an existing pin —
# that pin is what diff-test refuses to run against on mismatch.
list="$out.files.$$"
tmp="$out.new.$$"
trap 'rm -f "$list" "$tmp"' EXIT INT TERM
# LC_ALL=C: byte-order sort, so the same tree pins identically on every
# machine. -print0/-z: filenames with spaces or newlines survive intact.
find "$@" -type f -print0 | LC_ALL=C sort -z > "$list"
count=$(tr -dc '\0' < "$list" | wc -c | tr -d ' ')
if [ "$count" -eq 0 ]; then
# Was: `find ... | xargs -0 sha256sum > "$out"` — with no matches, xargs
# still ran sha256sum with zero file arguments, so it hashed STDIN (the
# empty pipe) and wrote a bogus "<sha-of-empty> -" entry, then reported
# SUCCESS. That is a pin of nothing that later "verifies" by reading
# stdin. Refuse instead, and leave any existing pin untouched.
echo "pin-testsuite: no regular files under: $* — refusing to write $out" >&2
exit 1
fi
# --text is GNU's default on Linux (a no-op there); it is spelled out because
# sha256sum defaults to *binary* mode on some platforms, emitting "<hash> *path"
# — a one-space-plus-asterisk form that diff-test's pin verifier (which splits
# on the two-space separator) does not parse. Paths stay repo-root-relative:
# find was handed repo-root-relative paths and we run from the repo root.
xargs -0 sha256sum --text -- < "$list" > "$tmp"
lines=$(wc -l < "$tmp" | tr -d ' ')
if [ "$lines" -ne "$count" ]; then
echo "pin-testsuite: hashed $lines of $count files — refusing to write a partial pin" >&2
exit 1
fi
mv "$tmp" "$out"
echo "wrote $out ($count files)"
echo "pinning a test suite is a deliberate human action (corpora/README.md):"
echo "commit the new pin with justification."