mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
gates: close the audit findings — enforce the gates the docs claimed
An adversarial audit found the enforcement story ahead of the code: gates that were documented as hard gates were advisory, manual, or absent, and several human-facing claims outran the machine record. This makes the claims true. High findings - Differential fuzz was documentation, not enforcement: verdict hardcoded fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and it is inherently baseline-vs-candidate, so it runs on the accept path and reports Skipped in the standalone `just gates` flow. New hard rule: a machine `accepted` verdict is impossible unless the fuzz gate actually passed — otherwise it caps at needs-human-review. - The agent's system prompt claimed FP/concurrency/unsafe patches were "routed to human review automatically". No such mechanism existed. Added a conservative lexical risk classifier (crates/verdict/src/risk.rs) that forces needs-human-review on any would-be accept carrying those signals; fp-tolerance equivalence always signals. - The local verdict path ran agent-influenced code unsandboxed. It now runs under scripts/no-net.sh (network namespace) and fails closed. Upstream test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when present (scripts/pin-testsuite.sh generates them). Also fixed, found while fixing the above - The pipeline script's `set -e` was inert (brace group as the left operand of `&&`), so a failing candidate build did not abort the run and verdict could bench a stale binary. Now a scoped subshell whose status is branched on at top level. - A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path) stranded RUNNING.lock and left read_verdict polling forever. Failure accounting now lives outside the wrapper. - harnessd's patch allowlist ignored git rename/copy headers and --no-prefix diffs; the rename-escape case is now a committed regression test. - The workspace did not compile on Windows (ungated Unix APIs). cfg-gated; 71 portable tests now run on both platforms. - pin-check regenerated the corpus manifest instead of verifying it; it is now verify-only, with a deliberate `just pin-corpus` to re-pin. - Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics (compare --perf-stat), never part of the accept decision. Docs now match the code, including what is NOT yet enforced: no target ships a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not wired to the fp-tolerance gate — each is stated as mechanized-but-inert rather than done. Corrected the stale "33 ledger rows" (34), the playbook's class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation claim (five accepted rows carry sanitizers_clean=false), the unsupported "40,000 requests" figure, and the blanket "historical rows record fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
+107
-23
@@ -4,7 +4,21 @@ produce byte-identical stdout (serialization + checksum) AND identical exit
|
||||
code on mutated JSON. cJSON is a deterministic single-threaded parser, so
|
||||
raw byte comparison needs no canonicalization.
|
||||
|
||||
python3 scripts/diff-fuzz-cjson.py <workdir> <iters> <candidate-bin>
|
||||
python3 scripts/diff-fuzz-cjson.py <workdir> <iters> <baseline-bin> <candidate-bin>
|
||||
|
||||
All four arguments are required and are supplied by the harness: diff-test
|
||||
substitutes {iters}, {baseline} and {candidate} into the target's
|
||||
[gates].fuzz template and runs it from the repo root. Both binary paths may
|
||||
be repo-root-relative OR absolute (the accept path rebuilds the pristine
|
||||
baseline to an absolute path) — relative ones resolve against the repo root.
|
||||
Nothing here may hardcode a baseline location: the harness owns where the
|
||||
pristine build lands, and a hardcoded path is a gate that always hard-fails.
|
||||
|
||||
Final stdout line is the machine contract diff-test parses:
|
||||
FUZZ-RESULT iters=<n> divergences=<m>. Exit 0 iff the run completed with
|
||||
zero divergences, 1 on divergence (the line is still printed, with the real
|
||||
count), 2 on a usage/setup error — which prints no FUZZ-RESULT and is
|
||||
therefore graded a hard gate FAILURE, never a pass.
|
||||
|
||||
Seeds: slices of the pinned corpus plus hand JSON fragments. Mutations
|
||||
target parser edges: structural tokens, string escapes, number forms
|
||||
@@ -15,13 +29,12 @@ import random
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import NoReturn
|
||||
|
||||
ROOT = Path("/home/user/peltier.io")
|
||||
BASE = ROOT / "targets/cjson/baseline/cjson-bench"
|
||||
CAND = ROOT / sys.argv[3] if len(sys.argv) > 3 else ROOT / "targets/cjson/baseline/cjson-bench"
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
CORPUS = ROOT / "corpora/cjson/input/big.json"
|
||||
WORK = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("/tmp/difffuzz-cjson")
|
||||
ITERS = int(sys.argv[2]) if len(sys.argv) > 2 else 10000
|
||||
|
||||
USAGE = "usage: diff-fuzz-cjson.py <workdir> <iters> <baseline-bin> <candidate-bin>"
|
||||
|
||||
TOKENS = [
|
||||
b"{", b"}", b"[", b"]", b'"', b":", b",", b"\\", b"\\n", b"\\u00e9", b"\\uD83D",
|
||||
@@ -38,6 +51,59 @@ SEEDS_STATIC = [
|
||||
]
|
||||
|
||||
|
||||
def die(msg: str) -> NoReturn:
|
||||
"""Fatal setup error: exit 2 having printed NO FUZZ-RESULT line. diff-test
|
||||
grades a fuzz command that never reports the line as a hard gate failure,
|
||||
which is the right posture for a gate that could not run at all — it must
|
||||
never be gradeable as a pass."""
|
||||
print(f"diff-fuzz-cjson: {msg}", file=sys.stderr)
|
||||
raise SystemExit(2)
|
||||
|
||||
|
||||
def parse_args(argv: list[str]) -> tuple[Path, int, Path, Path]:
|
||||
"""<workdir> <iters> <baseline-bin> <candidate-bin>, all positional."""
|
||||
if len(argv) != 5:
|
||||
die(f"expected 4 arguments, got {len(argv) - 1}\n{USAGE}")
|
||||
work, raw_iters, baseline, candidate = argv[1:5]
|
||||
if not raw_iters.isdigit():
|
||||
die(f"iters must be a non-negative integer, got {raw_iters!r}\n{USAGE}")
|
||||
iters = int(raw_iters)
|
||||
if iters < 1:
|
||||
# A zero-iteration run would print FUZZ-RESULT iters=0 divergences=0
|
||||
# and exit 0 — a fuzz gate that passes without fuzzing anything.
|
||||
die(f"iters must be >= 1, got {iters}\n{USAGE}")
|
||||
# `ROOT / p` keeps a repo-root-relative path anchored at the repo root and
|
||||
# leaves an absolute path untouched (pathlib drops the left operand when
|
||||
# the right side is absolute). The harness supplies either form.
|
||||
return ROOT / work, iters, ROOT / baseline, ROOT / candidate
|
||||
|
||||
|
||||
def need_binary(path: Path, label: str) -> Path:
|
||||
if not path.is_file():
|
||||
die(
|
||||
f"{label} binary not found: {path}\nthe harness substitutes "
|
||||
"{baseline}/{candidate} in [gates].fuzz — check targets/cjson/target.toml"
|
||||
)
|
||||
return path
|
||||
|
||||
|
||||
def keep_dir(work: Path, tag: str) -> Path:
|
||||
"""First unused divergence-<tag> directory under `work`.
|
||||
|
||||
The RNG is seeded, so re-running in the same work dir re-diverges at the
|
||||
same iteration: the kept-divergence directory already exists. The old
|
||||
plain `mkdir()` raised FileExistsError there and killed the run *before*
|
||||
the FUZZ-RESULT line, so diff-test reported a genuine divergence — the
|
||||
whole point of the gate — as "fuzz command did not report FUZZ-RESULT".
|
||||
This never raises; it picks the next free name instead.
|
||||
"""
|
||||
path, n = work / f"divergence-{tag}", 2
|
||||
while path.exists():
|
||||
path = work / f"divergence-{tag}-{n}"
|
||||
n += 1
|
||||
return path
|
||||
|
||||
|
||||
def mutate(data: bytes, rng: random.Random) -> bytes:
|
||||
b = bytearray(data)
|
||||
for _ in range(rng.randrange(1, 8)):
|
||||
@@ -65,36 +131,54 @@ def run(binary: Path, path: Path) -> tuple[bytes, int]:
|
||||
return r.stdout, r.returncode
|
||||
|
||||
|
||||
def main() -> int:
|
||||
def main(argv: list[str]) -> int:
|
||||
work, iters, baseline, candidate = parse_args(argv)
|
||||
need_binary(baseline, "baseline")
|
||||
need_binary(candidate, "candidate")
|
||||
if not CORPUS.is_file():
|
||||
die(f"corpus not found: {CORPUS} (generate it with corpora/cjson/gen-corpus.sh)")
|
||||
|
||||
rng = random.Random(0x1CE_B00C)
|
||||
corpus = CORPUS.read_bytes()
|
||||
seeds = list(SEEDS_STATIC)
|
||||
# windows of the real corpus (often invalid mid-slice — exercises the
|
||||
# error path, which must diverge identically or not at all)
|
||||
for o in range(0, len(corpus) - 4096, len(corpus) // 200):
|
||||
# error path, which must diverge identically or not at all). The max(1, …)
|
||||
# guards are no-ops on the pinned 5.6 MB corpus (identical seed set, so
|
||||
# the fuzz stays reproducible) and stop a degenerate tiny corpus from
|
||||
# raising a zero-step ValueError before FUZZ-RESULT is ever printed.
|
||||
for o in range(0, max(1, len(corpus) - 4096), max(1, len(corpus) // 200)):
|
||||
for n in (64, 512, 4096):
|
||||
seeds.append(corpus[o : o + n])
|
||||
WORK.mkdir(parents=True, exist_ok=True)
|
||||
case = WORK / "case.json"
|
||||
work.mkdir(parents=True, exist_ok=True)
|
||||
case = work / "case.json"
|
||||
divergences = 0
|
||||
for i in range(ITERS):
|
||||
for i in range(iters):
|
||||
payload = mutate(rng.choice(seeds), rng)
|
||||
case.write_bytes(payload)
|
||||
a_out, a_rc = run(BASE, case)
|
||||
b_out, b_rc = run(CAND, case)
|
||||
a_out, a_rc = run(baseline, case)
|
||||
b_out, b_rc = run(candidate, case)
|
||||
if a_out != b_out or a_rc != b_rc:
|
||||
divergences += 1
|
||||
keep = WORK / f"divergence-{i}"
|
||||
keep.mkdir(exist_ok=True)
|
||||
(keep / "input.json").write_bytes(payload)
|
||||
(keep / "baseline.out").write_bytes(a_out + f"\nrc={a_rc}".encode())
|
||||
(keep / "candidate.out").write_bytes(b_out + f"\nrc={b_rc}".encode())
|
||||
print(f"DIVERGENCE at iter {i} (rc {a_rc} vs {b_rc}); kept in {keep}")
|
||||
# Keeping the reproducer is best-effort: an I/O failure here must
|
||||
# never cost the FUZZ-RESULT line and turn a real divergence into
|
||||
# an unparseable gate error (see keep_dir).
|
||||
try:
|
||||
keep = keep_dir(work, str(i))
|
||||
keep.mkdir(parents=True, exist_ok=True)
|
||||
(keep / "input.json").write_bytes(payload)
|
||||
(keep / "baseline.out").write_bytes(a_out + f"\nrc={a_rc}".encode())
|
||||
(keep / "candidate.out").write_bytes(b_out + f"\nrc={b_rc}".encode())
|
||||
print(f"DIVERGENCE at iter {i} (rc {a_rc} vs {b_rc}); kept in {keep}")
|
||||
except OSError as e:
|
||||
print(f"DIVERGENCE at iter {i} (rc {a_rc} vs {b_rc}); "
|
||||
f"could not keep artifacts: {e}")
|
||||
if (i + 1) % 1000 == 0:
|
||||
print(f"{i + 1}/{ITERS}, {divergences} divergences", flush=True)
|
||||
print(f"done: {ITERS} inputs, {divergences} divergences")
|
||||
print(f"{i + 1}/{iters}, {divergences} divergences", flush=True)
|
||||
print(f"done: {iters} inputs, {divergences} divergences")
|
||||
# Machine contract, always the final stdout line on any completing run.
|
||||
print(f"FUZZ-RESULT iters={iters} divergences={divergences}", flush=True)
|
||||
return 1 if divergences else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
sys.exit(main(sys.argv))
|
||||
|
||||
+108
-22
@@ -4,7 +4,21 @@ produce byte-identical HTML (stdout), stderr, and exit code on mutated
|
||||
markdown. comrak is a single-threaded stdin->stdout filter, so raw byte
|
||||
comparison needs no canonicalization.
|
||||
|
||||
python3 scripts/diff-fuzz-comrak.py <workdir> <iters> <candidate-bin>
|
||||
python3 scripts/diff-fuzz-comrak.py <workdir> <iters> <baseline-bin> <candidate-bin>
|
||||
|
||||
All four arguments are required and are supplied by the harness: diff-test
|
||||
substitutes {iters}, {baseline} and {candidate} into the target's
|
||||
[gates].fuzz template and runs it from the repo root. Both binary paths may
|
||||
be repo-root-relative OR absolute (the accept path rebuilds the pristine
|
||||
baseline to an absolute path) — relative ones resolve against the repo root.
|
||||
Nothing here may hardcode a baseline location: the harness owns where the
|
||||
pristine build lands, and a hardcoded path is a gate that always hard-fails.
|
||||
|
||||
Final stdout line is the machine contract diff-test parses:
|
||||
FUZZ-RESULT iters=<n> divergences=<m>. Exit 0 iff the run completed with
|
||||
zero divergences, 1 on divergence (the line is still printed, with the real
|
||||
count), 2 on a usage/setup error — which prints no FUZZ-RESULT and is
|
||||
therefore graded a hard gate FAILURE, never a pass.
|
||||
|
||||
Seeds: windows of the pinned Pro Git corpus. Mutations focus on inline/
|
||||
block structure the parser and teardown paths exercise: emphasis, code
|
||||
@@ -15,13 +29,12 @@ import random
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import NoReturn
|
||||
|
||||
ROOT = Path("/home/user/peltier.io")
|
||||
BASE = ROOT / "targets/comrak/baseline/release/comrak"
|
||||
CAND = ROOT / sys.argv[3]
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
CORPUS = ROOT / "corpora/comrak/progit.md"
|
||||
WORK = Path(sys.argv[1])
|
||||
ITERS = int(sys.argv[2])
|
||||
|
||||
USAGE = "usage: diff-fuzz-comrak.py <workdir> <iters> <baseline-bin> <candidate-bin>"
|
||||
|
||||
TOKENS = [
|
||||
b"**", b"*", b"__", b"_", b"`", b"```", b"~~~", b"~~", b"[", b"](", b")",
|
||||
@@ -30,6 +43,59 @@ TOKENS = [
|
||||
]
|
||||
|
||||
|
||||
def die(msg: str) -> NoReturn:
|
||||
"""Fatal setup error: exit 2 having printed NO FUZZ-RESULT line. diff-test
|
||||
grades a fuzz command that never reports the line as a hard gate failure,
|
||||
which is the right posture for a gate that could not run at all — it must
|
||||
never be gradeable as a pass."""
|
||||
print(f"diff-fuzz-comrak: {msg}", file=sys.stderr)
|
||||
raise SystemExit(2)
|
||||
|
||||
|
||||
def parse_args(argv: list[str]) -> tuple[Path, int, Path, Path]:
|
||||
"""<workdir> <iters> <baseline-bin> <candidate-bin>, all positional."""
|
||||
if len(argv) != 5:
|
||||
die(f"expected 4 arguments, got {len(argv) - 1}\n{USAGE}")
|
||||
work, raw_iters, baseline, candidate = argv[1:5]
|
||||
if not raw_iters.isdigit():
|
||||
die(f"iters must be a non-negative integer, got {raw_iters!r}\n{USAGE}")
|
||||
iters = int(raw_iters)
|
||||
if iters < 1:
|
||||
# A zero-iteration run would print FUZZ-RESULT iters=0 divergences=0
|
||||
# and exit 0 — a fuzz gate that passes without fuzzing anything.
|
||||
die(f"iters must be >= 1, got {iters}\n{USAGE}")
|
||||
# `ROOT / p` keeps a repo-root-relative path anchored at the repo root and
|
||||
# leaves an absolute path untouched (pathlib drops the left operand when
|
||||
# the right side is absolute). The harness supplies either form.
|
||||
return ROOT / work, iters, ROOT / baseline, ROOT / candidate
|
||||
|
||||
|
||||
def need_binary(path: Path, label: str) -> Path:
|
||||
if not path.is_file():
|
||||
die(
|
||||
f"{label} binary not found: {path}\nthe harness substitutes "
|
||||
"{baseline}/{candidate} in [gates].fuzz — check targets/comrak/target.toml"
|
||||
)
|
||||
return path
|
||||
|
||||
|
||||
def keep_dir(work: Path, tag: str) -> Path:
|
||||
"""First unused divergence-<tag> directory under `work`.
|
||||
|
||||
The RNG is seeded, so re-running in the same work dir re-diverges at the
|
||||
same iteration: the kept-divergence directory already exists. The old
|
||||
plain `mkdir()` raised FileExistsError there and killed the run *before*
|
||||
the FUZZ-RESULT line, so diff-test reported a genuine divergence — the
|
||||
whole point of the gate — as "fuzz command did not report FUZZ-RESULT".
|
||||
This never raises; it picks the next free name instead.
|
||||
"""
|
||||
path, n = work / f"divergence-{tag}", 2
|
||||
while path.exists():
|
||||
path = work / f"divergence-{tag}-{n}"
|
||||
n += 1
|
||||
return path
|
||||
|
||||
|
||||
def mutate(data: bytes, rng: random.Random) -> bytes:
|
||||
b = bytearray(data)
|
||||
for _ in range(rng.randrange(1, 8)):
|
||||
@@ -57,34 +123,54 @@ def run(binary: Path, payload: bytes) -> tuple[bytes, bytes, int]:
|
||||
return r.stdout, r.stderr, r.returncode
|
||||
|
||||
|
||||
def main() -> int:
|
||||
def main(argv: list[str]) -> int:
|
||||
work, iters, baseline, candidate = parse_args(argv)
|
||||
need_binary(baseline, "baseline")
|
||||
need_binary(candidate, "candidate")
|
||||
if not CORPUS.is_file():
|
||||
die(f"corpus not found: {CORPUS} (generate it with corpora/comrak/gen-corpus.sh)")
|
||||
|
||||
rng = random.Random(0xC0FFEE)
|
||||
corpus = CORPUS.read_bytes()
|
||||
# Seed windows across the whole book, varied sizes.
|
||||
# Seed windows across the whole book, varied sizes. The max(1, …) guards
|
||||
# are no-ops on the pinned 11 MB corpus (identical seed set, so the fuzz
|
||||
# stays reproducible); they stop a degenerate tiny corpus from raising a
|
||||
# zero-step ValueError, or yielding no seeds at all, before FUZZ-RESULT
|
||||
# is ever printed.
|
||||
seeds = [
|
||||
corpus[o : o + n]
|
||||
for o in range(0, len(corpus) - 65536, len(corpus) // 300)
|
||||
for o in range(0, max(1, len(corpus) - 65536), max(1, len(corpus) // 300))
|
||||
for n in (512, 4096, 32768)
|
||||
]
|
||||
WORK.mkdir(parents=True, exist_ok=True)
|
||||
if not seeds:
|
||||
die(f"corpus is empty, no seeds to fuzz from: {CORPUS}")
|
||||
work.mkdir(parents=True, exist_ok=True)
|
||||
divergences = 0
|
||||
for i in range(ITERS):
|
||||
for i in range(iters):
|
||||
payload = mutate(rng.choice(seeds), rng)
|
||||
a = run(BASE, payload)
|
||||
b = run(CAND, payload)
|
||||
a = run(baseline, payload)
|
||||
b = run(candidate, payload)
|
||||
if a != b:
|
||||
divergences += 1
|
||||
keep = WORK / f"divergence-{i}"
|
||||
keep.mkdir()
|
||||
(keep / "input.md").write_bytes(payload)
|
||||
(keep / "baseline.out").write_bytes(a[0] + b"\n--stderr--\n" + a[1] + f"\nrc={a[2]}".encode())
|
||||
(keep / "candidate.out").write_bytes(b[0] + b"\n--stderr--\n" + b[1] + f"\nrc={b[2]}".encode())
|
||||
print(f"DIVERGENCE at iter {i}; kept in {keep}")
|
||||
# Keeping the reproducer is best-effort: an I/O failure here must
|
||||
# never cost the FUZZ-RESULT line and turn a real divergence into
|
||||
# an unparseable gate error (see keep_dir).
|
||||
try:
|
||||
keep = keep_dir(work, str(i))
|
||||
keep.mkdir(parents=True, exist_ok=True)
|
||||
(keep / "input.md").write_bytes(payload)
|
||||
(keep / "baseline.out").write_bytes(a[0] + b"\n--stderr--\n" + a[1] + f"\nrc={a[2]}".encode())
|
||||
(keep / "candidate.out").write_bytes(b[0] + b"\n--stderr--\n" + b[1] + f"\nrc={b[2]}".encode())
|
||||
print(f"DIVERGENCE at iter {i}; kept in {keep}")
|
||||
except OSError as e:
|
||||
print(f"DIVERGENCE at iter {i}; could not keep artifacts: {e}")
|
||||
if (i + 1) % 1000 == 0:
|
||||
print(f"{i + 1}/{ITERS}, {divergences} divergences", flush=True)
|
||||
print(f"done: {ITERS} inputs, {divergences} divergences")
|
||||
print(f"{i + 1}/{iters}, {divergences} divergences", flush=True)
|
||||
print(f"done: {iters} inputs, {divergences} divergences")
|
||||
# Machine contract, always the final stdout line on any completing run.
|
||||
print(f"FUZZ-RESULT iters={iters} divergences={divergences}", flush=True)
|
||||
return 1 if divergences else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
sys.exit(main(sys.argv))
|
||||
|
||||
+114
-24
@@ -1,8 +1,25 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Differential fuzz for phase2-tokei-001: pristine vs candidate tokei
|
||||
"""Differential fuzz for tokei attempts: pristine vs candidate tokei
|
||||
must produce byte-identical output on mutated source inputs targeting
|
||||
the patched code path (quote/comment token scanning).
|
||||
|
||||
python3 scripts/diff-fuzz-tokei.py <workdir> <iters> <baseline-bin> <candidate-bin>
|
||||
|
||||
All four arguments are required and are supplied by the harness: diff-test
|
||||
substitutes {iters}, {baseline} and {candidate} into the target's
|
||||
[gates].fuzz template and runs it from the repo root. Both binary paths may
|
||||
be repo-root-relative OR absolute (the accept path rebuilds the pristine
|
||||
baseline to an absolute path) — relative ones resolve against the repo root.
|
||||
Nothing here may hardcode a baseline location: the harness owns where the
|
||||
pristine build lands, and a hardcoded path is a gate that always hard-fails.
|
||||
|
||||
Final stdout line is the machine contract diff-test parses:
|
||||
FUZZ-RESULT iters=<n> divergences=<m>. Exit 0 iff the run completed with
|
||||
zero divergences, 1 on divergence (the line is still printed, with the real
|
||||
count), 2 on a usage/setup error — which prints no FUZZ-RESULT and is
|
||||
therefore graded a hard gate FAILURE, never a pass. `iters` counts mutated
|
||||
files (tokei scans a directory, so files are batched per invocation).
|
||||
|
||||
Seeds: files from the pinned corpus (many languages). Mutations focus on
|
||||
quote/comment structure: inserting/deleting/duplicating quote chars,
|
||||
comment openers/closers, backslashes, and random bytes; plus boundary
|
||||
@@ -13,13 +30,12 @@ import random
|
||||
import subprocess
|
||||
import sys
|
||||
from pathlib import Path
|
||||
from typing import NoReturn
|
||||
|
||||
ROOT = Path("/home/user/peltier.io")
|
||||
BASE = ROOT / "targets/tokei/baseline/release/tokei"
|
||||
CAND = ROOT / "targets/tokei/candidate-4e3da7e6bd1c/release/tokei"
|
||||
ROOT = Path(__file__).resolve().parents[1]
|
||||
CORPUS = ROOT / "corpora/tokei/input/copy1"
|
||||
WORK = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("/tmp/difffuzz")
|
||||
ITERS = int(sys.argv[2]) if len(sys.argv) > 2 else 10000
|
||||
|
||||
USAGE = "usage: diff-fuzz-tokei.py <workdir> <iters> <baseline-bin> <candidate-bin>"
|
||||
|
||||
TOKENS = [
|
||||
b'"', b"'", b'`', b'"""', b"'''", b"/*", b"*/", b"//", b"#", b"--",
|
||||
@@ -27,6 +43,60 @@ TOKENS = [
|
||||
b"r#\"", b"\"#", b"=begin", b"=end", b"'''", b'f"""', b"%{", b"}",
|
||||
]
|
||||
|
||||
|
||||
def die(msg: str) -> NoReturn:
|
||||
"""Fatal setup error: exit 2 having printed NO FUZZ-RESULT line. diff-test
|
||||
grades a fuzz command that never reports the line as a hard gate failure,
|
||||
which is the right posture for a gate that could not run at all — it must
|
||||
never be gradeable as a pass."""
|
||||
print(f"diff-fuzz-tokei: {msg}", file=sys.stderr)
|
||||
raise SystemExit(2)
|
||||
|
||||
|
||||
def parse_args(argv: list[str]) -> tuple[Path, int, Path, Path]:
|
||||
"""<workdir> <iters> <baseline-bin> <candidate-bin>, all positional."""
|
||||
if len(argv) != 5:
|
||||
die(f"expected 4 arguments, got {len(argv) - 1}\n{USAGE}")
|
||||
work, raw_iters, baseline, candidate = argv[1:5]
|
||||
if not raw_iters.isdigit():
|
||||
die(f"iters must be a non-negative integer, got {raw_iters!r}\n{USAGE}")
|
||||
iters = int(raw_iters)
|
||||
if iters < 1:
|
||||
# A zero-iteration run would print FUZZ-RESULT iters=0 divergences=0
|
||||
# and exit 0 — a fuzz gate that passes without fuzzing anything.
|
||||
die(f"iters must be >= 1, got {iters}\n{USAGE}")
|
||||
# `ROOT / p` keeps a repo-root-relative path anchored at the repo root and
|
||||
# leaves an absolute path untouched (pathlib drops the left operand when
|
||||
# the right side is absolute). The harness supplies either form.
|
||||
return ROOT / work, iters, ROOT / baseline, ROOT / candidate
|
||||
|
||||
|
||||
def need_binary(path: Path, label: str) -> Path:
|
||||
if not path.is_file():
|
||||
die(
|
||||
f"{label} binary not found: {path}\nthe harness substitutes "
|
||||
"{baseline}/{candidate} in [gates].fuzz — check targets/tokei/target.toml"
|
||||
)
|
||||
return path
|
||||
|
||||
|
||||
def keep_dir(work: Path, tag: str) -> Path:
|
||||
"""First unused divergence-<tag> directory under `work`.
|
||||
|
||||
The RNG is seeded, so re-running in the same work dir re-diverges at the
|
||||
same batch: the kept-divergence directory already exists. The old plain
|
||||
`rename()` raised (the destination is a non-empty directory) and killed
|
||||
the run *before* the FUZZ-RESULT line, so diff-test reported a genuine
|
||||
divergence — the whole point of the gate — as "fuzz command did not
|
||||
report FUZZ-RESULT". This never raises; it picks the next free name.
|
||||
"""
|
||||
path, n = work / f"divergence-{tag}", 2
|
||||
while path.exists():
|
||||
path = work / f"divergence-{tag}-{n}"
|
||||
n += 1
|
||||
return path
|
||||
|
||||
|
||||
def mutate(data: bytes, rng: random.Random) -> bytes:
|
||||
ops = rng.randrange(1, 6)
|
||||
b = bytearray(data)
|
||||
@@ -51,6 +121,7 @@ def mutate(data: bytes, rng: random.Random) -> bytes:
|
||||
b[pos:pos] = b"\n"
|
||||
return bytes(b)
|
||||
|
||||
|
||||
def canonicalize(payload: bytes) -> str:
|
||||
"""Parse tokei JSON and sort every list so benign output-order
|
||||
nondeterminism (parallel walker arrival order) doesn't count as a
|
||||
@@ -78,43 +149,62 @@ def run(binary: Path, d: Path) -> tuple[str, bytes, int]:
|
||||
canon = f"UNPARSEABLE:{e}:{r.stdout[:200]!r}"
|
||||
return canon + f"||rc={r.returncode}", r.stdout, r.returncode
|
||||
|
||||
def main() -> int:
|
||||
|
||||
def main(argv: list[str]) -> int:
|
||||
work, iters, baseline, candidate = parse_args(argv)
|
||||
need_binary(baseline, "baseline")
|
||||
need_binary(candidate, "candidate")
|
||||
if not CORPUS.is_dir():
|
||||
die(f"corpus not found: {CORPUS} (generate it with corpora/tokei/gen-corpus.sh)")
|
||||
|
||||
rng = random.Random(0xB16B00B5)
|
||||
seeds = sorted(CORPUS.rglob("*"))
|
||||
seeds = [p for p in seeds if p.is_file() and p.stat().st_size < 200_000][:400]
|
||||
if not seeds:
|
||||
print("no seeds found", file=sys.stderr)
|
||||
return 2
|
||||
WORK.mkdir(parents=True, exist_ok=True)
|
||||
fdir = WORK / "case"
|
||||
die(f"no seeds found under {CORPUS}")
|
||||
work.mkdir(parents=True, exist_ok=True)
|
||||
fdir = work / "case"
|
||||
divergences = 0
|
||||
done = 0
|
||||
runs = 0
|
||||
# Batch: mutate BATCH files per tokei invocation (tokei scans a dir),
|
||||
# keeping each seed's extension so language detection stays realistic.
|
||||
BATCH = 50
|
||||
while done < ITERS:
|
||||
while done < iters:
|
||||
if fdir.exists():
|
||||
for f in fdir.iterdir():
|
||||
f.unlink()
|
||||
fdir.mkdir(exist_ok=True)
|
||||
for j in range(min(BATCH, ITERS - done)):
|
||||
batch = min(BATCH, iters - done)
|
||||
for j in range(batch):
|
||||
seed = rng.choice(seeds)
|
||||
name = f"m{done + j}{seed.suffix or '.txt'}"
|
||||
(fdir / name).write_bytes(mutate(seed.read_bytes(), rng))
|
||||
a_canon, a_raw, a_rc = run(BASE, fdir)
|
||||
b_canon, b_raw, b_rc = run(CAND, fdir)
|
||||
a_canon, a_raw, a_rc = run(baseline, fdir)
|
||||
b_canon, b_raw, b_rc = run(candidate, fdir)
|
||||
if a_canon != b_canon:
|
||||
divergences += 1
|
||||
keep = WORK / f"divergence-{done}"
|
||||
fdir.rename(keep)
|
||||
(keep / "_baseline.out").write_bytes(a_raw + f"\nrc={a_rc}".encode())
|
||||
(keep / "_candidate.out").write_bytes(b_raw + f"\nrc={b_rc}".encode())
|
||||
print(f"DIVERGENCE at batch starting {done}; inputs+outputs kept in {keep}")
|
||||
done += BATCH
|
||||
# Keeping the reproducer is best-effort: an I/O failure here must
|
||||
# never cost the FUZZ-RESULT line and turn a real divergence into
|
||||
# an unparseable gate error (see keep_dir). If the rename fails,
|
||||
# fdir survives and the next batch cleans it out.
|
||||
try:
|
||||
keep = keep_dir(work, str(done))
|
||||
fdir.rename(keep)
|
||||
(keep / "_baseline.out").write_bytes(a_raw + f"\nrc={a_rc}".encode())
|
||||
(keep / "_candidate.out").write_bytes(b_raw + f"\nrc={b_rc}".encode())
|
||||
print(f"DIVERGENCE at batch starting {done}; inputs+outputs kept in {keep}")
|
||||
except OSError as e:
|
||||
print(f"DIVERGENCE at batch starting {done}; could not keep artifacts: {e}")
|
||||
done += batch
|
||||
runs += 1
|
||||
if done % 1000 == 0:
|
||||
print(f"{done}/{ITERS} mutated files, {divergences} divergences", flush=True)
|
||||
print(f"done: {done} mutated files across {done // BATCH} runs, {divergences} divergences")
|
||||
print(f"{done}/{iters} mutated files, {divergences} divergences", flush=True)
|
||||
print(f"done: {done} mutated files across {runs} runs, {divergences} divergences")
|
||||
# Machine contract, always the final stdout line on any completing run.
|
||||
print(f"FUZZ-RESULT iters={done} divergences={divergences}", flush=True)
|
||||
return 1 if divergences else 0
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
sys.exit(main())
|
||||
sys.exit(main(sys.argv))
|
||||
|
||||
Executable
+25
@@ -0,0 +1,25 @@
|
||||
#!/bin/sh
|
||||
# Network-isolation wrapper for the local verdict path (SPEC §10): exec the
|
||||
# given command inside a fresh network namespace so target code built and
|
||||
# run by the pipeline cannot reach the network. harnessd invokes it as
|
||||
# `no-net.sh sh -c <pipeline-script>` (HOTPATH_VERDICT_WRAPPER overrides).
|
||||
# CI gets the same guarantee from `docker --network=none` instead.
|
||||
set -eu
|
||||
if [ $# -eq 0 ]; then
|
||||
echo "usage: scripts/no-net.sh <command> [args...]" >&2
|
||||
exit 2
|
||||
fi
|
||||
if unshare --net --map-current-user true 2>/dev/null; then
|
||||
exec unshare --net --map-current-user -- "$@"
|
||||
fi
|
||||
if [ "${HOTPATH_ALLOW_UNISOLATED:-}" = "1" ]; then
|
||||
echo "no-net.sh: WARNING: unshare --net unavailable and HOTPATH_ALLOW_UNISOLATED=1 —" >&2
|
||||
echo "no-net.sh: WARNING: running WITHOUT network isolation. Target code can reach the network." >&2
|
||||
exec "$@"
|
||||
fi
|
||||
echo "no-net.sh: cannot create a network namespace (unshare --net --map-current-user failed)." >&2
|
||||
echo "no-net.sh: refusing to run target code unisolated. Options:" >&2
|
||||
echo "no-net.sh: - run on a host with unprivileged user namespaces enabled" >&2
|
||||
echo "no-net.sh: - in CI, run the whole pipeline under docker --network=none" >&2
|
||||
echo "no-net.sh: - set HOTPATH_ALLOW_UNISOLATED=1 to override (loud warning, not for accepts)" >&2
|
||||
exit 97
|
||||
Executable
+63
@@ -0,0 +1,63 @@
|
||||
#!/bin/sh
|
||||
# Pin an upstream test suite: hash every file under the given repo-root-
|
||||
# relative paths (typically targets/<t>/workspace/tests/...) into
|
||||
# corpora/<target>/TESTSUITE.sha256, sha256sum two-space format. diff-test
|
||||
# verifies this pin before running upstream tests; targets/fetch.sh
|
||||
# verifies it after every fetch. Run from repo root:
|
||||
# scripts/pin-testsuite.sh <target> <path>...
|
||||
set -eu
|
||||
if [ $# -lt 2 ]; then
|
||||
echo "usage: scripts/pin-testsuite.sh <target> <path>..." >&2
|
||||
exit 2
|
||||
fi
|
||||
target="$1"
|
||||
shift
|
||||
out="corpora/$target/TESTSUITE.sha256"
|
||||
if [ ! -d "corpora/$target" ]; then
|
||||
echo "pin-testsuite: corpora/$target does not exist (run from repo root?)" >&2
|
||||
exit 1
|
||||
fi
|
||||
for p in "$@"; do
|
||||
if [ ! -e "$p" ]; then
|
||||
echo "pin-testsuite: $p not found (paths are repo-root-relative)" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
# Build the pin out-of-place and install it only on success: a run that fails
|
||||
# anywhere below must not truncate, replace, or half-write an existing pin —
|
||||
# that pin is what diff-test refuses to run against on mismatch.
|
||||
list="$out.files.$$"
|
||||
tmp="$out.new.$$"
|
||||
trap 'rm -f "$list" "$tmp"' EXIT INT TERM
|
||||
|
||||
# LC_ALL=C: byte-order sort, so the same tree pins identically on every
|
||||
# machine. -print0/-z: filenames with spaces or newlines survive intact.
|
||||
find "$@" -type f -print0 | LC_ALL=C sort -z > "$list"
|
||||
count=$(tr -dc '\0' < "$list" | wc -c | tr -d ' ')
|
||||
if [ "$count" -eq 0 ]; then
|
||||
# Was: `find ... | xargs -0 sha256sum > "$out"` — with no matches, xargs
|
||||
# still ran sha256sum with zero file arguments, so it hashed STDIN (the
|
||||
# empty pipe) and wrote a bogus "<sha-of-empty> -" entry, then reported
|
||||
# SUCCESS. That is a pin of nothing that later "verifies" by reading
|
||||
# stdin. Refuse instead, and leave any existing pin untouched.
|
||||
echo "pin-testsuite: no regular files under: $* — refusing to write $out" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# --text is GNU's default on Linux (a no-op there); it is spelled out because
|
||||
# sha256sum defaults to *binary* mode on some platforms, emitting "<hash> *path"
|
||||
# — a one-space-plus-asterisk form that diff-test's pin verifier (which splits
|
||||
# on the two-space separator) does not parse. Paths stay repo-root-relative:
|
||||
# find was handed repo-root-relative paths and we run from the repo root.
|
||||
xargs -0 sha256sum --text -- < "$list" > "$tmp"
|
||||
|
||||
lines=$(wc -l < "$tmp" | tr -d ' ')
|
||||
if [ "$lines" -ne "$count" ]; then
|
||||
echo "pin-testsuite: hashed $lines of $count files — refusing to write a partial pin" >&2
|
||||
exit 1
|
||||
fi
|
||||
mv "$tmp" "$out"
|
||||
echo "wrote $out ($count files)"
|
||||
echo "pinning a test suite is a deliberate human action (corpora/README.md):"
|
||||
echo "commit the new pin with justification."
|
||||
Reference in New Issue
Block a user