mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
gates: close the audit findings — enforce the gates the docs claimed
An adversarial audit found the enforcement story ahead of the code: gates that were documented as hard gates were advisory, manual, or absent, and several human-facing claims outran the machine record. This makes the claims true. High findings - Differential fuzz was documentation, not enforcement: verdict hardcoded fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and it is inherently baseline-vs-candidate, so it runs on the accept path and reports Skipped in the standalone `just gates` flow. New hard rule: a machine `accepted` verdict is impossible unless the fuzz gate actually passed — otherwise it caps at needs-human-review. - The agent's system prompt claimed FP/concurrency/unsafe patches were "routed to human review automatically". No such mechanism existed. Added a conservative lexical risk classifier (crates/verdict/src/risk.rs) that forces needs-human-review on any would-be accept carrying those signals; fp-tolerance equivalence always signals. - The local verdict path ran agent-influenced code unsandboxed. It now runs under scripts/no-net.sh (network namespace) and fails closed. Upstream test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when present (scripts/pin-testsuite.sh generates them). Also fixed, found while fixing the above - The pipeline script's `set -e` was inert (brace group as the left operand of `&&`), so a failing candidate build did not abort the run and verdict could bench a stale binary. Now a scoped subshell whose status is branched on at top level. - A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path) stranded RUNNING.lock and left read_verdict polling forever. Failure accounting now lives outside the wrapper. - harnessd's patch allowlist ignored git rename/copy headers and --no-prefix diffs; the rename-escape case is now a committed regression test. - The workspace did not compile on Windows (ungated Unix APIs). cfg-gated; 71 portable tests now run on both platforms. - pin-check regenerated the corpus manifest instead of verifying it; it is now verify-only, with a deliberate `just pin-corpus` to re-pin. - Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics (compare --perf-stat), never part of the accept decision. Docs now match the code, including what is NOT yet enforced: no target ships a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not wired to the fp-tolerance gate — each is stated as mechanized-but-inert rather than done. Corrected the stale "33 ledger rows" (34), the playbook's class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation claim (five accepted rows carry sanitizers_clean=false), the unsupported "40,000 requests" figure, and the blanket "historical rows record fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
+63
-9
@@ -7,6 +7,22 @@ candidate must prove it does the same thing the baseline does. These are
|
||||
|
||||
## The layers
|
||||
|
||||
0. **Pin checks.** Before any gate runs, the corpus is verified against its
|
||||
committed `MANIFEST.sha256` and the harness **refuses to run** on a
|
||||
mismatch — nothing is measured on tampered inputs. This ships and is
|
||||
enforced for every target.
|
||||
|
||||
The *test suite* has the same machinery but is **not yet active**:
|
||||
`diff-test` verifies `corpora/<t>/TESTSUITE.sha256` when the file is
|
||||
present (mismatch = hard refusal) and prints an explicit "suite unpinned"
|
||||
warning when it is absent. **No target ships a suite pin today**, so this
|
||||
check currently constrains nothing. It matters because upstream suites
|
||||
are vendored under `targets/<t>/workspace/` — an agent-writable path — so
|
||||
"the agent cannot edit the tests" rests on a pin that still has to be
|
||||
generated (`scripts/pin-testsuite.sh`, a deliberate human action, run
|
||||
after a fetch). Until then, that specific guarantee is a mechanism, not
|
||||
a fact.
|
||||
|
||||
1. **Upstream test suite.** The target's own tests, run against the patched
|
||||
code. For cJSON that is the vendored Unity/CTest suite (19 tests:
|
||||
`parse_number`, `parse_string`, `print_*`, `misc_tests`, …); for the Rust
|
||||
@@ -20,15 +36,49 @@ candidate must prove it does the same thing the baseline does. These are
|
||||
mismatch.
|
||||
|
||||
3. **Differential fuzzing.** Pristine vs candidate on thousands of mutated
|
||||
inputs, comparing output. Every accepted win cleared **10,000 mutated
|
||||
inputs with 0 divergences** — the tokei wins after canonicalizing benign
|
||||
parallel-walker output ordering (a real finding: differential gates must
|
||||
compare *semantics*, not bytes, where the target is legitimately
|
||||
nondeterministic in presentation), the cJSON win on quote/number/escape
|
||||
edge cases.
|
||||
inputs, comparing output.
|
||||
|
||||
4. **Sanitizers.** ASan/LSan for Rust, ASan+UBSan for C. Run on **every
|
||||
would-be accept** — this is machine-enforced, not manual (see below).
|
||||
*Where it runs.* Fuzz is the one gate that is **not** a property of the
|
||||
candidate alone: it differs the candidate against a **pristine baseline**,
|
||||
so it needs both binaries. That means it runs **only on the accept path**
|
||||
(`just verdict`, the one flow that rebuilds a pristine baseline). The
|
||||
standalone `just gates` flow has no baseline, so the gate reports
|
||||
**Skipped, with that reason recorded** — it is never faked by comparing
|
||||
the candidate against itself. `just gates` is a fast equivalence check,
|
||||
**not** a fuzz run. The gate is graded strictly: `diff-test` runs the
|
||||
target's declared `[gates].fuzz` command and parses a
|
||||
`FUZZ-RESULT iters=<n> divergences=<m>` line; a run that never prints one
|
||||
**fails**, rather than passing by silence.
|
||||
|
||||
*The hard rule.* A machine `accepted` verdict is now **impossible**
|
||||
without this gate having actually passed. A skipped or failed fuzz gate
|
||||
caps the verdict at `needs-human-review` — because an accept minted
|
||||
without differential fuzz is an unverified claim.
|
||||
|
||||
*Honest history.* Through Phase 5 this ran as an **out-of-band
|
||||
per-attempt script** (`scripts/diff-fuzz-<target>.py`), not inside the
|
||||
pipeline — so **8 of the 10 accepted ledger rows record `fuzz_iters=0`**
|
||||
even where a 10,000-input audit run is documented in the case study. The
|
||||
two exceptions are the Phase 0 tokei rows, which record 4,332. Those
|
||||
out-of-band runs produced real findings — the tokei wins passed only
|
||||
after canonicalizing benign parallel-walker output ordering (differential
|
||||
gates must compare *semantics*, not bytes, where the target is
|
||||
legitimately nondeterministic in presentation), and the cJSON win cleared
|
||||
quote/number/escape edge cases with 0 divergences.
|
||||
|
||||
4. **Sanitizers.** ASan+UBSan (ASan/LSan on the Rust targets) is built and
|
||||
run against the pinned workload on **every would-be accept** — the
|
||||
sanitizer gate is machine-enforced, not manual (see below). A flagged
|
||||
run, *or* a target that declares no sanitizer build at all, caps the
|
||||
accept at `needs-human-review`.
|
||||
|
||||
A **TSan lane** exists on the same accept path, but it only runs when a
|
||||
target declares `[build].tsan`. **No target declares it today, so there
|
||||
is currently no TSan coverage** — the lane is wired and dormant, and a
|
||||
patch touching threads needs that opt-in added before it can be checked.
|
||||
(MIRI, named in the spec, was never implemented.) The concurrency safety
|
||||
net that *is* live is the review tier below, which routes any
|
||||
concurrency-token patch to a human.
|
||||
|
||||
## The tier rule (SPEC §8)
|
||||
|
||||
@@ -36,7 +86,11 @@ Some changes can pass every gate and still be unsafe to auto-ship. A change
|
||||
that touches floating-point ordering, concurrency primitives, `unsafe`, or
|
||||
carries a sanitizer flag is **never** auto-accepted — its verdict is capped
|
||||
at `needs-human-review`. The tier is decided by the *nature* of the change,
|
||||
never by the size of the number. The mimalloc allocator swap
|
||||
never by the size of the number. The routing is machine-enforced: `verdict`
|
||||
runs a conservative **lexical risk classifier** over the patch's changed
|
||||
lines (concurrency / unsafe / floating-point token lists — deliberately
|
||||
over-triggering, not a semantic analysis) and any signal, or the use of
|
||||
fp-tolerance equivalence mode at all, forces a would-be accept to review. The mimalloc allocator swap
|
||||
([case study](./case-studies/build-flags.md)) is the canonical example: a
|
||||
real +4.6% win, correctly held for a human ruling because a global
|
||||
allocator change's safety is not something one benchmark can clear.
|
||||
|
||||
Reference in New Issue
Block a user