gates: close the audit findings — enforce the gates the docs claimed

An adversarial audit found the enforcement story ahead of the code: gates
that were documented as hard gates were advisory, manual, or absent, and
several human-facing claims outran the machine record. This makes the
claims true.

High findings

- Differential fuzz was documentation, not enforcement: verdict hardcoded
  fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case
  studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real
  pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and
  it is inherently baseline-vs-candidate, so it runs on the accept path and
  reports Skipped in the standalone `just gates` flow. New hard rule: a
  machine `accepted` verdict is impossible unless the fuzz gate actually
  passed — otherwise it caps at needs-human-review.
- The agent's system prompt claimed FP/concurrency/unsafe patches were
  "routed to human review automatically". No such mechanism existed. Added
  a conservative lexical risk classifier (crates/verdict/src/risk.rs) that
  forces needs-human-review on any would-be accept carrying those signals;
  fp-tolerance equivalence always signals.
- The local verdict path ran agent-influenced code unsandboxed. It now runs
  under scripts/no-net.sh (network namespace) and fails closed. Upstream
  test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when
  present (scripts/pin-testsuite.sh generates them).

Also fixed, found while fixing the above

- The pipeline script's `set -e` was inert (brace group as the left operand
  of `&&`), so a failing candidate build did not abort the run and verdict
  could bench a stale binary. Now a scoped subshell whose status is branched
  on at top level.
- A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path)
  stranded RUNNING.lock and left read_verdict polling forever. Failure
  accounting now lives outside the wrapper.
- harnessd's patch allowlist ignored git rename/copy headers and --no-prefix
  diffs; the rename-escape case is now a committed regression test.
- The workspace did not compile on Windows (ungated Unix APIs). cfg-gated;
  71 portable tests now run on both platforms.
- pin-check regenerated the corpus manifest instead of verifying it; it is
  now verify-only, with a deliberate `just pin-corpus` to re-pin.
- Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics
  (compare --perf-stat), never part of the accept decision.

Docs now match the code, including what is NOT yet enforced: no target ships
a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not
wired to the fp-tolerance gate — each is stated as mechanized-but-inert
rather than done. Corrected the stale "33 ledger rows" (34), the playbook's
class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation
claim (five accepted rows carry sanitizers_clean=false), the unsupported
"40,000 requests" figure, and the blanket "historical rows record
fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-14 13:10:14 -04:00
co-authored by Claude Opus 4.8
parent 21352efe6c
commit 899d55f4b6
57 changed files with 3308 additions and 692 deletions
+71
View File
@@ -43,6 +43,49 @@ pub fn all_passed(results: &[(GateLayer, GateOutcome)]) -> bool {
.any(|(_, o)| matches!(o, GateOutcome::Passed))
}
/// Inputs to one gate run: the candidate binary under test, the pristine
/// baseline to differ it against, and the differential-fuzz iteration
/// budget (from AcceptConfig, default 10_000).
#[derive(Debug, Clone, Copy)]
pub struct GateInputs<'a> {
/// Patched binary (repo-root-relative): golden replay runs it, and it
/// is the `{candidate}` side of differential fuzz.
pub candidate_binary: &'a str,
/// Pristine baseline binary (repo-root-relative), the `{baseline}` side
/// of differential fuzz. `None` when the caller has no baseline to
/// differ against — the standalone `just gates` flow builds only the
/// working tree — and the fuzz gate is then skipped with that reason
/// rather than silently comparing the candidate to itself. The accept
/// path (`just verdict`) rebuilds a pristine baseline and passes it,
/// and only a fuzz gate that actually Passed can mint an accept.
pub baseline_binary: Option<&'a str>,
pub fuzz_iters: u64,
}
/// Everything one gate run produced: the four layers in fixed order
/// (UpstreamTests, GoldenReplay, DifferentialFuzz, Sanitizers) plus the
/// fuzz counts and equivalence mode that verdict records in the ledger.
#[derive(Debug, Clone)]
pub struct GateReport {
pub gates: Vec<(GateLayer, GateOutcome)>,
/// Fuzz iterations actually executed (parsed from the FUZZ-RESULT
/// line); 0 when the fuzz gate was skipped or never reported.
pub fuzz_iters: u64,
pub fuzz_divergence: bool,
/// "byte-identical" | "fp-tolerance"
pub equivalence_mode: String,
}
impl GateReport {
pub fn outcome(&self, layer: GateLayer) -> Option<&GateOutcome> {
self.gates.iter().find(|(l, _)| *l == layer).map(|(_, o)| o)
}
pub fn all_passed(&self) -> bool {
all_passed(&self.gates)
}
}
#[cfg(test)]
mod tests {
use super::*;
@@ -71,4 +114,32 @@ mod tests {
)];
assert!(!all_passed(&results));
}
#[test]
fn report_outcome_looks_up_by_layer() {
let report = GateReport {
gates: vec![
(GateLayer::UpstreamTests, GateOutcome::Passed),
(
GateLayer::GoldenReplay,
GateOutcome::Failed {
detail: "stdout diverged".into(),
},
),
],
fuzz_iters: 0,
fuzz_divergence: false,
equivalence_mode: "byte-identical".into(),
};
assert_eq!(
report.outcome(GateLayer::UpstreamTests),
Some(&GateOutcome::Passed)
);
assert!(matches!(
report.outcome(GateLayer::GoldenReplay),
Some(GateOutcome::Failed { .. })
));
assert_eq!(report.outcome(GateLayer::Sanitizers), None);
assert!(!report.all_passed());
}
}