mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
An adversarial audit found the enforcement story ahead of the code: gates that were documented as hard gates were advisory, manual, or absent, and several human-facing claims outran the machine record. This makes the claims true. High findings - Differential fuzz was documentation, not enforcement: verdict hardcoded fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and it is inherently baseline-vs-candidate, so it runs on the accept path and reports Skipped in the standalone `just gates` flow. New hard rule: a machine `accepted` verdict is impossible unless the fuzz gate actually passed — otherwise it caps at needs-human-review. - The agent's system prompt claimed FP/concurrency/unsafe patches were "routed to human review automatically". No such mechanism existed. Added a conservative lexical risk classifier (crates/verdict/src/risk.rs) that forces needs-human-review on any would-be accept carrying those signals; fp-tolerance equivalence always signals. - The local verdict path ran agent-influenced code unsandboxed. It now runs under scripts/no-net.sh (network namespace) and fails closed. Upstream test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when present (scripts/pin-testsuite.sh generates them). Also fixed, found while fixing the above - The pipeline script's `set -e` was inert (brace group as the left operand of `&&`), so a failing candidate build did not abort the run and verdict could bench a stale binary. Now a scoped subshell whose status is branched on at top level. - A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path) stranded RUNNING.lock and left read_verdict polling forever. Failure accounting now lives outside the wrapper. - harnessd's patch allowlist ignored git rename/copy headers and --no-prefix diffs; the rename-escape case is now a committed regression test. - The workspace did not compile on Windows (ungated Unix APIs). cfg-gated; 71 portable tests now run on both platforms. - pin-check regenerated the corpus manifest instead of verifying it; it is now verify-only, with a deliberate `just pin-corpus` to re-pin. - Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics (compare --perf-stat), never part of the accept decision. Docs now match the code, including what is NOT yet enforced: no target ships a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not wired to the fp-tolerance gate — each is stated as mechanized-but-inert rather than done. Corrected the stale "33 ledger rows" (34), the playbook's class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation claim (five accepted rows carry sanitizers_clean=false), the unsupported "40,000 requests" figure, and the blanket "historical rows record fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
146 lines
4.7 KiB
Rust
146 lines
4.7 KiB
Rust
//! Equivalence gates (SPEC §3.2) and corpus hash-pinning.
|
|
//!
|
|
//! The gate *sequencing* (upstream tests → golden replay → differential
|
|
//! fuzz → sanitizers) is orchestrated per-target; this crate owns the
|
|
//! pieces that must be trustworthy regardless of target: the pin
|
|
//! verifier that refuses to run on a tampered corpus, and the
|
|
//! per-target equivalence policy parser (byte-identical unless a target
|
|
//! declares an explicit FP tolerance).
|
|
|
|
pub mod orchestrate;
|
|
pub mod pin;
|
|
pub mod policy;
|
|
pub mod target;
|
|
|
|
use serde::{Deserialize, Serialize};
|
|
|
|
/// Outcome of one gate layer. `Skipped` is only legal where the SPEC
|
|
/// allows it (e.g. TSan when no concurrency primitives changed) and the
|
|
/// reason is always recorded.
|
|
#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
|
|
pub enum GateOutcome {
|
|
Passed,
|
|
Failed { detail: String },
|
|
Skipped { reason: String },
|
|
}
|
|
|
|
#[derive(Debug, Clone, Copy, PartialEq, Eq, Serialize, Deserialize)]
|
|
#[serde(rename_all = "kebab-case")]
|
|
pub enum GateLayer {
|
|
UpstreamTests,
|
|
GoldenReplay,
|
|
DifferentialFuzz,
|
|
Sanitizers,
|
|
}
|
|
|
|
/// All layers must pass; any failure is an auto-reject (SPEC §8).
|
|
pub fn all_passed(results: &[(GateLayer, GateOutcome)]) -> bool {
|
|
results
|
|
.iter()
|
|
.all(|(_, o)| matches!(o, GateOutcome::Passed | GateOutcome::Skipped { .. }))
|
|
&& results
|
|
.iter()
|
|
.any(|(_, o)| matches!(o, GateOutcome::Passed))
|
|
}
|
|
|
|
/// Inputs to one gate run: the candidate binary under test, the pristine
|
|
/// baseline to differ it against, and the differential-fuzz iteration
|
|
/// budget (from AcceptConfig, default 10_000).
|
|
#[derive(Debug, Clone, Copy)]
|
|
pub struct GateInputs<'a> {
|
|
/// Patched binary (repo-root-relative): golden replay runs it, and it
|
|
/// is the `{candidate}` side of differential fuzz.
|
|
pub candidate_binary: &'a str,
|
|
/// Pristine baseline binary (repo-root-relative), the `{baseline}` side
|
|
/// of differential fuzz. `None` when the caller has no baseline to
|
|
/// differ against — the standalone `just gates` flow builds only the
|
|
/// working tree — and the fuzz gate is then skipped with that reason
|
|
/// rather than silently comparing the candidate to itself. The accept
|
|
/// path (`just verdict`) rebuilds a pristine baseline and passes it,
|
|
/// and only a fuzz gate that actually Passed can mint an accept.
|
|
pub baseline_binary: Option<&'a str>,
|
|
pub fuzz_iters: u64,
|
|
}
|
|
|
|
/// Everything one gate run produced: the four layers in fixed order
|
|
/// (UpstreamTests, GoldenReplay, DifferentialFuzz, Sanitizers) plus the
|
|
/// fuzz counts and equivalence mode that verdict records in the ledger.
|
|
#[derive(Debug, Clone)]
|
|
pub struct GateReport {
|
|
pub gates: Vec<(GateLayer, GateOutcome)>,
|
|
/// Fuzz iterations actually executed (parsed from the FUZZ-RESULT
|
|
/// line); 0 when the fuzz gate was skipped or never reported.
|
|
pub fuzz_iters: u64,
|
|
pub fuzz_divergence: bool,
|
|
/// "byte-identical" | "fp-tolerance"
|
|
pub equivalence_mode: String,
|
|
}
|
|
|
|
impl GateReport {
|
|
pub fn outcome(&self, layer: GateLayer) -> Option<&GateOutcome> {
|
|
self.gates.iter().find(|(l, _)| *l == layer).map(|(_, o)| o)
|
|
}
|
|
|
|
pub fn all_passed(&self) -> bool {
|
|
all_passed(&self.gates)
|
|
}
|
|
}
|
|
|
|
#[cfg(test)]
|
|
mod tests {
|
|
use super::*;
|
|
|
|
#[test]
|
|
fn any_failure_rejects() {
|
|
let results = vec![
|
|
(GateLayer::UpstreamTests, GateOutcome::Passed),
|
|
(
|
|
GateLayer::GoldenReplay,
|
|
GateOutcome::Failed {
|
|
detail: "stdout diverged".into(),
|
|
},
|
|
),
|
|
];
|
|
assert!(!all_passed(&results));
|
|
}
|
|
|
|
#[test]
|
|
fn all_skipped_is_not_a_pass() {
|
|
let results = vec![(
|
|
GateLayer::Sanitizers,
|
|
GateOutcome::Skipped {
|
|
reason: "n/a".into(),
|
|
},
|
|
)];
|
|
assert!(!all_passed(&results));
|
|
}
|
|
|
|
#[test]
|
|
fn report_outcome_looks_up_by_layer() {
|
|
let report = GateReport {
|
|
gates: vec![
|
|
(GateLayer::UpstreamTests, GateOutcome::Passed),
|
|
(
|
|
GateLayer::GoldenReplay,
|
|
GateOutcome::Failed {
|
|
detail: "stdout diverged".into(),
|
|
},
|
|
),
|
|
],
|
|
fuzz_iters: 0,
|
|
fuzz_divergence: false,
|
|
equivalence_mode: "byte-identical".into(),
|
|
};
|
|
assert_eq!(
|
|
report.outcome(GateLayer::UpstreamTests),
|
|
Some(&GateOutcome::Passed)
|
|
);
|
|
assert!(matches!(
|
|
report.outcome(GateLayer::GoldenReplay),
|
|
Some(GateOutcome::Failed { .. })
|
|
));
|
|
assert_eq!(report.outcome(GateLayer::Sanitizers), None);
|
|
assert!(!report.all_passed());
|
|
}
|
|
}
|