mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
gates: close the audit findings — enforce the gates the docs claimed
An adversarial audit found the enforcement story ahead of the code: gates that were documented as hard gates were advisory, manual, or absent, and several human-facing claims outran the machine record. This makes the claims true. High findings - Differential fuzz was documentation, not enforcement: verdict hardcoded fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and it is inherently baseline-vs-candidate, so it runs on the accept path and reports Skipped in the standalone `just gates` flow. New hard rule: a machine `accepted` verdict is impossible unless the fuzz gate actually passed — otherwise it caps at needs-human-review. - The agent's system prompt claimed FP/concurrency/unsafe patches were "routed to human review automatically". No such mechanism existed. Added a conservative lexical risk classifier (crates/verdict/src/risk.rs) that forces needs-human-review on any would-be accept carrying those signals; fp-tolerance equivalence always signals. - The local verdict path ran agent-influenced code unsandboxed. It now runs under scripts/no-net.sh (network namespace) and fails closed. Upstream test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when present (scripts/pin-testsuite.sh generates them). Also fixed, found while fixing the above - The pipeline script's `set -e` was inert (brace group as the left operand of `&&`), so a failing candidate build did not abort the run and verdict could bench a stale binary. Now a scoped subshell whose status is branched on at top level. - A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path) stranded RUNNING.lock and left read_verdict polling forever. Failure accounting now lives outside the wrapper. - harnessd's patch allowlist ignored git rename/copy headers and --no-prefix diffs; the rename-escape case is now a committed regression test. - The workspace did not compile on Windows (ungated Unix APIs). cfg-gated; 71 portable tests now run on both platforms. - pin-check regenerated the corpus manifest instead of verifying it; it is now verify-only, with a deliberate `just pin-corpus` to re-pin. - Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics (compare --perf-stat), never part of the accept decision. Docs now match the code, including what is NOT yet enforced: no target ships a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not wired to the fp-tolerance gate — each is stated as mechanized-but-inert rather than done. Corrected the stale "33 ledger rows" (34), the playbook's class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation claim (five accepted rows carry sanitizers_clean=false), the unsupported "40,000 requests" figure, and the blanket "historical rows record fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -36,7 +36,10 @@ impl Verdict {
|
||||
}
|
||||
}
|
||||
|
||||
/// Gate outcomes for one attempt (SPEC §3.2).
|
||||
/// Gate outcomes for one attempt (SPEC §3.2). The trailing fields are
|
||||
/// defaulted so historical rows (written before they existed) keep
|
||||
/// deserializing; they live only in the `gates` JSON column — no schema
|
||||
/// change, append-only doctrine untouched.
|
||||
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
|
||||
pub struct GateResults {
|
||||
pub upstream_tests: bool,
|
||||
@@ -44,6 +47,15 @@ pub struct GateResults {
|
||||
pub fuzz_iters: u64,
|
||||
pub fuzz_divergence: bool,
|
||||
pub sanitizers_clean: bool,
|
||||
/// None = no TSan lane configured/run for this attempt.
|
||||
#[serde(default)]
|
||||
pub tsan_clean: Option<bool>,
|
||||
/// Lexical NHR-classifier hits (crates/verdict src/risk.rs).
|
||||
#[serde(default)]
|
||||
pub risk_signals: Vec<String>,
|
||||
/// "byte-identical" | "fp-tolerance"; None for historical rows.
|
||||
#[serde(default)]
|
||||
pub equivalence_mode: Option<String>,
|
||||
}
|
||||
|
||||
/// Benchmark evidence for one attempt. All times in seconds; CIs are
|
||||
@@ -334,6 +346,42 @@ mod tests {
|
||||
assert_eq!(ledger.attempted_classes("example").unwrap(), vec![2]);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn historical_gates_json_still_deserializes() {
|
||||
// Exact shape of the 34 pre-existing rows' gates column: no
|
||||
// tsan_clean / risk_signals / equivalence_mode keys.
|
||||
let old = r#"{"upstream_tests":true,"golden_replay":true,"fuzz_iters":4332,
|
||||
"fuzz_divergence":false,"sanitizers_clean":true}"#;
|
||||
let g: GateResults = serde_json::from_str(old).unwrap();
|
||||
assert!(g.upstream_tests && g.golden_replay && g.sanitizers_clean);
|
||||
assert_eq!(g.fuzz_iters, 4332);
|
||||
assert_eq!(g.tsan_clean, None);
|
||||
assert!(g.risk_signals.is_empty());
|
||||
assert_eq!(g.equivalence_mode, None);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn extended_gates_round_trip() {
|
||||
let g = GateResults {
|
||||
upstream_tests: true,
|
||||
golden_replay: true,
|
||||
fuzz_iters: 10_000,
|
||||
fuzz_divergence: false,
|
||||
sanitizers_clean: true,
|
||||
tsan_clean: Some(false),
|
||||
risk_signals: vec!["concurrency".into()],
|
||||
equivalence_mode: Some("fp-tolerance".into()),
|
||||
};
|
||||
let json = serde_json::to_string(&g).unwrap();
|
||||
let back: GateResults = serde_json::from_str(&json).unwrap();
|
||||
assert_eq!(back.tsan_clean, Some(false));
|
||||
assert_eq!(back.risk_signals, vec!["concurrency".to_string()]);
|
||||
assert_eq!(back.equivalence_mode.as_deref(), Some("fp-tolerance"));
|
||||
// The keys the defensive Value readers depend on must survive.
|
||||
let v: serde_json::Value = serde_json::from_str(&json).unwrap();
|
||||
assert_eq!(v.get("sanitizers_clean"), Some(&serde_json::json!(true)));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn append_only_is_enforced() {
|
||||
let ledger = Ledger::open_in_memory().unwrap();
|
||||
|
||||
Reference in New Issue
Block a user