mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
gates: close the audit findings — enforce the gates the docs claimed
An adversarial audit found the enforcement story ahead of the code: gates that were documented as hard gates were advisory, manual, or absent, and several human-facing claims outran the machine record. This makes the claims true. High findings - Differential fuzz was documentation, not enforcement: verdict hardcoded fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and it is inherently baseline-vs-candidate, so it runs on the accept path and reports Skipped in the standalone `just gates` flow. New hard rule: a machine `accepted` verdict is impossible unless the fuzz gate actually passed — otherwise it caps at needs-human-review. - The agent's system prompt claimed FP/concurrency/unsafe patches were "routed to human review automatically". No such mechanism existed. Added a conservative lexical risk classifier (crates/verdict/src/risk.rs) that forces needs-human-review on any would-be accept carrying those signals; fp-tolerance equivalence always signals. - The local verdict path ran agent-influenced code unsandboxed. It now runs under scripts/no-net.sh (network namespace) and fails closed. Upstream test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when present (scripts/pin-testsuite.sh generates them). Also fixed, found while fixing the above - The pipeline script's `set -e` was inert (brace group as the left operand of `&&`), so a failing candidate build did not abort the run and verdict could bench a stale binary. Now a scoped subshell whose status is branched on at top level. - A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path) stranded RUNNING.lock and left read_verdict polling forever. Failure accounting now lives outside the wrapper. - harnessd's patch allowlist ignored git rename/copy headers and --no-prefix diffs; the rename-escape case is now a committed regression test. - The workspace did not compile on Windows (ungated Unix APIs). cfg-gated; 71 portable tests now run on both platforms. - pin-check regenerated the corpus manifest instead of verifying it; it is now verify-only, with a deliberate `just pin-corpus` to re-pin. - Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics (compare --perf-stat), never part of the accept decision. Docs now match the code, including what is NOT yet enforced: no target ships a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not wired to the fp-tolerance gate — each is stated as mechanized-but-inert rather than done. Corrected the stale "33 ledger rows" (34), the playbook's class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation claim (five accepted rows carry sanitizers_clean=false), the unsupported "40,000 requests" figure, and the blanket "historical rows record fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -1,7 +1,7 @@
|
||||
"""Phase 2 agent loop (SPEC §3.5): Claude Agent SDK driving the six
|
||||
"""Phase 2 agent loop (SPEC §3.5): Claude Agent SDK driving the seven
|
||||
harness tools over a standalone stdio MCP server. The agent gets NO
|
||||
built-in tools — no shell, no file writes; every capability is one of
|
||||
the six IPC operations, and the trust layer's allowlist/gates remain
|
||||
the seven IPC operations, and the trust layer's allowlist/gates remain
|
||||
the hard boundary (SPEC §10).
|
||||
|
||||
Run one attempt:
|
||||
|
||||
@@ -1,4 +1,4 @@
|
||||
"""Standalone MCP stdio server exposing the six harness tools (SPEC §3.5).
|
||||
"""Standalone MCP stdio server exposing the seven harness tools (SPEC §3.5).
|
||||
|
||||
Spawned as a subprocess by the agent loop; speaks MCP over stdio. Wraps
|
||||
the harnessd IPC client — the trust-layer allowlist and gates remain the
|
||||
|
||||
@@ -24,8 +24,12 @@ Rules, in priority order:
|
||||
Never restructure a patch to influence measurement rather than to
|
||||
express the hypothesis.
|
||||
|
||||
5. Equivalence is not yours to relax. Floating-point flag changes,
|
||||
concurrency-primitive changes, and new unsafe code are routed to
|
||||
human review automatically; do not attempt to phrase a patch to
|
||||
avoid that routing.
|
||||
5. Equivalence is not yours to relax. The trust layer (crates/verdict)
|
||||
runs a conservative lexical risk classifier over every patch's
|
||||
changed lines: concurrency, unsafe, and floating-point signals force
|
||||
any would-be accept to needs-human-review, and fp-tolerance
|
||||
equivalence mode always routes to human review. The classifier is
|
||||
lexical and deliberately over-triggering, not a semantic analysis;
|
||||
a human --needs-human-review flag also exists. Do not attempt to
|
||||
phrase a patch to slip past that routing.
|
||||
"""
|
||||
|
||||
@@ -1,6 +1,6 @@
|
||||
"""Tool surface exposed to the model (SPEC §3.5).
|
||||
|
||||
Exactly six tools, all mediated by harnessd over line-delimited JSON —
|
||||
Exactly seven tools, all mediated by harnessd over line-delimited JSON —
|
||||
the agent process has no direct filesystem or shell capability beyond
|
||||
what these return. Deliberately NOT exposed: shell on the host, writes
|
||||
outside targets/<name>/, any access to crates/, config/, corpora/.
|
||||
|
||||
Reference in New Issue
Block a user