mirror of https://github.com/bdbrown4/peltier.io.git synced 2026-11-07 10:58:28 +00:00

Files

T

benbrownandClaude Opus 4.8 899d55f4b6 gates: close the audit findings — enforce the gates the docs claimed

An adversarial audit found the enforcement story ahead of the code: gates
that were documented as hard gates were advisory, manual, or absent, and
several human-facing claims outran the machine record. This makes the
claims true.

High findings

- Differential fuzz was documentation, not enforcement: verdict hardcoded
  fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case
  studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real
  pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and
  it is inherently baseline-vs-candidate, so it runs on the accept path and
  reports Skipped in the standalone `just gates` flow. New hard rule: a
  machine `accepted` verdict is impossible unless the fuzz gate actually
  passed — otherwise it caps at needs-human-review.
- The agent's system prompt claimed FP/concurrency/unsafe patches were
  "routed to human review automatically". No such mechanism existed. Added
  a conservative lexical risk classifier (crates/verdict/src/risk.rs) that
  forces needs-human-review on any would-be accept carrying those signals;
  fp-tolerance equivalence always signals.
- The local verdict path ran agent-influenced code unsandboxed. It now runs
  under scripts/no-net.sh (network namespace) and fails closed. Upstream
  test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when
  present (scripts/pin-testsuite.sh generates them).

Also fixed, found while fixing the above

- The pipeline script's `set -e` was inert (brace group as the left operand
  of `&&`), so a failing candidate build did not abort the run and verdict
  could bench a stale binary. Now a scoped subshell whose status is branched
  on at top level.
- A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path)
  stranded RUNNING.lock and left read_verdict polling forever. Failure
  accounting now lives outside the wrapper.
- harnessd's patch allowlist ignored git rename/copy headers and --no-prefix
  diffs; the rename-escape case is now a committed regression test.
- The workspace did not compile on Windows (ungated Unix APIs). cfg-gated;
  71 portable tests now run on both platforms.
- pin-check regenerated the corpus manifest instead of verifying it; it is
  now verify-only, with a deliberate `just pin-corpus` to re-pin.
- Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics
  (compare --perf-stat), never part of the accept decision.

Docs now match the code, including what is NOT yet enforced: no target ships
a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not
wired to the fp-tolerance gate — each is stated as mechanized-but-inert
rather than done. Corrected the stale "33 ledger rows" (34), the playbook's
class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation
claim (five accepted rows carry sanitizers_clean=false), the unsupported
"40,000 requests" figure, and the blanket "historical rows record
fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-07-14 13:10:14 -04:00

3.8 KiB

Raw Permalink Blame History

Adversarial review — whole package

A hostile pass over the entire project: every headline claim re-checked against the ledger, the trust boundary attacked directly, the newest code scrutinized hardest, and the statistics re-validated. Two real defects found and fixed; the load-bearing guarantees held.

Findings

1. FIXED — the ROI generator minted a clean pitch for an overturned patch

The report generator trusted the ledger's verdict field. phase2-comrak-010 is immutably accepted (it predates the machine-enforced sanitizer gate; the audit overturned it for a LeakSanitizer leak and it was never banked). report --run-id phase2-comrak-010 therefore produced a clean $16,886/year ROI report for a memory-leaking patch — exactly the kind of self-contradiction a hostile reviewer would seize on.

Fix: the report now treats an accepted row that is not sanitizers_clean as not shippable and prints a "Not shipped" warning before any dollar figure. Because the pipeline now enforces ASan/UBSan on every accept, all legitimate wins are sanitizers_clean: true; only the historical overturned row is accepted + false, so the rule cleanly separates them. Regression-checked: phase3-cjson-002 still renders clean.

2. FIXED — load generator could panic on degenerate inputs

bench-runner service --rate 0 (or --count 0) reached Duration::from_secs_f64(1.0/0.0) → panic. Fix: run_session now rejects a non-finite/non-positive rate and zero count/workers with a clear error.

Attacks that were correctly refused (no change needed)

Ledger append-only: UPDATE/DELETE on a copy both refused by the SQLite triggers.
Patch boundary: propose_patch refused ../ traversal and .git access; read_target_source refused traversal. A patch can only touch targets/<t>/workspace.
Accepted-row invariant: all 10 accepted rows have green upstream tests + golden replay and a bootstrap CI lower bound ≥ 1.02.
Statistics: every batch and service A/A calibration is a null result (0 false positives); every injected-5% regression is detected (≥95%). The service load generator is coordinated-omission-correct and its injection test resolves a known 5% latency regression 10/10.

Verification run

cargo test --workspace (all green), clippy -D warnings (clean), cargo fmt --check (clean). Every case-study and CLAUDE.md figure re-checked against results/ledger.sqlite and the calibrated bench — consistent, with the compound tokei figure stated as an approximation (CIs don't multiply) and p99 service latency explicitly not claimed.

Standing (not defects)

phase2-comrak-010 remains accepted in the ledger by design — the ledger is append-only and records what happened; the overturn lives in the case study and is now enforced by the report generator too.
phase0-comrak-002 (mimalloc) is needs-human-review, awaiting a ruling (see results/rulings/).

Erratum (2026-07-13)

Finding 1's separation claim above — "only the historical overturned row is accepted + false, so the rule cleanly separates them" — was too strong. Five accepted rows carry sanitizers_clean: false: phase1-verdict-smoke-002, phase1-ipc-smoke-001, phase2-tokei-001, phase2-tokei-003, and phase2-comrak-010. All five pre-date the machine-enforced sanitizer lane — the Phase 1 rows are harness smoke tests, and the two Phase 2 tokei wins were sanitizer-audited by hand during the 100% Phase 2 audit (see the Phase 2 case study) — but the ledger's machine record cannot distinguish "manually audited clean" from "never run", so the report generator's not-shippable rule flags all five, not just the overturned row. That is the correct conservative behavior; only the description of why the rule separates cleanly was wrong. The original text stands unedited above; this note is the correction.

3.8 KiB Raw Permalink Blame History