mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
benbrownandClaude Opus 4.8
899d55f4b6
An adversarial audit found the enforcement story ahead of the code: gates that were documented as hard gates were advisory, manual, or absent, and several human-facing claims outran the machine record. This makes the claims true. High findings - Differential fuzz was documentation, not enforcement: verdict hardcoded fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and it is inherently baseline-vs-candidate, so it runs on the accept path and reports Skipped in the standalone `just gates` flow. New hard rule: a machine `accepted` verdict is impossible unless the fuzz gate actually passed — otherwise it caps at needs-human-review. - The agent's system prompt claimed FP/concurrency/unsafe patches were "routed to human review automatically". No such mechanism existed. Added a conservative lexical risk classifier (crates/verdict/src/risk.rs) that forces needs-human-review on any would-be accept carrying those signals; fp-tolerance equivalence always signals. - The local verdict path ran agent-influenced code unsandboxed. It now runs under scripts/no-net.sh (network namespace) and fails closed. Upstream test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when present (scripts/pin-testsuite.sh generates them). Also fixed, found while fixing the above - The pipeline script's `set -e` was inert (brace group as the left operand of `&&`), so a failing candidate build did not abort the run and verdict could bench a stale binary. Now a scoped subshell whose status is branched on at top level. - A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path) stranded RUNNING.lock and left read_verdict polling forever. Failure accounting now lives outside the wrapper. - harnessd's patch allowlist ignored git rename/copy headers and --no-prefix diffs; the rename-escape case is now a committed regression test. - The workspace did not compile on Windows (ungated Unix APIs). cfg-gated; 71 portable tests now run on both platforms. - pin-check regenerated the corpus manifest instead of verifying it; it is now verify-only, with a deliberate `just pin-corpus` to re-pin. - Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics (compare --perf-stat), never part of the accept decision. Docs now match the code, including what is NOT yet enforced: no target ships a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not wired to the fp-tolerance gate — each is stated as mechanized-but-inert rather than done. Corrected the stale "33 ledger rows" (34), the playbook's class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation claim (five accepted rows carry sanitizers_clean=false), the unsupported "40,000 requests" figure, and the blanket "historical rows record fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
76 lines
3.8 KiB
Markdown
76 lines
3.8 KiB
Markdown
# Adversarial review — whole package
|
|
|
|
A hostile pass over the entire project: every headline claim re-checked
|
|
against the ledger, the trust boundary attacked directly, the newest
|
|
code scrutinized hardest, and the statistics re-validated. Two real
|
|
defects found and fixed; the load-bearing guarantees held.
|
|
|
|
## Findings
|
|
|
|
### 1. FIXED — the ROI generator minted a clean pitch for an overturned patch
|
|
The report generator trusted the ledger's `verdict` field. `phase2-comrak-010`
|
|
is immutably `accepted` (it predates the machine-enforced sanitizer gate;
|
|
the audit overturned it for a LeakSanitizer leak and it was never banked).
|
|
`report --run-id phase2-comrak-010` therefore produced a clean
|
|
**$16,886/year** ROI report for a memory-leaking patch — exactly the kind
|
|
of self-contradiction a hostile reviewer would seize on.
|
|
|
|
**Fix:** the report now treats an `accepted` row that is not
|
|
`sanitizers_clean` as **not shippable** and prints a "Not shipped" warning
|
|
before any dollar figure. Because the pipeline now enforces ASan/UBSan on
|
|
every accept, all legitimate wins are `sanitizers_clean: true`; only the
|
|
historical overturned row is `accepted + false`, so the rule cleanly
|
|
separates them. Regression-checked: `phase3-cjson-002` still renders clean.
|
|
|
|
### 2. FIXED — load generator could panic on degenerate inputs
|
|
`bench-runner service --rate 0` (or `--count 0`) reached
|
|
`Duration::from_secs_f64(1.0/0.0)` → panic. **Fix:** `run_session` now
|
|
rejects a non-finite/non-positive rate and zero count/workers with a clear
|
|
error.
|
|
|
|
## Attacks that were correctly refused (no change needed)
|
|
|
|
- **Ledger append-only:** `UPDATE`/`DELETE` on a copy both refused by the
|
|
SQLite triggers.
|
|
- **Patch boundary:** `propose_patch` refused `../` traversal and `.git`
|
|
access; `read_target_source` refused traversal. A patch can only touch
|
|
`targets/<t>/workspace`.
|
|
- **Accepted-row invariant:** all 10 `accepted` rows have green upstream
|
|
tests + golden replay and a bootstrap CI lower bound ≥ 1.02.
|
|
- **Statistics:** every batch and service A/A calibration is a null result
|
|
(0 false positives); every injected-5% regression is detected (≥95%).
|
|
The service load generator is coordinated-omission-correct and its
|
|
injection test resolves a known 5% latency regression 10/10.
|
|
|
|
## Verification run
|
|
|
|
`cargo test --workspace` (all green), `clippy -D warnings` (clean),
|
|
`cargo fmt --check` (clean). Every case-study and CLAUDE.md figure
|
|
re-checked against `results/ledger.sqlite` and the calibrated bench —
|
|
consistent, with the compound tokei figure stated as an approximation
|
|
(CIs don't multiply) and p99 service latency explicitly not claimed.
|
|
|
|
## Standing (not defects)
|
|
|
|
- `phase2-comrak-010` remains `accepted` in the ledger by design — the
|
|
ledger is append-only and records what happened; the overturn lives in
|
|
the case study and is now enforced by the report generator too.
|
|
- `phase0-comrak-002` (mimalloc) is `needs-human-review`, awaiting a ruling
|
|
(see `results/rulings/`).
|
|
|
|
## Erratum (2026-07-13)
|
|
|
|
Finding 1's separation claim above — "only the historical overturned row
|
|
is `accepted + false`, so the rule cleanly separates them" — was too
|
|
strong. **Five** accepted rows carry `sanitizers_clean: false`:
|
|
`phase1-verdict-smoke-002`, `phase1-ipc-smoke-001`, `phase2-tokei-001`,
|
|
`phase2-tokei-003`, and `phase2-comrak-010`. All five pre-date the
|
|
machine-enforced sanitizer lane — the Phase 1 rows are harness smoke
|
|
tests, and the two Phase 2 tokei wins were sanitizer-audited by hand
|
|
during the 100% Phase 2 audit (see the Phase 2 case study) — but the
|
|
ledger's machine record cannot distinguish "manually audited clean" from
|
|
"never run", so the report generator's not-shippable rule flags all five,
|
|
not just the overturned row. That is the correct conservative behavior;
|
|
only the description of *why* the rule separates cleanly was wrong. The
|
|
original text stands unedited above; this note is the correction.
|