mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
gates: close the audit findings — enforce the gates the docs claimed
An adversarial audit found the enforcement story ahead of the code: gates that were documented as hard gates were advisory, manual, or absent, and several human-facing claims outran the machine record. This makes the claims true. High findings - Differential fuzz was documentation, not enforcement: verdict hardcoded fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and it is inherently baseline-vs-candidate, so it runs on the accept path and reports Skipped in the standalone `just gates` flow. New hard rule: a machine `accepted` verdict is impossible unless the fuzz gate actually passed — otherwise it caps at needs-human-review. - The agent's system prompt claimed FP/concurrency/unsafe patches were "routed to human review automatically". No such mechanism existed. Added a conservative lexical risk classifier (crates/verdict/src/risk.rs) that forces needs-human-review on any would-be accept carrying those signals; fp-tolerance equivalence always signals. - The local verdict path ran agent-influenced code unsandboxed. It now runs under scripts/no-net.sh (network namespace) and fails closed. Upstream test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when present (scripts/pin-testsuite.sh generates them). Also fixed, found while fixing the above - The pipeline script's `set -e` was inert (brace group as the left operand of `&&`), so a failing candidate build did not abort the run and verdict could bench a stale binary. Now a scoped subshell whose status is branched on at top level. - A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path) stranded RUNNING.lock and left read_verdict polling forever. Failure accounting now lives outside the wrapper. - harnessd's patch allowlist ignored git rename/copy headers and --no-prefix diffs; the rename-escape case is now a committed regression test. - The workspace did not compile on Windows (ungated Unix APIs). cfg-gated; 71 portable tests now run on both platforms. - pin-check regenerated the corpus manifest instead of verifying it; it is now verify-only, with a deliberate `just pin-corpus` to re-pin. - Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics (compare --perf-stat), never part of the accept decision. Docs now match the code, including what is NOT yet enforced: no target ships a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not wired to the fp-tolerance gate — each is stated as mechanized-but-inert rather than done. Corrected the stale "33 ledger rows" (34), the playbook's class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation claim (five accepted rows carry sanitizers_clean=false), the unsupported "40,000 requests" figure, and the blanket "historical rows record fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -57,3 +57,19 @@ consistent, with the compound tokei figure stated as an approximation
|
||||
the case study and is now enforced by the report generator too.
|
||||
- `phase0-comrak-002` (mimalloc) is `needs-human-review`, awaiting a ruling
|
||||
(see `results/rulings/`).
|
||||
|
||||
## Erratum (2026-07-13)
|
||||
|
||||
Finding 1's separation claim above — "only the historical overturned row
|
||||
is `accepted + false`, so the rule cleanly separates them" — was too
|
||||
strong. **Five** accepted rows carry `sanitizers_clean: false`:
|
||||
`phase1-verdict-smoke-002`, `phase1-ipc-smoke-001`, `phase2-tokei-001`,
|
||||
`phase2-tokei-003`, and `phase2-comrak-010`. All five pre-date the
|
||||
machine-enforced sanitizer lane — the Phase 1 rows are harness smoke
|
||||
tests, and the two Phase 2 tokei wins were sanitizer-audited by hand
|
||||
during the 100% Phase 2 audit (see the Phase 2 case study) — but the
|
||||
ledger's machine record cannot distinguish "manually audited clean" from
|
||||
"never run", so the report generator's not-shippable rule flags all five,
|
||||
not just the overturned row. That is the correct conservative behavior;
|
||||
only the description of *why* the rule separates cleanly was wrong. The
|
||||
original text stands unedited above; this note is the correction.
|
||||
|
||||
@@ -230,3 +230,14 @@ remains the deployment shape.
|
||||
pipeline into machine-enforced sanitizer gating.
|
||||
- **OS-level tool-boundary isolation:** met — was the tracked hardening
|
||||
gap; shipped and verified this phase.
|
||||
|
||||
---
|
||||
|
||||
> **Note (2026-07-13).** The differential-fuzz and ASan/LSan runs
|
||||
> described above ran **out-of-band** — per-attempt scripts during the
|
||||
> 100% human audit, exactly as this case study records ("fuzz and
|
||||
> sanitizers are per-attempt manual in this phase") — so the Phase 2
|
||||
> ledger rows carry `fuzz_iters=0` in their machine record. The fuzz
|
||||
> gate is now pipeline-integrated (`diff-test` runs the target's
|
||||
> declared fuzz command); rows written after this date record the
|
||||
> iteration count actually executed.
|
||||
|
||||
@@ -62,3 +62,12 @@ increments, all on the same 40%-of-runtime function — and one honest
|
||||
rejection in between (`phase2-tokei-009`, a real +5.7% median whose
|
||||
noisy CI lower bound of +0.2% could not clear the bar, so it does not
|
||||
count).
|
||||
|
||||
---
|
||||
|
||||
> **Note (2026-07-13).** The 10,000-input differential fuzz cited above
|
||||
> ran **out-of-band** (`scripts/diff-fuzz-tokei.py`, per-attempt), not
|
||||
> inside the verdict pipeline — so these ledger rows record
|
||||
> `fuzz_iters=0` in their machine record. The fuzz gate is now
|
||||
> pipeline-integrated; rows written after this date record the
|
||||
> iteration count actually executed.
|
||||
|
||||
@@ -84,3 +84,12 @@ about the trust story is Rust-specific. The cheapest safe lever (skip a
|
||||
per-number allocation) wasn't enough; the algorithmic lever (stop doing a
|
||||
redundant heavyweight parse) was — and the pipeline proved it the same
|
||||
way it proves everything else.
|
||||
|
||||
---
|
||||
|
||||
> **Note (2026-07-13).** The 10,000-input differential fuzz cited above
|
||||
> ran **out-of-band** (`scripts/diff-fuzz-cjson.py`, per-attempt), not
|
||||
> inside the verdict pipeline — so the `phase3-cjson-002` ledger row
|
||||
> records `fuzz_iters=0` in its machine record. The fuzz gate is now
|
||||
> pipeline-integrated; rows written after this date record the
|
||||
> iteration count actually executed.
|
||||
|
||||
@@ -66,3 +66,13 @@ fuzz, and sanitizers. Every rejection is a real mechanism that simply
|
||||
didn't clear the 2% CI-lower-bound bar — and it is in the ledger too,
|
||||
because a rejection is a complete, valid outcome. Nothing shipped that
|
||||
the pipeline couldn't prove.
|
||||
|
||||
> **Note (2026-07-13).** "Every win survived … a 10k-input differential
|
||||
> fuzz, and sanitizers" is true of the **evidence**, not of the machine
|
||||
> record at the time: through Phase 5 the fuzz ran **out-of-band**
|
||||
> (`scripts/diff-fuzz-<target>.py`, per-attempt) and the Phase 3 rows
|
||||
> therefore carry `fuzz_iters=0`. The fuzz gate is now integrated into the
|
||||
> accept path, where a passed fuzz gate is a **precondition** for a machine
|
||||
> `accepted` verdict; rows written after this date record the iteration
|
||||
> count actually executed. See the per-case-study notes in
|
||||
> `01-tokei-algorithmic-rust.md` and `04-cjson-c-target.md`.
|
||||
|
||||
@@ -36,9 +36,11 @@ numbers.
|
||||
## The result (`phase4-cjson-service`, ledger)
|
||||
|
||||
The accepted batch win `phase3-cjson-002` (+8.85% throughput) was
|
||||
measured as a **service under a 150 rps coordinated-omission-correct
|
||||
measured as a **service under a 150 rps-target coordinated-omission-correct
|
||||
replay**, baseline (pristine cJSON v1.7.19) vs candidate (the banked
|
||||
win), 20 interleaved rounds, server pinned, 40,000 requests, **0 drops**:
|
||||
win), 20 interleaved rounds, server pinned, **0 drops** — the ledger row
|
||||
records `sessions=20`, `rate_rps=150`, `drop_rate=0.0`; the total request
|
||||
count was **not machine-recorded** (see errata below):
|
||||
|
||||
| metric | baseline | candidate | speedup | 95% CI |
|
||||
|---|---|---|---|---|
|
||||
@@ -63,7 +65,10 @@ project in one line.
|
||||
## The ROI, generated mechanically
|
||||
|
||||
`just report phase4-cjson-service --service-json results/phase4/cjson-service.json`
|
||||
→ [`cjson-service-roi.md`](cjson-service-roi.md). On a 500-core fleet at
|
||||
→ [`cjson-service-roi.md`](cjson-service-roi.md). *(Note, 2026-07-13: the
|
||||
raw `cjson-service.json` was not preserved — see errata below — so this
|
||||
command is not reproducible as written; the generated report and the
|
||||
ledger row are the surviving evidence.)* On a 500-core fleet at
|
||||
$0.04/core-hour, the +6.2% p50 win returns **27.5 cores / $9,621 per
|
||||
year at the CI lower bound** (median 29.4 cores / $10,290) — and the
|
||||
report quotes the lower bound, because the point estimate is not a
|
||||
@@ -79,3 +84,20 @@ that produced it, so the number survives hostile review.
|
||||
ledger row and the calibrated service JSON with no hand-editing;
|
||||
throughput→cores→dollars and latency percentiles, CIs and methodology
|
||||
inline.
|
||||
|
||||
## Errata (2026-07-13)
|
||||
|
||||
- **"40,000 requests" retracted.** This report originally stated the
|
||||
replay comprised 40,000 requests. The `phase4-cjson-service` ledger row
|
||||
records only `sessions=20`, `rate_rps=150`, and `drop_rate=0.0` — the
|
||||
per-run request count was never machine-recorded, so the 40,000 figure
|
||||
(an arithmetic reconstruction, not evidence) is withdrawn. The prose
|
||||
above now states the ledger-backed facts only. The claimed result is
|
||||
unaffected: p50 latency +6.2%, 95% CI [+5.8%, +7.2%], under the 150
|
||||
rps-target CO-correct replay, 20 interleaved rounds, 0 drops.
|
||||
- **`results/phase4/cjson-service.json` was not preserved.** The
|
||||
`.gitignore` rule `results/**/*.json` swept the raw service-latency
|
||||
JSON that the report command consumes, so it no longer exists in the
|
||||
repository. A `!results/phase4/*.json` carve-out has been added so
|
||||
future service-mode evidence JSONs are committed alongside the
|
||||
calibration JSONs.
|
||||
|
||||
@@ -137,3 +137,31 @@ it generalize?" objection with the two hardest cases the spec named:
|
||||
|
||||
Zero shipped false accepts, still. The product is trust, and trust
|
||||
transferred.
|
||||
|
||||
---
|
||||
|
||||
## Erratum (2026-07-13) — how far the kernel lane is actually wired
|
||||
|
||||
The write-up above describes the FP-tolerance gate as though `matmul` runs
|
||||
through the standard pipeline. It does not, and the distinction matters:
|
||||
|
||||
- **The policy is in the pipeline.** `EquivalencePolicy` is honored by
|
||||
`diff-test`'s gate sequence (an fp-tolerance target needs a committed
|
||||
`[corpus].golden_reference`; the gate fails closed without one), and
|
||||
`crates/verdict` treats fp-tolerance equivalence as an automatic
|
||||
`needs-human-review` signal. That machinery is real and unit-tested.
|
||||
- **`matmul` is not a pipeline target.** It ships `targets/matmul/kernel.c`
|
||||
and `targets/matmul/equivalence.toml` but **no `target.toml`** — the file
|
||||
`diff-test`/`verdict` load a target from. So the lane cannot currently be
|
||||
run via `just gates` or `just verdict` at all.
|
||||
- **Therefore the demonstration is script-driven.**
|
||||
`scripts/kernel-lane-demo.sh` invokes the `fp-compare` binary and
|
||||
`bench-runner compare` directly, and the `phase5-matmul-opt` ledger row
|
||||
was recorded **outside** the automated verdict path.
|
||||
|
||||
Nothing in the measured result changes — the 3.23× [3.16, 3.26] speedup, the
|
||||
244,901/262,144 differing values, the rejected +0.5 perturbation, and the
|
||||
`needs-human-review` verdict all stand as printed. What was overstated is the
|
||||
*integration*: the equivalence policy is closed; wiring this target into the
|
||||
pipeline is open work. The original text stands unedited above; this note is
|
||||
the correction.
|
||||
|
||||
Reference in New Issue
Block a user