mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
benbrownandClaude Opus 4.8
899d55f4b6
An adversarial audit found the enforcement story ahead of the code: gates that were documented as hard gates were advisory, manual, or absent, and several human-facing claims outran the machine record. This makes the claims true. High findings - Differential fuzz was documentation, not enforcement: verdict hardcoded fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and it is inherently baseline-vs-candidate, so it runs on the accept path and reports Skipped in the standalone `just gates` flow. New hard rule: a machine `accepted` verdict is impossible unless the fuzz gate actually passed — otherwise it caps at needs-human-review. - The agent's system prompt claimed FP/concurrency/unsafe patches were "routed to human review automatically". No such mechanism existed. Added a conservative lexical risk classifier (crates/verdict/src/risk.rs) that forces needs-human-review on any would-be accept carrying those signals; fp-tolerance equivalence always signals. - The local verdict path ran agent-influenced code unsandboxed. It now runs under scripts/no-net.sh (network namespace) and fails closed. Upstream test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when present (scripts/pin-testsuite.sh generates them). Also fixed, found while fixing the above - The pipeline script's `set -e` was inert (brace group as the left operand of `&&`), so a failing candidate build did not abort the run and verdict could bench a stale binary. Now a scoped subshell whose status is branched on at top level. - A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path) stranded RUNNING.lock and left read_verdict polling forever. Failure accounting now lives outside the wrapper. - harnessd's patch allowlist ignored git rename/copy headers and --no-prefix diffs; the rename-escape case is now a committed regression test. - The workspace did not compile on Windows (ungated Unix APIs). cfg-gated; 71 portable tests now run on both platforms. - pin-check regenerated the corpus manifest instead of verifying it; it is now verify-only, with a deliberate `just pin-corpus` to re-pin. - Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics (compare --perf-stat), never part of the accept decision. Docs now match the code, including what is NOT yet enforced: no target ships a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not wired to the fp-tolerance gate — each is stated as mechanized-but-inert rather than done. Corrected the stale "33 ledger rows" (34), the playbook's class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation claim (five accepted rows carry sanitizers_clean=false), the unsupported "40,000 requests" figure, and the blanket "historical rows record fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
168 lines
8.1 KiB
Markdown
168 lines
8.1 KiB
Markdown
# Phase 5 — the research forks (SPEC §13)
|
||
|
||
Phase 5 exercises the two frontier ideas the spec deliberately left out of
|
||
the core roadmap. Neither adds a new product; each is a stress test that
|
||
answers one question: **does the trust machinery still hold when the shape
|
||
of the work changes?** The answer, both times, is yes — the gates, the
|
||
interleaved A/B timer, and the append-only ledger transfer unchanged; only
|
||
the *gate policy* (5b) or the *ordering prior* (5a) is new.
|
||
|
||
Nothing here relaxes a non-negotiable. Both forks route their headline
|
||
result to `needs-human-review`, and the learned prior is advisory only —
|
||
the gates still decide every verdict.
|
||
|
||
---
|
||
|
||
## 5a — learned class-selection policy
|
||
|
||
**Artifact:** `crates/policy/`, output in `learned-policy.txt`.
|
||
|
||
The ledger is now 34 rows of `(playbook class, target) → verdict`. That is
|
||
a dataset. Phase 5a turns the *fixed* cheapest-first ordering of the
|
||
playbook into a *learned* prior: rank each optimization class by the
|
||
**Wilson score lower bound** of its observed shippable-win rate, and try the
|
||
best-evidenced class first on a new hotspot.
|
||
|
||
Why the Wilson lower bound and not the raw win rate: a class that went 1-for-1
|
||
should not outrank a class that went 3-for-16. The Wilson lower bound
|
||
penalizes small samples automatically — it is the same "be pessimistic
|
||
about thin evidence" instinct as the bootstrap CI lower bound the bench
|
||
uses to accept a speedup. Ties break *untried-before-failed* (an unexplored
|
||
class keeps its cheapest-first prior; a class with evidence *against* it
|
||
sinks), then cheapest-first.
|
||
|
||
Two guards keep it honest, both visible in the output footer:
|
||
|
||
- A "win" counts **only** a machine-sanitizer-verified accept. The
|
||
overturned `comrak-010` false-accept and every pre-sanitizer-gate accept
|
||
are excluded — the policy trusts the ledger's machine record, not the
|
||
audit narrative that caught them.
|
||
- The tier-gated allocator win (mimalloc) shows as `held for human review`,
|
||
not as a loss and not as a shippable win — an accurate third category.
|
||
|
||
Current ranking (all targets, 34 rows):
|
||
|
||
| rank | class | attempts | wins | wilson-lb | note |
|
||
|---|---|---|---|---|---|
|
||
| 1 | algorithmic | 16 | 3 | 0.066 | proven winner |
|
||
| 2 | alloc-churn | 4 | 1 | 0.046 | proven winner |
|
||
| 3 | build-config | 9 | 1 | 0.020 | proven winner |
|
||
| 4 | concurrency | 0 | 0 | 0.000 | untried — cheapest-first prior only |
|
||
| 5 | allocator | 2 | 0 | 0.000 | held for human review (tier-gated) |
|
||
| 6 | data-layout | 1 | 0 | 0.000 | evidence against |
|
||
| 7 | simd | 2 | 0 | 0.000 | evidence against |
|
||
|
||
The recommendation the agent would act on: **try algorithmic first** (best
|
||
evidence), then the next-ranked class whose profile preconditions match.
|
||
This is exactly the prior art direction (MLGO, AlphaDev, LLM Compiler) at
|
||
the scale the ledger currently supports — a ranked prior, sharpening as the
|
||
dataset grows, never overriding a gate.
|
||
|
||
Reproduce: `cargo run -p policy` (add `--target cjson` to scope to one
|
||
target).
|
||
|
||
---
|
||
|
||
## 5b — the kernel lane: FP-tolerance equivalence
|
||
|
||
**Artifact:** `targets/matmul/`, `crates/diff-test/src/bin/fp-compare.rs`,
|
||
demo in `kernel-lane.txt`.
|
||
|
||
Every target through Phase 4 was gated by **byte-identical** golden replay:
|
||
the optimized output must match the baseline bit for bit. That is the right
|
||
gate for a parser or an HTTP service. It is the *wrong* gate for a numerical
|
||
kernel, and the kernel lane is the case that proves it.
|
||
|
||
`targets/matmul/kernel.c` is a single-precision matrix multiply in two
|
||
implementations:
|
||
|
||
- `matmul_ref` — naive `i,j,k` with one sequential accumulator. The oracle.
|
||
- `matmul_opt` — transpose B (so the inner reduction walks contiguous
|
||
memory: the cache win) **and** eight independent accumulators tree-combined
|
||
at the end (breaking the float-add dependency chain: the ILP win). The
|
||
second lever **reorders the reduction**, so the last few ULPs differ *by
|
||
construction*. Bit-identical output is impossible.
|
||
|
||
The demo (`sh scripts/kernel-lane-demo.sh 512`) walks the whole argument:
|
||
|
||
1. **byte-identical FAILs** — 244,901 of 262,144 result values differ. Under
|
||
the Phase 0–4 gate, this correct 3× speedup would be *rejected*.
|
||
2. **the FP-tolerance policy is EQUIVALENT** — `equivalence.toml` declares
|
||
`mode = "fp-tolerance"`, `abs = 1e-4`, `rel = 1e-3`; every numeric token
|
||
agrees within `abs + rel·|ref|`.
|
||
3. **a genuine wrong result is still caught** — perturbing one element by
|
||
`+0.5` is REJECTED by the same tolerance. The gate is loose enough for
|
||
last-ULP reordering, tight enough for a real bug.
|
||
4. **the speedup is measured with the same machinery** — interleaved A/B,
|
||
bootstrap 95% CI: **median 3.23×, CI [3.16, 3.26]**.
|
||
|
||
The equivalence policy (`EquivalencePolicy::compare` in `diff-test`) is where
|
||
the new rule lives: byte-identical stays exact; fp-tolerance tokenizes,
|
||
compares numeric tokens within tolerance, handles NaN explicitly, and keeps
|
||
non-numeric tokens and token counts exact. `fp-compare` applies a target's
|
||
policy to two output files from the command line.
|
||
|
||
**Verdict: `needs-human-review`** (`phase5-matmul-opt` ledger row). Using
|
||
the FP-tolerance tier *at all* is a human-review signal per SPEC §8 — a
|
||
person confirms the declared tolerance is defensible for the workload,
|
||
exactly as with the mimalloc allocator swap. The machine measures and
|
||
gates; the human ratifies the tolerance.
|
||
|
||
### Why this is the GPU lane in miniature
|
||
|
||
The environment has no GPU, so the GPU fork is demonstrated on the CPU as
|
||
what it actually is: **the same trust machinery with a different timer.** A
|
||
Triton or CUDA kernel versus a reference kernel is the identical shape —
|
||
reference-kernel differential testing within a tolerance, interleaved timing
|
||
with a bootstrap CI, a ledger row routed to human review. Only the profiler
|
||
and the bench clock become kernel-time-aware, and the hardware changes. The
|
||
correctness story, the statistics, and the audit trail are already built and
|
||
already exercised here on real code.
|
||
|
||
Reproduce: `sh scripts/kernel-lane-demo.sh 512`.
|
||
|
||
---
|
||
|
||
## What Phase 5 settles
|
||
|
||
The core loop was already complete at Phase 4. Phase 5 answers the "but does
|
||
it generalize?" objection with the two hardest cases the spec named:
|
||
|
||
- A **learned prior** can be extracted from the ledger without weakening a
|
||
single gate — it only reorders *what to try first*.
|
||
- A **numerically-reordered optimization** — the class byte-identical replay
|
||
cannot handle — slots into the same harness by swapping the equivalence
|
||
policy, with the correctness bar preserved (wrong results still caught) and
|
||
the tolerance itself put under human review.
|
||
|
||
Zero shipped false accepts, still. The product is trust, and trust
|
||
transferred.
|
||
|
||
---
|
||
|
||
## Erratum (2026-07-13) — how far the kernel lane is actually wired
|
||
|
||
The write-up above describes the FP-tolerance gate as though `matmul` runs
|
||
through the standard pipeline. It does not, and the distinction matters:
|
||
|
||
- **The policy is in the pipeline.** `EquivalencePolicy` is honored by
|
||
`diff-test`'s gate sequence (an fp-tolerance target needs a committed
|
||
`[corpus].golden_reference`; the gate fails closed without one), and
|
||
`crates/verdict` treats fp-tolerance equivalence as an automatic
|
||
`needs-human-review` signal. That machinery is real and unit-tested.
|
||
- **`matmul` is not a pipeline target.** It ships `targets/matmul/kernel.c`
|
||
and `targets/matmul/equivalence.toml` but **no `target.toml`** — the file
|
||
`diff-test`/`verdict` load a target from. So the lane cannot currently be
|
||
run via `just gates` or `just verdict` at all.
|
||
- **Therefore the demonstration is script-driven.**
|
||
`scripts/kernel-lane-demo.sh` invokes the `fp-compare` binary and
|
||
`bench-runner compare` directly, and the `phase5-matmul-opt` ledger row
|
||
was recorded **outside** the automated verdict path.
|
||
|
||
Nothing in the measured result changes — the 3.23× [3.16, 3.26] speedup, the
|
||
244,901/262,144 differing values, the rejected +0.5 perturbation, and the
|
||
`needs-human-review` verdict all stand as printed. What was overstated is the
|
||
*integration*: the equivalence policy is closed; wiring this target into the
|
||
pipeline is open work. The original text stands unedited above; this note is
|
||
the correction.
|