peltier.io/results/phase5/README.md

Phase 5 — the research forks (SPEC §13)

Phase 5 exercises the two frontier ideas the spec deliberately left out of the core roadmap. Neither adds a new product; each is a stress test that answers one question: does the trust machinery still hold when the shape of the work changes? The answer, both times, is yes — the gates, the interleaved A/B timer, and the append-only ledger transfer unchanged; only the gate policy (5b) or the ordering prior (5a) is new.

Nothing here relaxes a non-negotiable. Both forks route their headline result to needs-human-review, and the learned prior is advisory only — the gates still decide every verdict.

---

5a — learned class-selection policy

Artifact: crates/policy/, output in learned-policy.txt.

The ledger is now 34 rows of (playbook class, target) → verdict. That is a dataset. Phase 5a turns the fixed cheapest-first ordering of the playbook into a learned prior: rank each optimization class by the Wilson score lower bound of its observed shippable-win rate, and try the best-evidenced class first on a new hotspot.

Why the Wilson lower bound and not the raw win rate: a class that went 1-for-1 should not outrank a class that went 3-for-16. The Wilson lower bound penalizes small samples automatically — it is the same "be pessimistic about thin evidence" instinct as the bootstrap CI lower bound the bench uses to accept a speedup. Ties break untried-before-failed (an unexplored class keeps its cheapest-first prior; a class with evidence against it sinks), then cheapest-first.

Two guards keep it honest, both visible in the output footer:

• A "win" counts only a machine-sanitizer-verified accept. The overturned comrak-010 false-accept and every pre-sanitizer-gate accept are excluded — the policy trusts the ledger's machine record, not the audit narrative that caught them.
• The tier-gated allocator win (mimalloc) shows as held for human review, not as a loss and not as a shippable win — an accurate third category.

Current ranking (all targets, 34 rows):

rank	class	attempts	wins	wilson-lb	note
1	algorithmic	16	3	0.066	proven winner
2	alloc-churn	4	1	0.046	proven winner
3	build-config	9	1	0.020	proven winner
4	concurrency	0	0	0.000	untried — cheapest-first prior only
5	allocator	2	0	0.000	held for human review (tier-gated)
6	data-layout	1	0	0.000	evidence against
7	simd	2	0	0.000	evidence against

The recommendation the agent would act on: try algorithmic first (best evidence), then the next-ranked class whose profile preconditions match. This is exactly the prior art direction (MLGO, AlphaDev, LLM Compiler) at the scale the ledger currently supports — a ranked prior, sharpening as the dataset grows, never overriding a gate.

Reproduce: cargo run -p policy (add --target cjson to scope to one target).

---

5b — the kernel lane: FP-tolerance equivalence

Artifact: targets/matmul/, crates/diff-test/src/bin/fp-compare.rs, demo in kernel-lane.txt.

Every target through Phase 4 was gated by byte-identical golden replay: the optimized output must match the baseline bit for bit. That is the right gate for a parser or an HTTP service. It is the wrong gate for a numerical kernel, and the kernel lane is the case that proves it.

targets/matmul/kernel.c is a single-precision matrix multiply in two implementations:

• matmul_ref — naive i,j,k with one sequential accumulator. The oracle.
• matmul_opt — transpose B (so the inner reduction walks contiguous memory: the cache win) and eight independent accumulators tree-combined at the end (breaking the float-add dependency chain: the ILP win). The second lever reorders the reduction, so the last few ULPs differ by construction. Bit-identical output is impossible.

The demo (sh scripts/kernel-lane-demo.sh 512) walks the whole argument:

1. byte-identical FAILs — 244,901 of 262,144 result values differ. Under the Phase 0–4 gate, this correct 3× speedup would be rejected.
2. the FP-tolerance policy is EQUIVALENT — equivalence.toml declares mode = "fp-tolerance", abs = 1e-4, rel = 1e-3; every numeric token agrees within abs + rel·|ref|.
3. a genuine wrong result is still caught — perturbing one element by +0.5 is REJECTED by the same tolerance. The gate is loose enough for last-ULP reordering, tight enough for a real bug.
4. the speedup is measured with the same machinery — interleaved A/B, bootstrap 95% CI: median 3.23×, CI [3.16, 3.26].

The equivalence policy (EquivalencePolicy::compare in diff-test) is where the new rule lives: byte-identical stays exact; fp-tolerance tokenizes, compares numeric tokens within tolerance, handles NaN explicitly, and keeps non-numeric tokens and token counts exact. fp-compare applies a target's policy to two output files from the command line.

Verdict: needs-human-review (phase5-matmul-opt ledger row). Using the FP-tolerance tier at all is a human-review signal per SPEC §8 — a person confirms the declared tolerance is defensible for the workload, exactly as with the mimalloc allocator swap. The machine measures and gates; the human ratifies the tolerance.

Why this is the GPU lane in miniature

The environment has no GPU, so the GPU fork is demonstrated on the CPU as what it actually is: the same trust machinery with a different timer. A Triton or CUDA kernel versus a reference kernel is the identical shape — reference-kernel differential testing within a tolerance, interleaved timing with a bootstrap CI, a ledger row routed to human review. Only the profiler and the bench clock become kernel-time-aware, and the hardware changes. The correctness story, the statistics, and the audit trail are already built and already exercised here on real code.

Reproduce: sh scripts/kernel-lane-demo.sh 512.

---

What Phase 5 settles

The core loop was already complete at Phase 4. Phase 5 answers the "but does it generalize?" objection with the two hardest cases the spec named:

• A learned prior can be extracted from the ledger without weakening a single gate — it only reorders what to try first.
• A numerically-reordered optimization — the class byte-identical replay cannot handle — slots into the same harness by swapping the equivalence policy, with the correctness bar preserved (wrong results still caught) and the tolerance itself put under human review.

Zero shipped false accepts, still. The product is trust, and trust transferred.