phase5: closeout — research-forks case study, docs chapter, roadmap + CLAUDE.md

Phase 5 write-up for the two SPEC §13 forks now that both are built:
  - results/phase5/README.md — the full argument for each fork, with the
    kernel-lane demo and learned-policy ranking tables and reproduce commands.
  - results/phase5/learned-policy.txt — captured `cargo run -p policy` output
    over the 34-row ledger.
  - docs/src/research-forks.md — new mdBook chapter, wired into SUMMARY.md.
  - docs/src/roadmap.md — Phase 5 row added; the Frontier section rewritten
    from "out of scope" to "built, both routed to human review", with the
    honest note that the actual GPU run needs GPU hardware absent here.
  - CLAUDE.md — Phase 5 COMPLETE status paragraph.

No trust-layer or gate change; this is documentation of work already
committed (learned policy b7297ad, kernel lane 5abeb60).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2

CLAUDE.md

@@ -30,7 +30,9 @@ Full engineering spec: `SPEC.md`. Read it before starting any phase.
 - `playbook/` — optimization classes, ordered, with preconditions and known risks
 - `results/` — generated reports and flamegraphs
 
-**Phase 4 COMPLETE — all SPEC §5 exit criteria met. Phases 0–4 are all done.** Service-mode latency benchmarking shipped: a coordinated-omission-correct, open-loop, fixed-rate load generator (`crates/bench-runner/src/service.rs`) measuring latency from each request's intended send time (queueing counted, not hidden), interleaved A/B, exact percentiles, bootstrap p50/p99 CIs. A real service target — `targets/cjson/service.c`, a minimal HTTP server wrapping the patched `cJSON.c` (trust-layer, outside the workspace allowlist). Service calibration PASS (0/10 A/A false-positive, 10/10 injected-5%-latency-regression detection; `results/calibration/cjson-service-aa.json`). **The verified batch win phase3-cjson-002 measured under a 150 rps replay (20 rounds, 40k requests, 0 drops): p50 latency +6.2%, 95% CI [+5.8%, +7.2%] — accepted (`phase4-cjson-service` ledger row); p99 NOT claimed (CI [0.07, 4.97], single-worker loopback tail jitter) — harness correctly rejected it.** Mechanical ROI report generator (`crates/report` bin, `just report`, SPEC §9): reads a ledger row + service-latency JSON → throughput→cores→dollars (27.5 cores / $9,621-per-year CI lower bound on a 500-core fleet) *and* latency percentiles, every figure with CI + workload + methodology inline; flags any non-accepted row. Case study: `results/phase4/`. **The roadmap is complete — the profile→hypothesize→patch→gated-verdict→ROI loop runs end to end across Rust and C, batch and service, with zero shipped false accepts across all phases. Open follow-ups: the standing human ruling on `phase0-comrak-002` (mimalloc, needs-human-review); SPEC §13 research forks (GPU lane, learned policies) remain out of scope.**
+**Phase 4 COMPLETE — all SPEC §5 exit criteria met. Phases 0–4 are all done.** Service-mode latency benchmarking shipped: a coordinated-omission-correct, open-loop, fixed-rate load generator (`crates/bench-runner/src/service.rs`) measuring latency from each request's intended send time (queueing counted, not hidden), interleaved A/B, exact percentiles, bootstrap p50/p99 CIs. A real service target — `targets/cjson/service.c`, a minimal HTTP server wrapping the patched `cJSON.c` (trust-layer, outside the workspace allowlist). Service calibration PASS (0/10 A/A false-positive, 10/10 injected-5%-latency-regression detection; `results/calibration/cjson-service-aa.json`). **The verified batch win phase3-cjson-002 measured under a 150 rps replay (20 rounds, 40k requests, 0 drops): p50 latency +6.2%, 95% CI [+5.8%, +7.2%] — accepted (`phase4-cjson-service` ledger row); p99 NOT claimed (CI [0.07, 4.97], single-worker loopback tail jitter) — harness correctly rejected it.** Mechanical ROI report generator (`crates/report` bin, `just report`, SPEC §9): reads a ledger row + service-latency JSON → throughput→cores→dollars (27.5 cores / $9,621-per-year CI lower bound on a 500-core fleet) *and* latency percentiles, every figure with CI + workload + methodology inline; flags any non-accepted row. Case study: `results/phase4/`. **The roadmap is complete — the profile→hypothesize→patch→gated-verdict→ROI loop runs end to end across Rust and C, batch and service, with zero shipped false accepts across all phases.**
+
+**Phase 5 COMPLETE — the two SPEC §13 research forks, both built and both routed to human review.** (a) *Learned class-selection policy* (`crates/policy`, `cargo run -p policy`): reads the append-only ledger and ranks optimization classes by the Wilson lower bound of their shippable-win rate — a learned prior over the fixed cheapest-first ordering, advisory only (gates still decide). A "win" counts only machine-sanitizer-verified accepts; the overturned comrak-010 and tier-gated mimalloc are excluded/held. On the current 34-row ledger it recommends algorithmic first (Wilson lb 0.066). (b) *Kernel lane* (`targets/matmul/kernel.c`, `crates/diff-test` FP-tolerance policy + `fp-compare` bin, `scripts/kernel-lane-demo.sh`): a matmul optimization (transpose-B cache locality + eight-accumulator ILP) that **reorders FP accumulation**, so byte-identical golden replay is the wrong gate — 244,901/262,144 values differ. The FP-tolerance gate (`abs 1e-4 + rel 1e-3`) accepts it, still REJECTS a +0.5 perturbation, and the bench measures 3.23× [3.16, 3.26] with the same interleaved A/B + bootstrap CI. Ledger row `phase5-matmul-opt` = needs-human-review (using the tolerance tier is a §8 signal, same posture as mimalloc). No GPU in this environment, so the GPU fork is the identical shape shown on CPU — only the timer and hardware change. Case study: `results/phase5/`; docs chapter `docs/src/research-forks.md`. Open follow-up: the standing accept-scoped ruling on `phase0-comrak-002` (mimalloc) is resolved (`results/rulings/`).**
 
 ## Commands
 

docs/src/SUMMARY.md

@@ -19,6 +19,7 @@
 - [The caught false-accept (comrak)](./case-studies/comrak-false-accept.md)
 - [cJSON — the cross-language proof (C)](./case-studies/cjson.md)
 - [Services & scale: latency under load](./case-studies/service.md)
+- [Research forks: kernel lane & learned policy](./research-forks.md)
 
 # Reference
 

docs/src/research-forks.md

@@ -0,0 +1,109 @@
+# Research forks: kernel lane & learned policy
+
+The core loop was complete at Phase 4 — profile → hypothesize → patch →
+gated-verdict → ROI, running unattended across Rust and C, batch and
+service, with zero shipped false accepts. SPEC §13 named two frontier ideas
+that the roadmap deliberately excluded, precisely because they change the
+*shape* of the work rather than extend the loop. Phase 5 built both, as
+stress tests with one question each: **does the trust machinery still hold?**
+
+Both times it does. Neither fork relaxes a non-negotiable; each routes its
+headline result to `needs-human-review`, and the learned prior is advisory
+only. What follows is the argument for each.
+
+## The kernel lane: when byte-identical is the wrong gate
+
+Every target through Phase 4 was gated by **byte-identical golden replay** —
+the optimized output must match the baseline bit for bit. For a parser or an
+HTTP service that is exactly right. For a numerical kernel it is wrong, and
+the kernel lane is the case that proves it.
+
+`targets/matmul/kernel.c` is a single-precision matrix multiply in two forms:
+
+- **`matmul_ref`** — naive `i,j,k`, one sequential accumulator. The oracle.
+- **`matmul_opt`** — transpose B so the inner reduction walks contiguous
+  memory (the cache win), plus eight independent accumulators tree-combined
+  at the end (the ILP win, breaking the float-add dependency chain). The
+  second lever **reorders the reduction**, so the last few ULPs differ *by
+  construction*. Bit-identical output is impossible.
+
+The demonstration (`sh scripts/kernel-lane-demo.sh 512`) walks the argument:
+
+1. **byte-identical FAILs** — 244,901 of 262,144 values differ. Under the
+   Phase 0–4 gate this correct 3× speedup would be *rejected*.
+2. **FP-tolerance is EQUIVALENT** — the target declares `mode =
+   "fp-tolerance"`, `abs = 1e-4`, `rel = 1e-3`; every numeric token agrees
+   within `abs + rel·|ref|`.
+3. **a wrong result is still caught** — perturbing one element by `+0.5` is
+   REJECTED by the same tolerance. Loose enough for last-ULP reordering,
+   tight enough for a real bug.
+4. **measured with the same machinery** — interleaved A/B, bootstrap 95% CI:
+   **median 3.23×, CI [3.16, 3.26]**.
+
+The new rule lives in one place — `EquivalencePolicy::compare` in
+`diff-test`. Byte-identical stays exact; fp-tolerance tokenizes, compares
+numeric tokens within tolerance, handles NaN explicitly, and keeps
+non-numeric tokens and token counts exact. The `fp-compare` binary applies a
+target's `equivalence.toml` to two output files from the command line.
+
+The verdict is **`needs-human-review`**. Using the FP-tolerance tier *at
+all* is a human-review signal (SPEC §8): a person confirms the declared
+tolerance is defensible for the workload, exactly as with the mimalloc
+allocator swap. The machine measures and gates; the human ratifies the
+tolerance.
+
+**This is the GPU lane in miniature.** The environment has no GPU, so the
+GPU fork is shown on the CPU as what it actually is — the same trust
+machinery with a different timer. A Triton or CUDA kernel versus a reference
+kernel is the identical shape: reference-kernel differential testing within
+a tolerance, interleaved timing with a bootstrap CI, a ledger row routed to
+human review. Only the profiler and the bench clock become
+kernel-time-aware, and the hardware changes. The correctness story, the
+statistics, and the audit trail are already built and exercised here on real
+code.
+
+## The learned policy: the ledger as a training set
+
+The playbook orders optimization classes cheapest-first — a fixed prior. But
+the ledger is now dozens of rows of `(class, target) → verdict`. That is a
+dataset, and it can turn the fixed ordering into a *learned* one.
+
+`crates/policy` ranks each class by the **Wilson score lower bound** of its
+observed shippable-win rate. The Wilson lower bound is the same instinct as
+the bootstrap CI lower bound the bench uses to accept a speedup: be
+pessimistic about thin evidence. A class that went 1-for-1 should not
+outrank one that went 3-for-16 — and it does not, because the lower bound
+penalizes small samples automatically. Ties break untried-before-failed (an
+unexplored class keeps its cheapest-first prior; a class with evidence
+*against* it sinks), then cheapest-first.
+
+Two guards keep it honest, both printed in the output footer:
+
+- A "win" counts **only** a machine-sanitizer-verified accept. The
+  overturned `comrak-010` false-accept and every pre-sanitizer-gate accept
+  are excluded — the policy trusts the ledger's machine record, not the
+  audit narrative that caught them.
+- The tier-gated mimalloc win shows as *held for human review* — a third
+  category, neither a loss nor a shippable win.
+
+On the current 34-row ledger the ranking recommends **algorithmic first**
+(best evidence, Wilson lb 0.066), then the next-ranked class whose profile
+preconditions match. It is advisory: the equivalence and significance gates
+still decide every verdict. This is the MLGO / AlphaDev / LLM-Compiler
+direction at the scale the ledger currently supports — a ranked prior that
+sharpens as the dataset grows and never overrides a gate.
+
+Run it with `cargo run -p policy` (add `--target <name>` to scope the
+evidence to one target).
+
+## What the forks settle
+
+- A **learned prior** can be extracted from the ledger without weakening a
+  single gate — it only reorders *what to try first*.
+- A **numerically-reordered optimization** — the class byte-identical replay
+  cannot handle — slots into the same harness by swapping the equivalence
+  policy, with the correctness bar preserved and the tolerance itself put
+  under human review.
+
+The full write-up, with reproduction commands and the raw tool output, is in
+[`results/phase5/`](https://github.com/bdbrown4/peltier.io/tree/main/results/phase5).

docs/src/roadmap.md

@@ -9,6 +9,7 @@
 | **2** | Unattended agent loop | 20 audited attempts, 3 verified wins, **0 shipped false accepts**, OS-isolated |
 | **3** | Playbook + proof, C/C++ | cross-language cJSON win (+8.85%), 4 case studies, 6 playbook classes |
 | **4** | Services & scale | cJSON service +6.2% p50 under CO-correct replay, mechanical ROI report |
+| **5** | Research forks (SPEC §13) | FP-tolerance kernel lane (matmul 3.23×, needs-human-review) + learned class-selection prior from the ledger |
 
 The `profile → hypothesize → patch → gated-verdict → ROI` loop runs
 unattended, OS-isolated, across **Rust and C**, **batch and service**, from
@@ -16,22 +17,27 @@ a pinned corpus to a dollar figure — with zero shipped false accepts across
 all phases, and the two pipeline over-accepts both caught by audit and
 turned into permanent gates.
 
-## Frontier — the research forks (SPEC §13)
+## Phase 5 — the research forks (SPEC §13)
 
-Deliberately out of the core roadmap, these are where the largest gaps
+Deliberately out of the core roadmap, these are the two hardest cases the
-live:
+spec named — both now built and exercised, both routed to human review.
+Full write-up: [Research forks](./research-forks.md).
 
-- **GPU kernel lane.** Triton/CUDA targets where 2–10× gaps are common;
+- **Kernel lane — done as a CPU demonstration.** A matmul optimization that
-  correctness via reference-kernel differential testing; reward = measured
+  reorders floating-point accumulation (transpose-B + eight-accumulator ILP)
-  kernel time. The trust machinery (differential equivalence, interleaved
+  runs 3.23× faster and is *impossible* to gate byte-identically. It slots
-  timing, append-only ledger) transfers directly; the profiler and bench
+  into the harness by swapping the equivalence policy for a declared
-  become kernel-time-aware, and it needs GPU hardware to run.
+  FP-tolerance (`abs + rel·|ref|`) — which still catches a genuine wrong
-- **Learned optimization policies.** The ledger is a growing dataset of
+  result — and routes to `needs-human-review` because using the tolerance
-  `(profile signature, class, hypothesis) → verdict`. That is training data
+  tier at all is a §8 signal. This is the GPU lane in miniature:
-  for a policy that *ranks* which class to try first on a new hotspot,
+  reference-kernel differential testing, interleaved timing, ledger row.
-  turning the fixed cheapest-first ordering into a learned prior. Prior art:
+  Only the timer and the hardware change; **the actual GPU run needs GPU
-  MLGO, AlphaDev, Meta's LLM Compiler; ProGraML-style IR graphs as the
+  hardware**, absent here.
-  program representation.
+- **Learned optimization policies — done.** `crates/policy` reads the ledger
+  and ranks optimization classes by the Wilson lower bound of their
+  shippable-win rate, turning the fixed cheapest-first ordering into a
+  learned prior (advisory only — the gates still decide). Prior art: MLGO,
+  AlphaDev, Meta's LLM Compiler.
 
 ## The real target
 

results/phase5/README.md

@@ -0,0 +1,139 @@
+# Phase 5 — the research forks (SPEC §13)
+
+Phase 5 exercises the two frontier ideas the spec deliberately left out of
+the core roadmap. Neither adds a new product; each is a stress test that
+answers one question: **does the trust machinery still hold when the shape
+of the work changes?** The answer, both times, is yes — the gates, the
+interleaved A/B timer, and the append-only ledger transfer unchanged; only
+the *gate policy* (5b) or the *ordering prior* (5a) is new.
+
+Nothing here relaxes a non-negotiable. Both forks route their headline
+result to `needs-human-review`, and the learned prior is advisory only —
+the gates still decide every verdict.
+
+---
+
+## 5a — learned class-selection policy
+
+**Artifact:** `crates/policy/`, output in `learned-policy.txt`.
+
+The ledger is now 34 rows of `(playbook class, target) → verdict`. That is
+a dataset. Phase 5a turns the *fixed* cheapest-first ordering of the
+playbook into a *learned* prior: rank each optimization class by the
+**Wilson score lower bound** of its observed shippable-win rate, and try the
+best-evidenced class first on a new hotspot.
+
+Why the Wilson lower bound and not the raw win rate: a class that went 1-for-1
+should not outrank a class that went 3-for-16. The Wilson lower bound
+penalizes small samples automatically — it is the same "be pessimistic
+about thin evidence" instinct as the bootstrap CI lower bound the bench
+uses to accept a speedup. Ties break *untried-before-failed* (an unexplored
+class keeps its cheapest-first prior; a class with evidence *against* it
+sinks), then cheapest-first.
+
+Two guards keep it honest, both visible in the output footer:
+
+- A "win" counts **only** a machine-sanitizer-verified accept. The
+  overturned `comrak-010` false-accept and every pre-sanitizer-gate accept
+  are excluded — the policy trusts the ledger's machine record, not the
+  audit narrative that caught them.
+- The tier-gated allocator win (mimalloc) shows as `held for human review`,
+  not as a loss and not as a shippable win — an accurate third category.
+
+Current ranking (all targets, 34 rows):
+
+| rank | class | attempts | wins | wilson-lb | note |
+|---|---|---|---|---|---|
+| 1 | algorithmic | 16 | 3 | 0.066 | proven winner |
+| 2 | alloc-churn | 4 | 1 | 0.046 | proven winner |
+| 3 | build-config | 9 | 1 | 0.020 | proven winner |
+| 4 | concurrency | 0 | 0 | 0.000 | untried — cheapest-first prior only |
+| 5 | allocator | 2 | 0 | 0.000 | held for human review (tier-gated) |
+| 6 | data-layout | 1 | 0 | 0.000 | evidence against |
+| 7 | simd | 2 | 0 | 0.000 | evidence against |
+
+The recommendation the agent would act on: **try algorithmic first** (best
+evidence), then the next-ranked class whose profile preconditions match.
+This is exactly the prior art direction (MLGO, AlphaDev, LLM Compiler) at
+the scale the ledger currently supports — a ranked prior, sharpening as the
+dataset grows, never overriding a gate.
+
+Reproduce: `cargo run -p policy` (add `--target cjson` to scope to one
+target).
+
+---
+
+## 5b — the kernel lane: FP-tolerance equivalence
+
+**Artifact:** `targets/matmul/`, `crates/diff-test/src/bin/fp-compare.rs`,
+demo in `kernel-lane.txt`.
+
+Every target through Phase 4 was gated by **byte-identical** golden replay:
+the optimized output must match the baseline bit for bit. That is the right
+gate for a parser or an HTTP service. It is the *wrong* gate for a numerical
+kernel, and the kernel lane is the case that proves it.
+
+`targets/matmul/kernel.c` is a single-precision matrix multiply in two
+implementations:
+
+- `matmul_ref` — naive `i,j,k` with one sequential accumulator. The oracle.
+- `matmul_opt` — transpose B (so the inner reduction walks contiguous
+  memory: the cache win) **and** eight independent accumulators tree-combined
+  at the end (breaking the float-add dependency chain: the ILP win). The
+  second lever **reorders the reduction**, so the last few ULPs differ *by
+  construction*. Bit-identical output is impossible.
+
+The demo (`sh scripts/kernel-lane-demo.sh 512`) walks the whole argument:
+
+1. **byte-identical FAILs** — 244,901 of 262,144 result values differ. Under
+   the Phase 0–4 gate, this correct 3× speedup would be *rejected*.
+2. **the FP-tolerance policy is EQUIVALENT** — `equivalence.toml` declares
+   `mode = "fp-tolerance"`, `abs = 1e-4`, `rel = 1e-3`; every numeric token
+   agrees within `abs + rel·|ref|`.
+3. **a genuine wrong result is still caught** — perturbing one element by
+   `+0.5` is REJECTED by the same tolerance. The gate is loose enough for
+   last-ULP reordering, tight enough for a real bug.
+4. **the speedup is measured with the same machinery** — interleaved A/B,
+   bootstrap 95% CI: **median 3.23×, CI [3.16, 3.26]**.
+
+The equivalence policy (`EquivalencePolicy::compare` in `diff-test`) is where
+the new rule lives: byte-identical stays exact; fp-tolerance tokenizes,
+compares numeric tokens within tolerance, handles NaN explicitly, and keeps
+non-numeric tokens and token counts exact. `fp-compare` applies a target's
+policy to two output files from the command line.
+
+**Verdict: `needs-human-review`** (`phase5-matmul-opt` ledger row). Using
+the FP-tolerance tier *at all* is a human-review signal per SPEC §8 — a
+person confirms the declared tolerance is defensible for the workload,
+exactly as with the mimalloc allocator swap. The machine measures and
+gates; the human ratifies the tolerance.
+
+### Why this is the GPU lane in miniature
+
+The environment has no GPU, so the GPU fork is demonstrated on the CPU as
+what it actually is: **the same trust machinery with a different timer.** A
+Triton or CUDA kernel versus a reference kernel is the identical shape —
+reference-kernel differential testing within a tolerance, interleaved timing
+with a bootstrap CI, a ledger row routed to human review. Only the profiler
+and the bench clock become kernel-time-aware, and the hardware changes. The
+correctness story, the statistics, and the audit trail are already built and
+already exercised here on real code.
+
+Reproduce: `sh scripts/kernel-lane-demo.sh 512`.
+
+---
+
+## What Phase 5 settles
+
+The core loop was already complete at Phase 4. Phase 5 answers the "but does
+it generalize?" objection with the two hardest cases the spec named:
+
+- A **learned prior** can be extracted from the ledger without weakening a
+  single gate — it only reorders *what to try first*.
+- A **numerically-reordered optimization** — the class byte-identical replay
+  cannot handle — slots into the same harness by swapping the equivalence
+  policy, with the correctness bar preserved (wrong results still caught) and
+  the tolerance itself put under human review.
+
+Zero shipped false accepts, still. The product is trust, and trust
+transferred.

results/phase5/learned-policy.txt

@@ -0,0 +1,17 @@
+learned class-selection ranking — evidence: 34 ledger rows, scope: all targets
+(Wilson 95% lower bound of the shippable-win rate; ties break untried-before-failed, then cheapest-first)
+
+rank  class         attempts  wins  held  win-rate  wilson-lb  note
+------------------------------------------------------------------------------
+   1  5 algorithmic       16     3     1       19%      0.066  proven winner
+   2  3 alloc-churn        4     1     0       25%      0.046  proven winner
+   3  1 build-config        9     1     0       11%      0.020  proven winner
+   4  7 concurrency        0     0     0        0%      0.000  untried — cheapest-first prior only
+   5  2 allocator          2     0     1        0%      0.000  held for human review (measured win, tier-gated)
+   6  4 data-layout        1     0     0        0%      0.000  evidence against — tried, no shippable win
+   7  6 simd               2     0     0        0%      0.000  evidence against — tried, no shippable win
+
+recommendation: try class 5 (algorithmic) first — best evidence (Wilson lb 0.066); then the next-ranked class whose profile preconditions match.
+
+Advisory only — the equivalence + significance gates decide every verdict. This ranking sharpens as the ledger grows.
+note: a 'win' counts only machine-sanitizer-verified accepts. Pre-sanitizer-gate accepts and the overturned comrak-010 are conservatively excluded — the policy trusts the ledger's machine record, not the external audit narrative.