diff --git a/CLAUDE.md b/CLAUDE.md index 286d0f1..372ded2 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -30,7 +30,9 @@ Full engineering spec: `SPEC.md`. Read it before starting any phase. - `playbook/` — optimization classes, ordered, with preconditions and known risks - `results/` — generated reports and flamegraphs -**Phase 4 COMPLETE — all SPEC §5 exit criteria met. Phases 0–4 are all done.** Service-mode latency benchmarking shipped: a coordinated-omission-correct, open-loop, fixed-rate load generator (`crates/bench-runner/src/service.rs`) measuring latency from each request's intended send time (queueing counted, not hidden), interleaved A/B, exact percentiles, bootstrap p50/p99 CIs. A real service target — `targets/cjson/service.c`, a minimal HTTP server wrapping the patched `cJSON.c` (trust-layer, outside the workspace allowlist). Service calibration PASS (0/10 A/A false-positive, 10/10 injected-5%-latency-regression detection; `results/calibration/cjson-service-aa.json`). **The verified batch win phase3-cjson-002 measured under a 150 rps replay (20 rounds, 40k requests, 0 drops): p50 latency +6.2%, 95% CI [+5.8%, +7.2%] — accepted (`phase4-cjson-service` ledger row); p99 NOT claimed (CI [0.07, 4.97], single-worker loopback tail jitter) — harness correctly rejected it.** Mechanical ROI report generator (`crates/report` bin, `just report`, SPEC §9): reads a ledger row + service-latency JSON → throughput→cores→dollars (27.5 cores / $9,621-per-year CI lower bound on a 500-core fleet) *and* latency percentiles, every figure with CI + workload + methodology inline; flags any non-accepted row. Case study: `results/phase4/`. **The roadmap is complete — the profile→hypothesize→patch→gated-verdict→ROI loop runs end to end across Rust and C, batch and service, with zero shipped false accepts across all phases. Open follow-ups: the standing human ruling on `phase0-comrak-002` (mimalloc, needs-human-review); SPEC §13 research forks (GPU lane, learned policies) remain out of scope.** +**Phase 4 COMPLETE — all SPEC §5 exit criteria met. Phases 0–4 are all done.** Service-mode latency benchmarking shipped: a coordinated-omission-correct, open-loop, fixed-rate load generator (`crates/bench-runner/src/service.rs`) measuring latency from each request's intended send time (queueing counted, not hidden), interleaved A/B, exact percentiles, bootstrap p50/p99 CIs. A real service target — `targets/cjson/service.c`, a minimal HTTP server wrapping the patched `cJSON.c` (trust-layer, outside the workspace allowlist). Service calibration PASS (0/10 A/A false-positive, 10/10 injected-5%-latency-regression detection; `results/calibration/cjson-service-aa.json`). **The verified batch win phase3-cjson-002 measured under a 150 rps replay (20 rounds, 40k requests, 0 drops): p50 latency +6.2%, 95% CI [+5.8%, +7.2%] — accepted (`phase4-cjson-service` ledger row); p99 NOT claimed (CI [0.07, 4.97], single-worker loopback tail jitter) — harness correctly rejected it.** Mechanical ROI report generator (`crates/report` bin, `just report`, SPEC §9): reads a ledger row + service-latency JSON → throughput→cores→dollars (27.5 cores / $9,621-per-year CI lower bound on a 500-core fleet) *and* latency percentiles, every figure with CI + workload + methodology inline; flags any non-accepted row. Case study: `results/phase4/`. **The roadmap is complete — the profile→hypothesize→patch→gated-verdict→ROI loop runs end to end across Rust and C, batch and service, with zero shipped false accepts across all phases.** + +**Phase 5 COMPLETE — the two SPEC §13 research forks, both built and both routed to human review.** (a) *Learned class-selection policy* (`crates/policy`, `cargo run -p policy`): reads the append-only ledger and ranks optimization classes by the Wilson lower bound of their shippable-win rate — a learned prior over the fixed cheapest-first ordering, advisory only (gates still decide). A "win" counts only machine-sanitizer-verified accepts; the overturned comrak-010 and tier-gated mimalloc are excluded/held. On the current 34-row ledger it recommends algorithmic first (Wilson lb 0.066). (b) *Kernel lane* (`targets/matmul/kernel.c`, `crates/diff-test` FP-tolerance policy + `fp-compare` bin, `scripts/kernel-lane-demo.sh`): a matmul optimization (transpose-B cache locality + eight-accumulator ILP) that **reorders FP accumulation**, so byte-identical golden replay is the wrong gate — 244,901/262,144 values differ. The FP-tolerance gate (`abs 1e-4 + rel 1e-3`) accepts it, still REJECTS a +0.5 perturbation, and the bench measures 3.23× [3.16, 3.26] with the same interleaved A/B + bootstrap CI. Ledger row `phase5-matmul-opt` = needs-human-review (using the tolerance tier is a §8 signal, same posture as mimalloc). No GPU in this environment, so the GPU fork is the identical shape shown on CPU — only the timer and hardware change. Case study: `results/phase5/`; docs chapter `docs/src/research-forks.md`. Open follow-up: the standing accept-scoped ruling on `phase0-comrak-002` (mimalloc) is resolved (`results/rulings/`).** ## Commands diff --git a/docs/src/SUMMARY.md b/docs/src/SUMMARY.md index e8e0b0a..a569428 100644 --- a/docs/src/SUMMARY.md +++ b/docs/src/SUMMARY.md @@ -19,6 +19,7 @@ - [The caught false-accept (comrak)](./case-studies/comrak-false-accept.md) - [cJSON — the cross-language proof (C)](./case-studies/cjson.md) - [Services & scale: latency under load](./case-studies/service.md) +- [Research forks: kernel lane & learned policy](./research-forks.md) # Reference diff --git a/docs/src/research-forks.md b/docs/src/research-forks.md new file mode 100644 index 0000000..a06b5b0 --- /dev/null +++ b/docs/src/research-forks.md @@ -0,0 +1,109 @@ +# Research forks: kernel lane & learned policy + +The core loop was complete at Phase 4 — profile → hypothesize → patch → +gated-verdict → ROI, running unattended across Rust and C, batch and +service, with zero shipped false accepts. SPEC §13 named two frontier ideas +that the roadmap deliberately excluded, precisely because they change the +*shape* of the work rather than extend the loop. Phase 5 built both, as +stress tests with one question each: **does the trust machinery still hold?** + +Both times it does. Neither fork relaxes a non-negotiable; each routes its +headline result to `needs-human-review`, and the learned prior is advisory +only. What follows is the argument for each. + +## The kernel lane: when byte-identical is the wrong gate + +Every target through Phase 4 was gated by **byte-identical golden replay** — +the optimized output must match the baseline bit for bit. For a parser or an +HTTP service that is exactly right. For a numerical kernel it is wrong, and +the kernel lane is the case that proves it. + +`targets/matmul/kernel.c` is a single-precision matrix multiply in two forms: + +- **`matmul_ref`** — naive `i,j,k`, one sequential accumulator. The oracle. +- **`matmul_opt`** — transpose B so the inner reduction walks contiguous + memory (the cache win), plus eight independent accumulators tree-combined + at the end (the ILP win, breaking the float-add dependency chain). The + second lever **reorders the reduction**, so the last few ULPs differ *by + construction*. Bit-identical output is impossible. + +The demonstration (`sh scripts/kernel-lane-demo.sh 512`) walks the argument: + +1. **byte-identical FAILs** — 244,901 of 262,144 values differ. Under the + Phase 0–4 gate this correct 3× speedup would be *rejected*. +2. **FP-tolerance is EQUIVALENT** — the target declares `mode = + "fp-tolerance"`, `abs = 1e-4`, `rel = 1e-3`; every numeric token agrees + within `abs + rel·|ref|`. +3. **a wrong result is still caught** — perturbing one element by `+0.5` is + REJECTED by the same tolerance. Loose enough for last-ULP reordering, + tight enough for a real bug. +4. **measured with the same machinery** — interleaved A/B, bootstrap 95% CI: + **median 3.23×, CI [3.16, 3.26]**. + +The new rule lives in one place — `EquivalencePolicy::compare` in +`diff-test`. Byte-identical stays exact; fp-tolerance tokenizes, compares +numeric tokens within tolerance, handles NaN explicitly, and keeps +non-numeric tokens and token counts exact. The `fp-compare` binary applies a +target's `equivalence.toml` to two output files from the command line. + +The verdict is **`needs-human-review`**. Using the FP-tolerance tier *at +all* is a human-review signal (SPEC §8): a person confirms the declared +tolerance is defensible for the workload, exactly as with the mimalloc +allocator swap. The machine measures and gates; the human ratifies the +tolerance. + +**This is the GPU lane in miniature.** The environment has no GPU, so the +GPU fork is shown on the CPU as what it actually is — the same trust +machinery with a different timer. A Triton or CUDA kernel versus a reference +kernel is the identical shape: reference-kernel differential testing within +a tolerance, interleaved timing with a bootstrap CI, a ledger row routed to +human review. Only the profiler and the bench clock become +kernel-time-aware, and the hardware changes. The correctness story, the +statistics, and the audit trail are already built and exercised here on real +code. + +## The learned policy: the ledger as a training set + +The playbook orders optimization classes cheapest-first — a fixed prior. But +the ledger is now dozens of rows of `(class, target) → verdict`. That is a +dataset, and it can turn the fixed ordering into a *learned* one. + +`crates/policy` ranks each class by the **Wilson score lower bound** of its +observed shippable-win rate. The Wilson lower bound is the same instinct as +the bootstrap CI lower bound the bench uses to accept a speedup: be +pessimistic about thin evidence. A class that went 1-for-1 should not +outrank one that went 3-for-16 — and it does not, because the lower bound +penalizes small samples automatically. Ties break untried-before-failed (an +unexplored class keeps its cheapest-first prior; a class with evidence +*against* it sinks), then cheapest-first. + +Two guards keep it honest, both printed in the output footer: + +- A "win" counts **only** a machine-sanitizer-verified accept. The + overturned `comrak-010` false-accept and every pre-sanitizer-gate accept + are excluded — the policy trusts the ledger's machine record, not the + audit narrative that caught them. +- The tier-gated mimalloc win shows as *held for human review* — a third + category, neither a loss nor a shippable win. + +On the current 34-row ledger the ranking recommends **algorithmic first** +(best evidence, Wilson lb 0.066), then the next-ranked class whose profile +preconditions match. It is advisory: the equivalence and significance gates +still decide every verdict. This is the MLGO / AlphaDev / LLM-Compiler +direction at the scale the ledger currently supports — a ranked prior that +sharpens as the dataset grows and never overrides a gate. + +Run it with `cargo run -p policy` (add `--target ` to scope the +evidence to one target). + +## What the forks settle + +- A **learned prior** can be extracted from the ledger without weakening a + single gate — it only reorders *what to try first*. +- A **numerically-reordered optimization** — the class byte-identical replay + cannot handle — slots into the same harness by swapping the equivalence + policy, with the correctness bar preserved and the tolerance itself put + under human review. + +The full write-up, with reproduction commands and the raw tool output, is in +[`results/phase5/`](https://github.com/bdbrown4/peltier.io/tree/main/results/phase5). diff --git a/docs/src/roadmap.md b/docs/src/roadmap.md index 82100ce..9ee2048 100644 --- a/docs/src/roadmap.md +++ b/docs/src/roadmap.md @@ -9,6 +9,7 @@ | **2** | Unattended agent loop | 20 audited attempts, 3 verified wins, **0 shipped false accepts**, OS-isolated | | **3** | Playbook + proof, C/C++ | cross-language cJSON win (+8.85%), 4 case studies, 6 playbook classes | | **4** | Services & scale | cJSON service +6.2% p50 under CO-correct replay, mechanical ROI report | +| **5** | Research forks (SPEC §13) | FP-tolerance kernel lane (matmul 3.23×, needs-human-review) + learned class-selection prior from the ledger | The `profile → hypothesize → patch → gated-verdict → ROI` loop runs unattended, OS-isolated, across **Rust and C**, **batch and service**, from @@ -16,22 +17,27 @@ a pinned corpus to a dollar figure — with zero shipped false accepts across all phases, and the two pipeline over-accepts both caught by audit and turned into permanent gates. -## Frontier — the research forks (SPEC §13) +## Phase 5 — the research forks (SPEC §13) -Deliberately out of the core roadmap, these are where the largest gaps -live: +Deliberately out of the core roadmap, these are the two hardest cases the +spec named — both now built and exercised, both routed to human review. +Full write-up: [Research forks](./research-forks.md). -- **GPU kernel lane.** Triton/CUDA targets where 2–10× gaps are common; - correctness via reference-kernel differential testing; reward = measured - kernel time. The trust machinery (differential equivalence, interleaved - timing, append-only ledger) transfers directly; the profiler and bench - become kernel-time-aware, and it needs GPU hardware to run. -- **Learned optimization policies.** The ledger is a growing dataset of - `(profile signature, class, hypothesis) → verdict`. That is training data - for a policy that *ranks* which class to try first on a new hotspot, - turning the fixed cheapest-first ordering into a learned prior. Prior art: - MLGO, AlphaDev, Meta's LLM Compiler; ProGraML-style IR graphs as the - program representation. +- **Kernel lane — done as a CPU demonstration.** A matmul optimization that + reorders floating-point accumulation (transpose-B + eight-accumulator ILP) + runs 3.23× faster and is *impossible* to gate byte-identically. It slots + into the harness by swapping the equivalence policy for a declared + FP-tolerance (`abs + rel·|ref|`) — which still catches a genuine wrong + result — and routes to `needs-human-review` because using the tolerance + tier at all is a §8 signal. This is the GPU lane in miniature: + reference-kernel differential testing, interleaved timing, ledger row. + Only the timer and the hardware change; **the actual GPU run needs GPU + hardware**, absent here. +- **Learned optimization policies — done.** `crates/policy` reads the ledger + and ranks optimization classes by the Wilson lower bound of their + shippable-win rate, turning the fixed cheapest-first ordering into a + learned prior (advisory only — the gates still decide). Prior art: MLGO, + AlphaDev, Meta's LLM Compiler. ## The real target diff --git a/results/phase5/README.md b/results/phase5/README.md new file mode 100644 index 0000000..ba83e73 --- /dev/null +++ b/results/phase5/README.md @@ -0,0 +1,139 @@ +# Phase 5 — the research forks (SPEC §13) + +Phase 5 exercises the two frontier ideas the spec deliberately left out of +the core roadmap. Neither adds a new product; each is a stress test that +answers one question: **does the trust machinery still hold when the shape +of the work changes?** The answer, both times, is yes — the gates, the +interleaved A/B timer, and the append-only ledger transfer unchanged; only +the *gate policy* (5b) or the *ordering prior* (5a) is new. + +Nothing here relaxes a non-negotiable. Both forks route their headline +result to `needs-human-review`, and the learned prior is advisory only — +the gates still decide every verdict. + +--- + +## 5a — learned class-selection policy + +**Artifact:** `crates/policy/`, output in `learned-policy.txt`. + +The ledger is now 34 rows of `(playbook class, target) → verdict`. That is +a dataset. Phase 5a turns the *fixed* cheapest-first ordering of the +playbook into a *learned* prior: rank each optimization class by the +**Wilson score lower bound** of its observed shippable-win rate, and try the +best-evidenced class first on a new hotspot. + +Why the Wilson lower bound and not the raw win rate: a class that went 1-for-1 +should not outrank a class that went 3-for-16. The Wilson lower bound +penalizes small samples automatically — it is the same "be pessimistic +about thin evidence" instinct as the bootstrap CI lower bound the bench +uses to accept a speedup. Ties break *untried-before-failed* (an unexplored +class keeps its cheapest-first prior; a class with evidence *against* it +sinks), then cheapest-first. + +Two guards keep it honest, both visible in the output footer: + +- A "win" counts **only** a machine-sanitizer-verified accept. The + overturned `comrak-010` false-accept and every pre-sanitizer-gate accept + are excluded — the policy trusts the ledger's machine record, not the + audit narrative that caught them. +- The tier-gated allocator win (mimalloc) shows as `held for human review`, + not as a loss and not as a shippable win — an accurate third category. + +Current ranking (all targets, 34 rows): + +| rank | class | attempts | wins | wilson-lb | note | +|---|---|---|---|---|---| +| 1 | algorithmic | 16 | 3 | 0.066 | proven winner | +| 2 | alloc-churn | 4 | 1 | 0.046 | proven winner | +| 3 | build-config | 9 | 1 | 0.020 | proven winner | +| 4 | concurrency | 0 | 0 | 0.000 | untried — cheapest-first prior only | +| 5 | allocator | 2 | 0 | 0.000 | held for human review (tier-gated) | +| 6 | data-layout | 1 | 0 | 0.000 | evidence against | +| 7 | simd | 2 | 0 | 0.000 | evidence against | + +The recommendation the agent would act on: **try algorithmic first** (best +evidence), then the next-ranked class whose profile preconditions match. +This is exactly the prior art direction (MLGO, AlphaDev, LLM Compiler) at +the scale the ledger currently supports — a ranked prior, sharpening as the +dataset grows, never overriding a gate. + +Reproduce: `cargo run -p policy` (add `--target cjson` to scope to one +target). + +--- + +## 5b — the kernel lane: FP-tolerance equivalence + +**Artifact:** `targets/matmul/`, `crates/diff-test/src/bin/fp-compare.rs`, +demo in `kernel-lane.txt`. + +Every target through Phase 4 was gated by **byte-identical** golden replay: +the optimized output must match the baseline bit for bit. That is the right +gate for a parser or an HTTP service. It is the *wrong* gate for a numerical +kernel, and the kernel lane is the case that proves it. + +`targets/matmul/kernel.c` is a single-precision matrix multiply in two +implementations: + +- `matmul_ref` — naive `i,j,k` with one sequential accumulator. The oracle. +- `matmul_opt` — transpose B (so the inner reduction walks contiguous + memory: the cache win) **and** eight independent accumulators tree-combined + at the end (breaking the float-add dependency chain: the ILP win). The + second lever **reorders the reduction**, so the last few ULPs differ *by + construction*. Bit-identical output is impossible. + +The demo (`sh scripts/kernel-lane-demo.sh 512`) walks the whole argument: + +1. **byte-identical FAILs** — 244,901 of 262,144 result values differ. Under + the Phase 0–4 gate, this correct 3× speedup would be *rejected*. +2. **the FP-tolerance policy is EQUIVALENT** — `equivalence.toml` declares + `mode = "fp-tolerance"`, `abs = 1e-4`, `rel = 1e-3`; every numeric token + agrees within `abs + rel·|ref|`. +3. **a genuine wrong result is still caught** — perturbing one element by + `+0.5` is REJECTED by the same tolerance. The gate is loose enough for + last-ULP reordering, tight enough for a real bug. +4. **the speedup is measured with the same machinery** — interleaved A/B, + bootstrap 95% CI: **median 3.23×, CI [3.16, 3.26]**. + +The equivalence policy (`EquivalencePolicy::compare` in `diff-test`) is where +the new rule lives: byte-identical stays exact; fp-tolerance tokenizes, +compares numeric tokens within tolerance, handles NaN explicitly, and keeps +non-numeric tokens and token counts exact. `fp-compare` applies a target's +policy to two output files from the command line. + +**Verdict: `needs-human-review`** (`phase5-matmul-opt` ledger row). Using +the FP-tolerance tier *at all* is a human-review signal per SPEC §8 — a +person confirms the declared tolerance is defensible for the workload, +exactly as with the mimalloc allocator swap. The machine measures and +gates; the human ratifies the tolerance. + +### Why this is the GPU lane in miniature + +The environment has no GPU, so the GPU fork is demonstrated on the CPU as +what it actually is: **the same trust machinery with a different timer.** A +Triton or CUDA kernel versus a reference kernel is the identical shape — +reference-kernel differential testing within a tolerance, interleaved timing +with a bootstrap CI, a ledger row routed to human review. Only the profiler +and the bench clock become kernel-time-aware, and the hardware changes. The +correctness story, the statistics, and the audit trail are already built and +already exercised here on real code. + +Reproduce: `sh scripts/kernel-lane-demo.sh 512`. + +--- + +## What Phase 5 settles + +The core loop was already complete at Phase 4. Phase 5 answers the "but does +it generalize?" objection with the two hardest cases the spec named: + +- A **learned prior** can be extracted from the ledger without weakening a + single gate — it only reorders *what to try first*. +- A **numerically-reordered optimization** — the class byte-identical replay + cannot handle — slots into the same harness by swapping the equivalence + policy, with the correctness bar preserved (wrong results still caught) and + the tolerance itself put under human review. + +Zero shipped false accepts, still. The product is trust, and trust +transferred. diff --git a/results/phase5/learned-policy.txt b/results/phase5/learned-policy.txt new file mode 100644 index 0000000..d783510 --- /dev/null +++ b/results/phase5/learned-policy.txt @@ -0,0 +1,17 @@ +learned class-selection ranking — evidence: 34 ledger rows, scope: all targets +(Wilson 95% lower bound of the shippable-win rate; ties break untried-before-failed, then cheapest-first) + +rank class attempts wins held win-rate wilson-lb note +------------------------------------------------------------------------------ + 1 5 algorithmic 16 3 1 19% 0.066 proven winner + 2 3 alloc-churn 4 1 0 25% 0.046 proven winner + 3 1 build-config 9 1 0 11% 0.020 proven winner + 4 7 concurrency 0 0 0 0% 0.000 untried — cheapest-first prior only + 5 2 allocator 2 0 1 0% 0.000 held for human review (measured win, tier-gated) + 6 4 data-layout 1 0 0 0% 0.000 evidence against — tried, no shippable win + 7 6 simd 2 0 0 0% 0.000 evidence against — tried, no shippable win + +recommendation: try class 5 (algorithmic) first — best evidence (Wilson lb 0.066); then the next-ranked class whose profile preconditions match. + +Advisory only — the equivalence + significance gates decide every verdict. This ranking sharpens as the ledger grows. +note: a 'win' counts only machine-sanitizer-verified accepts. Pre-sanitizer-gate accepts and the overturned comrak-010 are conservatively excluded — the policy trusts the ledger's machine record, not the external audit narrative.