phase4: COMPLETE — service latency win + mechanical ROI report; roadmap done

phase4-cjson-service (ledger): the accepted batch win phase3-cjson-002
measured under a 150 rps coordinated-omission-correct replay (20 rounds,
40k requests, 0 drops). p50 latency +6.2%, 95% CI [+5.8%, +7.2%] —
accepted; p99 CI [0.07, 4.97] too noisy (single-worker loopback tail),
harness correctly rejected. ROI (500-core fleet, $0.04/core-hr): 27.5
cores / $9,621/year at the CI lower bound, report generated mechanically
(results/phase4/cjson-service-roi.md).

- results/phase4/: case study + README + the generated ROI report.
- CLAUDE.md: Phase 4 complete; Phases 0-4 all done.

All SPEC §5 exit criteria met across every phase; zero shipped false
accepts across the whole project.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2

CLAUDE.md

@@ -30,9 +30,7 @@ Full engineering spec: `SPEC.md`. Read it before starting any phase.
 - `playbook/` — optimization classes, ordered, with preconditions and known risks
 - `results/` — generated reports and flamegraphs
 
-## Current phase
-
-**Phase 3 COMPLETE — all SPEC §5 exit criteria met.** The trust layer went language-agnostic: build isolation moved from a cargo-specific `CARGO_TARGET_DIR` injection to an explicit `{out}` placeholder + optional `sanitizer`/`sanitizer_binary` templates (`crates/diff-test/src/target.rs`), proven zero-regression on tokei before any C target. **A C target joined the roster** — cJSON v1.7.19 (MIT): trust-layer harness driver compiled with the patched `cJSON.c`, hermetic Unity/CTest gate (19 tests), ASan+UBSan sanitizers, deterministic pinned corpus, own A/A calibration (0/20 FP, 20/20 injection), CI-gated. **First verified cross-language win:** phase3-cjson-002 (class 5+3, stack-buffer number parse + `strtod` round-trip) +8.85%, 95% CI [+7.5%, +10.0%] — banked, byte-identical golden replay, independent ASan+UBSan clean, 10k-input diff-fuzz (`scripts/diff-fuzz-cjson.py`) 0 divergences. **Four public case studies** in `results/phase3/` (tokei class-5 wins; build flags + allocators; the comrak caught false-accept; the cJSON cross-language proof). **Six playbook classes exercised (1–6)**; wins on classes 1/2(review)/3/5 across Rust *and* C. coz wired (`-DHOTPATH_COZ`, `just coz`, `scripts/coz-*`); its apt runtime aborts in init on this container's glibc, so callgrind cache-sim is the working profiler (documented). Zero regressions shipped. **Next: Phase 4** (load replay, latency mode under replayed traffic, mechanical ROI reports — SPEC.md §5; `crates/report` has the ROI math, needs the generator).
+**Phase 4 COMPLETE — all SPEC §5 exit criteria met. Phases 0–4 are all done.** Service-mode latency benchmarking shipped: a coordinated-omission-correct, open-loop, fixed-rate load generator (`crates/bench-runner/src/service.rs`) measuring latency from each request's intended send time (queueing counted, not hidden), interleaved A/B, exact percentiles, bootstrap p50/p99 CIs. A real service target — `targets/cjson/service.c`, a minimal HTTP server wrapping the patched `cJSON.c` (trust-layer, outside the workspace allowlist). Service calibration PASS (0/10 A/A false-positive, 10/10 injected-5%-latency-regression detection; `results/calibration/cjson-service-aa.json`). **The verified batch win phase3-cjson-002 measured under a 150 rps replay (20 rounds, 40k requests, 0 drops): p50 latency +6.2%, 95% CI [+5.8%, +7.2%] — accepted (`phase4-cjson-service` ledger row); p99 NOT claimed (CI [0.07, 4.97], single-worker loopback tail jitter) — harness correctly rejected it.** Mechanical ROI report generator (`crates/report` bin, `just report`, SPEC §9): reads a ledger row + service-latency JSON → throughput→cores→dollars (27.5 cores / $9,621-per-year CI lower bound on a 500-core fleet) *and* latency percentiles, every figure with CI + workload + methodology inline; flags any non-accepted row. Case study: `results/phase4/`. **The roadmap is complete — the profile→hypothesize→patch→gated-verdict→ROI loop runs end to end across Rust and C, batch and service, with zero shipped false accepts across all phases. Open follow-ups: the standing human ruling on `phase0-comrak-002` (mimalloc, needs-human-review); SPEC §13 research forks (GPU lane, learned policies) remain out of scope.**
 
 ## Commands
 

results/phase4/README.md

@@ -0,0 +1,81 @@
+# Phase 4 — services & scale
+
+**Exit criteria (SPEC §5): one real service workload optimized under
+replayed traffic; report generated mechanically.** Both met.
+
+Phase 4 takes the trust story off the batch stopwatch and onto a running
+service: the same verified optimization, measured as a latency delta
+under replayed load, then turned into a dollar figure by a generator
+that reads only the ledger and the calibrated bench — no hand-typed
+numbers.
+
+## What was built
+
+- **Service-mode latency bench** (`crates/bench-runner/src/service.rs`,
+  SPEC §3.1 mode c): a **coordinated-omission-correct**, open-loop,
+  fixed-rate load generator. Each request's latency is measured from its
+  *intended* send time, so when the single worker falls behind, the
+  requests queued behind it carry the full delay a real client would see
+  — the exact error closed-loop tools hide. Interleaved A/B of two
+  server binaries, exact percentiles (bounded sample; no HdrHistogram
+  bucketing error), bootstrap p50/p99 CIs through the same ratio-CI
+  machine the wall-clock bench uses.
+- **A real service target**: `targets/cjson/service.c`, a minimal HTTP
+  server wrapping the (patched) `cJSON.c` — trust-layer code outside the
+  workspace allowlist, so a patch changes the parser under load but not
+  the measurement rig.
+- **Service-mode calibration** (`bench-runner service-calibrate`): the
+  latency measurement is a trusted anchor only after it passes A/A
+  (0/10 false positives) and injected-5%-latency-regression detection
+  (10/10), recorded in `results/calibration/cjson-service-aa.json`.
+- **Mechanical ROI report** (`crates/report` binary, SPEC §9): reads a
+  ledger row + the service-latency JSON and renders
+  throughput→cores→dollars *and* latency percentiles, every figure with
+  its 95% CI, the pricing source, and the methodology printed inline.
+
+## The result (`phase4-cjson-service`, ledger)
+
+The accepted batch win `phase3-cjson-002` (+8.85% throughput) was
+measured as a **service under a 150 rps coordinated-omission-correct
+replay**, baseline (pristine cJSON v1.7.19) vs candidate (the banked
+win), 20 interleaved rounds, server pinned, 40,000 requests, **0 drops**:
+
+| metric | baseline | candidate | speedup | 95% CI |
+|---|---|---|---|---|
+| **p50 latency** | 3.332 ms | 3.136 ms | **1.062** | **[1.058, 1.072]** |
+| p99 latency | 24.7 ms | 23.9 ms | 1.033 | [0.07, 4.97] |
+
+**The p50 win is the claim: +6.2% service-latency reduction under load,
+CI [+5.8%, +7.2%]** — the batch parse speedup translating to lower
+per-request latency. It is a touch smaller than the +8.85% batch number
+because under load the latency also carries fixed connection and HTTP
+overhead the parse optimization doesn't touch — an honest dilution, not
+a measurement error.
+
+**The p99 was *not* claimed.** Its CI is [0.07, 4.97] — uselessly wide,
+because tail latency on a single-worker loopback service is dominated by
+rare OS-scheduling spikes that swamp the ~0.2 ms parse difference. The
+harness resolved this correctly: `p99 verdict: rejected-bench`. A number
+the measurement can't pin down does not get reported as a win, even
+though the median points the right way. That refusal is the whole
+project in one line.
+
+## The ROI, generated mechanically
+
+`just report phase4-cjson-service --service-json results/phase4/cjson-service.json`
+→ [`cjson-service-roi.md`](cjson-service-roi.md). On a 500-core fleet at
+$0.04/core-hour, the +6.2% p50 win returns **27.5 cores / $9,621 per
+year at the CI lower bound** (median 29.4 cores / $10,290) — and the
+report quotes the lower bound, because the point estimate is not a
+promise. Every figure carries its CI, its workload, and the methodology
+that produced it, so the number survives hostile review.
+
+## Exit criteria (SPEC §5) — met
+
+- **One real service workload optimized under replayed traffic:** the
+  cJSON HTTP service under CO-correct open-loop replay; the verified win
+  shows a CI-significant +6.2% p50 latency improvement.
+- **Report generated mechanically:** the ROI report is produced from the
+  ledger row and the calibrated service JSON with no hand-editing;
+  throughput→cores→dollars and latency percentiles, CIs and methodology
+  inline.

results/phase4/cjson-service-roi.md

@@ -0,0 +1,50 @@
+# ROI report — `phase4-cjson-service`
+
+**Target:** cjson · **Playbook class:** 5 · **Verdict:** `accepted`
+
+**Workload:** corpora/cjson/input/service.json, 1 parse+print/req, 150 rps open-loop, CO-correct
+
+## Throughput → cores → dollars
+
+Measured speedup (baseline/candidate): **1.0624**, 95% bootstrap CI **[1.0581, 1.0725]**.
+
+On a **500-core** fleet running this workload:
+
+| metric | median | 95% CI |
+|---|---|---|
+| cores returned | 29.4 | [27.5, 33.8] |
+| annualized saving | $10290 | [$9621, $11838] |
+
+Pricing: **$0.04/core-hour**, **8760 h/year**. Source: _example: AWS m7i on-demand us-east-1, per-vCPU approximation — replace per engagement_
+
+The **CI lower bound is the number to quote** — 27.5 cores / $9621 per year is
+the saving that survives the 95% interval; the point estimate is not a promise.
+
+## Latency under replayed load
+
+Workload: corpora/cjson/input/service.json, 1 parse+print/req, 150 rps open-loop, CO-correct. Coordinated-omission-correct open-loop replay.
+
+| percentile | baseline | candidate | speedup | 95% CI |
+|---|---|---|---|---|
+| p50 | 3.332 ms | 3.136 ms | 1.0624 | [1.0581, 1.0725] |
+| p99 | 24.726 ms | 23.947 ms | 1.0325 | [0.0715, 4.9714] |
+
+## Methodology (ships with the number)
+
+- **Interleaved A/B**, baseline rebuilt from a pristine checkout, never the agent's
+workspace. Candidate and baseline alternate to control thermal/background drift.
+- **Bootstrap 95% CI** of the ratio-of-medians; an effect is accepted only if the CI
+lower bound clears the threshold in `config/accept.toml` (2%).
+- **Equivalence gates** passed before any number was trusted: upstream test suite,
+byte-identical golden replay, differential fuzzing, and ASan/UBSan sanitizers.
+- **Calibrated hardware**: the workload passed A/A (false-positive <5%) and injected-
+regression (≥95% detection) self-tests, recorded in `results/calibration/`.
+- **Coordinated-omission correct** latency (service mode): each request's latency is
+measured from its intended send time, so queueing is counted, not hidden.
+
+## Caveats
+
+Every figure is specific to the workload and hardware named above. Throughput→cores
+assumes the fleet is CPU-bound on this path and scales linearly; latency figures are
+for the stated arrival rate. The saving to commit to is the **CI lower bound**, not
+the median.