From 0ec111663b08152573eba95d2a9bc65fec050dc1 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 11 Jul 2026 23:48:22 +0000 Subject: [PATCH] =?UTF-8?q?phase4:=20COMPLETE=20=E2=80=94=20service=20late?= =?UTF-8?q?ncy=20win=20+=20mechanical=20ROI=20report;=20roadmap=20done?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit phase4-cjson-service (ledger): the accepted batch win phase3-cjson-002 measured under a 150 rps coordinated-omission-correct replay (20 rounds, 40k requests, 0 drops). p50 latency +6.2%, 95% CI [+5.8%, +7.2%] — accepted; p99 CI [0.07, 4.97] too noisy (single-worker loopback tail), harness correctly rejected. ROI (500-core fleet, $0.04/core-hr): 27.5 cores / $9,621/year at the CI lower bound, report generated mechanically (results/phase4/cjson-service-roi.md). - results/phase4/: case study + README + the generated ROI report. - CLAUDE.md: Phase 4 complete; Phases 0-4 all done. All SPEC §5 exit criteria met across every phase; zero shipped false accepts across the whole project. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2 --- CLAUDE.md | 4 +- results/ledger.sqlite | Bin 167936 -> 167936 bytes results/phase4/README.md | 81 ++++++++++++++++++++++++++++ results/phase4/cjson-service-roi.md | 50 +++++++++++++++++ 4 files changed, 132 insertions(+), 3 deletions(-) create mode 100644 results/phase4/README.md create mode 100644 results/phase4/cjson-service-roi.md diff --git a/CLAUDE.md b/CLAUDE.md index 40e0b2a..286d0f1 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -30,9 +30,7 @@ Full engineering spec: `SPEC.md`. Read it before starting any phase. - `playbook/` — optimization classes, ordered, with preconditions and known risks - `results/` — generated reports and flamegraphs -## Current phase - -**Phase 3 COMPLETE — all SPEC §5 exit criteria met.** The trust layer went language-agnostic: build isolation moved from a cargo-specific `CARGO_TARGET_DIR` injection to an explicit `{out}` placeholder + optional `sanitizer`/`sanitizer_binary` templates (`crates/diff-test/src/target.rs`), proven zero-regression on tokei before any C target. **A C target joined the roster** — cJSON v1.7.19 (MIT): trust-layer harness driver compiled with the patched `cJSON.c`, hermetic Unity/CTest gate (19 tests), ASan+UBSan sanitizers, deterministic pinned corpus, own A/A calibration (0/20 FP, 20/20 injection), CI-gated. **First verified cross-language win:** phase3-cjson-002 (class 5+3, stack-buffer number parse + `strtod` round-trip) +8.85%, 95% CI [+7.5%, +10.0%] — banked, byte-identical golden replay, independent ASan+UBSan clean, 10k-input diff-fuzz (`scripts/diff-fuzz-cjson.py`) 0 divergences. **Four public case studies** in `results/phase3/` (tokei class-5 wins; build flags + allocators; the comrak caught false-accept; the cJSON cross-language proof). **Six playbook classes exercised (1–6)**; wins on classes 1/2(review)/3/5 across Rust *and* C. coz wired (`-DHOTPATH_COZ`, `just coz`, `scripts/coz-*`); its apt runtime aborts in init on this container's glibc, so callgrind cache-sim is the working profiler (documented). Zero regressions shipped. **Next: Phase 4** (load replay, latency mode under replayed traffic, mechanical ROI reports — SPEC.md §5; `crates/report` has the ROI math, needs the generator). +**Phase 4 COMPLETE — all SPEC §5 exit criteria met. Phases 0–4 are all done.** Service-mode latency benchmarking shipped: a coordinated-omission-correct, open-loop, fixed-rate load generator (`crates/bench-runner/src/service.rs`) measuring latency from each request's intended send time (queueing counted, not hidden), interleaved A/B, exact percentiles, bootstrap p50/p99 CIs. A real service target — `targets/cjson/service.c`, a minimal HTTP server wrapping the patched `cJSON.c` (trust-layer, outside the workspace allowlist). Service calibration PASS (0/10 A/A false-positive, 10/10 injected-5%-latency-regression detection; `results/calibration/cjson-service-aa.json`). **The verified batch win phase3-cjson-002 measured under a 150 rps replay (20 rounds, 40k requests, 0 drops): p50 latency +6.2%, 95% CI [+5.8%, +7.2%] — accepted (`phase4-cjson-service` ledger row); p99 NOT claimed (CI [0.07, 4.97], single-worker loopback tail jitter) — harness correctly rejected it.** Mechanical ROI report generator (`crates/report` bin, `just report`, SPEC §9): reads a ledger row + service-latency JSON → throughput→cores→dollars (27.5 cores / $9,621-per-year CI lower bound on a 500-core fleet) *and* latency percentiles, every figure with CI + workload + methodology inline; flags any non-accepted row. Case study: `results/phase4/`. **The roadmap is complete — the profile→hypothesize→patch→gated-verdict→ROI loop runs end to end across Rust and C, batch and service, with zero shipped false accepts across all phases. Open follow-ups: the standing human ruling on `phase0-comrak-002` (mimalloc, needs-human-review); SPEC §13 research forks (GPU lane, learned policies) remain out of scope.** ## Commands diff --git a/results/ledger.sqlite b/results/ledger.sqlite index f866981275e88694244752204c91190d606b0801..f5be65cedd213dcb0d5a93d180752169274351bf 100644 GIT binary patch delta 1421 zcmb7^&x;&I6vxw(EDp(LE(yj&2WuJvyKMJNRsZe@jBNIzg07m4f+37GRW+G5-Ce!a z-MegXm@EoGf}jkt*9e{h3Skd{{1HO%>cKxHZ+;~^As(zlS5sB*Tkn0|dv*WO+53;q z{`{_X^_Tayw5w0P`ThOVjq~;KTWGU#zJB~U`g`^AJEvVO8_--hCi^y*shJD47ctQ| z_RPO3tBu<3soJl#A8W(f?r{y%+IZMGr!~GGb{-5n+S6gD@?f)CS$THq+~bvJPY(ZF z89huI7hLz!Z7tD$5Qkfb?57Xv)#{tBu(0Ks*f$B;vFx_vw+Z=r>H~AVfH)%T2Z9mI zc)+>e^TM#tDGhAW_eki5c3_`jAK%#i6lGN9Vk1*gTA*^2a-q~bZuQCESj-o{OA=y%{Z?{ZPlDAwEhJhkWz;5Tp3 zv~}}3d^h^#y?f@KP*Fbuw#x~a@z)JK5m1^8G*48)v~)&LJ7{eYY3pS)>jv7|{?AGT z#R;zeP5kzY;PHtQy1qv&*C&v3-*B2CtdYQVywI_*?M~Fe z@aTD6yEw93cv3Lr-obEIpHS|q4Dw6dDeGKJn++B$Aipow-AP#jAt?P!D%zU9KCLK) rerU~`w>fzf4Ma}9GA(FuSc2vP^;a!>=_EgWI9znd)5i}hU!HjZmaWA- delta 87 zcmV-d0I2_fpbCJX3XmHCB#|6L0VJ_tq#pwy4K|YpAVITHFzyVqHZ9Hq3Lp-B01lN7 ti4J|U5HMj5kx&kUeLjbMJ^{CVJ_3h91_2AI01K+KQ6QiTx2lT*Rs?_t8j}D3 diff --git a/results/phase4/README.md b/results/phase4/README.md new file mode 100644 index 0000000..217012f --- /dev/null +++ b/results/phase4/README.md @@ -0,0 +1,81 @@ +# Phase 4 — services & scale + +**Exit criteria (SPEC §5): one real service workload optimized under +replayed traffic; report generated mechanically.** Both met. + +Phase 4 takes the trust story off the batch stopwatch and onto a running +service: the same verified optimization, measured as a latency delta +under replayed load, then turned into a dollar figure by a generator +that reads only the ledger and the calibrated bench — no hand-typed +numbers. + +## What was built + +- **Service-mode latency bench** (`crates/bench-runner/src/service.rs`, + SPEC §3.1 mode c): a **coordinated-omission-correct**, open-loop, + fixed-rate load generator. Each request's latency is measured from its + *intended* send time, so when the single worker falls behind, the + requests queued behind it carry the full delay a real client would see + — the exact error closed-loop tools hide. Interleaved A/B of two + server binaries, exact percentiles (bounded sample; no HdrHistogram + bucketing error), bootstrap p50/p99 CIs through the same ratio-CI + machine the wall-clock bench uses. +- **A real service target**: `targets/cjson/service.c`, a minimal HTTP + server wrapping the (patched) `cJSON.c` — trust-layer code outside the + workspace allowlist, so a patch changes the parser under load but not + the measurement rig. +- **Service-mode calibration** (`bench-runner service-calibrate`): the + latency measurement is a trusted anchor only after it passes A/A + (0/10 false positives) and injected-5%-latency-regression detection + (10/10), recorded in `results/calibration/cjson-service-aa.json`. +- **Mechanical ROI report** (`crates/report` binary, SPEC §9): reads a + ledger row + the service-latency JSON and renders + throughput→cores→dollars *and* latency percentiles, every figure with + its 95% CI, the pricing source, and the methodology printed inline. + +## The result (`phase4-cjson-service`, ledger) + +The accepted batch win `phase3-cjson-002` (+8.85% throughput) was +measured as a **service under a 150 rps coordinated-omission-correct +replay**, baseline (pristine cJSON v1.7.19) vs candidate (the banked +win), 20 interleaved rounds, server pinned, 40,000 requests, **0 drops**: + +| metric | baseline | candidate | speedup | 95% CI | +|---|---|---|---|---| +| **p50 latency** | 3.332 ms | 3.136 ms | **1.062** | **[1.058, 1.072]** | +| p99 latency | 24.7 ms | 23.9 ms | 1.033 | [0.07, 4.97] | + +**The p50 win is the claim: +6.2% service-latency reduction under load, +CI [+5.8%, +7.2%]** — the batch parse speedup translating to lower +per-request latency. It is a touch smaller than the +8.85% batch number +because under load the latency also carries fixed connection and HTTP +overhead the parse optimization doesn't touch — an honest dilution, not +a measurement error. + +**The p99 was *not* claimed.** Its CI is [0.07, 4.97] — uselessly wide, +because tail latency on a single-worker loopback service is dominated by +rare OS-scheduling spikes that swamp the ~0.2 ms parse difference. The +harness resolved this correctly: `p99 verdict: rejected-bench`. A number +the measurement can't pin down does not get reported as a win, even +though the median points the right way. That refusal is the whole +project in one line. + +## The ROI, generated mechanically + +`just report phase4-cjson-service --service-json results/phase4/cjson-service.json` +→ [`cjson-service-roi.md`](cjson-service-roi.md). On a 500-core fleet at +$0.04/core-hour, the +6.2% p50 win returns **27.5 cores / $9,621 per +year at the CI lower bound** (median 29.4 cores / $10,290) — and the +report quotes the lower bound, because the point estimate is not a +promise. Every figure carries its CI, its workload, and the methodology +that produced it, so the number survives hostile review. + +## Exit criteria (SPEC §5) — met + +- **One real service workload optimized under replayed traffic:** the + cJSON HTTP service under CO-correct open-loop replay; the verified win + shows a CI-significant +6.2% p50 latency improvement. +- **Report generated mechanically:** the ROI report is produced from the + ledger row and the calibrated service JSON with no hand-editing; + throughput→cores→dollars and latency percentiles, CIs and methodology + inline. diff --git a/results/phase4/cjson-service-roi.md b/results/phase4/cjson-service-roi.md new file mode 100644 index 0000000..65314c8 --- /dev/null +++ b/results/phase4/cjson-service-roi.md @@ -0,0 +1,50 @@ +# ROI report — `phase4-cjson-service` + +**Target:** cjson · **Playbook class:** 5 · **Verdict:** `accepted` + +**Workload:** corpora/cjson/input/service.json, 1 parse+print/req, 150 rps open-loop, CO-correct + +## Throughput → cores → dollars + +Measured speedup (baseline/candidate): **1.0624**, 95% bootstrap CI **[1.0581, 1.0725]**. + +On a **500-core** fleet running this workload: + +| metric | median | 95% CI | +|---|---|---| +| cores returned | 29.4 | [27.5, 33.8] | +| annualized saving | $10290 | [$9621, $11838] | + +Pricing: **$0.04/core-hour**, **8760 h/year**. Source: _example: AWS m7i on-demand us-east-1, per-vCPU approximation — replace per engagement_ + +The **CI lower bound is the number to quote** — 27.5 cores / $9621 per year is +the saving that survives the 95% interval; the point estimate is not a promise. + +## Latency under replayed load + +Workload: corpora/cjson/input/service.json, 1 parse+print/req, 150 rps open-loop, CO-correct. Coordinated-omission-correct open-loop replay. + +| percentile | baseline | candidate | speedup | 95% CI | +|---|---|---|---|---| +| p50 | 3.332 ms | 3.136 ms | 1.0624 | [1.0581, 1.0725] | +| p99 | 24.726 ms | 23.947 ms | 1.0325 | [0.0715, 4.9714] | + +## Methodology (ships with the number) + +- **Interleaved A/B**, baseline rebuilt from a pristine checkout, never the agent's +workspace. Candidate and baseline alternate to control thermal/background drift. +- **Bootstrap 95% CI** of the ratio-of-medians; an effect is accepted only if the CI +lower bound clears the threshold in `config/accept.toml` (2%). +- **Equivalence gates** passed before any number was trusted: upstream test suite, +byte-identical golden replay, differential fuzzing, and ASan/UBSan sanitizers. +- **Calibrated hardware**: the workload passed A/A (false-positive <5%) and injected- +regression (≥95% detection) self-tests, recorded in `results/calibration/`. +- **Coordinated-omission correct** latency (service mode): each request's latency is +measured from its intended send time, so queueing is counted, not hidden. + +## Caveats + +Every figure is specific to the workload and hardware named above. Throughput→cores +assumes the fleet is CPU-bound on this path and scales linearly; latency figures are +for the stated arrival rate. The saving to commit to is the **CI lower bound**, not +the median.