mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
benbrownandClaude Opus 4.8
899d55f4b6
An adversarial audit found the enforcement story ahead of the code: gates that were documented as hard gates were advisory, manual, or absent, and several human-facing claims outran the machine record. This makes the claims true. High findings - Differential fuzz was documentation, not enforcement: verdict hardcoded fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and it is inherently baseline-vs-candidate, so it runs on the accept path and reports Skipped in the standalone `just gates` flow. New hard rule: a machine `accepted` verdict is impossible unless the fuzz gate actually passed — otherwise it caps at needs-human-review. - The agent's system prompt claimed FP/concurrency/unsafe patches were "routed to human review automatically". No such mechanism existed. Added a conservative lexical risk classifier (crates/verdict/src/risk.rs) that forces needs-human-review on any would-be accept carrying those signals; fp-tolerance equivalence always signals. - The local verdict path ran agent-influenced code unsandboxed. It now runs under scripts/no-net.sh (network namespace) and fails closed. Upstream test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when present (scripts/pin-testsuite.sh generates them). Also fixed, found while fixing the above - The pipeline script's `set -e` was inert (brace group as the left operand of `&&`), so a failing candidate build did not abort the run and verdict could bench a stale binary. Now a scoped subshell whose status is branched on at top level. - A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path) stranded RUNNING.lock and left read_verdict polling forever. Failure accounting now lives outside the wrapper. - harnessd's patch allowlist ignored git rename/copy headers and --no-prefix diffs; the rename-escape case is now a committed regression test. - The workspace did not compile on Windows (ungated Unix APIs). cfg-gated; 71 portable tests now run on both platforms. - pin-check regenerated the corpus manifest instead of verifying it; it is now verify-only, with a deliberate `just pin-corpus` to re-pin. - Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics (compare --perf-stat), never part of the accept decision. Docs now match the code, including what is NOT yet enforced: no target ships a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not wired to the fp-tolerance gate — each is stated as mechanized-but-inert rather than done. Corrected the stale "33 ledger rows" (34), the playbook's class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation claim (five accepted rows carry sanitizers_clean=false), the unsupported "40,000 requests" figure, and the blanket "historical rows record fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
104 lines
5.4 KiB
Markdown
104 lines
5.4 KiB
Markdown
# Phase 4 — services & scale
|
|
|
|
**Exit criteria (SPEC §5): one real service workload optimized under
|
|
replayed traffic; report generated mechanically.** Both met.
|
|
|
|
Phase 4 takes the trust story off the batch stopwatch and onto a running
|
|
service: the same verified optimization, measured as a latency delta
|
|
under replayed load, then turned into a dollar figure by a generator
|
|
that reads only the ledger and the calibrated bench — no hand-typed
|
|
numbers.
|
|
|
|
## What was built
|
|
|
|
- **Service-mode latency bench** (`crates/bench-runner/src/service.rs`,
|
|
SPEC §3.1 mode c): a **coordinated-omission-correct**, open-loop,
|
|
fixed-rate load generator. Each request's latency is measured from its
|
|
*intended* send time, so when the single worker falls behind, the
|
|
requests queued behind it carry the full delay a real client would see
|
|
— the exact error closed-loop tools hide. Interleaved A/B of two
|
|
server binaries, exact percentiles (bounded sample; no HdrHistogram
|
|
bucketing error), bootstrap p50/p99 CIs through the same ratio-CI
|
|
machine the wall-clock bench uses.
|
|
- **A real service target**: `targets/cjson/service.c`, a minimal HTTP
|
|
server wrapping the (patched) `cJSON.c` — trust-layer code outside the
|
|
workspace allowlist, so a patch changes the parser under load but not
|
|
the measurement rig.
|
|
- **Service-mode calibration** (`bench-runner service-calibrate`): the
|
|
latency measurement is a trusted anchor only after it passes A/A
|
|
(0/10 false positives) and injected-5%-latency-regression detection
|
|
(10/10), recorded in `results/calibration/cjson-service-aa.json`.
|
|
- **Mechanical ROI report** (`crates/report` binary, SPEC §9): reads a
|
|
ledger row + the service-latency JSON and renders
|
|
throughput→cores→dollars *and* latency percentiles, every figure with
|
|
its 95% CI, the pricing source, and the methodology printed inline.
|
|
|
|
## The result (`phase4-cjson-service`, ledger)
|
|
|
|
The accepted batch win `phase3-cjson-002` (+8.85% throughput) was
|
|
measured as a **service under a 150 rps-target coordinated-omission-correct
|
|
replay**, baseline (pristine cJSON v1.7.19) vs candidate (the banked
|
|
win), 20 interleaved rounds, server pinned, **0 drops** — the ledger row
|
|
records `sessions=20`, `rate_rps=150`, `drop_rate=0.0`; the total request
|
|
count was **not machine-recorded** (see errata below):
|
|
|
|
| metric | baseline | candidate | speedup | 95% CI |
|
|
|---|---|---|---|---|
|
|
| **p50 latency** | 3.332 ms | 3.136 ms | **1.062** | **[1.058, 1.072]** |
|
|
| p99 latency | 24.7 ms | 23.9 ms | 1.033 | [0.07, 4.97] |
|
|
|
|
**The p50 win is the claim: +6.2% service-latency reduction under load,
|
|
CI [+5.8%, +7.2%]** — the batch parse speedup translating to lower
|
|
per-request latency. It is a touch smaller than the +8.85% batch number
|
|
because under load the latency also carries fixed connection and HTTP
|
|
overhead the parse optimization doesn't touch — an honest dilution, not
|
|
a measurement error.
|
|
|
|
**The p99 was *not* claimed.** Its CI is [0.07, 4.97] — uselessly wide,
|
|
because tail latency on a single-worker loopback service is dominated by
|
|
rare OS-scheduling spikes that swamp the ~0.2 ms parse difference. The
|
|
harness resolved this correctly: `p99 verdict: rejected-bench`. A number
|
|
the measurement can't pin down does not get reported as a win, even
|
|
though the median points the right way. That refusal is the whole
|
|
project in one line.
|
|
|
|
## The ROI, generated mechanically
|
|
|
|
`just report phase4-cjson-service --service-json results/phase4/cjson-service.json`
|
|
→ [`cjson-service-roi.md`](cjson-service-roi.md). *(Note, 2026-07-13: the
|
|
raw `cjson-service.json` was not preserved — see errata below — so this
|
|
command is not reproducible as written; the generated report and the
|
|
ledger row are the surviving evidence.)* On a 500-core fleet at
|
|
$0.04/core-hour, the +6.2% p50 win returns **27.5 cores / $9,621 per
|
|
year at the CI lower bound** (median 29.4 cores / $10,290) — and the
|
|
report quotes the lower bound, because the point estimate is not a
|
|
promise. Every figure carries its CI, its workload, and the methodology
|
|
that produced it, so the number survives hostile review.
|
|
|
|
## Exit criteria (SPEC §5) — met
|
|
|
|
- **One real service workload optimized under replayed traffic:** the
|
|
cJSON HTTP service under CO-correct open-loop replay; the verified win
|
|
shows a CI-significant +6.2% p50 latency improvement.
|
|
- **Report generated mechanically:** the ROI report is produced from the
|
|
ledger row and the calibrated service JSON with no hand-editing;
|
|
throughput→cores→dollars and latency percentiles, CIs and methodology
|
|
inline.
|
|
|
|
## Errata (2026-07-13)
|
|
|
|
- **"40,000 requests" retracted.** This report originally stated the
|
|
replay comprised 40,000 requests. The `phase4-cjson-service` ledger row
|
|
records only `sessions=20`, `rate_rps=150`, and `drop_rate=0.0` — the
|
|
per-run request count was never machine-recorded, so the 40,000 figure
|
|
(an arithmetic reconstruction, not evidence) is withdrawn. The prose
|
|
above now states the ledger-backed facts only. The claimed result is
|
|
unaffected: p50 latency +6.2%, 95% CI [+5.8%, +7.2%], under the 150
|
|
rps-target CO-correct replay, 20 interleaved rounds, 0 drops.
|
|
- **`results/phase4/cjson-service.json` was not preserved.** The
|
|
`.gitignore` rule `results/**/*.json` swept the raw service-latency
|
|
JSON that the report command consumes, so it no longer exists in the
|
|
repository. A `!results/phase4/*.json` carve-out has been added so
|
|
future service-mode evidence JSONs are committed alongside the
|
|
calibration JSONs.
|