benbrown/peltier.io
2
1
Fork 0
mirror of https://github.com/bdbrown4/peltier.io.git synced 2026-11-07 10:58:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

bench: automated A/A + regression-injection calibration (Phase 1)

Browse Source 
Adds 'bench-runner calibrate': N A/A sessions counted against a <5%
false-positive bar, and N sessions with a synthetic slowdown injected
as a real busy-wait inside the candidate's timed window, counted
against a >=95% detection bar (detected = ratio CI entirely below 1.0).
Bootstrap seed varies per session so resampling noise is not
correlated across the very statistic being validated. JSON evidence
includes per-session CIs, injected wait, acceptance verdict, and env
fingerprint.

First run against the real comrak workload (20+20 sessions, pinned
core): false-positive rate 0.000, detection rate 0.950 -> PASS.
Evidence: results/calibration/2026-07-06-comrak-full-calibration.json.
Closes the SPEC 3.1 acceptance-criteria gap; remaining Phase 1 gaps:
no-net containers, perf profiling, diff-test orchestration, agent IPC.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
This commit is contained in:
Claude
2026-07-06 20:20:40 +00:00
parent e69d2c46ed
commit 73763321c6
6 changed files with 347 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+1
View File
@@ -54,6 +54,7 @@ just build / test / lint # trust-layer workspace: cargo build/test/clippy+fmt
just aa [cmd] # A/A self-test of bench-runner on a shell command
just compare <a> <b> # interleaved A/B of two shell commands, bootstrap CI
just pin-check <target> # verify corpus MANIFEST.sha256
just calibrate <cmd> <out> # automated A/A + regression-injection calibration
```
## Definition of done — one optimization attempt