mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
Claude
73763321c6
Adds 'bench-runner calibrate': N A/A sessions counted against a <5% false-positive bar, and N sessions with a synthetic slowdown injected as a real busy-wait inside the candidate's timed window, counted against a >=95% detection bar (detected = ratio CI entirely below 1.0). Bootstrap seed varies per session so resampling noise is not correlated across the very statistic being validated. JSON evidence includes per-session CIs, injected wait, acceptance verdict, and env fingerprint. First run against the real comrak workload (20+20 sessions, pinned core): false-positive rate 0.000, detection rate 0.950 -> PASS. Evidence: results/calibration/2026-07-06-comrak-full-calibration.json. Closes the SPEC 3.1 acceptance-criteria gap; remaining Phase 1 gaps: no-net containers, perf profiling, diff-test orchestration, agent IPC. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
results/calibration/
Evidence for the bench-runner acceptance criteria (SPEC §3.1):
- A/A self-tests: same binary as both sides must yield a null verdict; false-positive rate < 5% across sessions.
- Regression injection: a synthetic 5% slowdown must be detected ≥ 95% of the time.
One JSON file per calibration session, including the full env fingerprint. These files are the license to trust every number the system reports afterward; they are referenced from case studies and ROI reports.