benbrown/peltier.io
2
1
Fork 0
mirror of https://github.com/bdbrown4/peltier.io.git synced 2026-11-07 10:58:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

bench: automated A/A + regression-injection calibration (Phase 1)

Browse Source 
Adds 'bench-runner calibrate': N A/A sessions counted against a <5%
false-positive bar, and N sessions with a synthetic slowdown injected
as a real busy-wait inside the candidate's timed window, counted
against a >=95% detection bar (detected = ratio CI entirely below 1.0).
Bootstrap seed varies per session so resampling noise is not
correlated across the very statistic being validated. JSON evidence
includes per-session CIs, injected wait, acceptance verdict, and env
fingerprint.

First run against the real comrak workload (20+20 sessions, pinned
core): false-positive rate 0.000, detection rate 0.950 -> PASS.
Evidence: results/calibration/2026-07-06-comrak-full-calibration.json.
Closes the SPEC 3.1 acceptance-criteria gap; remaining Phase 1 gaps:
no-net containers, perf profiling, diff-test orchestration, agent IPC.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
This commit is contained in:
Claude
2026-07-06 20:20:40 +00:00
parent e69d2c46ed
commit 73763321c6
6 changed files with 347 additions and 7 deletions
Download Patch File Download Diff File Expand all files Collapse all files
justfile
+6
View File
@@ -27,3 +27,9 @@ compare baseline candidate:
# (wired through diff-test CLI once it exists; manual check for now)
pin-check target:
cd corpora/{{target}} && sh gen-corpus.sh
# Automated calibration: N A/A sessions (<5% false-positive required) +
# N injected-slowdown sessions (>=95% detection required), JSON evidence
calibrate cmd out sessions="20":
cargo run -p bench-runner -- --config config/accept.toml calibrate \
--cmd "{{cmd}}" --sessions {{sessions}} --out "{{out}}"