mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
Claude
73763321c6
Adds 'bench-runner calibrate': N A/A sessions counted against a <5% false-positive bar, and N sessions with a synthetic slowdown injected as a real busy-wait inside the candidate's timed window, counted against a >=95% detection bar (detected = ratio CI entirely below 1.0). Bootstrap seed varies per session so resampling noise is not correlated across the very statistic being validated. JSON evidence includes per-session CIs, injected wait, acceptance verdict, and env fingerprint. First run against the real comrak workload (20+20 sessions, pinned core): false-positive rate 0.000, detection rate 0.950 -> PASS. Evidence: results/calibration/2026-07-06-comrak-full-calibration.json. Closes the SPEC 3.1 acceptance-criteria gap; remaining Phase 1 gaps: no-net containers, perf profiling, diff-test orchestration, agent IPC. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
36 lines
1.2 KiB
Makefile
36 lines
1.2 KiB
Makefile
# hotpath task runner. `just --list` for a summary.
|
|
|
|
# Build the trust-layer workspace
|
|
build:
|
|
cargo build --workspace
|
|
|
|
# Run all trust-layer unit tests
|
|
test:
|
|
cargo test --workspace
|
|
|
|
# Lint: warnings are errors in the trust layer
|
|
lint:
|
|
cargo clippy --workspace --all-targets -- -D warnings
|
|
cargo fmt --all --check
|
|
|
|
# A/A self-test of the bench harness against a sample command.
|
|
# Must print a null verdict; an "accept" is a calibration failure.
|
|
aa cmd="sleep 0.05":
|
|
cargo run -p bench-runner -- --config config/accept.toml aa --cmd "{{cmd}}"
|
|
|
|
# Interleaved A/B comparison of two shell commands
|
|
compare baseline candidate:
|
|
cargo run -p bench-runner -- --config config/accept.toml compare \
|
|
--baseline "{{baseline}}" --candidate "{{candidate}}"
|
|
|
|
# Verify a target corpus against its pinned manifest
|
|
# (wired through diff-test CLI once it exists; manual check for now)
|
|
pin-check target:
|
|
cd corpora/{{target}} && sh gen-corpus.sh
|
|
|
|
# Automated calibration: N A/A sessions (<5% false-positive required) +
|
|
# N injected-slowdown sessions (>=95% detection required), JSON evidence
|
|
calibrate cmd out sessions="20":
|
|
cargo run -p bench-runner -- --config config/accept.toml calibrate \
|
|
--cmd "{{cmd}}" --sessions {{sessions}} --out "{{out}}"
|