mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
Claude
baa6c4e967
Adds crates/verdict ('just verdict'): runs the core gates on the
candidate binary, then an interleaved A/B bench vs the baseline with
the CI-lower-bound rule, and writes the append-only ledger row with
per-side median CIs, workload statement, and env fingerprint. Gate
failure short-circuits to rejected-gate; --needs-human-review caps an
accepted verdict (never upgrades). CPU pinning moves into accept.toml
(pin_prefix, machine-specific, applied to both sides identically).
The smoke test caught a real trust gap: pointing --baseline-bin at the
workspace build (phase1-verdict-smoke-001) measured patched-vs-patched
and returned null, because gate runs rebuild the workspace binary from
the patched tree. Fix: --rebuild-baseline stashes the patch, builds
the baseline into isolated targets/<name>/baseline/, and restores the
patch — SPEC 3.1's pristine-checkout rule, now enforced by the
harness. Re-run (phase1-verdict-smoke-002) reproduces the Phase 0
result mechanically: +10.4% median, 95% CI [1.0932, 1.1137], accepted.
Both smoke rows are in the ledger, including the null one.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
50 lines
1.9 KiB
Makefile
50 lines
1.9 KiB
Makefile
# hotpath task runner. `just --list` for a summary.
|
|
|
|
# Build the trust-layer workspace
|
|
build:
|
|
cargo build --workspace
|
|
|
|
# Run all trust-layer unit tests
|
|
test:
|
|
cargo test --workspace
|
|
|
|
# Lint: warnings are errors in the trust layer
|
|
lint:
|
|
cargo clippy --workspace --all-targets -- -D warnings
|
|
cargo fmt --all --check
|
|
|
|
# A/A self-test of the bench harness against a sample command.
|
|
# Must print a null verdict; an "accept" is a calibration failure.
|
|
aa cmd="sleep 0.05":
|
|
cargo run -p bench-runner -- --config config/accept.toml aa --cmd "{{cmd}}"
|
|
|
|
# Interleaved A/B comparison of two shell commands
|
|
compare baseline candidate:
|
|
cargo run -p bench-runner -- --config config/accept.toml compare \
|
|
--baseline "{{baseline}}" --candidate "{{candidate}}"
|
|
|
|
# Verify a target corpus against its pinned manifest
|
|
# (wired through diff-test CLI once it exists; manual check for now)
|
|
pin-check target:
|
|
cd corpora/{{target}} && sh gen-corpus.sh
|
|
|
|
# Automated calibration: N A/A sessions (<5% false-positive required) +
|
|
# N injected-slowdown sessions (>=95% detection required), JSON evidence
|
|
calibrate cmd out sessions="20":
|
|
cargo run -p bench-runner -- --config config/accept.toml calibrate \
|
|
--cmd "{{cmd}}" --sessions {{sessions}} --out "{{out}}"
|
|
|
|
# Equivalence gates for a target: corpus pin (refuses on mismatch) ->
|
|
# upstream tests -> golden replay. Fuzz/sanitizers stay per-attempt.
|
|
gates target:
|
|
cargo run -p diff-test -- {{target}}
|
|
|
|
# One-shot attempt verdict: gates on candidate -> interleaved A/B vs a
|
|
# pristine-rebuilt baseline -> ledger row. Extra flags pass through
|
|
# (--patch-file, --needs-human-review, --baseline-bin).
|
|
verdict target candidate run_id class hypothesis hotspot *flags:
|
|
cargo run -p verdict -- {{target}} --rebuild-baseline \
|
|
--candidate-bin "{{candidate}}" --run-id "{{run_id}}" \
|
|
--playbook-class {{class}} --hypothesis "{{hypothesis}}" \
|
|
--hotspot "{{hotspot}}" {{flags}}
|