mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
benbrownandClaude Opus 4.8
af3fc3dccd
The reusable artifact here was never the vendored targets — it is the discipline: prove equivalence, prove significance, or claim nothing. This packages it as a Claude Code skill usable from any repo. Two modes. `verify` works anywhere, on any two shell commands in any language: equivalence before timing -> A/A calibration -> interleaved A/B with a bootstrap CI -> a verdict decided by the CI lower bound, never the median. `attempt` drives the full in-repo loop (gates, pristine-baseline verdict, ledger row, ROI report). It drives the real bench-runner binary and refuses rather than degrade: on an unsupported host, or with no trust layer reachable, preflight stops and says so instead of falling back to a hand-rolled timing loop. A second copy of the statistics is a second thing that can lie. A refusal ends the claim, not the work. An adversarial review of the skill caught six high-severity defects, all fixed here — several of them the very errors the skill exists to prevent: - verify mode had no risk classification at all. bench-runner emits only accepted/rejected-bench; needs-human-review is minted by the in-repo verdict path, which verify does not run. A concurrency change with byte-identical output would have sailed through as "accepted". The skill now makes the agent the classifier and says so explicitly. - the accept template exposed `pin_prefix`, which is inert for compare/aa/calibrate (only `verdict` reads it) — an agent would have set it and reported pinning that never happened. Now documents wrapping both sides in taskset instead. - the equivalence gate compared stdout only, so two commands that write to stderr and a dist/ tree would "GOLDEN PASS" on two empty files. Now hashes stdout + stderr + exit code + artifact tree, and an empty diff over empty output is a failed gate, not a passed one. - nothing told the agent to build two coexisting immutable artifacts, so a `cargo run` baseline could rebuild from the patched tree mid-bench — the exact trap phase1-verdict-smoke-001 recorded. - service mode was documented as generic HTTP. It is not: the server must take <port> <doc> <iters>, print READY, and close the connection. A stock Express app hangs the harness forever. Contract now stated. - the CI refusal test could not pass: preflight also walks up from its own location, so running it in-place found this very repo. The test now copies the skill out of the tree first. CI job `skill-preflight` asserts the contract that matters — resolve when a trust layer is reachable, REFUSE when it is not, and work when consumed from another repo via PELTIER_HOME. It deliberately does not assert a statistical outcome: a shared runner is too noisy for that, which is precisely why the skill calibrates before it claims. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
42 lines
1.6 KiB
TOML
42 lines
1.6 KiB
TOML
# peltier accept thresholds — consumer template.
|
|
#
|
|
# Copy this next to the code you are measuring and pass it with
|
|
# `bench-runner --config <this-file>`. The defaults are peltier's own
|
|
# published bar; loosening them to make a result pass is not measurement,
|
|
# it is negotiation.
|
|
|
|
# The bootstrap CI *lower bound* of the speedup ratio must clear 1 + threshold.
|
|
# The median is not the claim. The lower bound is the claim.
|
|
threshold = 0.02
|
|
|
|
# Measured runs per side, interleaved ABAB (SPEC minimum: 30).
|
|
runs_per_side = 30
|
|
|
|
# Warm-up runs per side, discarded — they pay for cold caches and JIT.
|
|
warmup_runs = 3
|
|
|
|
# Bootstrap resamples for the ratio CI.
|
|
bootstrap_iters = 10000
|
|
|
|
# Confidence level for the ratio CI.
|
|
confidence = 0.95
|
|
|
|
# Fixed PRNG seed so a calibration run is reproducible by anyone.
|
|
bootstrap_seed = 1887343589
|
|
|
|
# CPU pinning. WARNING: this key is INERT for `compare`, `aa`, and `calibrate`
|
|
# — it is read only by the in-repo `verdict` pipeline. Setting it here and then
|
|
# reporting "pinned to core N" would be a claim about a run that never happened.
|
|
#
|
|
# To actually pin a compare/aa/calibrate run, wrap BOTH sides identically in
|
|
# the command itself: --baseline 'taskset -c 2 <cmd>' --candidate 'taskset -c 2 <cmd>'
|
|
#
|
|
# Pinning tightens the A/A confidence interval ~5x on peltier's calibration
|
|
# hardware. An unpinned host is not disqualified; a NOISY one is, and the A/A
|
|
# self-test decides that empirically. Leave this empty.
|
|
pin_prefix = ""
|
|
|
|
# Differential-fuzz iterations requested per gate run. Only used by the
|
|
# in-repo `verdict` pipeline, which requires a target to declare [gates].fuzz.
|
|
# fuzz_iters = 10000
|