benbrownandClaude Opus 4.8
af3fc3dccd
The reusable artifact here was never the vendored targets — it is the discipline: prove equivalence, prove significance, or claim nothing. This packages it as a Claude Code skill usable from any repo. Two modes. `verify` works anywhere, on any two shell commands in any language: equivalence before timing -> A/A calibration -> interleaved A/B with a bootstrap CI -> a verdict decided by the CI lower bound, never the median. `attempt` drives the full in-repo loop (gates, pristine-baseline verdict, ledger row, ROI report). It drives the real bench-runner binary and refuses rather than degrade: on an unsupported host, or with no trust layer reachable, preflight stops and says so instead of falling back to a hand-rolled timing loop. A second copy of the statistics is a second thing that can lie. A refusal ends the claim, not the work. An adversarial review of the skill caught six high-severity defects, all fixed here — several of them the very errors the skill exists to prevent: - verify mode had no risk classification at all. bench-runner emits only accepted/rejected-bench; needs-human-review is minted by the in-repo verdict path, which verify does not run. A concurrency change with byte-identical output would have sailed through as "accepted". The skill now makes the agent the classifier and says so explicitly. - the accept template exposed `pin_prefix`, which is inert for compare/aa/calibrate (only `verdict` reads it) — an agent would have set it and reported pinning that never happened. Now documents wrapping both sides in taskset instead. - the equivalence gate compared stdout only, so two commands that write to stderr and a dist/ tree would "GOLDEN PASS" on two empty files. Now hashes stdout + stderr + exit code + artifact tree, and an empty diff over empty output is a failed gate, not a passed one. - nothing told the agent to build two coexisting immutable artifacts, so a `cargo run` baseline could rebuild from the patched tree mid-bench — the exact trap phase1-verdict-smoke-001 recorded. - service mode was documented as generic HTTP. It is not: the server must take <port> <doc> <iters>, print READY, and close the connection. A stock Express app hangs the harness forever. Contract now stated. - the CI refusal test could not pass: preflight also walks up from its own location, so running it in-place found this very repo. The test now copies the skill out of the tree first. CI job `skill-preflight` asserts the contract that matters — resolve when a trust layer is reachable, REFUSE when it is not, and work when consumed from another repo via PELTIER_HOME. It deliberately does not assert a statistical outcome: a shared runner is too noisy for that, which is precisely why the skill calibrates before it claims. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
9.4 KiB
verify — prove a speedup in any repo
You have two things that should do the same work, one of which you think is faster. Find out. Works on any pair of shell commands, in any language.
0. Preflight
sh <skill-dir>/scripts/preflight.sh
Read the key=value output; you need BENCH_RUNNER and PIN_SUPPORTED. A
refusal ends the claim, not the work — report the reason verbatim and do
not substitute another timing tool.
BR=<BENCH_RUNNER from preflight>
1. Config
cp <skill-dir>/assets/accept.template.toml ./peltier-accept.toml
CFG=./peltier-accept.toml
Do not loosen threshold, runs_per_side, or confidence to make a
result pass. That is not measurement, it is negotiation.
Pinning is not done through this file. pin_prefix is read only by the
in-repo verdict pipeline — compare, aa, and calibrate ignore it
entirely (bench-runner's own fingerprint says "caller-provided (wrap cmd in taskset)"). If PIN_SUPPORTED=yes and you know a core is free, pin by
wrapping both sides identically:
--baseline 'taskset -c 2 <cmd>' --candidate 'taskset -c 2 <cmd>'
An unpinned host is not disqualified. A noisy one is, and step 3 is what decides that.
2. Two immutable artifacts — not two states of one tree
The harness runs the two sides alternately, 66 times. That is impossible if "baseline" and "candidate" are the same working tree in different states. Before measuring:
- Build each side into its own location — a
git worktreeper side, or two separate build/output directories. Nevergit stashbetween runs. - Pass the built artifact paths as the commands. Never pass a wrapper that
can rebuild:
cargo run,npm start,go run,mvn execwill happily rebuild from whatever is in the tree mid-benchmark, so you can end up timing the same binary twice — and paying build cost inside the timed window. - Confirm the two artifacts actually differ:
sha256sum <baseline> <candidate>.
Peltier learned this the hard way (phase1-verdict-smoke-001: "workspace paths
get silently rebuilt from the patched tree by test runs"), which is why the
in-repo pipeline rebuilds a pristine baseline from the pinned upstream commit
rather than trusting the tree.
3. Equivalence — before you time anything
Capture everything each side produces, on a fixed, representative workload:
run_side() { # $1 = tag, $2 = command
rm -rf "out.$1"; mkdir -p "out.$1"
( eval "$2" ) > "out.$1/stdout" 2> "out.$1/stderr"; echo "$?" > "out.$1/exit"
}
run_side base "<baseline-cmd>"
run_side cand "<candidate-cmd>"
diff -r out.base out.cand && echo "GOLDEN PASS" || echo "DIVERGENT"
If the command's real output is a file tree (a bundler, a compiler, an image pipeline), hash that too — stdout is not the artifact:
( cd dist && find . -type f | sort | xargs sha256sum ) > out.<tag>/tree
An empty diff over empty output is a failed gate, not a passed one. If both sides wrote nothing to stdout, nothing to stderr, and you captured no artifact tree, you have compared nothing — stop and find the real output before going further. Then run the project's own test suite against the candidate; it must pass.
- Identical output + tests green → proceed.
- Divergent → stop. No speedup may be claimed from a change you cannot show
is behavior-preserving. The one exception — a change that legitimately
alters output, floating-point reassociation being the common case — is
needs-human-review, never an auto-accept. Quantify the divergence (how many values, by how much) and hand it to a human.
4. Calibration — can this host measure this change?
The A/A self-test runs the same command on both sides. It must find nothing:
$BR --config "$CFG" aa --cmd '<baseline-cmd>'
- stdout
A/A self-test passed (null verdict)→ the harness is not inventing speedups. Proceed. - Nonzero exit, with
Error: A/A self-test FAILED: harness claims a speedup from identical binarieson stderr → stop. This host is too noisy to measure this change; every number it produces afterwards is fiction. Report the noise floor, then quiet the machine (pin a core, disable turbo/SMT, close everything else) or move to calibrated hardware.
For a claim anyone will act on, run the full calibration — a false-positive rate over many A/A sessions plus proof the harness can detect a known injected regression:
$BR --config "$CFG" calibrate --cmd '<baseline-cmd>' \
--sessions 20 --slowdown 0.05 --out calibration.json
Required: <5% A/A false-positive rate and ≥95% detection of the injected
5% slowdown. A harness that cannot see a real 5% regression cannot be trusted
to confirm a real 5% win. Keep calibration.json — it is the evidence that the
measurement itself was sound.
Budget the runtime before you start
compare executes the workload (30 + 3) × 2 = 66 times. calibrate --sessions 20 executes it roughly 2,650 times. Size the workload to
~0.2–2 s of wall time: shorter and you measure process startup, longer and
calibration takes hours. Run calibrate in the background — it will blow
through a foreground command timeout.
5. Measure — interleaved A/B, bootstrap CI
$BR --config "$CFG" compare --baseline '<baseline-cmd>' --candidate '<candidate-cmd>'
Runs are interleaved (ABAB…), not blocked, so thermal drift and cache state hit both sides equally. Output:
speedup (baseline/candidate): median 1.0884, 95% CI [1.0745, 1.1001]
verdict: accepted
The ratio is baseline/candidate, so >1 means the candidate is faster.
--perf-stat (Linux) adds cycles/instructions/cache-miss medians — useful for
explaining why something got faster, never part of the accept decision,
and usually unavailable in VMs.
6. Risk — you are the classifier here
bench-runner emits only accepted or rejected-bench. It cannot route a
dangerous change to review, because in verify mode nothing is inspecting your
diff. So inspect it yourself, and override the tool:
Read the patch. If it touches any of these, the outcome is needs-human-review
regardless of what compare printed:
- Concurrency — threads, locks, atomics, memory ordering,
volatile, work queues, async pools,rayon/pthread/std::sync. - Unsafe / raw memory —
unsafe, pointer casts,transmute, FFI, manual allocation, uninitialized memory. - Floating-point reassociation — reordering accumulation,
-ffast-math, FP SIMD, changed summation order.
These changes can be correct on your workload and wrong in production, and a green bench says nothing about that. A byte-identical output does not clear them: a data race produces identical output right up until it doesn't.
7. Verdict — read it honestly
| What you see | What it means | What you may say |
|---|---|---|
verdict: accepted |
CI lower bound clears the threshold. | Claim it — with the CI lower bound, the interval, and the workload. |
verdict: rejected-bench, CI spans 1.0 (e.g. [0.983, 1.001]) |
Indistinguishable from no change. | "No shippable win." A real, complete result. |
verdict: rejected-bench, CI entirely below 1.0 (e.g. [0.91, 0.96]) |
A measured regression. | Report it as one, with the interval. Do not call it "no win." |
| Any risk signal from step 6 | Out of the machine's hands. | needs-human-review. Never auto-accept. |
Claim the lower bound, not the median: a median of 1.088 with CI [1.074, 1.100] supports "≥7.4% faster", not "8.8% faster".
If the result is a rejection, that is the answer. Re-running until it passes is p-hacking. Two failed hypotheses on the same hot path means you need a new hypothesis, not another run.
Services (latency, not throughput)
For a server, wall-clock-per-run is the wrong instrument. Service mode is an open-loop, fixed-rate load generator that measures each request from its intended send time, so queueing delay is counted rather than hidden (coordinated omission). It reports p50/p99 with bootstrap CIs.
It is not a generic HTTP load generator. The server binary must:
- take
<port> <doc> <iters>as positional argv; - print a line beginning with
READYon stdout, flushed, once listening; - answer
GET /and close the connection (Connection: close) — the client reads until EOF with no timeout, so a keep-alive server hangs the harness forever.
Point it at a stock Express or Flask app and it will either fail the READY
handshake or block indefinitely. Peltier wrote a thin adapter binary
(targets/cjson/service.c) for exactly this reason; you will need one too.
$BR --config "$CFG" service --baseline-bin '<bin>' --candidate-bin '<bin>' \
--doc <workload> --rate 150 --sessions 20
Calibrate it the same way first (service-calibrate). Tail percentiles on a
single-worker loopback setup are jittery — if the p99 CI is wide, do not claim
p99, even when p50 is solid. Peltier's own shipped service result claims p50
(+6.2%, CI [+5.8%, +7.2%]) and explicitly declines to claim p99 (CI [0.07, 4.97]).
Record the attempt — including the failures
Outside peltier there is no ledger, so write the evidence where the next person will find it (a results file, the PR body): hypothesis, what changed, equivalence result, calibration evidence, the CI, the workload, and the verdict. Failures are the most valuable rows — they stop someone from re-running your dead end in six months.