benbrownandClaude Opus 4.8
af3fc3dccd
The reusable artifact here was never the vendored targets — it is the discipline: prove equivalence, prove significance, or claim nothing. This packages it as a Claude Code skill usable from any repo. Two modes. `verify` works anywhere, on any two shell commands in any language: equivalence before timing -> A/A calibration -> interleaved A/B with a bootstrap CI -> a verdict decided by the CI lower bound, never the median. `attempt` drives the full in-repo loop (gates, pristine-baseline verdict, ledger row, ROI report). It drives the real bench-runner binary and refuses rather than degrade: on an unsupported host, or with no trust layer reachable, preflight stops and says so instead of falling back to a hand-rolled timing loop. A second copy of the statistics is a second thing that can lie. A refusal ends the claim, not the work. An adversarial review of the skill caught six high-severity defects, all fixed here — several of them the very errors the skill exists to prevent: - verify mode had no risk classification at all. bench-runner emits only accepted/rejected-bench; needs-human-review is minted by the in-repo verdict path, which verify does not run. A concurrency change with byte-identical output would have sailed through as "accepted". The skill now makes the agent the classifier and says so explicitly. - the accept template exposed `pin_prefix`, which is inert for compare/aa/calibrate (only `verdict` reads it) — an agent would have set it and reported pinning that never happened. Now documents wrapping both sides in taskset instead. - the equivalence gate compared stdout only, so two commands that write to stderr and a dist/ tree would "GOLDEN PASS" on two empty files. Now hashes stdout + stderr + exit code + artifact tree, and an empty diff over empty output is a failed gate, not a passed one. - nothing told the agent to build two coexisting immutable artifacts, so a `cargo run` baseline could rebuild from the patched tree mid-bench — the exact trap phase1-verdict-smoke-001 recorded. - service mode was documented as generic HTTP. It is not: the server must take <port> <doc> <iters>, print READY, and close the connection. A stock Express app hangs the harness forever. Contract now stated. - the CI refusal test could not pass: preflight also walks up from its own location, so running it in-place found this very repo. The test now copies the skill out of the tree first. CI job `skill-preflight` asserts the contract that matters — resolve when a trust layer is reachable, REFUSE when it is not, and work when consumed from another repo via PELTIER_HOME. It deliberately does not assert a statistical outcome: a shared runner is too noisy for that, which is precisely why the skill calibrates before it claims. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
5.6 KiB
peltier.io
Spot cooling for hot paths. A profile-guided optimization agent that only ships wins it can prove.
peltier.io profiles real binaries and services, isolates hot paths, proposes optimizations, and accepts a change only when two independent bars are cleared: behavioral equivalence (upstream tests + byte-identical golden replay + differential fuzzing + sanitizers) and statistical significance (the speedup's bootstrap 95% CI lower bound clears a threshold on calibrated hardware). A change without both is discarded and logged.
Consultants sell surveys; compilers sell flags. Peltier sells verified deltas with the methodology attached — a stopwatch on pinned hardware, a proof of equivalence, and an ROI figure whose confidence interval survives hostile review.
📖 Full documentation: the peltier.io book · Design spec: SPEC.md · Charter: CLAUDE.md
Verified results
Every number carries its 95% CI and its workload, and every one survived the gates above. Across five phases and 34 ledger rows, zero shipped false accepts — including two pipeline over-accepts the audit caught before anything shipped, each becoming a permanent new gate.
| Target | Win | Verified |
|---|---|---|
| tokei (Rust) — LTO (class 1) | +10.4% | CI [+8.5%, +12.0%] |
| tokei — three class-5 algorithmic wins | +10.7% / +3.7% / +9.9% | each CI-significant |
| cJSON (C) — number-handling rewrite (class 5) | +8.85% | CI [+7.5%, +10.0%] |
| cJSON HTTP service — p50 latency under replay | +6.2% | CI [+5.8%, +7.2%] |
| comrak (Rust) — mimalloc (class 2) | +4.6% | CI [+3.3%, +5.8%], human-ruled |
Layout
crates/ Trust layer — agent has NO write access (SPEC §10)
bench-runner/ Interleaved A/B, bootstrap CIs, A/A calibration, service mode
diff-test/ Equivalence gates, corpus hash-pinning, per-target spec
ledger/ Append-only SQLite attempt ledger (mutation-refusing triggers)
report/ ROI: speedup CI → cores → dollars, methodology inline
verdict/ The pipeline in one command: gates → bench → ledger row
harnessd/ The one door the agent talks through (7 JSON ops)
agent/ Untrusted proposer (Claude Agent SDK, Python)
playbook/ Optimization classes 1–7, tried strictly cheapest-first
config/ accept.toml (thresholds), pricing.toml (ROI inputs)
targets/ Vendored OSS targets — the only agent-writable path
corpora/ Hash-pinned golden-replay inputs (read-only to agent)
results/ Calibration evidence, case studies, generated ROI reports
docs/ The mdBook site (deployed to GitHub Pages)
Quick start
just build / test / lint # trust-layer workspace
just aa # A/A self-test — must yield a null verdict
just gates <target> # corpus pin + upstream tests + golden replay
# (fuzz needs a baseline — it runs in `just verdict`)
just verdict <t> <bin> ... # gates + fuzz + sanitizers + bench vs pristine baseline
just report <run-id> # ROI report from a ledger row
just isolation-check # 19 OS-boundary checks (both modes)
The pipeline is Linux/POSIX-only at runtime (Unix sockets, setsid,
sh-based gates). Windows builds the workspace and runs the portable unit
tests, but cannot run the pipeline.
Full command reference and reproduction steps: Reproduce it yourself.
Use it on your own code
The trust layer is not specific to these targets — it will judge any two shell
commands. .claude/skills/peltier/ packages it as a Claude Code skill:
# from any repo, with a peltier checkout reachable
export PELTIER_HOME=/path/to/peltier.io
mkdir -p .claude/skills
cp -r /path/to/peltier.io/.claude/skills/peltier .claude/skills/
Then ask Claude to verify a speedup. The skill enforces the order that makes a performance claim mean something: equivalence before timing (a faster program that computes something different is a bug, not an optimization) → A/A calibration (a host that "finds" a speedup between a binary and itself cannot measure yours) → interleaved A/B with a bootstrap CI → a verdict decided by the CI lower bound, never the median.
It drives the real bench-runner binary and refuses rather than degrade:
on an unsupported host, or with no trust layer reachable, it stops and says so
instead of falling back to a hand-rolled timing loop. A second copy of the
statistics is a second thing that can lie.
How it stays honest
- The agent speaks to the trust layer through seven read-only-plus-one- write JSON operations; it has no shell, cannot write outside a target workspace, and cannot touch the ledger or thresholds.
- The baseline is rebuilt from a pristine checkout every session — a patch never becomes its own comparison.
- The ledger is append-only (SQLite triggers refuse UPDATE/DELETE);
accepted rows are written only by the
verdictbinary after the gates. - The agent process runs under OS-level isolation (read-only mount namespace or an unprivileged uid) — the boundary is filesystem permissions, not a prompt.
See the case studies — especially the caught false-accept, where the pipeline over-accepted a leaking patch and the audit overturned it before it shipped.
License
GPL-3.0-or-later — see LICENSE.