mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
skill: make the peltier skill consumable across agent harnesses
The SKILL.md Agent Skills standard (agentskills.io) is now read natively by Codex, Copilot, VS Code, Cursor, Gemini CLI, opencode, Goose, Amp, Zed, Factory, and ZeroClaw — so the portable move is not per-harness adapter files (forked content that drifts) but one canonical skill placed where each harness actually looks. - Spec-compliant frontmatter: license, compatibility (the Linux-only measurement story, stated up front), version/author/tags top-level (ZeroClaw's runtime reads those; the spec ignores unknown fields). reference/ renamed to references/ per the spec's conventional layout. - .agents/skills/peltier/ mirror: the cross-tool convergence path (Codex, Zed, Cursor, Gemini CLI, Copilot, opencode, Goose, Amp). Byte-identical to the .claude canonical, enforced by CI — drift between the two copies is a red build, not a footnote. - AGENTS.md at the repo root for the ~20 harnesses that read it (Claude Code does not; CLAUDE.md remains its file). Deliberately thin: it points at CLAUDE.md and the skill instead of restating rules. - scripts/install-skill.sh: stamps a consumer repo at both discovery paths (--layout claude|agents|both), or emits a script-less --zeroclaw-variant. ZeroClaw's skill security policy blocks script files by default (skills.allow_scripts off) — its audit rejected the full skill and passes the variant, verified against the real zeroclaw binary. The variant loses nothing: preflight.sh still exists exactly once, in the peltier checkout the skill already requires, and SKILL.md now documents running it from there. Re-implementing preflight inline is named as a refusal condition. - CI (skill-preflight): frontmatter upgraded to a full Agent Skills spec conformance check on BOTH copies (name rules, dir match, description and body limits, metadata string-map), plus mirror equality and a functional installer test (both layouts identical to canonical, variant carries no scripts/, nonexistent target refused). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
Claude Fable 5
@@ -0,0 +1,142 @@
|
||||
---
|
||||
name: peltier
|
||||
description: "Prove a performance change is real — or refuse to claim it. Use when optimizing code for speed, verifying a claimed speedup ('is this actually faster?', 'did my optimization work?'), checking for a performance regression ('did this PR make things slower?'), or comparing the performance of two implementations. Runs peltier's trust layer — equivalence gates, A/A environment calibration, interleaved A/B with bootstrap confidence intervals. Never reports a naked percentage and never claims a win that is not statistically significant. Do NOT use for correctness-only work."
|
||||
license: "GPL-3.0-or-later"
|
||||
compatibility: "Needs a peltier checkout (https://github.com/bdbrown4/peltier.io; set PELTIER_HOME) with cargo to build bench-runner. Measurement requires a Linux/POSIX host; on unsupported hosts the skill refuses to produce numbers rather than degrade."
|
||||
version: "1.1.0"
|
||||
author: "peltier.io (Ben Brown)"
|
||||
tags: ["performance", "benchmarking", "statistics", "verification"]
|
||||
metadata:
|
||||
homepage: "https://bdbrown4.github.io/peltier.io/"
|
||||
source: "https://github.com/bdbrown4/peltier.io"
|
||||
---
|
||||
|
||||
# peltier — prove it, or claim nothing
|
||||
|
||||
A performance change that has not passed equivalence gates **and** shown a
|
||||
statistically significant improvement **does not exist**. You may discard it
|
||||
or log it. You may not claim it.
|
||||
|
||||
Most benchmarking is wrong in ways that flatter the author: run-to-run noise
|
||||
read as a win, a faster binary that computes something subtly different, one
|
||||
lucky run out of five, a percentage with no interval attached. This skill
|
||||
exists to make those failures impossible rather than unlikely.
|
||||
|
||||
## Preflight — never skip, never work around
|
||||
|
||||
Run `scripts/preflight.sh` **relative to this skill's directory** (resolve its
|
||||
absolute path first — the skill may be installed at `~/.claude/skills/`, as a
|
||||
plugin, or copied into another project):
|
||||
|
||||
sh <skill-dir>/scripts/preflight.sh
|
||||
|
||||
Some harnesses block script files inside skills, so your installation may
|
||||
carry no `scripts/` directory. The script still exists exactly once — in the
|
||||
peltier checkout this skill drives (which you need anyway; it is where
|
||||
`bench-runner` lives). Run it from there:
|
||||
|
||||
sh "$PELTIER_HOME/.claude/skills/peltier/scripts/preflight.sh"
|
||||
|
||||
Never transcribe or re-implement the preflight logic inline; if you cannot
|
||||
locate a checkout to run it from, that IS a preflight refusal — report it.
|
||||
|
||||
It locates the trust layer, builds the real `bench-runner`, and checks this
|
||||
host can run it. `STATUS=ok` does **not** mean the host can measure your
|
||||
change — only the A/A self-test decides that.
|
||||
|
||||
**If it refuses, that ends the claim, not the work.** You may still write the
|
||||
patch and explain why you think it is faster. You may not attach a number to
|
||||
it. Do not fall back to `time`, `hyperfine`, or a stopwatch loop — a number
|
||||
you cannot stand behind is worse than no number, because someone will act on
|
||||
it. The statistics live in `bench-runner` and are used from there; a second
|
||||
copy of the math is a second thing that can lie.
|
||||
|
||||
## Two modes
|
||||
|
||||
| Mode | Use when | Read |
|
||||
|---|---|---|
|
||||
| **verify** | You have a baseline and a candidate — any repo, any language. | [references/verify.md](references/verify.md) |
|
||||
| **attempt** | You are inside a peltier checkout and want the full loop: profile → hypothesize → patch → gated verdict → ledger row. | [references/attempt.md](references/attempt.md) |
|
||||
|
||||
Default to **verify**. It is the portable one, and it is what almost every
|
||||
request actually wants.
|
||||
|
||||
### No candidate yet? Profile first — do not guess.
|
||||
|
||||
"Measure before optimizing" is not advice, it is a gate. If you have one
|
||||
program and a hunch, get profile data *before* writing a patch: `perf record`
|
||||
(native), `py-spy` / `cProfile` (Python), `node --prof` / `--cpu-prof` (Node),
|
||||
`cargo flamegraph` (Rust), or `just coz` for causal profiling inside a peltier
|
||||
checkout. Then form a hypothesis about a *named* hot path, and only then patch.
|
||||
Guessing where the time goes is how people spend a week on 0.3% of the runtime.
|
||||
|
||||
## The order of the gates is the point
|
||||
|
||||
Each gate can end the job, and ending early is a success, not a failure.
|
||||
|
||||
1. **Equivalence — before you time anything.** A faster program that does
|
||||
something different is not an optimization, it is a bug with good
|
||||
benchmarks. Prove baseline and candidate produce identical output on a fixed
|
||||
workload. Comparing two *empty* outputs is a failed gate, not a passed one.
|
||||
2. **Calibration — can this machine measure this change at all?** The A/A
|
||||
self-test runs the same command against itself and must find **nothing**. A
|
||||
harness that "discovers" a speedup between a binary and itself is measuring
|
||||
noise, and every number it gives you afterwards is fiction.
|
||||
3. **Measurement — interleaved A/B, bootstrap CI.** Not "run A five times, then
|
||||
B five times" — that hands any thermal or cache drift to whichever side ran
|
||||
last.
|
||||
4. **Risk — what did the patch actually touch?** Concurrency primitives,
|
||||
`unsafe`/raw memory, floating-point reassociation: `needs-human-review`,
|
||||
**regardless of how good the numbers are**. In verify mode *you* are the
|
||||
classifier — `bench-runner` only ever emits `accepted` or `rejected-bench`,
|
||||
so it cannot make this call for you.
|
||||
5. **Verdict — the CI lower bound decides.** Accept only if the bootstrap 95%
|
||||
CI lower bound of the speedup clears the threshold (default 2%). The median
|
||||
is not the claim. The lower bound is the claim.
|
||||
|
||||
## Non-negotiables
|
||||
|
||||
- **Cheapest wins first:** build flags → LTO → PGO → allocator swap → *then*
|
||||
code changes. Do not hand-write SIMD before you have tried `-O3`.
|
||||
- **Every attempt is recorded, including failures.** A rejected hypothesis is a
|
||||
result — it stops the next person from re-running your dead end.
|
||||
- **A failed gate is a complete, valid outcome.** Write it up and move on. Do
|
||||
not iterate on a rejected patch more than twice without a new hypothesis.
|
||||
- **The thing being measured never grades itself.** The patch may not touch the
|
||||
harness, the corpus, or the tests.
|
||||
|
||||
## Never
|
||||
|
||||
- **Never a naked percentage.** Every number carries its confidence interval
|
||||
and the workload it was measured on. "12% faster" is not a result.
|
||||
- **Never re-run a benchmark until it passes.** That is p-hacking with extra
|
||||
steps. The first calibrated run is the answer.
|
||||
- **Never claim a win the CI lower bound does not support**, however good the
|
||||
median looks and however "obviously" faster the change is.
|
||||
- **Never present a measurement from a host that failed calibration.**
|
||||
|
||||
## What good looks like
|
||||
|
||||
**A claim, fully dressed** (`phase3-cjson-002`, in the ledger):
|
||||
|
||||
> Accepted. Speedup 1.088 (candidate is 8.8% faster), 95% CI [1.074, 1.100] —
|
||||
> so the *defensible* claim is ≥7.4%, not 8.8%. Workload: 5.6 MB synthetic
|
||||
> JSON, 20k records, parsed + serialized 2×, single thread. Gates: upstream
|
||||
> suite green, golden replay byte-identical, ASan + UBSan clean.
|
||||
|
||||
Note what that write-up does **not** say. It does not claim a differential-fuzz
|
||||
run, because that row records `fuzz_iters=0` — the fuzzing of that era ran
|
||||
out-of-band and the machine never witnessed it. Fuzz is a pipeline gate now, and
|
||||
new rows carry a real count. Say what the record supports, not what you
|
||||
remember doing.
|
||||
|
||||
**Declining to claim what the data won't support** (`phase4-cjson-service`):
|
||||
p50 latency improved 6.2% [95% CI +5.8%, +7.2%] at 150 rps — claimed. The p99
|
||||
CI came back [0.07, 4.97] — uselessly wide, so **p99 was not claimed at all**,
|
||||
even though its median looked like a win.
|
||||
|
||||
**Good numbers on a bad change** (`phase2-comrak-010`): 1.107 speedup, 95% CI
|
||||
[1.079, 1.121], on 11 MB of markdown. Tight interval, real improvement,
|
||||
**accepted — and wrong.** LeakSanitizer caught it afterwards and the row was
|
||||
overturned. It never shipped. The bench was never the problem; skipping a gate
|
||||
was. This is why the order above is not negotiable.
|
||||
@@ -0,0 +1,41 @@
|
||||
# peltier accept thresholds — consumer template.
|
||||
#
|
||||
# Copy this next to the code you are measuring and pass it with
|
||||
# `bench-runner --config <this-file>`. The defaults are peltier's own
|
||||
# published bar; loosening them to make a result pass is not measurement,
|
||||
# it is negotiation.
|
||||
|
||||
# The bootstrap CI *lower bound* of the speedup ratio must clear 1 + threshold.
|
||||
# The median is not the claim. The lower bound is the claim.
|
||||
threshold = 0.02
|
||||
|
||||
# Measured runs per side, interleaved ABAB (SPEC minimum: 30).
|
||||
runs_per_side = 30
|
||||
|
||||
# Warm-up runs per side, discarded — they pay for cold caches and JIT.
|
||||
warmup_runs = 3
|
||||
|
||||
# Bootstrap resamples for the ratio CI.
|
||||
bootstrap_iters = 10000
|
||||
|
||||
# Confidence level for the ratio CI.
|
||||
confidence = 0.95
|
||||
|
||||
# Fixed PRNG seed so a calibration run is reproducible by anyone.
|
||||
bootstrap_seed = 1887343589
|
||||
|
||||
# CPU pinning. WARNING: this key is INERT for `compare`, `aa`, and `calibrate`
|
||||
# — it is read only by the in-repo `verdict` pipeline. Setting it here and then
|
||||
# reporting "pinned to core N" would be a claim about a run that never happened.
|
||||
#
|
||||
# To actually pin a compare/aa/calibrate run, wrap BOTH sides identically in
|
||||
# the command itself: --baseline 'taskset -c 2 <cmd>' --candidate 'taskset -c 2 <cmd>'
|
||||
#
|
||||
# Pinning tightens the A/A confidence interval ~5x on peltier's calibration
|
||||
# hardware. An unpinned host is not disqualified; a NOISY one is, and the A/A
|
||||
# self-test decides that empirically. Leave this empty.
|
||||
pin_prefix = ""
|
||||
|
||||
# Differential-fuzz iterations requested per gate run. Only used by the
|
||||
# in-repo `verdict` pipeline, which requires a target to declare [gates].fuzz.
|
||||
# fuzz_iters = 10000
|
||||
+2
@@ -6,6 +6,8 @@ faster. Find out. Works on any pair of shell commands, in any language.
|
||||
## 0. Preflight
|
||||
|
||||
sh <skill-dir>/scripts/preflight.sh
|
||||
# or, if this installation carries no scripts/ (script-blocking harness):
|
||||
sh "$PELTIER_HOME/.claude/skills/peltier/scripts/preflight.sh"
|
||||
|
||||
Read the `key=value` output; you need `BENCH_RUNNER` and `PIN_SUPPORTED`. A
|
||||
refusal ends the **claim**, not the work — report the reason verbatim and do
|
||||
@@ -0,0 +1,91 @@
|
||||
#!/bin/sh
|
||||
# peltier preflight — locate the trust layer, check this host can RUN it, and
|
||||
# build the real bench-runner.
|
||||
#
|
||||
# STATUS=ok means the harness is present and runnable here. It does NOT mean
|
||||
# this host can measure your change: only the A/A self-test decides that, and
|
||||
# it decides it empirically. Do not read STATUS=ok as "calibrated".
|
||||
#
|
||||
# A refusal (STATUS=refuse, nonzero exit) is a complete, valid outcome. It
|
||||
# ends the *claim*, not the work: you may still write and explain a patch, you
|
||||
# just may not attach a number to it. Never work around a refusal with a
|
||||
# hand-rolled timing loop.
|
||||
set -eu
|
||||
|
||||
say() { printf '%s\n' "$1"; }
|
||||
refuse() {
|
||||
say "STATUS=refuse"
|
||||
say "REASON=$1"
|
||||
exit 1
|
||||
}
|
||||
abspath() { ( CDPATH= cd -- "$1" 2>/dev/null && pwd ); }
|
||||
|
||||
# --- host -------------------------------------------------------------
|
||||
# bench-runner shells every timed run through `sh -c`, and the in-repo
|
||||
# pipeline (harnessd, service mode, the sh-based gates) is POSIX-only.
|
||||
# Windows compiles the workspace but cannot run it.
|
||||
uname_s=$(uname -s 2>/dev/null || echo unknown)
|
||||
case "$uname_s" in
|
||||
Linux) host=linux ;;
|
||||
Darwin) host=darwin ;;
|
||||
*) refuse "unsupported host '$uname_s' — peltier's harness is Linux/POSIX-only at runtime. Run it on Linux (or macOS for verify mode). Do not substitute another benchmark." ;;
|
||||
esac
|
||||
|
||||
# --- trust layer ------------------------------------------------------
|
||||
is_peltier() { [ -f "$1/crates/bench-runner/Cargo.toml" ] && [ -f "$1/config/accept.toml" ]; }
|
||||
|
||||
# Walk up from a starting dir looking for a checkout. Absolute POSIX paths
|
||||
# always terminate at "/", so this cannot spin.
|
||||
walk_up() {
|
||||
d=$1
|
||||
while [ -n "$d" ] && [ "$d" != "/" ]; do
|
||||
if is_peltier "$d"; then printf '%s' "$d"; return 0; fi
|
||||
d=$(dirname "$d")
|
||||
done
|
||||
is_peltier / && printf '/'
|
||||
return 0
|
||||
}
|
||||
|
||||
home=""
|
||||
if [ -n "${PELTIER_HOME:-}" ]; then
|
||||
canon=$(abspath "$PELTIER_HOME" || true)
|
||||
[ -n "$canon" ] || refuse "PELTIER_HOME='$PELTIER_HOME' does not exist"
|
||||
is_peltier "$canon" || refuse "PELTIER_HOME='$PELTIER_HOME' is not a peltier checkout (no crates/bench-runner/Cargo.toml + config/accept.toml)"
|
||||
home=$canon
|
||||
else
|
||||
# cwd first (working inside a checkout), then this script's own location
|
||||
# (the skill still living inside one). A skill copied into another project
|
||||
# finds neither — which is why PELTIER_HOME exists.
|
||||
home=$(walk_up "$(pwd)")
|
||||
[ -n "$home" ] || home=$(walk_up "$(abspath "$(dirname -- "$0")")")
|
||||
fi
|
||||
[ -n "$home" ] || refuse "no peltier checkout found — clone https://github.com/bdbrown4/peltier.io and set PELTIER_HOME=/path/to/peltier.io. The statistics live in bench-runner; do not reimplement them."
|
||||
|
||||
command -v cargo >/dev/null 2>&1 || refuse "cargo not on PATH — needed to build bench-runner from $home"
|
||||
|
||||
# --- build the real harness -------------------------------------------
|
||||
# Always build. cargo is a no-op when current, and skipping this because a
|
||||
# binary happens to exist would silently measure with a stale copy of the
|
||||
# statistics after a `git pull`.
|
||||
br="$home/target/release/bench-runner"
|
||||
( cd "$home" && cargo build --release -q -p bench-runner ) \
|
||||
|| refuse "bench-runner failed to build in $home — fix the build before claiming any number"
|
||||
{ [ -f "$br" ] && [ -x "$br" ]; } || refuse "bench-runner did not appear at $br after a successful build"
|
||||
|
||||
# --- pinning ----------------------------------------------------------
|
||||
# NOTE: config/accept.toml's `pin_prefix` is read ONLY by the in-repo verdict
|
||||
# pipeline. For compare/aa/calibrate you must wrap the commands yourself
|
||||
# (`taskset -c N <cmd>` on both sides). PIN_SUPPORTED just says whether that
|
||||
# tool is available here.
|
||||
if [ "$host" = linux ] && command -v taskset >/dev/null 2>&1; then
|
||||
pin=yes
|
||||
else
|
||||
pin=no
|
||||
fi
|
||||
|
||||
say "STATUS=ok"
|
||||
say "PELTIER_HOME=$home"
|
||||
say "BENCH_RUNNER=$br"
|
||||
say "HOST=$host"
|
||||
say "PIN_SUPPORTED=$pin"
|
||||
say "NOTE=harness runnable here; the A/A self-test decides whether it can measure your change"
|
||||
@@ -1,6 +1,14 @@
|
||||
---
|
||||
name: peltier
|
||||
description: "Prove a performance change is real — or refuse to claim it. Use when optimizing code for speed, verifying a claimed speedup ('is this actually faster?', 'did my optimization work?'), checking for a performance regression ('did this PR make things slower?'), or comparing the performance of two implementations. Runs peltier's trust layer — equivalence gates, A/A environment calibration, interleaved A/B with bootstrap confidence intervals. Never reports a naked percentage and never claims a win that is not statistically significant. Do NOT use for correctness-only work."
|
||||
license: "GPL-3.0-or-later"
|
||||
compatibility: "Needs a peltier checkout (https://github.com/bdbrown4/peltier.io; set PELTIER_HOME) with cargo to build bench-runner. Measurement requires a Linux/POSIX host; on unsupported hosts the skill refuses to produce numbers rather than degrade."
|
||||
version: "1.1.0"
|
||||
author: "peltier.io (Ben Brown)"
|
||||
tags: ["performance", "benchmarking", "statistics", "verification"]
|
||||
metadata:
|
||||
homepage: "https://bdbrown4.github.io/peltier.io/"
|
||||
source: "https://github.com/bdbrown4/peltier.io"
|
||||
---
|
||||
|
||||
# peltier — prove it, or claim nothing
|
||||
@@ -22,6 +30,16 @@ plugin, or copied into another project):
|
||||
|
||||
sh <skill-dir>/scripts/preflight.sh
|
||||
|
||||
Some harnesses block script files inside skills, so your installation may
|
||||
carry no `scripts/` directory. The script still exists exactly once — in the
|
||||
peltier checkout this skill drives (which you need anyway; it is where
|
||||
`bench-runner` lives). Run it from there:
|
||||
|
||||
sh "$PELTIER_HOME/.claude/skills/peltier/scripts/preflight.sh"
|
||||
|
||||
Never transcribe or re-implement the preflight logic inline; if you cannot
|
||||
locate a checkout to run it from, that IS a preflight refusal — report it.
|
||||
|
||||
It locates the trust layer, builds the real `bench-runner`, and checks this
|
||||
host can run it. `STATUS=ok` does **not** mean the host can measure your
|
||||
change — only the A/A self-test decides that.
|
||||
@@ -37,8 +55,8 @@ copy of the math is a second thing that can lie.
|
||||
|
||||
| Mode | Use when | Read |
|
||||
|---|---|---|
|
||||
| **verify** | You have a baseline and a candidate — any repo, any language. | [reference/verify.md](reference/verify.md) |
|
||||
| **attempt** | You are inside a peltier checkout and want the full loop: profile → hypothesize → patch → gated verdict → ledger row. | [reference/attempt.md](reference/attempt.md) |
|
||||
| **verify** | You have a baseline and a candidate — any repo, any language. | [references/verify.md](references/verify.md) |
|
||||
| **attempt** | You are inside a peltier checkout and want the full loop: profile → hypothesize → patch → gated verdict → ledger row. | [references/attempt.md](references/attempt.md) |
|
||||
|
||||
Default to **verify**. It is the portable one, and it is what almost every
|
||||
request actually wants.
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
# attempt — the full loop, inside a peltier checkout
|
||||
|
||||
Profile → hypothesize → patch → gated verdict → ledger row. This mode only
|
||||
applies when you are working **inside a peltier checkout** on a target that has
|
||||
a `targets/<name>/target.toml`. Everywhere else, use
|
||||
[verify.md](verify.md).
|
||||
|
||||
Requires Linux (harnessd uses Unix sockets and `setsid`; every gate shells
|
||||
through `sh`).
|
||||
|
||||
## Before you touch anything
|
||||
|
||||
**Read the ledger first.** It records every attempt ever made on this target,
|
||||
including the failures — which is exactly what stops you re-running a dead
|
||||
end:
|
||||
|
||||
cargo run -p policy # ranks playbook classes by their evidence
|
||||
|
||||
`policy` reads the ledger and ranks optimization classes by the Wilson lower
|
||||
bound of their shippable-win rate. It is **advisory** — a learned prior over
|
||||
the fixed cheapest-first order. The gates still decide every verdict.
|
||||
|
||||
**Then respect the order.** Cheapest wins first: build flags → LTO → PGO →
|
||||
allocator swap → *then* code changes. Do not hand-write SIMD before trying
|
||||
`-O3`. The playbook (`playbook/`) has the preconditions and known risks for
|
||||
each class.
|
||||
|
||||
## The boundary you may not cross
|
||||
|
||||
The agent may modify **only** allowlisted paths under `targets/<name>/`.
|
||||
`crates/`, `config/`, `corpora/`, and the upstream test suites are read-only:
|
||||
the thing being measured does not get to grade itself. `harnessd` enforces
|
||||
this on every proposed diff (path allowlist over every diff header, including
|
||||
`rename`/`copy` — a patch that tries to escape is rejected, not sanitized).
|
||||
|
||||
## Log the hypothesis *before* patching
|
||||
|
||||
A hypothesis written after seeing the numbers is not a hypothesis. State what
|
||||
you think is slow, why, and what you expect the change to buy — then patch.
|
||||
|
||||
## Run the gates
|
||||
|
||||
just gates <target>
|
||||
|
||||
Corpus pin → test-suite pin (when `corpora/<t>/TESTSUITE.sha256` exists) →
|
||||
upstream tests → golden replay. Note what this **does not** do: differential
|
||||
fuzz needs a pristine baseline to differ against, so it reports `Skipped` here
|
||||
and runs for real on the accept path below. `just gates` is a fast equivalence
|
||||
check, not a fuzz run — do not report it as one.
|
||||
|
||||
## The verdict — the only thing that can accept a change
|
||||
|
||||
just verdict <target> <candidate-bin> <run-id> <class> "<hypothesis>" "<hotspot>" \
|
||||
--patch-file <diff>
|
||||
|
||||
This rebuilds a **pristine baseline** from the pinned upstream commit (not
|
||||
whatever is lying around in your tree), runs the full gate set against the
|
||||
candidate, benches interleaved A/B against that baseline, and writes one
|
||||
append-only ledger row with the verdict and the complete evidence.
|
||||
|
||||
It will refuse to hand you an `accepted` unless:
|
||||
|
||||
- the upstream test suite is green **and** golden replay is byte-identical (or
|
||||
within the target's declared FP tolerance);
|
||||
- the **differential-fuzz gate actually passed** — a real run reporting zero
|
||||
divergences. No fuzz run, no machine accept. Ever;
|
||||
- ASan/UBSan are clean (and TSan, when the target configures it);
|
||||
- the bootstrap 95% CI lower bound of the speedup clears the threshold in
|
||||
`config/accept.toml` (default 2%);
|
||||
- and a conservative lexical risk classifier finds **no** concurrency, unsafe,
|
||||
or floating-point signals in the patch. If it does, the verdict is capped at
|
||||
`needs-human-review` no matter how good the numbers are.
|
||||
|
||||
Any of these failing produces a rejected row. **That is a complete, valid
|
||||
outcome** — the row is the deliverable. Write it and move on. Do not iterate on
|
||||
a rejected patch more than twice without a genuinely new hypothesis.
|
||||
|
||||
## Report
|
||||
|
||||
just report <run-id>
|
||||
|
||||
Mechanically generates the ROI from the ledger row: throughput → cores →
|
||||
dollars, and/or latency percentiles — every figure with its CI, workload, and
|
||||
methodology printed inline. It flags any row that was not accepted, so a
|
||||
rejected attempt cannot be laundered into a dollar figure.
|
||||
|
||||
## Unattended
|
||||
|
||||
just agent-attempt <target> <run-id>
|
||||
|
||||
Runs one full attempt behind the OS boundary: `harnessd` as the trusted uid on
|
||||
a Unix socket, the agent loop as an unprivileged user, the verdict pipeline
|
||||
wrapped in a network namespace (`scripts/no-net.sh`) that **fails closed** — if
|
||||
namespaces are unavailable it refuses to run rather than quietly running with
|
||||
the network up. `just isolation-check` verifies the boundary from the agent's
|
||||
side.
|
||||
|
||||
## When the numbers are good and the change is scary
|
||||
|
||||
FP-ordering, concurrency primitives, anything UB-adjacent: `needs-human-review`,
|
||||
always, regardless of the speedup. This is not a formality. The one false
|
||||
accept in peltier's history (`phase2-comrak-010`) passed its bench and was
|
||||
caught by a human audit afterwards — which is precisely why the sanitizer lane
|
||||
is now machine-enforced and why this rule does not bend.
|
||||
@@ -0,0 +1,205 @@
|
||||
# verify — prove a speedup in any repo
|
||||
|
||||
You have two things that should do the same work, one of which you think is
|
||||
faster. Find out. Works on any pair of shell commands, in any language.
|
||||
|
||||
## 0. Preflight
|
||||
|
||||
sh <skill-dir>/scripts/preflight.sh
|
||||
# or, if this installation carries no scripts/ (script-blocking harness):
|
||||
sh "$PELTIER_HOME/.claude/skills/peltier/scripts/preflight.sh"
|
||||
|
||||
Read the `key=value` output; you need `BENCH_RUNNER` and `PIN_SUPPORTED`. A
|
||||
refusal ends the **claim**, not the work — report the reason verbatim and do
|
||||
not substitute another timing tool.
|
||||
|
||||
BR=<BENCH_RUNNER from preflight>
|
||||
|
||||
## 1. Config
|
||||
|
||||
cp <skill-dir>/assets/accept.template.toml ./peltier-accept.toml
|
||||
CFG=./peltier-accept.toml
|
||||
|
||||
Do **not** loosen `threshold`, `runs_per_side`, or `confidence` to make a
|
||||
result pass. That is not measurement, it is negotiation.
|
||||
|
||||
**Pinning is not done through this file.** `pin_prefix` is read only by the
|
||||
in-repo `verdict` pipeline — `compare`, `aa`, and `calibrate` ignore it
|
||||
entirely (bench-runner's own fingerprint says `"caller-provided (wrap cmd in
|
||||
taskset)"`). If `PIN_SUPPORTED=yes` and you know a core is free, pin by
|
||||
wrapping **both sides identically**:
|
||||
|
||||
--baseline 'taskset -c 2 <cmd>' --candidate 'taskset -c 2 <cmd>'
|
||||
|
||||
An unpinned host is not disqualified. A *noisy* one is, and step 3 is what
|
||||
decides that.
|
||||
|
||||
## 2. Two immutable artifacts — not two states of one tree
|
||||
|
||||
The harness runs the two sides **alternately, 66 times**. That is impossible if
|
||||
"baseline" and "candidate" are the same working tree in different states.
|
||||
Before measuring:
|
||||
|
||||
- Build each side into its **own** location — a `git worktree` per side, or two
|
||||
separate build/output directories. Never `git stash` between runs.
|
||||
- Pass the **built artifact paths** as the commands. Never pass a wrapper that
|
||||
can rebuild: `cargo run`, `npm start`, `go run`, `mvn exec` will happily
|
||||
rebuild from whatever is in the tree *mid-benchmark*, so you can end up
|
||||
timing the same binary twice — and paying build cost inside the timed window.
|
||||
- Confirm the two artifacts actually differ: `sha256sum <baseline> <candidate>`.
|
||||
|
||||
Peltier learned this the hard way (`phase1-verdict-smoke-001`: "workspace paths
|
||||
get silently rebuilt from the patched tree by test runs"), which is why the
|
||||
in-repo pipeline rebuilds a pristine baseline from the pinned upstream commit
|
||||
rather than trusting the tree.
|
||||
|
||||
## 3. Equivalence — before you time anything
|
||||
|
||||
Capture **everything** each side produces, on a fixed, representative workload:
|
||||
|
||||
run_side() { # $1 = tag, $2 = command
|
||||
rm -rf "out.$1"; mkdir -p "out.$1"
|
||||
( eval "$2" ) > "out.$1/stdout" 2> "out.$1/stderr"; echo "$?" > "out.$1/exit"
|
||||
}
|
||||
run_side base "<baseline-cmd>"
|
||||
run_side cand "<candidate-cmd>"
|
||||
diff -r out.base out.cand && echo "GOLDEN PASS" || echo "DIVERGENT"
|
||||
|
||||
If the command's real output is a **file tree** (a bundler, a compiler, an
|
||||
image pipeline), hash that too — stdout is not the artifact:
|
||||
|
||||
( cd dist && find . -type f | sort | xargs sha256sum ) > out.<tag>/tree
|
||||
|
||||
**An empty diff over empty output is a failed gate, not a passed one.** If both
|
||||
sides wrote nothing to stdout, nothing to stderr, and you captured no artifact
|
||||
tree, you have compared nothing — stop and find the real output before going
|
||||
further. Then run the project's own test suite against the candidate; it must
|
||||
pass.
|
||||
|
||||
- **Identical output + tests green** → proceed.
|
||||
- **Divergent** → stop. No speedup may be claimed from a change you cannot show
|
||||
is behavior-preserving. The one exception — a change that *legitimately*
|
||||
alters output, floating-point reassociation being the common case — is
|
||||
`needs-human-review`, never an auto-accept. Quantify the divergence (how many
|
||||
values, by how much) and hand it to a human.
|
||||
|
||||
## 4. Calibration — can this host measure this change?
|
||||
|
||||
The A/A self-test runs the **same command on both sides**. It must find nothing:
|
||||
|
||||
$BR --config "$CFG" aa --cmd '<baseline-cmd>'
|
||||
|
||||
- stdout `A/A self-test passed (null verdict)` → the harness is not inventing
|
||||
speedups. Proceed.
|
||||
- **Nonzero exit**, with `Error: A/A self-test FAILED: harness claims a speedup
|
||||
from identical binaries` on **stderr** → stop. This host is too noisy to
|
||||
measure this change; every number it produces afterwards is fiction. Report
|
||||
the noise floor, then quiet the machine (pin a core, disable turbo/SMT, close
|
||||
everything else) or move to calibrated hardware.
|
||||
|
||||
For a claim anyone will act on, run the full calibration — a false-positive
|
||||
rate over many A/A sessions *plus* proof the harness can detect a **known
|
||||
injected regression**:
|
||||
|
||||
$BR --config "$CFG" calibrate --cmd '<baseline-cmd>' \
|
||||
--sessions 20 --slowdown 0.05 --out calibration.json
|
||||
|
||||
Required: **<5% A/A false-positive rate** and **≥95% detection** of the injected
|
||||
5% slowdown. A harness that cannot see a real 5% regression cannot be trusted
|
||||
to confirm a real 5% win. Keep `calibration.json` — it is the evidence that the
|
||||
measurement itself was sound.
|
||||
|
||||
### Budget the runtime before you start
|
||||
|
||||
`compare` executes the workload **(30 + 3) × 2 = 66 times**. `calibrate
|
||||
--sessions 20` executes it roughly **2,650 times**. Size the workload to
|
||||
**~0.2–2 s** of wall time: shorter and you measure process startup, longer and
|
||||
calibration takes hours. Run `calibrate` in the background — it will blow
|
||||
through a foreground command timeout.
|
||||
|
||||
## 5. Measure — interleaved A/B, bootstrap CI
|
||||
|
||||
$BR --config "$CFG" compare --baseline '<baseline-cmd>' --candidate '<candidate-cmd>'
|
||||
|
||||
Runs are **interleaved** (ABAB…), not blocked, so thermal drift and cache state
|
||||
hit both sides equally. Output:
|
||||
|
||||
speedup (baseline/candidate): median 1.0884, 95% CI [1.0745, 1.1001]
|
||||
verdict: accepted
|
||||
|
||||
The ratio is `baseline/candidate`, so **>1 means the candidate is faster**.
|
||||
|
||||
`--perf-stat` (Linux) adds cycles/instructions/cache-miss medians — useful for
|
||||
explaining *why* something got faster, **never** part of the accept decision,
|
||||
and usually unavailable in VMs.
|
||||
|
||||
## 6. Risk — you are the classifier here
|
||||
|
||||
`bench-runner` emits only `accepted` or `rejected-bench`. It **cannot** route a
|
||||
dangerous change to review, because in verify mode nothing is inspecting your
|
||||
diff. So inspect it yourself, and override the tool:
|
||||
|
||||
Read the patch. If it touches any of these, the outcome is **`needs-human-review`
|
||||
regardless of what `compare` printed**:
|
||||
|
||||
- **Concurrency** — threads, locks, atomics, memory ordering, `volatile`, work
|
||||
queues, async pools, `rayon`/`pthread`/`std::sync`.
|
||||
- **Unsafe / raw memory** — `unsafe`, pointer casts, `transmute`, FFI,
|
||||
manual allocation, uninitialized memory.
|
||||
- **Floating-point reassociation** — reordering accumulation, `-ffast-math`,
|
||||
FP SIMD, changed summation order.
|
||||
|
||||
These changes can be *correct on your workload and wrong in production*, and a
|
||||
green bench says nothing about that. A byte-identical output does not clear
|
||||
them: a data race produces identical output right up until it doesn't.
|
||||
|
||||
## 7. Verdict — read it honestly
|
||||
|
||||
| What you see | What it means | What you may say |
|
||||
|---|---|---|
|
||||
| `verdict: accepted` | CI lower bound clears the threshold. | Claim it — with the **CI lower bound**, the interval, and the workload. |
|
||||
| `verdict: rejected-bench`, CI spans 1.0 (e.g. `[0.983, 1.001]`) | Indistinguishable from no change. | "No shippable win." A real, complete result. |
|
||||
| `verdict: rejected-bench`, CI **entirely below** 1.0 (e.g. `[0.91, 0.96]`) | A measured **regression**. | Report it as one, with the interval. Do not call it "no win." |
|
||||
| Any risk signal from step 6 | Out of the machine's hands. | `needs-human-review`. Never auto-accept. |
|
||||
|
||||
Claim the **lower bound**, not the median: a median of 1.088 with CI
|
||||
[1.074, 1.100] supports "≥7.4% faster", not "8.8% faster".
|
||||
|
||||
If the result is a rejection, **that is the answer**. Re-running until it passes
|
||||
is p-hacking. Two failed hypotheses on the same hot path means you need a new
|
||||
hypothesis, not another run.
|
||||
|
||||
## Services (latency, not throughput)
|
||||
|
||||
For a server, wall-clock-per-run is the wrong instrument. Service mode is an
|
||||
open-loop, fixed-rate load generator that measures each request from its
|
||||
**intended** send time, so queueing delay is counted rather than hidden
|
||||
(coordinated omission). It reports p50/p99 with bootstrap CIs.
|
||||
|
||||
**It is not a generic HTTP load generator.** The server binary must:
|
||||
|
||||
- take `<port> <doc> <iters>` as positional argv;
|
||||
- print a line beginning with `READY` on stdout, **flushed**, once listening;
|
||||
- answer `GET /` and **close the connection** (`Connection: close`) — the client
|
||||
reads until EOF with no timeout, so a keep-alive server hangs the harness
|
||||
forever.
|
||||
|
||||
Point it at a stock Express or Flask app and it will either fail the READY
|
||||
handshake or block indefinitely. Peltier wrote a thin adapter binary
|
||||
(`targets/cjson/service.c`) for exactly this reason; you will need one too.
|
||||
|
||||
$BR --config "$CFG" service --baseline-bin '<bin>' --candidate-bin '<bin>' \
|
||||
--doc <workload> --rate 150 --sessions 20
|
||||
|
||||
Calibrate it the same way first (`service-calibrate`). Tail percentiles on a
|
||||
single-worker loopback setup are jittery — **if the p99 CI is wide, do not claim
|
||||
p99**, even when p50 is solid. Peltier's own shipped service result claims p50
|
||||
(+6.2%, CI [+5.8%, +7.2%]) and explicitly declines to claim p99 (CI [0.07, 4.97]).
|
||||
|
||||
## Record the attempt — including the failures
|
||||
|
||||
Outside peltier there is no ledger, so write the evidence where the next person
|
||||
will find it (a results file, the PR body): hypothesis, what changed,
|
||||
equivalence result, calibration evidence, the CI, the workload, and the verdict.
|
||||
**Failures are the most valuable rows** — they stop someone from re-running your
|
||||
dead end in six months.
|
||||
+48
-13
@@ -105,23 +105,58 @@ jobs:
|
||||
- uses: dtolnay/rust-toolchain@stable
|
||||
- uses: Swatinem/rust-cache@v2
|
||||
|
||||
- name: SKILL.md frontmatter is loadable
|
||||
- name: SKILL.md conforms to the Agent Skills spec (both copies)
|
||||
run: |
|
||||
python3 - <<'PY'
|
||||
import sys, yaml, pathlib
|
||||
p = pathlib.Path(".claude/skills/peltier/SKILL.md")
|
||||
text = p.read_text(encoding="utf-8")
|
||||
if not text.startswith("---\n"):
|
||||
sys.exit("SKILL.md must open with a --- frontmatter block")
|
||||
fm = yaml.safe_load(text.split("---", 2)[1]) # raises on malformed YAML
|
||||
for key in ("name", "description"):
|
||||
if not isinstance(fm.get(key), str) or not fm[key].strip():
|
||||
sys.exit(f"frontmatter '{key}' must be a non-empty string")
|
||||
if fm["name"] != p.parent.name:
|
||||
sys.exit(f"frontmatter name '{fm['name']}' != directory '{p.parent.name}'")
|
||||
print(f"frontmatter OK: name={fm['name']}, description={len(fm['description'])} chars")
|
||||
import re, sys, yaml, pathlib
|
||||
for skill in (".claude/skills/peltier", ".agents/skills/peltier"):
|
||||
p = pathlib.Path(skill) / "SKILL.md"
|
||||
text = p.read_text(encoding="utf-8")
|
||||
if not text.startswith("---\n"):
|
||||
sys.exit(f"{p}: must open with a --- frontmatter block")
|
||||
fm = yaml.safe_load(text.split("---", 2)[1]) # raises on malformed YAML
|
||||
body = text.split("---", 2)[2]
|
||||
for key in ("name", "description"):
|
||||
if not isinstance(fm.get(key), str) or not fm[key].strip():
|
||||
sys.exit(f"{p}: frontmatter '{key}' must be a non-empty string")
|
||||
# agentskills.io spec: 1-64 chars, [a-z0-9-], no edge/double hyphens,
|
||||
# name == parent dir; description <= 1024 chars; body <= 500 lines.
|
||||
n = fm["name"]
|
||||
if not re.fullmatch(r"[a-z0-9]+(-[a-z0-9]+)*", n) or len(n) > 64:
|
||||
sys.exit(f"{p}: name '{n}' violates the Agent Skills name rules")
|
||||
if n != p.parent.name:
|
||||
sys.exit(f"{p}: name '{n}' != directory '{p.parent.name}'")
|
||||
if len(fm["description"]) > 1024:
|
||||
sys.exit(f"{p}: description exceeds 1024 chars")
|
||||
if len(body.splitlines()) > 500:
|
||||
sys.exit(f"{p}: body exceeds 500 lines")
|
||||
if "metadata" in fm and not (
|
||||
isinstance(fm["metadata"], dict)
|
||||
and all(isinstance(k, str) and isinstance(v, str) for k, v in fm["metadata"].items())
|
||||
):
|
||||
sys.exit(f"{p}: metadata must be a string-to-string map")
|
||||
print(f"{p}: spec OK (name={n}, description={len(fm['description'])} chars)")
|
||||
PY
|
||||
|
||||
- name: .agents mirror is byte-identical to .claude canonical
|
||||
run: |
|
||||
diff -r .claude/skills/peltier .agents/skills/peltier
|
||||
echo "mirror identical"
|
||||
|
||||
- name: Installer stamps both layouts + the ZeroClaw variant correctly
|
||||
run: |
|
||||
if sh scripts/install-skill.sh /tmp/no-such-dir 2>/dev/null; then
|
||||
echo "FAIL: installer accepted a nonexistent target"; exit 1
|
||||
fi
|
||||
mkdir -p /tmp/consumer && sh scripts/install-skill.sh /tmp/consumer
|
||||
diff -r .claude/skills/peltier /tmp/consumer/.claude/skills/peltier
|
||||
diff -r .claude/skills/peltier /tmp/consumer/.agents/skills/peltier
|
||||
sh scripts/install-skill.sh --zeroclaw-variant /tmp/zc-variant
|
||||
test ! -e /tmp/zc-variant/scripts # script-less: ZeroClaw blocks script files
|
||||
test -f /tmp/zc-variant/SKILL.md
|
||||
diff -r .claude/skills/peltier/references /tmp/zc-variant/references
|
||||
echo "installer OK: both layouts identical to canonical; zeroclaw variant has no scripts/"
|
||||
|
||||
- name: Preflight resolves inside a peltier checkout
|
||||
run: |
|
||||
sh .claude/skills/peltier/scripts/preflight.sh | tee /tmp/pf.txt
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
# peltier.io — agent entry point
|
||||
|
||||
This file exists for AGENTS.md-reading harnesses (Codex, Jules, Cursor, Zed,
|
||||
Amp, Copilot coding agent, Gemini CLI, …). It is deliberately thin: the
|
||||
binding project rules live in one place, not here.
|
||||
|
||||
## Working on this repo
|
||||
|
||||
Read **[CLAUDE.md](CLAUDE.md)** first and treat it as binding regardless of
|
||||
which harness you are. It holds the non-negotiables (no unverified performance
|
||||
claims, measure-before-optimizing, hard equivalence gates, the append-only
|
||||
ledger, the trust-layer write boundary) and the honest current state of every
|
||||
subsystem. The engineering spec is [SPEC.md](SPEC.md). Commands live in the
|
||||
[justfile](justfile). The pipeline is Linux/POSIX-only at runtime; Windows
|
||||
compiles the workspace but cannot run it.
|
||||
|
||||
Two hard rules that bear repeating even in a pointer file:
|
||||
|
||||
- **Never claim a performance number without its confidence interval and
|
||||
workload.** If you did not run the gates, the number does not exist.
|
||||
- **The trust layer (`crates/`, `config/`, `corpora/`) is read-only to
|
||||
optimization agents.** Patches touch only allowlisted paths under
|
||||
`targets/<name>/`.
|
||||
|
||||
## Verifying performance changes (any repo, any harness)
|
||||
|
||||
This repo ships its trust layer as an Agent Skill:
|
||||
**[.claude/skills/peltier/SKILL.md](.claude/skills/peltier/SKILL.md)** —
|
||||
mirrored byte-identically at `.agents/skills/peltier/` for harnesses that
|
||||
discover skills there (CI enforces the mirror). If your harness auto-loads
|
||||
skills from either path, use it whenever a task involves proving or refuting
|
||||
a speedup. If not, read that SKILL.md and follow it manually — starting with
|
||||
`scripts/preflight.sh`, whose refusal is final.
|
||||
@@ -29,7 +29,7 @@ Full engineering spec: `SPEC.md`. Read it before starting any phase.
|
||||
- `targets/` — vendored OSS targets for case studies (permissive licenses only)
|
||||
- `playbook/` — optimization classes, ordered, with preconditions and known risks
|
||||
- `results/` — generated reports and flamegraphs
|
||||
- `.claude/skills/peltier/` — the trust layer packaged as a consumable Claude Code skill: `verify` mode (equivalence → A/A calibration → interleaved A/B + bootstrap CI) works in any repo against any two shell commands; `attempt` mode drives the full in-repo loop. Drives the real `bench-runner` binary and refuses on unsupported hosts rather than falling back to a second, forkable copy of the statistics. Preflight refusal is asserted in CI (`skill-preflight`).
|
||||
- `.claude/skills/peltier/` — the trust layer packaged as a consumable **Agent Skill** (agentskills.io SKILL.md standard, spec-compliant frontmatter): `verify` mode (equivalence → A/A calibration → interleaved A/B + bootstrap CI) works in any repo against any two shell commands; `attempt` mode drives the full in-repo loop. Drives the real `bench-runner` binary and refuses on unsupported hosts rather than falling back to a second, forkable copy of the statistics. Cross-harness: mirrored byte-identically at `.agents/skills/peltier/` (the Codex/Zed/Cursor/Gemini/Copilot convergence path — CI enforces the mirror), `AGENTS.md` at the repo root points non-Claude harnesses here, and `scripts/install-skill.sh` stamps consumer repos (plus a script-less `--zeroclaw-variant`, since ZeroClaw blocks script files in skills by default — its preflight runs from the checkout). Preflight refusal, spec conformance, mirror equality, and the installer are all asserted in CI (`skill-preflight`).
|
||||
|
||||
**Phase 4 COMPLETE — all SPEC §5 exit criteria met. Phases 0–4 are all done.** Service-mode latency benchmarking shipped: a coordinated-omission-correct, open-loop, fixed-rate load generator (`crates/bench-runner/src/service.rs`) measuring latency from each request's intended send time (queueing counted, not hidden), interleaved A/B, exact percentiles, bootstrap p50/p99 CIs. A real service target — `targets/cjson/service.c`, a minimal HTTP server wrapping the patched `cJSON.c` (trust-layer, outside the workspace allowlist). Service calibration PASS (0/10 A/A false-positive, 10/10 injected-5%-latency-regression detection; `results/calibration/cjson-service-aa.json`). **The verified batch win phase3-cjson-002 measured under a 150 rps-target replay (20 interleaved rounds, 0 drops — the `phase4-cjson-service` ledger row records sessions=20, rate_rps=150, drop_rate=0.0; total request count is not machine-recorded): p50 latency +6.2%, 95% CI [+5.8%, +7.2%] — accepted; p99 NOT claimed (CI [0.07, 4.97], single-worker loopback tail jitter) — harness correctly rejected it.** Mechanical ROI report generator (`crates/report` bin, `just report`, SPEC §9): reads a ledger row + service-latency JSON → throughput→cores→dollars (27.5 cores / $9,621-per-year CI lower bound on a 500-core fleet) *and* latency percentiles, every figure with CI + workload + methodology inline; flags any non-accepted row. Case study: `results/phase4/`. **The roadmap is complete — the profile→hypothesize→patch→gated-verdict→ROI loop runs end to end across Rust and C, batch and service, with zero shipped false accepts across all phases.**
|
||||
|
||||
|
||||
@@ -71,24 +71,40 @@ tests, but cannot run the pipeline.
|
||||
Full command reference and reproduction steps:
|
||||
[Reproduce it yourself](https://bdbrown4.github.io/peltier.io/reproduce.html).
|
||||
|
||||
## Use it on your own code
|
||||
## Use it on your own code — from any agent harness
|
||||
|
||||
The trust layer is not specific to these targets — it will judge any two shell
|
||||
commands. `.claude/skills/peltier/` packages it as a Claude Code skill:
|
||||
commands. It ships as an **[Agent Skill](https://agentskills.io)** (the open
|
||||
SKILL.md standard), which Claude Code, OpenAI Codex, GitHub Copilot, VS Code,
|
||||
Cursor, Gemini CLI, opencode, Goose, Amp, Zed, and ZeroClaw all load natively.
|
||||
One installer stamps it into your repo at both discovery paths:
|
||||
|
||||
```sh
|
||||
# from any repo, with a peltier checkout reachable
|
||||
export PELTIER_HOME=/path/to/peltier.io
|
||||
mkdir -p .claude/skills
|
||||
cp -r /path/to/peltier.io/.claude/skills/peltier .claude/skills/
|
||||
# from your repo, with a peltier checkout at /path/to/peltier.io
|
||||
sh /path/to/peltier.io/scripts/install-skill.sh .
|
||||
export PELTIER_HOME=/path/to/peltier.io # the skill drives bench-runner from here
|
||||
```
|
||||
|
||||
Then ask Claude to verify a speedup. The skill enforces the order that makes a
|
||||
performance claim mean something: **equivalence before timing** (a faster
|
||||
program that computes something different is a bug, not an optimization) →
|
||||
**A/A calibration** (a host that "finds" a speedup between a binary and itself
|
||||
cannot measure yours) → **interleaved A/B with a bootstrap CI** → a verdict
|
||||
decided by the CI *lower bound*, never the median.
|
||||
That writes `.claude/skills/peltier/` (Claude Code; also read by Copilot,
|
||||
VS Code, opencode, Amp) and `.agents/skills/peltier/` (the cross-tool path:
|
||||
Codex, Zed, Cursor, Gemini CLI, Goose, and others). This repo carries both,
|
||||
CI-enforced byte-identical, so the two copies cannot drift.
|
||||
|
||||
**ZeroClaw** blocks script files inside skills by default, so it gets a
|
||||
script-less variant (preflight runs from your peltier checkout instead —
|
||||
same one copy of the script):
|
||||
|
||||
```sh
|
||||
sh /path/to/peltier.io/scripts/install-skill.sh --zeroclaw-variant /tmp/peltier-skill
|
||||
zeroclaw skills install /tmp/peltier-skill
|
||||
```
|
||||
|
||||
Then ask your agent to verify a speedup. The skill enforces the order that
|
||||
makes a performance claim mean something: **equivalence before timing** (a
|
||||
faster program that computes something different is a bug, not an
|
||||
optimization) → **A/A calibration** (a host that "finds" a speedup between a
|
||||
binary and itself cannot measure yours) → **interleaved A/B with a bootstrap
|
||||
CI** → a verdict decided by the CI *lower bound*, never the median.
|
||||
|
||||
It drives the real `bench-runner` binary and **refuses rather than degrade**:
|
||||
on an unsupported host, or with no trust layer reachable, it stops and says so
|
||||
|
||||
Executable
+86
@@ -0,0 +1,86 @@
|
||||
#!/bin/sh
|
||||
# install-skill.sh — stamp the peltier Agent Skill into a consumer repo.
|
||||
#
|
||||
# sh scripts/install-skill.sh <target-repo> [--layout claude|agents|both]
|
||||
# sh scripts/install-skill.sh --zeroclaw-variant <out-dir>
|
||||
#
|
||||
# The skill is one directory (SKILL.md + references/ + scripts/ + assets/),
|
||||
# already in the Agent Skills format (agentskills.io) that Claude Code, Codex,
|
||||
# Copilot, VS Code, Cursor, Gemini CLI, opencode, Goose, Amp, Zed and others
|
||||
# load natively. All this script does is copy that one canonical directory to
|
||||
# the discovery path(s) a harness actually reads:
|
||||
#
|
||||
# .claude/skills/peltier/ Claude Code (also read by Copilot, VS Code,
|
||||
# opencode, Amp, and Cursor in compat mode)
|
||||
# .agents/skills/peltier/ the cross-tool convergence path (Codex, Zed,
|
||||
# Cursor, Copilot, VS Code, Gemini CLI, opencode,
|
||||
# Goose, Amp)
|
||||
#
|
||||
# --layout both (the default) writes both. That is two copies in YOUR repo;
|
||||
# keep them identical (re-run this script to update both; peltier's own CI
|
||||
# shows the diff -r pattern for enforcing it).
|
||||
#
|
||||
# --zeroclaw-variant writes a copy WITHOUT scripts/ to <out-dir>: ZeroClaw's
|
||||
# skill security policy blocks script files by default (skills.allow_scripts
|
||||
# is off), so its variant runs preflight from the peltier checkout instead —
|
||||
# the skill's SKILL.md already explains that path. Then install it with:
|
||||
# zeroclaw skills install <out-dir>
|
||||
#
|
||||
# The skill still needs a peltier checkout at runtime (bench-runner lives
|
||||
# there). Set PELTIER_HOME in the environment your agent runs in.
|
||||
set -eu
|
||||
|
||||
usage() {
|
||||
sed -n '2,30p' "$0" | sed 's/^# \{0,1\}//'
|
||||
exit 2
|
||||
}
|
||||
|
||||
here=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
|
||||
src="$here/../.claude/skills/peltier"
|
||||
[ -f "$src/SKILL.md" ] || { echo "install-skill: cannot find the skill at $src" >&2; exit 1; }
|
||||
|
||||
copy_skill() { # $1 = dest dir, $2 = "full" | "noscripts"
|
||||
rm -rf "$1"
|
||||
mkdir -p "$1"
|
||||
cp -R "$src/." "$1/"
|
||||
if [ "$2" = "noscripts" ]; then rm -rf "$1/scripts"; fi
|
||||
}
|
||||
|
||||
case "${1:-}" in
|
||||
""|-h|--help) usage ;;
|
||||
--zeroclaw-variant)
|
||||
out=${2:-}; [ -n "$out" ] || usage
|
||||
copy_skill "$out" noscripts
|
||||
echo "wrote script-less variant to $out (ZeroClaw blocks script files in skills by default)"
|
||||
echo "next: zeroclaw skills install \"$out\" # optionally --bundle <alias>, then add the"
|
||||
echo " bundle to agents.<alias>.skill_bundles so your agent loads it"
|
||||
echo "note: at runtime the skill runs preflight from your peltier checkout —"
|
||||
echo " set PELTIER_HOME to that checkout for the agent's environment"
|
||||
exit 0
|
||||
;;
|
||||
esac
|
||||
|
||||
target=$1; shift
|
||||
layout=both
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--layout) layout=${2:-}; shift 2 ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
done
|
||||
case "$layout" in claude|agents|both) ;; *) usage ;; esac
|
||||
[ -d "$target" ] || { echo "install-skill: target repo '$target' is not a directory" >&2; exit 1; }
|
||||
|
||||
wrote=""
|
||||
if [ "$layout" = claude ] || [ "$layout" = both ]; then
|
||||
copy_skill "$target/.claude/skills/peltier" full
|
||||
wrote="$wrote $target/.claude/skills/peltier"
|
||||
fi
|
||||
if [ "$layout" = agents ] || [ "$layout" = both ]; then
|
||||
copy_skill "$target/.agents/skills/peltier" full
|
||||
wrote="$wrote $target/.agents/skills/peltier"
|
||||
fi
|
||||
|
||||
echo "installed:$wrote"
|
||||
echo "next: set PELTIER_HOME=/path/to/peltier.io in the environment your agent runs in"
|
||||
echo " (the skill drives the real bench-runner from that checkout; preflight refuses without it)"
|
||||
Reference in New Issue
Block a user