mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
Merge pull request #19 from bdbrown4/skill/cross-harness
skill: make the peltier skill consumable across agent harnesses
This commit is contained in:
@@ -0,0 +1,142 @@
|
||||
---
|
||||
name: peltier
|
||||
description: "Prove a performance change is real — or refuse to claim it. Use when optimizing code for speed, verifying a claimed speedup ('is this actually faster?', 'did my optimization work?'), checking for a performance regression ('did this PR make things slower?'), or comparing the performance of two implementations. Runs peltier's trust layer — equivalence gates, A/A environment calibration, interleaved A/B with bootstrap confidence intervals. Never reports a naked percentage and never claims a win that is not statistically significant. Do NOT use for correctness-only work."
|
||||
license: "GPL-3.0-or-later"
|
||||
compatibility: "Needs a peltier checkout (https://github.com/bdbrown4/peltier.io; set PELTIER_HOME) with cargo to build bench-runner. Measurement requires a Linux/POSIX host; on unsupported hosts the skill refuses to produce numbers rather than degrade."
|
||||
version: "1.1.0"
|
||||
author: "peltier.io (Ben Brown)"
|
||||
tags: ["performance", "benchmarking", "statistics", "verification"]
|
||||
metadata:
|
||||
homepage: "https://bdbrown4.github.io/peltier.io/"
|
||||
source: "https://github.com/bdbrown4/peltier.io"
|
||||
---
|
||||
|
||||
# peltier — prove it, or claim nothing
|
||||
|
||||
A performance change that has not passed equivalence gates **and** shown a
|
||||
statistically significant improvement **does not exist**. You may discard it
|
||||
or log it. You may not claim it.
|
||||
|
||||
Most benchmarking is wrong in ways that flatter the author: run-to-run noise
|
||||
read as a win, a faster binary that computes something subtly different, one
|
||||
lucky run out of five, a percentage with no interval attached. This skill
|
||||
exists to make those failures impossible rather than unlikely.
|
||||
|
||||
## Preflight — never skip, never work around
|
||||
|
||||
Run `scripts/preflight.sh` **relative to this skill's directory** (resolve its
|
||||
absolute path first — the skill may be installed at `~/.claude/skills/`, as a
|
||||
plugin, or copied into another project):
|
||||
|
||||
sh <skill-dir>/scripts/preflight.sh
|
||||
|
||||
Some harnesses block script files inside skills, so your installation may
|
||||
carry no `scripts/` directory. The script still exists exactly once — in the
|
||||
peltier checkout this skill drives (which you need anyway; it is where
|
||||
`bench-runner` lives). Run it from there:
|
||||
|
||||
sh "$PELTIER_HOME/.claude/skills/peltier/scripts/preflight.sh"
|
||||
|
||||
Never transcribe or re-implement the preflight logic inline; if you cannot
|
||||
locate a checkout to run it from, that IS a preflight refusal — report it.
|
||||
|
||||
It locates the trust layer, builds the real `bench-runner`, and checks this
|
||||
host can run it. `STATUS=ok` does **not** mean the host can measure your
|
||||
change — only the A/A self-test decides that.
|
||||
|
||||
**If it refuses, that ends the claim, not the work.** You may still write the
|
||||
patch and explain why you think it is faster. You may not attach a number to
|
||||
it. Do not fall back to `time`, `hyperfine`, or a stopwatch loop — a number
|
||||
you cannot stand behind is worse than no number, because someone will act on
|
||||
it. The statistics live in `bench-runner` and are used from there; a second
|
||||
copy of the math is a second thing that can lie.
|
||||
|
||||
## Two modes
|
||||
|
||||
| Mode | Use when | Read |
|
||||
|---|---|---|
|
||||
| **verify** | You have a baseline and a candidate — any repo, any language. | [references/verify.md](references/verify.md) |
|
||||
| **attempt** | You are inside a peltier checkout and want the full loop: profile → hypothesize → patch → gated verdict → ledger row. | [references/attempt.md](references/attempt.md) |
|
||||
|
||||
Default to **verify**. It is the portable one, and it is what almost every
|
||||
request actually wants.
|
||||
|
||||
### No candidate yet? Profile first — do not guess.
|
||||
|
||||
"Measure before optimizing" is not advice, it is a gate. If you have one
|
||||
program and a hunch, get profile data *before* writing a patch: `perf record`
|
||||
(native), `py-spy` / `cProfile` (Python), `node --prof` / `--cpu-prof` (Node),
|
||||
`cargo flamegraph` (Rust), or `just coz` for causal profiling inside a peltier
|
||||
checkout. Then form a hypothesis about a *named* hot path, and only then patch.
|
||||
Guessing where the time goes is how people spend a week on 0.3% of the runtime.
|
||||
|
||||
## The order of the gates is the point
|
||||
|
||||
Each gate can end the job, and ending early is a success, not a failure.
|
||||
|
||||
1. **Equivalence — before you time anything.** A faster program that does
|
||||
something different is not an optimization, it is a bug with good
|
||||
benchmarks. Prove baseline and candidate produce identical output on a fixed
|
||||
workload. Comparing two *empty* outputs is a failed gate, not a passed one.
|
||||
2. **Calibration — can this machine measure this change at all?** The A/A
|
||||
self-test runs the same command against itself and must find **nothing**. A
|
||||
harness that "discovers" a speedup between a binary and itself is measuring
|
||||
noise, and every number it gives you afterwards is fiction.
|
||||
3. **Measurement — interleaved A/B, bootstrap CI.** Not "run A five times, then
|
||||
B five times" — that hands any thermal or cache drift to whichever side ran
|
||||
last.
|
||||
4. **Risk — what did the patch actually touch?** Concurrency primitives,
|
||||
`unsafe`/raw memory, floating-point reassociation: `needs-human-review`,
|
||||
**regardless of how good the numbers are**. In verify mode *you* are the
|
||||
classifier — `bench-runner` only ever emits `accepted` or `rejected-bench`,
|
||||
so it cannot make this call for you.
|
||||
5. **Verdict — the CI lower bound decides.** Accept only if the bootstrap 95%
|
||||
CI lower bound of the speedup clears the threshold (default 2%). The median
|
||||
is not the claim. The lower bound is the claim.
|
||||
|
||||
## Non-negotiables
|
||||
|
||||
- **Cheapest wins first:** build flags → LTO → PGO → allocator swap → *then*
|
||||
code changes. Do not hand-write SIMD before you have tried `-O3`.
|
||||
- **Every attempt is recorded, including failures.** A rejected hypothesis is a
|
||||
result — it stops the next person from re-running your dead end.
|
||||
- **A failed gate is a complete, valid outcome.** Write it up and move on. Do
|
||||
not iterate on a rejected patch more than twice without a new hypothesis.
|
||||
- **The thing being measured never grades itself.** The patch may not touch the
|
||||
harness, the corpus, or the tests.
|
||||
|
||||
## Never
|
||||
|
||||
- **Never a naked percentage.** Every number carries its confidence interval
|
||||
and the workload it was measured on. "12% faster" is not a result.
|
||||
- **Never re-run a benchmark until it passes.** That is p-hacking with extra
|
||||
steps. The first calibrated run is the answer.
|
||||
- **Never claim a win the CI lower bound does not support**, however good the
|
||||
median looks and however "obviously" faster the change is.
|
||||
- **Never present a measurement from a host that failed calibration.**
|
||||
|
||||
## What good looks like
|
||||
|
||||
**A claim, fully dressed** (`phase3-cjson-002`, in the ledger):
|
||||
|
||||
> Accepted. Speedup 1.088 (candidate is 8.8% faster), 95% CI [1.074, 1.100] —
|
||||
> so the *defensible* claim is ≥7.4%, not 8.8%. Workload: 5.6 MB synthetic
|
||||
> JSON, 20k records, parsed + serialized 2×, single thread. Gates: upstream
|
||||
> suite green, golden replay byte-identical, ASan + UBSan clean.
|
||||
|
||||
Note what that write-up does **not** say. It does not claim a differential-fuzz
|
||||
run, because that row records `fuzz_iters=0` — the fuzzing of that era ran
|
||||
out-of-band and the machine never witnessed it. Fuzz is a pipeline gate now, and
|
||||
new rows carry a real count. Say what the record supports, not what you
|
||||
remember doing.
|
||||
|
||||
**Declining to claim what the data won't support** (`phase4-cjson-service`):
|
||||
p50 latency improved 6.2% [95% CI +5.8%, +7.2%] at 150 rps — claimed. The p99
|
||||
CI came back [0.07, 4.97] — uselessly wide, so **p99 was not claimed at all**,
|
||||
even though its median looked like a win.
|
||||
|
||||
**Good numbers on a bad change** (`phase2-comrak-010`): 1.107 speedup, 95% CI
|
||||
[1.079, 1.121], on 11 MB of markdown. Tight interval, real improvement,
|
||||
**accepted — and wrong.** LeakSanitizer caught it afterwards and the row was
|
||||
overturned. It never shipped. The bench was never the problem; skipping a gate
|
||||
was. This is why the order above is not negotiable.
|
||||
@@ -0,0 +1,41 @@
|
||||
# peltier accept thresholds — consumer template.
|
||||
#
|
||||
# Copy this next to the code you are measuring and pass it with
|
||||
# `bench-runner --config <this-file>`. The defaults are peltier's own
|
||||
# published bar; loosening them to make a result pass is not measurement,
|
||||
# it is negotiation.
|
||||
|
||||
# The bootstrap CI *lower bound* of the speedup ratio must clear 1 + threshold.
|
||||
# The median is not the claim. The lower bound is the claim.
|
||||
threshold = 0.02
|
||||
|
||||
# Measured runs per side, interleaved ABAB (SPEC minimum: 30).
|
||||
runs_per_side = 30
|
||||
|
||||
# Warm-up runs per side, discarded — they pay for cold caches and JIT.
|
||||
warmup_runs = 3
|
||||
|
||||
# Bootstrap resamples for the ratio CI.
|
||||
bootstrap_iters = 10000
|
||||
|
||||
# Confidence level for the ratio CI.
|
||||
confidence = 0.95
|
||||
|
||||
# Fixed PRNG seed so a calibration run is reproducible by anyone.
|
||||
bootstrap_seed = 1887343589
|
||||
|
||||
# CPU pinning. WARNING: this key is INERT for `compare`, `aa`, and `calibrate`
|
||||
# — it is read only by the in-repo `verdict` pipeline. Setting it here and then
|
||||
# reporting "pinned to core N" would be a claim about a run that never happened.
|
||||
#
|
||||
# To actually pin a compare/aa/calibrate run, wrap BOTH sides identically in
|
||||
# the command itself: --baseline 'taskset -c 2 <cmd>' --candidate 'taskset -c 2 <cmd>'
|
||||
#
|
||||
# Pinning tightens the A/A confidence interval ~5x on peltier's calibration
|
||||
# hardware. An unpinned host is not disqualified; a NOISY one is, and the A/A
|
||||
# self-test decides that empirically. Leave this empty.
|
||||
pin_prefix = ""
|
||||
|
||||
# Differential-fuzz iterations requested per gate run. Only used by the
|
||||
# in-repo `verdict` pipeline, which requires a target to declare [gates].fuzz.
|
||||
# fuzz_iters = 10000
|
||||
+2
@@ -6,6 +6,8 @@ faster. Find out. Works on any pair of shell commands, in any language.
|
||||
## 0. Preflight
|
||||
|
||||
sh <skill-dir>/scripts/preflight.sh
|
||||
# or, if this installation carries no scripts/ (script-blocking harness):
|
||||
sh "$PELTIER_HOME/.claude/skills/peltier/scripts/preflight.sh"
|
||||
|
||||
Read the `key=value` output; you need `BENCH_RUNNER` and `PIN_SUPPORTED`. A
|
||||
refusal ends the **claim**, not the work — report the reason verbatim and do
|
||||
Executable
+91
@@ -0,0 +1,91 @@
|
||||
#!/bin/sh
|
||||
# peltier preflight — locate the trust layer, check this host can RUN it, and
|
||||
# build the real bench-runner.
|
||||
#
|
||||
# STATUS=ok means the harness is present and runnable here. It does NOT mean
|
||||
# this host can measure your change: only the A/A self-test decides that, and
|
||||
# it decides it empirically. Do not read STATUS=ok as "calibrated".
|
||||
#
|
||||
# A refusal (STATUS=refuse, nonzero exit) is a complete, valid outcome. It
|
||||
# ends the *claim*, not the work: you may still write and explain a patch, you
|
||||
# just may not attach a number to it. Never work around a refusal with a
|
||||
# hand-rolled timing loop.
|
||||
set -eu
|
||||
|
||||
say() { printf '%s\n' "$1"; }
|
||||
refuse() {
|
||||
say "STATUS=refuse"
|
||||
say "REASON=$1"
|
||||
exit 1
|
||||
}
|
||||
abspath() { ( CDPATH= cd -- "$1" 2>/dev/null && pwd ); }
|
||||
|
||||
# --- host -------------------------------------------------------------
|
||||
# bench-runner shells every timed run through `sh -c`, and the in-repo
|
||||
# pipeline (harnessd, service mode, the sh-based gates) is POSIX-only.
|
||||
# Windows compiles the workspace but cannot run it.
|
||||
uname_s=$(uname -s 2>/dev/null || echo unknown)
|
||||
case "$uname_s" in
|
||||
Linux) host=linux ;;
|
||||
Darwin) host=darwin ;;
|
||||
*) refuse "unsupported host '$uname_s' — peltier's harness is Linux/POSIX-only at runtime. Run it on Linux (or macOS for verify mode). Do not substitute another benchmark." ;;
|
||||
esac
|
||||
|
||||
# --- trust layer ------------------------------------------------------
|
||||
is_peltier() { [ -f "$1/crates/bench-runner/Cargo.toml" ] && [ -f "$1/config/accept.toml" ]; }
|
||||
|
||||
# Walk up from a starting dir looking for a checkout. Absolute POSIX paths
|
||||
# always terminate at "/", so this cannot spin.
|
||||
walk_up() {
|
||||
d=$1
|
||||
while [ -n "$d" ] && [ "$d" != "/" ]; do
|
||||
if is_peltier "$d"; then printf '%s' "$d"; return 0; fi
|
||||
d=$(dirname "$d")
|
||||
done
|
||||
is_peltier / && printf '/'
|
||||
return 0
|
||||
}
|
||||
|
||||
home=""
|
||||
if [ -n "${PELTIER_HOME:-}" ]; then
|
||||
canon=$(abspath "$PELTIER_HOME" || true)
|
||||
[ -n "$canon" ] || refuse "PELTIER_HOME='$PELTIER_HOME' does not exist"
|
||||
is_peltier "$canon" || refuse "PELTIER_HOME='$PELTIER_HOME' is not a peltier checkout (no crates/bench-runner/Cargo.toml + config/accept.toml)"
|
||||
home=$canon
|
||||
else
|
||||
# cwd first (working inside a checkout), then this script's own location
|
||||
# (the skill still living inside one). A skill copied into another project
|
||||
# finds neither — which is why PELTIER_HOME exists.
|
||||
home=$(walk_up "$(pwd)")
|
||||
[ -n "$home" ] || home=$(walk_up "$(abspath "$(dirname -- "$0")")")
|
||||
fi
|
||||
[ -n "$home" ] || refuse "no peltier checkout found — clone https://github.com/bdbrown4/peltier.io and set PELTIER_HOME=/path/to/peltier.io. The statistics live in bench-runner; do not reimplement them."
|
||||
|
||||
command -v cargo >/dev/null 2>&1 || refuse "cargo not on PATH — needed to build bench-runner from $home"
|
||||
|
||||
# --- build the real harness -------------------------------------------
|
||||
# Always build. cargo is a no-op when current, and skipping this because a
|
||||
# binary happens to exist would silently measure with a stale copy of the
|
||||
# statistics after a `git pull`.
|
||||
br="$home/target/release/bench-runner"
|
||||
( cd "$home" && cargo build --release -q -p bench-runner ) \
|
||||
|| refuse "bench-runner failed to build in $home — fix the build before claiming any number"
|
||||
{ [ -f "$br" ] && [ -x "$br" ]; } || refuse "bench-runner did not appear at $br after a successful build"
|
||||
|
||||
# --- pinning ----------------------------------------------------------
|
||||
# NOTE: config/accept.toml's `pin_prefix` is read ONLY by the in-repo verdict
|
||||
# pipeline. For compare/aa/calibrate you must wrap the commands yourself
|
||||
# (`taskset -c N <cmd>` on both sides). PIN_SUPPORTED just says whether that
|
||||
# tool is available here.
|
||||
if [ "$host" = linux ] && command -v taskset >/dev/null 2>&1; then
|
||||
pin=yes
|
||||
else
|
||||
pin=no
|
||||
fi
|
||||
|
||||
say "STATUS=ok"
|
||||
say "PELTIER_HOME=$home"
|
||||
say "BENCH_RUNNER=$br"
|
||||
say "HOST=$host"
|
||||
say "PIN_SUPPORTED=$pin"
|
||||
say "NOTE=harness runnable here; the A/A self-test decides whether it can measure your change"
|
||||
@@ -1,6 +1,14 @@
|
||||
---
|
||||
name: peltier
|
||||
description: "Prove a performance change is real — or refuse to claim it. Use when optimizing code for speed, verifying a claimed speedup ('is this actually faster?', 'did my optimization work?'), checking for a performance regression ('did this PR make things slower?'), or comparing the performance of two implementations. Runs peltier's trust layer — equivalence gates, A/A environment calibration, interleaved A/B with bootstrap confidence intervals. Never reports a naked percentage and never claims a win that is not statistically significant. Do NOT use for correctness-only work."
|
||||
license: "GPL-3.0-or-later"
|
||||
compatibility: "Needs a peltier checkout (https://github.com/bdbrown4/peltier.io; set PELTIER_HOME) with cargo to build bench-runner. Measurement requires a Linux/POSIX host; on unsupported hosts the skill refuses to produce numbers rather than degrade."
|
||||
version: "1.1.0"
|
||||
author: "peltier.io (Ben Brown)"
|
||||
tags: ["performance", "benchmarking", "statistics", "verification"]
|
||||
metadata:
|
||||
homepage: "https://bdbrown4.github.io/peltier.io/"
|
||||
source: "https://github.com/bdbrown4/peltier.io"
|
||||
---
|
||||
|
||||
# peltier — prove it, or claim nothing
|
||||
@@ -22,6 +30,16 @@ plugin, or copied into another project):
|
||||
|
||||
sh <skill-dir>/scripts/preflight.sh
|
||||
|
||||
Some harnesses block script files inside skills, so your installation may
|
||||
carry no `scripts/` directory. The script still exists exactly once — in the
|
||||
peltier checkout this skill drives (which you need anyway; it is where
|
||||
`bench-runner` lives). Run it from there:
|
||||
|
||||
sh "$PELTIER_HOME/.claude/skills/peltier/scripts/preflight.sh"
|
||||
|
||||
Never transcribe or re-implement the preflight logic inline; if you cannot
|
||||
locate a checkout to run it from, that IS a preflight refusal — report it.
|
||||
|
||||
It locates the trust layer, builds the real `bench-runner`, and checks this
|
||||
host can run it. `STATUS=ok` does **not** mean the host can measure your
|
||||
change — only the A/A self-test decides that.
|
||||
@@ -37,8 +55,8 @@ copy of the math is a second thing that can lie.
|
||||
|
||||
| Mode | Use when | Read |
|
||||
|---|---|---|
|
||||
| **verify** | You have a baseline and a candidate — any repo, any language. | [reference/verify.md](reference/verify.md) |
|
||||
| **attempt** | You are inside a peltier checkout and want the full loop: profile → hypothesize → patch → gated verdict → ledger row. | [reference/attempt.md](reference/attempt.md) |
|
||||
| **verify** | You have a baseline and a candidate — any repo, any language. | [references/verify.md](references/verify.md) |
|
||||
| **attempt** | You are inside a peltier checkout and want the full loop: profile → hypothesize → patch → gated verdict → ledger row. | [references/attempt.md](references/attempt.md) |
|
||||
|
||||
Default to **verify**. It is the portable one, and it is what almost every
|
||||
request actually wants.
|
||||
|
||||
@@ -0,0 +1,104 @@
|
||||
# attempt — the full loop, inside a peltier checkout
|
||||
|
||||
Profile → hypothesize → patch → gated verdict → ledger row. This mode only
|
||||
applies when you are working **inside a peltier checkout** on a target that has
|
||||
a `targets/<name>/target.toml`. Everywhere else, use
|
||||
[verify.md](verify.md).
|
||||
|
||||
Requires Linux (harnessd uses Unix sockets and `setsid`; every gate shells
|
||||
through `sh`).
|
||||
|
||||
## Before you touch anything
|
||||
|
||||
**Read the ledger first.** It records every attempt ever made on this target,
|
||||
including the failures — which is exactly what stops you re-running a dead
|
||||
end:
|
||||
|
||||
cargo run -p policy # ranks playbook classes by their evidence
|
||||
|
||||
`policy` reads the ledger and ranks optimization classes by the Wilson lower
|
||||
bound of their shippable-win rate. It is **advisory** — a learned prior over
|
||||
the fixed cheapest-first order. The gates still decide every verdict.
|
||||
|
||||
**Then respect the order.** Cheapest wins first: build flags → LTO → PGO →
|
||||
allocator swap → *then* code changes. Do not hand-write SIMD before trying
|
||||
`-O3`. The playbook (`playbook/`) has the preconditions and known risks for
|
||||
each class.
|
||||
|
||||
## The boundary you may not cross
|
||||
|
||||
The agent may modify **only** allowlisted paths under `targets/<name>/`.
|
||||
`crates/`, `config/`, `corpora/`, and the upstream test suites are read-only:
|
||||
the thing being measured does not get to grade itself. `harnessd` enforces
|
||||
this on every proposed diff (path allowlist over every diff header, including
|
||||
`rename`/`copy` — a patch that tries to escape is rejected, not sanitized).
|
||||
|
||||
## Log the hypothesis *before* patching
|
||||
|
||||
A hypothesis written after seeing the numbers is not a hypothesis. State what
|
||||
you think is slow, why, and what you expect the change to buy — then patch.
|
||||
|
||||
## Run the gates
|
||||
|
||||
just gates <target>
|
||||
|
||||
Corpus pin → test-suite pin (when `corpora/<t>/TESTSUITE.sha256` exists) →
|
||||
upstream tests → golden replay. Note what this **does not** do: differential
|
||||
fuzz needs a pristine baseline to differ against, so it reports `Skipped` here
|
||||
and runs for real on the accept path below. `just gates` is a fast equivalence
|
||||
check, not a fuzz run — do not report it as one.
|
||||
|
||||
## The verdict — the only thing that can accept a change
|
||||
|
||||
just verdict <target> <candidate-bin> <run-id> <class> "<hypothesis>" "<hotspot>" \
|
||||
--patch-file <diff>
|
||||
|
||||
This rebuilds a **pristine baseline** from the pinned upstream commit (not
|
||||
whatever is lying around in your tree), runs the full gate set against the
|
||||
candidate, benches interleaved A/B against that baseline, and writes one
|
||||
append-only ledger row with the verdict and the complete evidence.
|
||||
|
||||
It will refuse to hand you an `accepted` unless:
|
||||
|
||||
- the upstream test suite is green **and** golden replay is byte-identical (or
|
||||
within the target's declared FP tolerance);
|
||||
- the **differential-fuzz gate actually passed** — a real run reporting zero
|
||||
divergences. No fuzz run, no machine accept. Ever;
|
||||
- ASan/UBSan are clean (and TSan, when the target configures it);
|
||||
- the bootstrap 95% CI lower bound of the speedup clears the threshold in
|
||||
`config/accept.toml` (default 2%);
|
||||
- and a conservative lexical risk classifier finds **no** concurrency, unsafe,
|
||||
or floating-point signals in the patch. If it does, the verdict is capped at
|
||||
`needs-human-review` no matter how good the numbers are.
|
||||
|
||||
Any of these failing produces a rejected row. **That is a complete, valid
|
||||
outcome** — the row is the deliverable. Write it and move on. Do not iterate on
|
||||
a rejected patch more than twice without a genuinely new hypothesis.
|
||||
|
||||
## Report
|
||||
|
||||
just report <run-id>
|
||||
|
||||
Mechanically generates the ROI from the ledger row: throughput → cores →
|
||||
dollars, and/or latency percentiles — every figure with its CI, workload, and
|
||||
methodology printed inline. It flags any row that was not accepted, so a
|
||||
rejected attempt cannot be laundered into a dollar figure.
|
||||
|
||||
## Unattended
|
||||
|
||||
just agent-attempt <target> <run-id>
|
||||
|
||||
Runs one full attempt behind the OS boundary: `harnessd` as the trusted uid on
|
||||
a Unix socket, the agent loop as an unprivileged user, the verdict pipeline
|
||||
wrapped in a network namespace (`scripts/no-net.sh`) that **fails closed** — if
|
||||
namespaces are unavailable it refuses to run rather than quietly running with
|
||||
the network up. `just isolation-check` verifies the boundary from the agent's
|
||||
side.
|
||||
|
||||
## When the numbers are good and the change is scary
|
||||
|
||||
FP-ordering, concurrency primitives, anything UB-adjacent: `needs-human-review`,
|
||||
always, regardless of the speedup. This is not a formality. The one false
|
||||
accept in peltier's history (`phase2-comrak-010`) passed its bench and was
|
||||
caught by a human audit afterwards — which is precisely why the sanitizer lane
|
||||
is now machine-enforced and why this rule does not bend.
|
||||
@@ -0,0 +1,205 @@
|
||||
# verify — prove a speedup in any repo
|
||||
|
||||
You have two things that should do the same work, one of which you think is
|
||||
faster. Find out. Works on any pair of shell commands, in any language.
|
||||
|
||||
## 0. Preflight
|
||||
|
||||
sh <skill-dir>/scripts/preflight.sh
|
||||
# or, if this installation carries no scripts/ (script-blocking harness):
|
||||
sh "$PELTIER_HOME/.claude/skills/peltier/scripts/preflight.sh"
|
||||
|
||||
Read the `key=value` output; you need `BENCH_RUNNER` and `PIN_SUPPORTED`. A
|
||||
refusal ends the **claim**, not the work — report the reason verbatim and do
|
||||
not substitute another timing tool.
|
||||
|
||||
BR=<BENCH_RUNNER from preflight>
|
||||
|
||||
## 1. Config
|
||||
|
||||
cp <skill-dir>/assets/accept.template.toml ./peltier-accept.toml
|
||||
CFG=./peltier-accept.toml
|
||||
|
||||
Do **not** loosen `threshold`, `runs_per_side`, or `confidence` to make a
|
||||
result pass. That is not measurement, it is negotiation.
|
||||
|
||||
**Pinning is not done through this file.** `pin_prefix` is read only by the
|
||||
in-repo `verdict` pipeline — `compare`, `aa`, and `calibrate` ignore it
|
||||
entirely (bench-runner's own fingerprint says `"caller-provided (wrap cmd in
|
||||
taskset)"`). If `PIN_SUPPORTED=yes` and you know a core is free, pin by
|
||||
wrapping **both sides identically**:
|
||||
|
||||
--baseline 'taskset -c 2 <cmd>' --candidate 'taskset -c 2 <cmd>'
|
||||
|
||||
An unpinned host is not disqualified. A *noisy* one is, and step 3 is what
|
||||
decides that.
|
||||
|
||||
## 2. Two immutable artifacts — not two states of one tree
|
||||
|
||||
The harness runs the two sides **alternately, 66 times**. That is impossible if
|
||||
"baseline" and "candidate" are the same working tree in different states.
|
||||
Before measuring:
|
||||
|
||||
- Build each side into its **own** location — a `git worktree` per side, or two
|
||||
separate build/output directories. Never `git stash` between runs.
|
||||
- Pass the **built artifact paths** as the commands. Never pass a wrapper that
|
||||
can rebuild: `cargo run`, `npm start`, `go run`, `mvn exec` will happily
|
||||
rebuild from whatever is in the tree *mid-benchmark*, so you can end up
|
||||
timing the same binary twice — and paying build cost inside the timed window.
|
||||
- Confirm the two artifacts actually differ: `sha256sum <baseline> <candidate>`.
|
||||
|
||||
Peltier learned this the hard way (`phase1-verdict-smoke-001`: "workspace paths
|
||||
get silently rebuilt from the patched tree by test runs"), which is why the
|
||||
in-repo pipeline rebuilds a pristine baseline from the pinned upstream commit
|
||||
rather than trusting the tree.
|
||||
|
||||
## 3. Equivalence — before you time anything
|
||||
|
||||
Capture **everything** each side produces, on a fixed, representative workload:
|
||||
|
||||
run_side() { # $1 = tag, $2 = command
|
||||
rm -rf "out.$1"; mkdir -p "out.$1"
|
||||
( eval "$2" ) > "out.$1/stdout" 2> "out.$1/stderr"; echo "$?" > "out.$1/exit"
|
||||
}
|
||||
run_side base "<baseline-cmd>"
|
||||
run_side cand "<candidate-cmd>"
|
||||
diff -r out.base out.cand && echo "GOLDEN PASS" || echo "DIVERGENT"
|
||||
|
||||
If the command's real output is a **file tree** (a bundler, a compiler, an
|
||||
image pipeline), hash that too — stdout is not the artifact:
|
||||
|
||||
( cd dist && find . -type f | sort | xargs sha256sum ) > out.<tag>/tree
|
||||
|
||||
**An empty diff over empty output is a failed gate, not a passed one.** If both
|
||||
sides wrote nothing to stdout, nothing to stderr, and you captured no artifact
|
||||
tree, you have compared nothing — stop and find the real output before going
|
||||
further. Then run the project's own test suite against the candidate; it must
|
||||
pass.
|
||||
|
||||
- **Identical output + tests green** → proceed.
|
||||
- **Divergent** → stop. No speedup may be claimed from a change you cannot show
|
||||
is behavior-preserving. The one exception — a change that *legitimately*
|
||||
alters output, floating-point reassociation being the common case — is
|
||||
`needs-human-review`, never an auto-accept. Quantify the divergence (how many
|
||||
values, by how much) and hand it to a human.
|
||||
|
||||
## 4. Calibration — can this host measure this change?
|
||||
|
||||
The A/A self-test runs the **same command on both sides**. It must find nothing:
|
||||
|
||||
$BR --config "$CFG" aa --cmd '<baseline-cmd>'
|
||||
|
||||
- stdout `A/A self-test passed (null verdict)` → the harness is not inventing
|
||||
speedups. Proceed.
|
||||
- **Nonzero exit**, with `Error: A/A self-test FAILED: harness claims a speedup
|
||||
from identical binaries` on **stderr** → stop. This host is too noisy to
|
||||
measure this change; every number it produces afterwards is fiction. Report
|
||||
the noise floor, then quiet the machine (pin a core, disable turbo/SMT, close
|
||||
everything else) or move to calibrated hardware.
|
||||
|
||||
For a claim anyone will act on, run the full calibration — a false-positive
|
||||
rate over many A/A sessions *plus* proof the harness can detect a **known
|
||||
injected regression**:
|
||||
|
||||
$BR --config "$CFG" calibrate --cmd '<baseline-cmd>' \
|
||||
--sessions 20 --slowdown 0.05 --out calibration.json
|
||||
|
||||
Required: **<5% A/A false-positive rate** and **≥95% detection** of the injected
|
||||
5% slowdown. A harness that cannot see a real 5% regression cannot be trusted
|
||||
to confirm a real 5% win. Keep `calibration.json` — it is the evidence that the
|
||||
measurement itself was sound.
|
||||
|
||||
### Budget the runtime before you start
|
||||
|
||||
`compare` executes the workload **(30 + 3) × 2 = 66 times**. `calibrate
|
||||
--sessions 20` executes it roughly **2,650 times**. Size the workload to
|
||||
**~0.2–2 s** of wall time: shorter and you measure process startup, longer and
|
||||
calibration takes hours. Run `calibrate` in the background — it will blow
|
||||
through a foreground command timeout.
|
||||
|
||||
## 5. Measure — interleaved A/B, bootstrap CI
|
||||
|
||||
$BR --config "$CFG" compare --baseline '<baseline-cmd>' --candidate '<candidate-cmd>'
|
||||
|
||||
Runs are **interleaved** (ABAB…), not blocked, so thermal drift and cache state
|
||||
hit both sides equally. Output:
|
||||
|
||||
speedup (baseline/candidate): median 1.0884, 95% CI [1.0745, 1.1001]
|
||||
verdict: accepted
|
||||
|
||||
The ratio is `baseline/candidate`, so **>1 means the candidate is faster**.
|
||||
|
||||
`--perf-stat` (Linux) adds cycles/instructions/cache-miss medians — useful for
|
||||
explaining *why* something got faster, **never** part of the accept decision,
|
||||
and usually unavailable in VMs.
|
||||
|
||||
## 6. Risk — you are the classifier here
|
||||
|
||||
`bench-runner` emits only `accepted` or `rejected-bench`. It **cannot** route a
|
||||
dangerous change to review, because in verify mode nothing is inspecting your
|
||||
diff. So inspect it yourself, and override the tool:
|
||||
|
||||
Read the patch. If it touches any of these, the outcome is **`needs-human-review`
|
||||
regardless of what `compare` printed**:
|
||||
|
||||
- **Concurrency** — threads, locks, atomics, memory ordering, `volatile`, work
|
||||
queues, async pools, `rayon`/`pthread`/`std::sync`.
|
||||
- **Unsafe / raw memory** — `unsafe`, pointer casts, `transmute`, FFI,
|
||||
manual allocation, uninitialized memory.
|
||||
- **Floating-point reassociation** — reordering accumulation, `-ffast-math`,
|
||||
FP SIMD, changed summation order.
|
||||
|
||||
These changes can be *correct on your workload and wrong in production*, and a
|
||||
green bench says nothing about that. A byte-identical output does not clear
|
||||
them: a data race produces identical output right up until it doesn't.
|
||||
|
||||
## 7. Verdict — read it honestly
|
||||
|
||||
| What you see | What it means | What you may say |
|
||||
|---|---|---|
|
||||
| `verdict: accepted` | CI lower bound clears the threshold. | Claim it — with the **CI lower bound**, the interval, and the workload. |
|
||||
| `verdict: rejected-bench`, CI spans 1.0 (e.g. `[0.983, 1.001]`) | Indistinguishable from no change. | "No shippable win." A real, complete result. |
|
||||
| `verdict: rejected-bench`, CI **entirely below** 1.0 (e.g. `[0.91, 0.96]`) | A measured **regression**. | Report it as one, with the interval. Do not call it "no win." |
|
||||
| Any risk signal from step 6 | Out of the machine's hands. | `needs-human-review`. Never auto-accept. |
|
||||
|
||||
Claim the **lower bound**, not the median: a median of 1.088 with CI
|
||||
[1.074, 1.100] supports "≥7.4% faster", not "8.8% faster".
|
||||
|
||||
If the result is a rejection, **that is the answer**. Re-running until it passes
|
||||
is p-hacking. Two failed hypotheses on the same hot path means you need a new
|
||||
hypothesis, not another run.
|
||||
|
||||
## Services (latency, not throughput)
|
||||
|
||||
For a server, wall-clock-per-run is the wrong instrument. Service mode is an
|
||||
open-loop, fixed-rate load generator that measures each request from its
|
||||
**intended** send time, so queueing delay is counted rather than hidden
|
||||
(coordinated omission). It reports p50/p99 with bootstrap CIs.
|
||||
|
||||
**It is not a generic HTTP load generator.** The server binary must:
|
||||
|
||||
- take `<port> <doc> <iters>` as positional argv;
|
||||
- print a line beginning with `READY` on stdout, **flushed**, once listening;
|
||||
- answer `GET /` and **close the connection** (`Connection: close`) — the client
|
||||
reads until EOF with no timeout, so a keep-alive server hangs the harness
|
||||
forever.
|
||||
|
||||
Point it at a stock Express or Flask app and it will either fail the READY
|
||||
handshake or block indefinitely. Peltier wrote a thin adapter binary
|
||||
(`targets/cjson/service.c`) for exactly this reason; you will need one too.
|
||||
|
||||
$BR --config "$CFG" service --baseline-bin '<bin>' --candidate-bin '<bin>' \
|
||||
--doc <workload> --rate 150 --sessions 20
|
||||
|
||||
Calibrate it the same way first (`service-calibrate`). Tail percentiles on a
|
||||
single-worker loopback setup are jittery — **if the p99 CI is wide, do not claim
|
||||
p99**, even when p50 is solid. Peltier's own shipped service result claims p50
|
||||
(+6.2%, CI [+5.8%, +7.2%]) and explicitly declines to claim p99 (CI [0.07, 4.97]).
|
||||
|
||||
## Record the attempt — including the failures
|
||||
|
||||
Outside peltier there is no ledger, so write the evidence where the next person
|
||||
will find it (a results file, the PR body): hypothesis, what changed,
|
||||
equivalence result, calibration evidence, the CI, the workload, and the verdict.
|
||||
**Failures are the most valuable rows** — they stop someone from re-running your
|
||||
dead end in six months.
|
||||
+48
-13
@@ -105,23 +105,58 @@ jobs:
|
||||
- uses: dtolnay/rust-toolchain@stable
|
||||
- uses: Swatinem/rust-cache@v2
|
||||
|
||||
- name: SKILL.md frontmatter is loadable
|
||||
- name: SKILL.md conforms to the Agent Skills spec (both copies)
|
||||
run: |
|
||||
python3 - <<'PY'
|
||||
import sys, yaml, pathlib
|
||||
p = pathlib.Path(".claude/skills/peltier/SKILL.md")
|
||||
text = p.read_text(encoding="utf-8")
|
||||
if not text.startswith("---\n"):
|
||||
sys.exit("SKILL.md must open with a --- frontmatter block")
|
||||
fm = yaml.safe_load(text.split("---", 2)[1]) # raises on malformed YAML
|
||||
for key in ("name", "description"):
|
||||
if not isinstance(fm.get(key), str) or not fm[key].strip():
|
||||
sys.exit(f"frontmatter '{key}' must be a non-empty string")
|
||||
if fm["name"] != p.parent.name:
|
||||
sys.exit(f"frontmatter name '{fm['name']}' != directory '{p.parent.name}'")
|
||||
print(f"frontmatter OK: name={fm['name']}, description={len(fm['description'])} chars")
|
||||
import re, sys, yaml, pathlib
|
||||
for skill in (".claude/skills/peltier", ".agents/skills/peltier"):
|
||||
p = pathlib.Path(skill) / "SKILL.md"
|
||||
text = p.read_text(encoding="utf-8")
|
||||
if not text.startswith("---\n"):
|
||||
sys.exit(f"{p}: must open with a --- frontmatter block")
|
||||
fm = yaml.safe_load(text.split("---", 2)[1]) # raises on malformed YAML
|
||||
body = text.split("---", 2)[2]
|
||||
for key in ("name", "description"):
|
||||
if not isinstance(fm.get(key), str) or not fm[key].strip():
|
||||
sys.exit(f"{p}: frontmatter '{key}' must be a non-empty string")
|
||||
# agentskills.io spec: 1-64 chars, [a-z0-9-], no edge/double hyphens,
|
||||
# name == parent dir; description <= 1024 chars; body <= 500 lines.
|
||||
n = fm["name"]
|
||||
if not re.fullmatch(r"[a-z0-9]+(-[a-z0-9]+)*", n) or len(n) > 64:
|
||||
sys.exit(f"{p}: name '{n}' violates the Agent Skills name rules")
|
||||
if n != p.parent.name:
|
||||
sys.exit(f"{p}: name '{n}' != directory '{p.parent.name}'")
|
||||
if len(fm["description"]) > 1024:
|
||||
sys.exit(f"{p}: description exceeds 1024 chars")
|
||||
if len(body.splitlines()) > 500:
|
||||
sys.exit(f"{p}: body exceeds 500 lines")
|
||||
if "metadata" in fm and not (
|
||||
isinstance(fm["metadata"], dict)
|
||||
and all(isinstance(k, str) and isinstance(v, str) for k, v in fm["metadata"].items())
|
||||
):
|
||||
sys.exit(f"{p}: metadata must be a string-to-string map")
|
||||
print(f"{p}: spec OK (name={n}, description={len(fm['description'])} chars)")
|
||||
PY
|
||||
|
||||
- name: .agents mirror is byte-identical to .claude canonical
|
||||
run: |
|
||||
diff -r .claude/skills/peltier .agents/skills/peltier
|
||||
echo "mirror identical"
|
||||
|
||||
- name: Installer stamps both layouts + the ZeroClaw variant correctly
|
||||
run: |
|
||||
if sh scripts/install-skill.sh /tmp/no-such-dir 2>/dev/null; then
|
||||
echo "FAIL: installer accepted a nonexistent target"; exit 1
|
||||
fi
|
||||
mkdir -p /tmp/consumer && sh scripts/install-skill.sh /tmp/consumer
|
||||
diff -r .claude/skills/peltier /tmp/consumer/.claude/skills/peltier
|
||||
diff -r .claude/skills/peltier /tmp/consumer/.agents/skills/peltier
|
||||
sh scripts/install-skill.sh --zeroclaw-variant /tmp/zc-variant
|
||||
test ! -e /tmp/zc-variant/scripts # script-less: ZeroClaw blocks script files
|
||||
test -f /tmp/zc-variant/SKILL.md
|
||||
diff -r .claude/skills/peltier/references /tmp/zc-variant/references
|
||||
echo "installer OK: both layouts identical to canonical; zeroclaw variant has no scripts/"
|
||||
|
||||
- name: Preflight resolves inside a peltier checkout
|
||||
run: |
|
||||
sh .claude/skills/peltier/scripts/preflight.sh | tee /tmp/pf.txt
|
||||
|
||||
@@ -0,0 +1,33 @@
|
||||
# peltier.io — agent entry point
|
||||
|
||||
This file exists for AGENTS.md-reading harnesses (Codex, Jules, Cursor, Zed,
|
||||
Amp, Copilot coding agent, Gemini CLI, …). It is deliberately thin: the
|
||||
binding project rules live in one place, not here.
|
||||
|
||||
## Working on this repo
|
||||
|
||||
Read **[CLAUDE.md](CLAUDE.md)** first and treat it as binding regardless of
|
||||
which harness you are. It holds the non-negotiables (no unverified performance
|
||||
claims, measure-before-optimizing, hard equivalence gates, the append-only
|
||||
ledger, the trust-layer write boundary) and the honest current state of every
|
||||
subsystem. The engineering spec is [SPEC.md](SPEC.md). Commands live in the
|
||||
[justfile](justfile). The pipeline is Linux/POSIX-only at runtime; Windows
|
||||
compiles the workspace but cannot run it.
|
||||
|
||||
Two hard rules that bear repeating even in a pointer file:
|
||||
|
||||
- **Never claim a performance number without its confidence interval and
|
||||
workload.** If you did not run the gates, the number does not exist.
|
||||
- **The trust layer (`crates/`, `config/`, `corpora/`) is read-only to
|
||||
optimization agents.** Patches touch only allowlisted paths under
|
||||
`targets/<name>/`.
|
||||
|
||||
## Verifying performance changes (any repo, any harness)
|
||||
|
||||
This repo ships its trust layer as an Agent Skill:
|
||||
**[.claude/skills/peltier/SKILL.md](.claude/skills/peltier/SKILL.md)** —
|
||||
mirrored byte-identically at `.agents/skills/peltier/` for harnesses that
|
||||
discover skills there (CI enforces the mirror). If your harness auto-loads
|
||||
skills from either path, use it whenever a task involves proving or refuting
|
||||
a speedup. If not, read that SKILL.md and follow it manually — starting with
|
||||
`scripts/preflight.sh`, whose refusal is final.
|
||||
@@ -29,7 +29,7 @@ Full engineering spec: `SPEC.md`. Read it before starting any phase.
|
||||
- `targets/` — vendored OSS targets for case studies (permissive licenses only)
|
||||
- `playbook/` — optimization classes, ordered, with preconditions and known risks
|
||||
- `results/` — generated reports and flamegraphs
|
||||
- `.claude/skills/peltier/` — the trust layer packaged as a consumable Claude Code skill: `verify` mode (equivalence → A/A calibration → interleaved A/B + bootstrap CI) works in any repo against any two shell commands; `attempt` mode drives the full in-repo loop. Drives the real `bench-runner` binary and refuses on unsupported hosts rather than falling back to a second, forkable copy of the statistics. Preflight refusal is asserted in CI (`skill-preflight`).
|
||||
- `.claude/skills/peltier/` — the trust layer packaged as a consumable **Agent Skill** (agentskills.io SKILL.md standard, spec-compliant frontmatter): `verify` mode (equivalence → A/A calibration → interleaved A/B + bootstrap CI) works in any repo against any two shell commands; `attempt` mode drives the full in-repo loop. Drives the real `bench-runner` binary and refuses on unsupported hosts rather than falling back to a second, forkable copy of the statistics. Cross-harness: mirrored byte-identically at `.agents/skills/peltier/` (the Codex/Zed/Cursor/Gemini/Copilot convergence path — CI enforces the mirror), `AGENTS.md` at the repo root points non-Claude harnesses here, and `scripts/install-skill.sh` stamps consumer repos (plus a script-less `--zeroclaw-variant`, since ZeroClaw blocks script files in skills by default — its preflight runs from the checkout). Preflight refusal, spec conformance, mirror equality, and the installer are all asserted in CI (`skill-preflight`).
|
||||
|
||||
**Phase 4 COMPLETE — all SPEC §5 exit criteria met. Phases 0–4 are all done.** Service-mode latency benchmarking shipped: a coordinated-omission-correct, open-loop, fixed-rate load generator (`crates/bench-runner/src/service.rs`) measuring latency from each request's intended send time (queueing counted, not hidden), interleaved A/B, exact percentiles, bootstrap p50/p99 CIs. A real service target — `targets/cjson/service.c`, a minimal HTTP server wrapping the patched `cJSON.c` (trust-layer, outside the workspace allowlist). Service calibration PASS (0/10 A/A false-positive, 10/10 injected-5%-latency-regression detection; `results/calibration/cjson-service-aa.json`). **The verified batch win phase3-cjson-002 measured under a 150 rps-target replay (20 interleaved rounds, 0 drops — the `phase4-cjson-service` ledger row records sessions=20, rate_rps=150, drop_rate=0.0; total request count is not machine-recorded): p50 latency +6.2%, 95% CI [+5.8%, +7.2%] — accepted; p99 NOT claimed (CI [0.07, 4.97], single-worker loopback tail jitter) — harness correctly rejected it.** Mechanical ROI report generator (`crates/report` bin, `just report`, SPEC §9): reads a ledger row + service-latency JSON → throughput→cores→dollars (27.5 cores / $9,621-per-year CI lower bound on a 500-core fleet) *and* latency percentiles, every figure with CI + workload + methodology inline; flags any non-accepted row. Case study: `results/phase4/`. **The roadmap is complete — the profile→hypothesize→patch→gated-verdict→ROI loop runs end to end across Rust and C, batch and service, with zero shipped false accepts across all phases.**
|
||||
|
||||
|
||||
@@ -71,24 +71,40 @@ tests, but cannot run the pipeline.
|
||||
Full command reference and reproduction steps:
|
||||
[Reproduce it yourself](https://bdbrown4.github.io/peltier.io/reproduce.html).
|
||||
|
||||
## Use it on your own code
|
||||
## Use it on your own code — from any agent harness
|
||||
|
||||
The trust layer is not specific to these targets — it will judge any two shell
|
||||
commands. `.claude/skills/peltier/` packages it as a Claude Code skill:
|
||||
commands. It ships as an **[Agent Skill](https://agentskills.io)** (the open
|
||||
SKILL.md standard), which Claude Code, OpenAI Codex, GitHub Copilot, VS Code,
|
||||
Cursor, Gemini CLI, opencode, Goose, Amp, Zed, and ZeroClaw all load natively.
|
||||
One installer stamps it into your repo at both discovery paths:
|
||||
|
||||
```sh
|
||||
# from any repo, with a peltier checkout reachable
|
||||
export PELTIER_HOME=/path/to/peltier.io
|
||||
mkdir -p .claude/skills
|
||||
cp -r /path/to/peltier.io/.claude/skills/peltier .claude/skills/
|
||||
# from your repo, with a peltier checkout at /path/to/peltier.io
|
||||
sh /path/to/peltier.io/scripts/install-skill.sh .
|
||||
export PELTIER_HOME=/path/to/peltier.io # the skill drives bench-runner from here
|
||||
```
|
||||
|
||||
Then ask Claude to verify a speedup. The skill enforces the order that makes a
|
||||
performance claim mean something: **equivalence before timing** (a faster
|
||||
program that computes something different is a bug, not an optimization) →
|
||||
**A/A calibration** (a host that "finds" a speedup between a binary and itself
|
||||
cannot measure yours) → **interleaved A/B with a bootstrap CI** → a verdict
|
||||
decided by the CI *lower bound*, never the median.
|
||||
That writes `.claude/skills/peltier/` (Claude Code; also read by Copilot,
|
||||
VS Code, opencode, Amp) and `.agents/skills/peltier/` (the cross-tool path:
|
||||
Codex, Zed, Cursor, Gemini CLI, Goose, and others). This repo carries both,
|
||||
CI-enforced byte-identical, so the two copies cannot drift.
|
||||
|
||||
**ZeroClaw** blocks script files inside skills by default, so it gets a
|
||||
script-less variant (preflight runs from your peltier checkout instead —
|
||||
same one copy of the script):
|
||||
|
||||
```sh
|
||||
sh /path/to/peltier.io/scripts/install-skill.sh --zeroclaw-variant /tmp/peltier-skill
|
||||
zeroclaw skills install /tmp/peltier-skill
|
||||
```
|
||||
|
||||
Then ask your agent to verify a speedup. The skill enforces the order that
|
||||
makes a performance claim mean something: **equivalence before timing** (a
|
||||
faster program that computes something different is a bug, not an
|
||||
optimization) → **A/A calibration** (a host that "finds" a speedup between a
|
||||
binary and itself cannot measure yours) → **interleaved A/B with a bootstrap
|
||||
CI** → a verdict decided by the CI *lower bound*, never the median.
|
||||
|
||||
It drives the real `bench-runner` binary and **refuses rather than degrade**:
|
||||
on an unsupported host, or with no trust layer reachable, it stops and says so
|
||||
|
||||
Executable
+86
@@ -0,0 +1,86 @@
|
||||
#!/bin/sh
|
||||
# install-skill.sh — stamp the peltier Agent Skill into a consumer repo.
|
||||
#
|
||||
# sh scripts/install-skill.sh <target-repo> [--layout claude|agents|both]
|
||||
# sh scripts/install-skill.sh --zeroclaw-variant <out-dir>
|
||||
#
|
||||
# The skill is one directory (SKILL.md + references/ + scripts/ + assets/),
|
||||
# already in the Agent Skills format (agentskills.io) that Claude Code, Codex,
|
||||
# Copilot, VS Code, Cursor, Gemini CLI, opencode, Goose, Amp, Zed and others
|
||||
# load natively. All this script does is copy that one canonical directory to
|
||||
# the discovery path(s) a harness actually reads:
|
||||
#
|
||||
# .claude/skills/peltier/ Claude Code (also read by Copilot, VS Code,
|
||||
# opencode, Amp, and Cursor in compat mode)
|
||||
# .agents/skills/peltier/ the cross-tool convergence path (Codex, Zed,
|
||||
# Cursor, Copilot, VS Code, Gemini CLI, opencode,
|
||||
# Goose, Amp)
|
||||
#
|
||||
# --layout both (the default) writes both. That is two copies in YOUR repo;
|
||||
# keep them identical (re-run this script to update both; peltier's own CI
|
||||
# shows the diff -r pattern for enforcing it).
|
||||
#
|
||||
# --zeroclaw-variant writes a copy WITHOUT scripts/ to <out-dir>: ZeroClaw's
|
||||
# skill security policy blocks script files by default (skills.allow_scripts
|
||||
# is off), so its variant runs preflight from the peltier checkout instead —
|
||||
# the skill's SKILL.md already explains that path. Then install it with:
|
||||
# zeroclaw skills install <out-dir>
|
||||
#
|
||||
# The skill still needs a peltier checkout at runtime (bench-runner lives
|
||||
# there). Set PELTIER_HOME in the environment your agent runs in.
|
||||
set -eu
|
||||
|
||||
usage() {
|
||||
sed -n '2,30p' "$0" | sed 's/^# \{0,1\}//'
|
||||
exit 2
|
||||
}
|
||||
|
||||
here=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd)
|
||||
src="$here/../.claude/skills/peltier"
|
||||
[ -f "$src/SKILL.md" ] || { echo "install-skill: cannot find the skill at $src" >&2; exit 1; }
|
||||
|
||||
copy_skill() { # $1 = dest dir, $2 = "full" | "noscripts"
|
||||
rm -rf "$1"
|
||||
mkdir -p "$1"
|
||||
cp -R "$src/." "$1/"
|
||||
if [ "$2" = "noscripts" ]; then rm -rf "$1/scripts"; fi
|
||||
}
|
||||
|
||||
case "${1:-}" in
|
||||
""|-h|--help) usage ;;
|
||||
--zeroclaw-variant)
|
||||
out=${2:-}; [ -n "$out" ] || usage
|
||||
copy_skill "$out" noscripts
|
||||
echo "wrote script-less variant to $out (ZeroClaw blocks script files in skills by default)"
|
||||
echo "next: zeroclaw skills install \"$out\" # optionally --bundle <alias>, then add the"
|
||||
echo " bundle to agents.<alias>.skill_bundles so your agent loads it"
|
||||
echo "note: at runtime the skill runs preflight from your peltier checkout —"
|
||||
echo " set PELTIER_HOME to that checkout for the agent's environment"
|
||||
exit 0
|
||||
;;
|
||||
esac
|
||||
|
||||
target=$1; shift
|
||||
layout=both
|
||||
while [ $# -gt 0 ]; do
|
||||
case "$1" in
|
||||
--layout) layout=${2:-}; shift 2 ;;
|
||||
*) usage ;;
|
||||
esac
|
||||
done
|
||||
case "$layout" in claude|agents|both) ;; *) usage ;; esac
|
||||
[ -d "$target" ] || { echo "install-skill: target repo '$target' is not a directory" >&2; exit 1; }
|
||||
|
||||
wrote=""
|
||||
if [ "$layout" = claude ] || [ "$layout" = both ]; then
|
||||
copy_skill "$target/.claude/skills/peltier" full
|
||||
wrote="$wrote $target/.claude/skills/peltier"
|
||||
fi
|
||||
if [ "$layout" = agents ] || [ "$layout" = both ]; then
|
||||
copy_skill "$target/.agents/skills/peltier" full
|
||||
wrote="$wrote $target/.agents/skills/peltier"
|
||||
fi
|
||||
|
||||
echo "installed:$wrote"
|
||||
echo "next: set PELTIER_HOME=/path/to/peltier.io in the environment your agent runs in"
|
||||
echo " (the skill drives the real bench-runner from that checkout; preflight refuses without it)"
|
||||
Reference in New Issue
Block a user