From 477c5472dd9d95b14172405ac31ca9c51dafba00 Mon Sep 17 00:00:00 2001 From: Ben Brown Date: Wed, 15 Jul 2026 11:59:26 -0400 Subject: [PATCH 1/2] skill: make the peltier skill consumable across agent harnesses MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The SKILL.md Agent Skills standard (agentskills.io) is now read natively by Codex, Copilot, VS Code, Cursor, Gemini CLI, opencode, Goose, Amp, Zed, Factory, and ZeroClaw — so the portable move is not per-harness adapter files (forked content that drifts) but one canonical skill placed where each harness actually looks. - Spec-compliant frontmatter: license, compatibility (the Linux-only measurement story, stated up front), version/author/tags top-level (ZeroClaw's runtime reads those; the spec ignores unknown fields). reference/ renamed to references/ per the spec's conventional layout. - .agents/skills/peltier/ mirror: the cross-tool convergence path (Codex, Zed, Cursor, Gemini CLI, Copilot, opencode, Goose, Amp). Byte-identical to the .claude canonical, enforced by CI — drift between the two copies is a red build, not a footnote. - AGENTS.md at the repo root for the ~20 harnesses that read it (Claude Code does not; CLAUDE.md remains its file). Deliberately thin: it points at CLAUDE.md and the skill instead of restating rules. - scripts/install-skill.sh: stamps a consumer repo at both discovery paths (--layout claude|agents|both), or emits a script-less --zeroclaw-variant. ZeroClaw's skill security policy blocks script files by default (skills.allow_scripts off) — its audit rejected the full skill and passes the variant, verified against the real zeroclaw binary. The variant loses nothing: preflight.sh still exists exactly once, in the peltier checkout the skill already requires, and SKILL.md now documents running it from there. Re-implementing preflight inline is named as a refusal condition. - CI (skill-preflight): frontmatter upgraded to a full Agent Skills spec conformance check on BOTH copies (name rules, dir match, description and body limits, metadata string-map), plus mirror equality and a functional installer test (both layouts identical to canonical, variant carries no scripts/, nonexistent target refused). Co-Authored-By: Claude Fable 5 --- .agents/skills/peltier/SKILL.md | 142 ++++++++++++ .../peltier/assets/accept.template.toml | 41 ++++ .../skills/peltier/references}/attempt.md | 0 .../skills/peltier/references}/verify.md | 2 + .agents/skills/peltier/scripts/preflight.sh | 91 ++++++++ .claude/skills/peltier/SKILL.md | 22 +- .claude/skills/peltier/references/attempt.md | 104 +++++++++ .claude/skills/peltier/references/verify.md | 205 ++++++++++++++++++ .github/workflows/ci.yml | 61 ++++-- AGENTS.md | 33 +++ CLAUDE.md | 2 +- README.md | 40 +++- scripts/install-skill.sh | 86 ++++++++ 13 files changed, 801 insertions(+), 28 deletions(-) create mode 100644 .agents/skills/peltier/SKILL.md create mode 100644 .agents/skills/peltier/assets/accept.template.toml rename {.claude/skills/peltier/reference => .agents/skills/peltier/references}/attempt.md (100%) rename {.claude/skills/peltier/reference => .agents/skills/peltier/references}/verify.md (98%) create mode 100644 .agents/skills/peltier/scripts/preflight.sh create mode 100644 .claude/skills/peltier/references/attempt.md create mode 100644 .claude/skills/peltier/references/verify.md create mode 100644 AGENTS.md create mode 100755 scripts/install-skill.sh diff --git a/.agents/skills/peltier/SKILL.md b/.agents/skills/peltier/SKILL.md new file mode 100644 index 0000000..5b1ce6b --- /dev/null +++ b/.agents/skills/peltier/SKILL.md @@ -0,0 +1,142 @@ +--- +name: peltier +description: "Prove a performance change is real — or refuse to claim it. Use when optimizing code for speed, verifying a claimed speedup ('is this actually faster?', 'did my optimization work?'), checking for a performance regression ('did this PR make things slower?'), or comparing the performance of two implementations. Runs peltier's trust layer — equivalence gates, A/A environment calibration, interleaved A/B with bootstrap confidence intervals. Never reports a naked percentage and never claims a win that is not statistically significant. Do NOT use for correctness-only work." +license: "GPL-3.0-or-later" +compatibility: "Needs a peltier checkout (https://github.com/bdbrown4/peltier.io; set PELTIER_HOME) with cargo to build bench-runner. Measurement requires a Linux/POSIX host; on unsupported hosts the skill refuses to produce numbers rather than degrade." +version: "1.1.0" +author: "peltier.io (Ben Brown)" +tags: ["performance", "benchmarking", "statistics", "verification"] +metadata: + homepage: "https://bdbrown4.github.io/peltier.io/" + source: "https://github.com/bdbrown4/peltier.io" +--- + +# peltier — prove it, or claim nothing + +A performance change that has not passed equivalence gates **and** shown a +statistically significant improvement **does not exist**. You may discard it +or log it. You may not claim it. + +Most benchmarking is wrong in ways that flatter the author: run-to-run noise +read as a win, a faster binary that computes something subtly different, one +lucky run out of five, a percentage with no interval attached. This skill +exists to make those failures impossible rather than unlikely. + +## Preflight — never skip, never work around + +Run `scripts/preflight.sh` **relative to this skill's directory** (resolve its +absolute path first — the skill may be installed at `~/.claude/skills/`, as a +plugin, or copied into another project): + + sh /scripts/preflight.sh + +Some harnesses block script files inside skills, so your installation may +carry no `scripts/` directory. The script still exists exactly once — in the +peltier checkout this skill drives (which you need anyway; it is where +`bench-runner` lives). Run it from there: + + sh "$PELTIER_HOME/.claude/skills/peltier/scripts/preflight.sh" + +Never transcribe or re-implement the preflight logic inline; if you cannot +locate a checkout to run it from, that IS a preflight refusal — report it. + +It locates the trust layer, builds the real `bench-runner`, and checks this +host can run it. `STATUS=ok` does **not** mean the host can measure your +change — only the A/A self-test decides that. + +**If it refuses, that ends the claim, not the work.** You may still write the +patch and explain why you think it is faster. You may not attach a number to +it. Do not fall back to `time`, `hyperfine`, or a stopwatch loop — a number +you cannot stand behind is worse than no number, because someone will act on +it. The statistics live in `bench-runner` and are used from there; a second +copy of the math is a second thing that can lie. + +## Two modes + +| Mode | Use when | Read | +|---|---|---| +| **verify** | You have a baseline and a candidate — any repo, any language. | [references/verify.md](references/verify.md) | +| **attempt** | You are inside a peltier checkout and want the full loop: profile → hypothesize → patch → gated verdict → ledger row. | [references/attempt.md](references/attempt.md) | + +Default to **verify**. It is the portable one, and it is what almost every +request actually wants. + +### No candidate yet? Profile first — do not guess. + +"Measure before optimizing" is not advice, it is a gate. If you have one +program and a hunch, get profile data *before* writing a patch: `perf record` +(native), `py-spy` / `cProfile` (Python), `node --prof` / `--cpu-prof` (Node), +`cargo flamegraph` (Rust), or `just coz` for causal profiling inside a peltier +checkout. Then form a hypothesis about a *named* hot path, and only then patch. +Guessing where the time goes is how people spend a week on 0.3% of the runtime. + +## The order of the gates is the point + +Each gate can end the job, and ending early is a success, not a failure. + +1. **Equivalence — before you time anything.** A faster program that does + something different is not an optimization, it is a bug with good + benchmarks. Prove baseline and candidate produce identical output on a fixed + workload. Comparing two *empty* outputs is a failed gate, not a passed one. +2. **Calibration — can this machine measure this change at all?** The A/A + self-test runs the same command against itself and must find **nothing**. A + harness that "discovers" a speedup between a binary and itself is measuring + noise, and every number it gives you afterwards is fiction. +3. **Measurement — interleaved A/B, bootstrap CI.** Not "run A five times, then + B five times" — that hands any thermal or cache drift to whichever side ran + last. +4. **Risk — what did the patch actually touch?** Concurrency primitives, + `unsafe`/raw memory, floating-point reassociation: `needs-human-review`, + **regardless of how good the numbers are**. In verify mode *you* are the + classifier — `bench-runner` only ever emits `accepted` or `rejected-bench`, + so it cannot make this call for you. +5. **Verdict — the CI lower bound decides.** Accept only if the bootstrap 95% + CI lower bound of the speedup clears the threshold (default 2%). The median + is not the claim. The lower bound is the claim. + +## Non-negotiables + +- **Cheapest wins first:** build flags → LTO → PGO → allocator swap → *then* + code changes. Do not hand-write SIMD before you have tried `-O3`. +- **Every attempt is recorded, including failures.** A rejected hypothesis is a + result — it stops the next person from re-running your dead end. +- **A failed gate is a complete, valid outcome.** Write it up and move on. Do + not iterate on a rejected patch more than twice without a new hypothesis. +- **The thing being measured never grades itself.** The patch may not touch the + harness, the corpus, or the tests. + +## Never + +- **Never a naked percentage.** Every number carries its confidence interval + and the workload it was measured on. "12% faster" is not a result. +- **Never re-run a benchmark until it passes.** That is p-hacking with extra + steps. The first calibrated run is the answer. +- **Never claim a win the CI lower bound does not support**, however good the + median looks and however "obviously" faster the change is. +- **Never present a measurement from a host that failed calibration.** + +## What good looks like + +**A claim, fully dressed** (`phase3-cjson-002`, in the ledger): + +> Accepted. Speedup 1.088 (candidate is 8.8% faster), 95% CI [1.074, 1.100] — +> so the *defensible* claim is ≥7.4%, not 8.8%. Workload: 5.6 MB synthetic +> JSON, 20k records, parsed + serialized 2×, single thread. Gates: upstream +> suite green, golden replay byte-identical, ASan + UBSan clean. + +Note what that write-up does **not** say. It does not claim a differential-fuzz +run, because that row records `fuzz_iters=0` — the fuzzing of that era ran +out-of-band and the machine never witnessed it. Fuzz is a pipeline gate now, and +new rows carry a real count. Say what the record supports, not what you +remember doing. + +**Declining to claim what the data won't support** (`phase4-cjson-service`): +p50 latency improved 6.2% [95% CI +5.8%, +7.2%] at 150 rps — claimed. The p99 +CI came back [0.07, 4.97] — uselessly wide, so **p99 was not claimed at all**, +even though its median looked like a win. + +**Good numbers on a bad change** (`phase2-comrak-010`): 1.107 speedup, 95% CI +[1.079, 1.121], on 11 MB of markdown. Tight interval, real improvement, +**accepted — and wrong.** LeakSanitizer caught it afterwards and the row was +overturned. It never shipped. The bench was never the problem; skipping a gate +was. This is why the order above is not negotiable. diff --git a/.agents/skills/peltier/assets/accept.template.toml b/.agents/skills/peltier/assets/accept.template.toml new file mode 100644 index 0000000..58f32ed --- /dev/null +++ b/.agents/skills/peltier/assets/accept.template.toml @@ -0,0 +1,41 @@ +# peltier accept thresholds — consumer template. +# +# Copy this next to the code you are measuring and pass it with +# `bench-runner --config `. The defaults are peltier's own +# published bar; loosening them to make a result pass is not measurement, +# it is negotiation. + +# The bootstrap CI *lower bound* of the speedup ratio must clear 1 + threshold. +# The median is not the claim. The lower bound is the claim. +threshold = 0.02 + +# Measured runs per side, interleaved ABAB (SPEC minimum: 30). +runs_per_side = 30 + +# Warm-up runs per side, discarded — they pay for cold caches and JIT. +warmup_runs = 3 + +# Bootstrap resamples for the ratio CI. +bootstrap_iters = 10000 + +# Confidence level for the ratio CI. +confidence = 0.95 + +# Fixed PRNG seed so a calibration run is reproducible by anyone. +bootstrap_seed = 1887343589 + +# CPU pinning. WARNING: this key is INERT for `compare`, `aa`, and `calibrate` +# — it is read only by the in-repo `verdict` pipeline. Setting it here and then +# reporting "pinned to core N" would be a claim about a run that never happened. +# +# To actually pin a compare/aa/calibrate run, wrap BOTH sides identically in +# the command itself: --baseline 'taskset -c 2 ' --candidate 'taskset -c 2 ' +# +# Pinning tightens the A/A confidence interval ~5x on peltier's calibration +# hardware. An unpinned host is not disqualified; a NOISY one is, and the A/A +# self-test decides that empirically. Leave this empty. +pin_prefix = "" + +# Differential-fuzz iterations requested per gate run. Only used by the +# in-repo `verdict` pipeline, which requires a target to declare [gates].fuzz. +# fuzz_iters = 10000 diff --git a/.claude/skills/peltier/reference/attempt.md b/.agents/skills/peltier/references/attempt.md similarity index 100% rename from .claude/skills/peltier/reference/attempt.md rename to .agents/skills/peltier/references/attempt.md diff --git a/.claude/skills/peltier/reference/verify.md b/.agents/skills/peltier/references/verify.md similarity index 98% rename from .claude/skills/peltier/reference/verify.md rename to .agents/skills/peltier/references/verify.md index df90ba8..190819d 100644 --- a/.claude/skills/peltier/reference/verify.md +++ b/.agents/skills/peltier/references/verify.md @@ -6,6 +6,8 @@ faster. Find out. Works on any pair of shell commands, in any language. ## 0. Preflight sh /scripts/preflight.sh + # or, if this installation carries no scripts/ (script-blocking harness): + sh "$PELTIER_HOME/.claude/skills/peltier/scripts/preflight.sh" Read the `key=value` output; you need `BENCH_RUNNER` and `PIN_SUPPORTED`. A refusal ends the **claim**, not the work — report the reason verbatim and do diff --git a/.agents/skills/peltier/scripts/preflight.sh b/.agents/skills/peltier/scripts/preflight.sh new file mode 100644 index 0000000..c30dbc9 --- /dev/null +++ b/.agents/skills/peltier/scripts/preflight.sh @@ -0,0 +1,91 @@ +#!/bin/sh +# peltier preflight — locate the trust layer, check this host can RUN it, and +# build the real bench-runner. +# +# STATUS=ok means the harness is present and runnable here. It does NOT mean +# this host can measure your change: only the A/A self-test decides that, and +# it decides it empirically. Do not read STATUS=ok as "calibrated". +# +# A refusal (STATUS=refuse, nonzero exit) is a complete, valid outcome. It +# ends the *claim*, not the work: you may still write and explain a patch, you +# just may not attach a number to it. Never work around a refusal with a +# hand-rolled timing loop. +set -eu + +say() { printf '%s\n' "$1"; } +refuse() { + say "STATUS=refuse" + say "REASON=$1" + exit 1 +} +abspath() { ( CDPATH= cd -- "$1" 2>/dev/null && pwd ); } + +# --- host ------------------------------------------------------------- +# bench-runner shells every timed run through `sh -c`, and the in-repo +# pipeline (harnessd, service mode, the sh-based gates) is POSIX-only. +# Windows compiles the workspace but cannot run it. +uname_s=$(uname -s 2>/dev/null || echo unknown) +case "$uname_s" in + Linux) host=linux ;; + Darwin) host=darwin ;; + *) refuse "unsupported host '$uname_s' — peltier's harness is Linux/POSIX-only at runtime. Run it on Linux (or macOS for verify mode). Do not substitute another benchmark." ;; +esac + +# --- trust layer ------------------------------------------------------ +is_peltier() { [ -f "$1/crates/bench-runner/Cargo.toml" ] && [ -f "$1/config/accept.toml" ]; } + +# Walk up from a starting dir looking for a checkout. Absolute POSIX paths +# always terminate at "/", so this cannot spin. +walk_up() { + d=$1 + while [ -n "$d" ] && [ "$d" != "/" ]; do + if is_peltier "$d"; then printf '%s' "$d"; return 0; fi + d=$(dirname "$d") + done + is_peltier / && printf '/' + return 0 +} + +home="" +if [ -n "${PELTIER_HOME:-}" ]; then + canon=$(abspath "$PELTIER_HOME" || true) + [ -n "$canon" ] || refuse "PELTIER_HOME='$PELTIER_HOME' does not exist" + is_peltier "$canon" || refuse "PELTIER_HOME='$PELTIER_HOME' is not a peltier checkout (no crates/bench-runner/Cargo.toml + config/accept.toml)" + home=$canon +else + # cwd first (working inside a checkout), then this script's own location + # (the skill still living inside one). A skill copied into another project + # finds neither — which is why PELTIER_HOME exists. + home=$(walk_up "$(pwd)") + [ -n "$home" ] || home=$(walk_up "$(abspath "$(dirname -- "$0")")") +fi +[ -n "$home" ] || refuse "no peltier checkout found — clone https://github.com/bdbrown4/peltier.io and set PELTIER_HOME=/path/to/peltier.io. The statistics live in bench-runner; do not reimplement them." + +command -v cargo >/dev/null 2>&1 || refuse "cargo not on PATH — needed to build bench-runner from $home" + +# --- build the real harness ------------------------------------------- +# Always build. cargo is a no-op when current, and skipping this because a +# binary happens to exist would silently measure with a stale copy of the +# statistics after a `git pull`. +br="$home/target/release/bench-runner" +( cd "$home" && cargo build --release -q -p bench-runner ) \ + || refuse "bench-runner failed to build in $home — fix the build before claiming any number" +{ [ -f "$br" ] && [ -x "$br" ]; } || refuse "bench-runner did not appear at $br after a successful build" + +# --- pinning ---------------------------------------------------------- +# NOTE: config/accept.toml's `pin_prefix` is read ONLY by the in-repo verdict +# pipeline. For compare/aa/calibrate you must wrap the commands yourself +# (`taskset -c N ` on both sides). PIN_SUPPORTED just says whether that +# tool is available here. +if [ "$host" = linux ] && command -v taskset >/dev/null 2>&1; then + pin=yes +else + pin=no +fi + +say "STATUS=ok" +say "PELTIER_HOME=$home" +say "BENCH_RUNNER=$br" +say "HOST=$host" +say "PIN_SUPPORTED=$pin" +say "NOTE=harness runnable here; the A/A self-test decides whether it can measure your change" diff --git a/.claude/skills/peltier/SKILL.md b/.claude/skills/peltier/SKILL.md index 24b9542..5b1ce6b 100644 --- a/.claude/skills/peltier/SKILL.md +++ b/.claude/skills/peltier/SKILL.md @@ -1,6 +1,14 @@ --- name: peltier description: "Prove a performance change is real — or refuse to claim it. Use when optimizing code for speed, verifying a claimed speedup ('is this actually faster?', 'did my optimization work?'), checking for a performance regression ('did this PR make things slower?'), or comparing the performance of two implementations. Runs peltier's trust layer — equivalence gates, A/A environment calibration, interleaved A/B with bootstrap confidence intervals. Never reports a naked percentage and never claims a win that is not statistically significant. Do NOT use for correctness-only work." +license: "GPL-3.0-or-later" +compatibility: "Needs a peltier checkout (https://github.com/bdbrown4/peltier.io; set PELTIER_HOME) with cargo to build bench-runner. Measurement requires a Linux/POSIX host; on unsupported hosts the skill refuses to produce numbers rather than degrade." +version: "1.1.0" +author: "peltier.io (Ben Brown)" +tags: ["performance", "benchmarking", "statistics", "verification"] +metadata: + homepage: "https://bdbrown4.github.io/peltier.io/" + source: "https://github.com/bdbrown4/peltier.io" --- # peltier — prove it, or claim nothing @@ -22,6 +30,16 @@ plugin, or copied into another project): sh /scripts/preflight.sh +Some harnesses block script files inside skills, so your installation may +carry no `scripts/` directory. The script still exists exactly once — in the +peltier checkout this skill drives (which you need anyway; it is where +`bench-runner` lives). Run it from there: + + sh "$PELTIER_HOME/.claude/skills/peltier/scripts/preflight.sh" + +Never transcribe or re-implement the preflight logic inline; if you cannot +locate a checkout to run it from, that IS a preflight refusal — report it. + It locates the trust layer, builds the real `bench-runner`, and checks this host can run it. `STATUS=ok` does **not** mean the host can measure your change — only the A/A self-test decides that. @@ -37,8 +55,8 @@ copy of the math is a second thing that can lie. | Mode | Use when | Read | |---|---|---| -| **verify** | You have a baseline and a candidate — any repo, any language. | [reference/verify.md](reference/verify.md) | -| **attempt** | You are inside a peltier checkout and want the full loop: profile → hypothesize → patch → gated verdict → ledger row. | [reference/attempt.md](reference/attempt.md) | +| **verify** | You have a baseline and a candidate — any repo, any language. | [references/verify.md](references/verify.md) | +| **attempt** | You are inside a peltier checkout and want the full loop: profile → hypothesize → patch → gated verdict → ledger row. | [references/attempt.md](references/attempt.md) | Default to **verify**. It is the portable one, and it is what almost every request actually wants. diff --git a/.claude/skills/peltier/references/attempt.md b/.claude/skills/peltier/references/attempt.md new file mode 100644 index 0000000..2b00b5d --- /dev/null +++ b/.claude/skills/peltier/references/attempt.md @@ -0,0 +1,104 @@ +# attempt — the full loop, inside a peltier checkout + +Profile → hypothesize → patch → gated verdict → ledger row. This mode only +applies when you are working **inside a peltier checkout** on a target that has +a `targets//target.toml`. Everywhere else, use +[verify.md](verify.md). + +Requires Linux (harnessd uses Unix sockets and `setsid`; every gate shells +through `sh`). + +## Before you touch anything + +**Read the ledger first.** It records every attempt ever made on this target, +including the failures — which is exactly what stops you re-running a dead +end: + + cargo run -p policy # ranks playbook classes by their evidence + +`policy` reads the ledger and ranks optimization classes by the Wilson lower +bound of their shippable-win rate. It is **advisory** — a learned prior over +the fixed cheapest-first order. The gates still decide every verdict. + +**Then respect the order.** Cheapest wins first: build flags → LTO → PGO → +allocator swap → *then* code changes. Do not hand-write SIMD before trying +`-O3`. The playbook (`playbook/`) has the preconditions and known risks for +each class. + +## The boundary you may not cross + +The agent may modify **only** allowlisted paths under `targets//`. +`crates/`, `config/`, `corpora/`, and the upstream test suites are read-only: +the thing being measured does not get to grade itself. `harnessd` enforces +this on every proposed diff (path allowlist over every diff header, including +`rename`/`copy` — a patch that tries to escape is rejected, not sanitized). + +## Log the hypothesis *before* patching + +A hypothesis written after seeing the numbers is not a hypothesis. State what +you think is slow, why, and what you expect the change to buy — then patch. + +## Run the gates + + just gates + +Corpus pin → test-suite pin (when `corpora//TESTSUITE.sha256` exists) → +upstream tests → golden replay. Note what this **does not** do: differential +fuzz needs a pristine baseline to differ against, so it reports `Skipped` here +and runs for real on the accept path below. `just gates` is a fast equivalence +check, not a fuzz run — do not report it as one. + +## The verdict — the only thing that can accept a change + + just verdict "" "" \ + --patch-file + +This rebuilds a **pristine baseline** from the pinned upstream commit (not +whatever is lying around in your tree), runs the full gate set against the +candidate, benches interleaved A/B against that baseline, and writes one +append-only ledger row with the verdict and the complete evidence. + +It will refuse to hand you an `accepted` unless: + +- the upstream test suite is green **and** golden replay is byte-identical (or + within the target's declared FP tolerance); +- the **differential-fuzz gate actually passed** — a real run reporting zero + divergences. No fuzz run, no machine accept. Ever; +- ASan/UBSan are clean (and TSan, when the target configures it); +- the bootstrap 95% CI lower bound of the speedup clears the threshold in + `config/accept.toml` (default 2%); +- and a conservative lexical risk classifier finds **no** concurrency, unsafe, + or floating-point signals in the patch. If it does, the verdict is capped at + `needs-human-review` no matter how good the numbers are. + +Any of these failing produces a rejected row. **That is a complete, valid +outcome** — the row is the deliverable. Write it and move on. Do not iterate on +a rejected patch more than twice without a genuinely new hypothesis. + +## Report + + just report + +Mechanically generates the ROI from the ledger row: throughput → cores → +dollars, and/or latency percentiles — every figure with its CI, workload, and +methodology printed inline. It flags any row that was not accepted, so a +rejected attempt cannot be laundered into a dollar figure. + +## Unattended + + just agent-attempt + +Runs one full attempt behind the OS boundary: `harnessd` as the trusted uid on +a Unix socket, the agent loop as an unprivileged user, the verdict pipeline +wrapped in a network namespace (`scripts/no-net.sh`) that **fails closed** — if +namespaces are unavailable it refuses to run rather than quietly running with +the network up. `just isolation-check` verifies the boundary from the agent's +side. + +## When the numbers are good and the change is scary + +FP-ordering, concurrency primitives, anything UB-adjacent: `needs-human-review`, +always, regardless of the speedup. This is not a formality. The one false +accept in peltier's history (`phase2-comrak-010`) passed its bench and was +caught by a human audit afterwards — which is precisely why the sanitizer lane +is now machine-enforced and why this rule does not bend. diff --git a/.claude/skills/peltier/references/verify.md b/.claude/skills/peltier/references/verify.md new file mode 100644 index 0000000..190819d --- /dev/null +++ b/.claude/skills/peltier/references/verify.md @@ -0,0 +1,205 @@ +# verify — prove a speedup in any repo + +You have two things that should do the same work, one of which you think is +faster. Find out. Works on any pair of shell commands, in any language. + +## 0. Preflight + + sh /scripts/preflight.sh + # or, if this installation carries no scripts/ (script-blocking harness): + sh "$PELTIER_HOME/.claude/skills/peltier/scripts/preflight.sh" + +Read the `key=value` output; you need `BENCH_RUNNER` and `PIN_SUPPORTED`. A +refusal ends the **claim**, not the work — report the reason verbatim and do +not substitute another timing tool. + + BR= + +## 1. Config + + cp /assets/accept.template.toml ./peltier-accept.toml + CFG=./peltier-accept.toml + +Do **not** loosen `threshold`, `runs_per_side`, or `confidence` to make a +result pass. That is not measurement, it is negotiation. + +**Pinning is not done through this file.** `pin_prefix` is read only by the +in-repo `verdict` pipeline — `compare`, `aa`, and `calibrate` ignore it +entirely (bench-runner's own fingerprint says `"caller-provided (wrap cmd in +taskset)"`). If `PIN_SUPPORTED=yes` and you know a core is free, pin by +wrapping **both sides identically**: + + --baseline 'taskset -c 2 ' --candidate 'taskset -c 2 ' + +An unpinned host is not disqualified. A *noisy* one is, and step 3 is what +decides that. + +## 2. Two immutable artifacts — not two states of one tree + +The harness runs the two sides **alternately, 66 times**. That is impossible if +"baseline" and "candidate" are the same working tree in different states. +Before measuring: + +- Build each side into its **own** location — a `git worktree` per side, or two + separate build/output directories. Never `git stash` between runs. +- Pass the **built artifact paths** as the commands. Never pass a wrapper that + can rebuild: `cargo run`, `npm start`, `go run`, `mvn exec` will happily + rebuild from whatever is in the tree *mid-benchmark*, so you can end up + timing the same binary twice — and paying build cost inside the timed window. +- Confirm the two artifacts actually differ: `sha256sum `. + +Peltier learned this the hard way (`phase1-verdict-smoke-001`: "workspace paths +get silently rebuilt from the patched tree by test runs"), which is why the +in-repo pipeline rebuilds a pristine baseline from the pinned upstream commit +rather than trusting the tree. + +## 3. Equivalence — before you time anything + +Capture **everything** each side produces, on a fixed, representative workload: + + run_side() { # $1 = tag, $2 = command + rm -rf "out.$1"; mkdir -p "out.$1" + ( eval "$2" ) > "out.$1/stdout" 2> "out.$1/stderr"; echo "$?" > "out.$1/exit" + } + run_side base "" + run_side cand "" + diff -r out.base out.cand && echo "GOLDEN PASS" || echo "DIVERGENT" + +If the command's real output is a **file tree** (a bundler, a compiler, an +image pipeline), hash that too — stdout is not the artifact: + + ( cd dist && find . -type f | sort | xargs sha256sum ) > out./tree + +**An empty diff over empty output is a failed gate, not a passed one.** If both +sides wrote nothing to stdout, nothing to stderr, and you captured no artifact +tree, you have compared nothing — stop and find the real output before going +further. Then run the project's own test suite against the candidate; it must +pass. + +- **Identical output + tests green** → proceed. +- **Divergent** → stop. No speedup may be claimed from a change you cannot show + is behavior-preserving. The one exception — a change that *legitimately* + alters output, floating-point reassociation being the common case — is + `needs-human-review`, never an auto-accept. Quantify the divergence (how many + values, by how much) and hand it to a human. + +## 4. Calibration — can this host measure this change? + +The A/A self-test runs the **same command on both sides**. It must find nothing: + + $BR --config "$CFG" aa --cmd '' + +- stdout `A/A self-test passed (null verdict)` → the harness is not inventing + speedups. Proceed. +- **Nonzero exit**, with `Error: A/A self-test FAILED: harness claims a speedup + from identical binaries` on **stderr** → stop. This host is too noisy to + measure this change; every number it produces afterwards is fiction. Report + the noise floor, then quiet the machine (pin a core, disable turbo/SMT, close + everything else) or move to calibrated hardware. + +For a claim anyone will act on, run the full calibration — a false-positive +rate over many A/A sessions *plus* proof the harness can detect a **known +injected regression**: + + $BR --config "$CFG" calibrate --cmd '' \ + --sessions 20 --slowdown 0.05 --out calibration.json + +Required: **<5% A/A false-positive rate** and **≥95% detection** of the injected +5% slowdown. A harness that cannot see a real 5% regression cannot be trusted +to confirm a real 5% win. Keep `calibration.json` — it is the evidence that the +measurement itself was sound. + +### Budget the runtime before you start + +`compare` executes the workload **(30 + 3) × 2 = 66 times**. `calibrate +--sessions 20` executes it roughly **2,650 times**. Size the workload to +**~0.2–2 s** of wall time: shorter and you measure process startup, longer and +calibration takes hours. Run `calibrate` in the background — it will blow +through a foreground command timeout. + +## 5. Measure — interleaved A/B, bootstrap CI + + $BR --config "$CFG" compare --baseline '' --candidate '' + +Runs are **interleaved** (ABAB…), not blocked, so thermal drift and cache state +hit both sides equally. Output: + + speedup (baseline/candidate): median 1.0884, 95% CI [1.0745, 1.1001] + verdict: accepted + +The ratio is `baseline/candidate`, so **>1 means the candidate is faster**. + +`--perf-stat` (Linux) adds cycles/instructions/cache-miss medians — useful for +explaining *why* something got faster, **never** part of the accept decision, +and usually unavailable in VMs. + +## 6. Risk — you are the classifier here + +`bench-runner` emits only `accepted` or `rejected-bench`. It **cannot** route a +dangerous change to review, because in verify mode nothing is inspecting your +diff. So inspect it yourself, and override the tool: + +Read the patch. If it touches any of these, the outcome is **`needs-human-review` +regardless of what `compare` printed**: + +- **Concurrency** — threads, locks, atomics, memory ordering, `volatile`, work + queues, async pools, `rayon`/`pthread`/`std::sync`. +- **Unsafe / raw memory** — `unsafe`, pointer casts, `transmute`, FFI, + manual allocation, uninitialized memory. +- **Floating-point reassociation** — reordering accumulation, `-ffast-math`, + FP SIMD, changed summation order. + +These changes can be *correct on your workload and wrong in production*, and a +green bench says nothing about that. A byte-identical output does not clear +them: a data race produces identical output right up until it doesn't. + +## 7. Verdict — read it honestly + +| What you see | What it means | What you may say | +|---|---|---| +| `verdict: accepted` | CI lower bound clears the threshold. | Claim it — with the **CI lower bound**, the interval, and the workload. | +| `verdict: rejected-bench`, CI spans 1.0 (e.g. `[0.983, 1.001]`) | Indistinguishable from no change. | "No shippable win." A real, complete result. | +| `verdict: rejected-bench`, CI **entirely below** 1.0 (e.g. `[0.91, 0.96]`) | A measured **regression**. | Report it as one, with the interval. Do not call it "no win." | +| Any risk signal from step 6 | Out of the machine's hands. | `needs-human-review`. Never auto-accept. | + +Claim the **lower bound**, not the median: a median of 1.088 with CI +[1.074, 1.100] supports "≥7.4% faster", not "8.8% faster". + +If the result is a rejection, **that is the answer**. Re-running until it passes +is p-hacking. Two failed hypotheses on the same hot path means you need a new +hypothesis, not another run. + +## Services (latency, not throughput) + +For a server, wall-clock-per-run is the wrong instrument. Service mode is an +open-loop, fixed-rate load generator that measures each request from its +**intended** send time, so queueing delay is counted rather than hidden +(coordinated omission). It reports p50/p99 with bootstrap CIs. + +**It is not a generic HTTP load generator.** The server binary must: + +- take ` ` as positional argv; +- print a line beginning with `READY` on stdout, **flushed**, once listening; +- answer `GET /` and **close the connection** (`Connection: close`) — the client + reads until EOF with no timeout, so a keep-alive server hangs the harness + forever. + +Point it at a stock Express or Flask app and it will either fail the READY +handshake or block indefinitely. Peltier wrote a thin adapter binary +(`targets/cjson/service.c`) for exactly this reason; you will need one too. + + $BR --config "$CFG" service --baseline-bin '' --candidate-bin '' \ + --doc --rate 150 --sessions 20 + +Calibrate it the same way first (`service-calibrate`). Tail percentiles on a +single-worker loopback setup are jittery — **if the p99 CI is wide, do not claim +p99**, even when p50 is solid. Peltier's own shipped service result claims p50 +(+6.2%, CI [+5.8%, +7.2%]) and explicitly declines to claim p99 (CI [0.07, 4.97]). + +## Record the attempt — including the failures + +Outside peltier there is no ledger, so write the evidence where the next person +will find it (a results file, the PR body): hypothesis, what changed, +equivalence result, calibration evidence, the CI, the workload, and the verdict. +**Failures are the most valuable rows** — they stop someone from re-running your +dead end in six months. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index efc13a8..0744fd3 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -105,23 +105,58 @@ jobs: - uses: dtolnay/rust-toolchain@stable - uses: Swatinem/rust-cache@v2 - - name: SKILL.md frontmatter is loadable + - name: SKILL.md conforms to the Agent Skills spec (both copies) run: | python3 - <<'PY' - import sys, yaml, pathlib - p = pathlib.Path(".claude/skills/peltier/SKILL.md") - text = p.read_text(encoding="utf-8") - if not text.startswith("---\n"): - sys.exit("SKILL.md must open with a --- frontmatter block") - fm = yaml.safe_load(text.split("---", 2)[1]) # raises on malformed YAML - for key in ("name", "description"): - if not isinstance(fm.get(key), str) or not fm[key].strip(): - sys.exit(f"frontmatter '{key}' must be a non-empty string") - if fm["name"] != p.parent.name: - sys.exit(f"frontmatter name '{fm['name']}' != directory '{p.parent.name}'") - print(f"frontmatter OK: name={fm['name']}, description={len(fm['description'])} chars") + import re, sys, yaml, pathlib + for skill in (".claude/skills/peltier", ".agents/skills/peltier"): + p = pathlib.Path(skill) / "SKILL.md" + text = p.read_text(encoding="utf-8") + if not text.startswith("---\n"): + sys.exit(f"{p}: must open with a --- frontmatter block") + fm = yaml.safe_load(text.split("---", 2)[1]) # raises on malformed YAML + body = text.split("---", 2)[2] + for key in ("name", "description"): + if not isinstance(fm.get(key), str) or not fm[key].strip(): + sys.exit(f"{p}: frontmatter '{key}' must be a non-empty string") + # agentskills.io spec: 1-64 chars, [a-z0-9-], no edge/double hyphens, + # name == parent dir; description <= 1024 chars; body <= 500 lines. + n = fm["name"] + if not re.fullmatch(r"[a-z0-9]+(-[a-z0-9]+)*", n) or len(n) > 64: + sys.exit(f"{p}: name '{n}' violates the Agent Skills name rules") + if n != p.parent.name: + sys.exit(f"{p}: name '{n}' != directory '{p.parent.name}'") + if len(fm["description"]) > 1024: + sys.exit(f"{p}: description exceeds 1024 chars") + if len(body.splitlines()) > 500: + sys.exit(f"{p}: body exceeds 500 lines") + if "metadata" in fm and not ( + isinstance(fm["metadata"], dict) + and all(isinstance(k, str) and isinstance(v, str) for k, v in fm["metadata"].items()) + ): + sys.exit(f"{p}: metadata must be a string-to-string map") + print(f"{p}: spec OK (name={n}, description={len(fm['description'])} chars)") PY + - name: .agents mirror is byte-identical to .claude canonical + run: | + diff -r .claude/skills/peltier .agents/skills/peltier + echo "mirror identical" + + - name: Installer stamps both layouts + the ZeroClaw variant correctly + run: | + if sh scripts/install-skill.sh /tmp/no-such-dir 2>/dev/null; then + echo "FAIL: installer accepted a nonexistent target"; exit 1 + fi + mkdir -p /tmp/consumer && sh scripts/install-skill.sh /tmp/consumer + diff -r .claude/skills/peltier /tmp/consumer/.claude/skills/peltier + diff -r .claude/skills/peltier /tmp/consumer/.agents/skills/peltier + sh scripts/install-skill.sh --zeroclaw-variant /tmp/zc-variant + test ! -e /tmp/zc-variant/scripts # script-less: ZeroClaw blocks script files + test -f /tmp/zc-variant/SKILL.md + diff -r .claude/skills/peltier/references /tmp/zc-variant/references + echo "installer OK: both layouts identical to canonical; zeroclaw variant has no scripts/" + - name: Preflight resolves inside a peltier checkout run: | sh .claude/skills/peltier/scripts/preflight.sh | tee /tmp/pf.txt diff --git a/AGENTS.md b/AGENTS.md new file mode 100644 index 0000000..996a5d0 --- /dev/null +++ b/AGENTS.md @@ -0,0 +1,33 @@ +# peltier.io — agent entry point + +This file exists for AGENTS.md-reading harnesses (Codex, Jules, Cursor, Zed, +Amp, Copilot coding agent, Gemini CLI, …). It is deliberately thin: the +binding project rules live in one place, not here. + +## Working on this repo + +Read **[CLAUDE.md](CLAUDE.md)** first and treat it as binding regardless of +which harness you are. It holds the non-negotiables (no unverified performance +claims, measure-before-optimizing, hard equivalence gates, the append-only +ledger, the trust-layer write boundary) and the honest current state of every +subsystem. The engineering spec is [SPEC.md](SPEC.md). Commands live in the +[justfile](justfile). The pipeline is Linux/POSIX-only at runtime; Windows +compiles the workspace but cannot run it. + +Two hard rules that bear repeating even in a pointer file: + +- **Never claim a performance number without its confidence interval and + workload.** If you did not run the gates, the number does not exist. +- **The trust layer (`crates/`, `config/`, `corpora/`) is read-only to + optimization agents.** Patches touch only allowlisted paths under + `targets//`. + +## Verifying performance changes (any repo, any harness) + +This repo ships its trust layer as an Agent Skill: +**[.claude/skills/peltier/SKILL.md](.claude/skills/peltier/SKILL.md)** — +mirrored byte-identically at `.agents/skills/peltier/` for harnesses that +discover skills there (CI enforces the mirror). If your harness auto-loads +skills from either path, use it whenever a task involves proving or refuting +a speedup. If not, read that SKILL.md and follow it manually — starting with +`scripts/preflight.sh`, whose refusal is final. diff --git a/CLAUDE.md b/CLAUDE.md index 7b09c30..20eb0e5 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -29,7 +29,7 @@ Full engineering spec: `SPEC.md`. Read it before starting any phase. - `targets/` — vendored OSS targets for case studies (permissive licenses only) - `playbook/` — optimization classes, ordered, with preconditions and known risks - `results/` — generated reports and flamegraphs -- `.claude/skills/peltier/` — the trust layer packaged as a consumable Claude Code skill: `verify` mode (equivalence → A/A calibration → interleaved A/B + bootstrap CI) works in any repo against any two shell commands; `attempt` mode drives the full in-repo loop. Drives the real `bench-runner` binary and refuses on unsupported hosts rather than falling back to a second, forkable copy of the statistics. Preflight refusal is asserted in CI (`skill-preflight`). +- `.claude/skills/peltier/` — the trust layer packaged as a consumable **Agent Skill** (agentskills.io SKILL.md standard, spec-compliant frontmatter): `verify` mode (equivalence → A/A calibration → interleaved A/B + bootstrap CI) works in any repo against any two shell commands; `attempt` mode drives the full in-repo loop. Drives the real `bench-runner` binary and refuses on unsupported hosts rather than falling back to a second, forkable copy of the statistics. Cross-harness: mirrored byte-identically at `.agents/skills/peltier/` (the Codex/Zed/Cursor/Gemini/Copilot convergence path — CI enforces the mirror), `AGENTS.md` at the repo root points non-Claude harnesses here, and `scripts/install-skill.sh` stamps consumer repos (plus a script-less `--zeroclaw-variant`, since ZeroClaw blocks script files in skills by default — its preflight runs from the checkout). Preflight refusal, spec conformance, mirror equality, and the installer are all asserted in CI (`skill-preflight`). **Phase 4 COMPLETE — all SPEC §5 exit criteria met. Phases 0–4 are all done.** Service-mode latency benchmarking shipped: a coordinated-omission-correct, open-loop, fixed-rate load generator (`crates/bench-runner/src/service.rs`) measuring latency from each request's intended send time (queueing counted, not hidden), interleaved A/B, exact percentiles, bootstrap p50/p99 CIs. A real service target — `targets/cjson/service.c`, a minimal HTTP server wrapping the patched `cJSON.c` (trust-layer, outside the workspace allowlist). Service calibration PASS (0/10 A/A false-positive, 10/10 injected-5%-latency-regression detection; `results/calibration/cjson-service-aa.json`). **The verified batch win phase3-cjson-002 measured under a 150 rps-target replay (20 interleaved rounds, 0 drops — the `phase4-cjson-service` ledger row records sessions=20, rate_rps=150, drop_rate=0.0; total request count is not machine-recorded): p50 latency +6.2%, 95% CI [+5.8%, +7.2%] — accepted; p99 NOT claimed (CI [0.07, 4.97], single-worker loopback tail jitter) — harness correctly rejected it.** Mechanical ROI report generator (`crates/report` bin, `just report`, SPEC §9): reads a ledger row + service-latency JSON → throughput→cores→dollars (27.5 cores / $9,621-per-year CI lower bound on a 500-core fleet) *and* latency percentiles, every figure with CI + workload + methodology inline; flags any non-accepted row. Case study: `results/phase4/`. **The roadmap is complete — the profile→hypothesize→patch→gated-verdict→ROI loop runs end to end across Rust and C, batch and service, with zero shipped false accepts across all phases.** diff --git a/README.md b/README.md index 3750919..37e371b 100644 --- a/README.md +++ b/README.md @@ -71,24 +71,40 @@ tests, but cannot run the pipeline. Full command reference and reproduction steps: [Reproduce it yourself](https://bdbrown4.github.io/peltier.io/reproduce.html). -## Use it on your own code +## Use it on your own code — from any agent harness The trust layer is not specific to these targets — it will judge any two shell -commands. `.claude/skills/peltier/` packages it as a Claude Code skill: +commands. It ships as an **[Agent Skill](https://agentskills.io)** (the open +SKILL.md standard), which Claude Code, OpenAI Codex, GitHub Copilot, VS Code, +Cursor, Gemini CLI, opencode, Goose, Amp, Zed, and ZeroClaw all load natively. +One installer stamps it into your repo at both discovery paths: ```sh -# from any repo, with a peltier checkout reachable -export PELTIER_HOME=/path/to/peltier.io -mkdir -p .claude/skills -cp -r /path/to/peltier.io/.claude/skills/peltier .claude/skills/ +# from your repo, with a peltier checkout at /path/to/peltier.io +sh /path/to/peltier.io/scripts/install-skill.sh . +export PELTIER_HOME=/path/to/peltier.io # the skill drives bench-runner from here ``` -Then ask Claude to verify a speedup. The skill enforces the order that makes a -performance claim mean something: **equivalence before timing** (a faster -program that computes something different is a bug, not an optimization) → -**A/A calibration** (a host that "finds" a speedup between a binary and itself -cannot measure yours) → **interleaved A/B with a bootstrap CI** → a verdict -decided by the CI *lower bound*, never the median. +That writes `.claude/skills/peltier/` (Claude Code; also read by Copilot, +VS Code, opencode, Amp) and `.agents/skills/peltier/` (the cross-tool path: +Codex, Zed, Cursor, Gemini CLI, Goose, and others). This repo carries both, +CI-enforced byte-identical, so the two copies cannot drift. + +**ZeroClaw** blocks script files inside skills by default, so it gets a +script-less variant (preflight runs from your peltier checkout instead — +same one copy of the script): + +```sh +sh /path/to/peltier.io/scripts/install-skill.sh --zeroclaw-variant /tmp/peltier-skill +zeroclaw skills install /tmp/peltier-skill +``` + +Then ask your agent to verify a speedup. The skill enforces the order that +makes a performance claim mean something: **equivalence before timing** (a +faster program that computes something different is a bug, not an +optimization) → **A/A calibration** (a host that "finds" a speedup between a +binary and itself cannot measure yours) → **interleaved A/B with a bootstrap +CI** → a verdict decided by the CI *lower bound*, never the median. It drives the real `bench-runner` binary and **refuses rather than degrade**: on an unsupported host, or with no trust layer reachable, it stops and says so diff --git a/scripts/install-skill.sh b/scripts/install-skill.sh new file mode 100755 index 0000000..986fb43 --- /dev/null +++ b/scripts/install-skill.sh @@ -0,0 +1,86 @@ +#!/bin/sh +# install-skill.sh — stamp the peltier Agent Skill into a consumer repo. +# +# sh scripts/install-skill.sh [--layout claude|agents|both] +# sh scripts/install-skill.sh --zeroclaw-variant +# +# The skill is one directory (SKILL.md + references/ + scripts/ + assets/), +# already in the Agent Skills format (agentskills.io) that Claude Code, Codex, +# Copilot, VS Code, Cursor, Gemini CLI, opencode, Goose, Amp, Zed and others +# load natively. All this script does is copy that one canonical directory to +# the discovery path(s) a harness actually reads: +# +# .claude/skills/peltier/ Claude Code (also read by Copilot, VS Code, +# opencode, Amp, and Cursor in compat mode) +# .agents/skills/peltier/ the cross-tool convergence path (Codex, Zed, +# Cursor, Copilot, VS Code, Gemini CLI, opencode, +# Goose, Amp) +# +# --layout both (the default) writes both. That is two copies in YOUR repo; +# keep them identical (re-run this script to update both; peltier's own CI +# shows the diff -r pattern for enforcing it). +# +# --zeroclaw-variant writes a copy WITHOUT scripts/ to : ZeroClaw's +# skill security policy blocks script files by default (skills.allow_scripts +# is off), so its variant runs preflight from the peltier checkout instead — +# the skill's SKILL.md already explains that path. Then install it with: +# zeroclaw skills install +# +# The skill still needs a peltier checkout at runtime (bench-runner lives +# there). Set PELTIER_HOME in the environment your agent runs in. +set -eu + +usage() { + sed -n '2,30p' "$0" | sed 's/^# \{0,1\}//' + exit 2 +} + +here=$(CDPATH= cd -- "$(dirname -- "$0")" && pwd) +src="$here/../.claude/skills/peltier" +[ -f "$src/SKILL.md" ] || { echo "install-skill: cannot find the skill at $src" >&2; exit 1; } + +copy_skill() { # $1 = dest dir, $2 = "full" | "noscripts" + rm -rf "$1" + mkdir -p "$1" + cp -R "$src/." "$1/" + if [ "$2" = "noscripts" ]; then rm -rf "$1/scripts"; fi +} + +case "${1:-}" in + ""|-h|--help) usage ;; + --zeroclaw-variant) + out=${2:-}; [ -n "$out" ] || usage + copy_skill "$out" noscripts + echo "wrote script-less variant to $out (ZeroClaw blocks script files in skills by default)" + echo "next: zeroclaw skills install \"$out\" # optionally --bundle , then add the" + echo " bundle to agents..skill_bundles so your agent loads it" + echo "note: at runtime the skill runs preflight from your peltier checkout —" + echo " set PELTIER_HOME to that checkout for the agent's environment" + exit 0 + ;; +esac + +target=$1; shift +layout=both +while [ $# -gt 0 ]; do + case "$1" in + --layout) layout=${2:-}; shift 2 ;; + *) usage ;; + esac +done +case "$layout" in claude|agents|both) ;; *) usage ;; esac +[ -d "$target" ] || { echo "install-skill: target repo '$target' is not a directory" >&2; exit 1; } + +wrote="" +if [ "$layout" = claude ] || [ "$layout" = both ]; then + copy_skill "$target/.claude/skills/peltier" full + wrote="$wrote $target/.claude/skills/peltier" +fi +if [ "$layout" = agents ] || [ "$layout" = both ]; then + copy_skill "$target/.agents/skills/peltier" full + wrote="$wrote $target/.agents/skills/peltier" +fi + +echo "installed:$wrote" +echo "next: set PELTIER_HOME=/path/to/peltier.io in the environment your agent runs in" +echo " (the skill drives the real bench-runner from that checkout; preflight refuses without it)" From 7c2177c5c69f55765d4aea860e5217296f14aa56 Mon Sep 17 00:00:00 2001 From: Ben Brown Date: Wed, 15 Jul 2026 11:59:47 -0400 Subject: [PATCH 2/2] skill: executable bit on the mirrored preflight.sh MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The .agents mirror staged preflight.sh as a new path, so it missed the canonical copy's 755. diff -r compares content, not modes — index modes must match too. Co-Authored-By: Claude Fable 5 --- .agents/skills/peltier/scripts/preflight.sh | 0 1 file changed, 0 insertions(+), 0 deletions(-) mode change 100644 => 100755 .agents/skills/peltier/scripts/preflight.sh diff --git a/.agents/skills/peltier/scripts/preflight.sh b/.agents/skills/peltier/scripts/preflight.sh old mode 100644 new mode 100755