skill: expose the trust layer as a consumable peltier skill

The reusable artifact here was never the vendored targets — it is the
discipline: prove equivalence, prove significance, or claim nothing. This
packages it as a Claude Code skill usable from any repo.

Two modes. `verify` works anywhere, on any two shell commands in any
language: equivalence before timing -> A/A calibration -> interleaved A/B
with a bootstrap CI -> a verdict decided by the CI lower bound, never the
median. `attempt` drives the full in-repo loop (gates, pristine-baseline
verdict, ledger row, ROI report).

It drives the real bench-runner binary and refuses rather than degrade: on
an unsupported host, or with no trust layer reachable, preflight stops and
says so instead of falling back to a hand-rolled timing loop. A second copy
of the statistics is a second thing that can lie. A refusal ends the claim,
not the work.

An adversarial review of the skill caught six high-severity defects, all
fixed here — several of them the very errors the skill exists to prevent:

- verify mode had no risk classification at all. bench-runner emits only
  accepted/rejected-bench; needs-human-review is minted by the in-repo
  verdict path, which verify does not run. A concurrency change with
  byte-identical output would have sailed through as "accepted". The skill
  now makes the agent the classifier and says so explicitly.
- the accept template exposed `pin_prefix`, which is inert for
  compare/aa/calibrate (only `verdict` reads it) — an agent would have set
  it and reported pinning that never happened. Now documents wrapping both
  sides in taskset instead.
- the equivalence gate compared stdout only, so two commands that write to
  stderr and a dist/ tree would "GOLDEN PASS" on two empty files. Now hashes
  stdout + stderr + exit code + artifact tree, and an empty diff over empty
  output is a failed gate, not a passed one.
- nothing told the agent to build two coexisting immutable artifacts, so a
  `cargo run` baseline could rebuild from the patched tree mid-bench — the
  exact trap phase1-verdict-smoke-001 recorded.
- service mode was documented as generic HTTP. It is not: the server must
  take <port> <doc> <iters>, print READY, and close the connection. A stock
  Express app hangs the harness forever. Contract now stated.
- the CI refusal test could not pass: preflight also walks up from its own
  location, so running it in-place found this very repo. The test now copies
  the skill out of the tree first.

CI job `skill-preflight` asserts the contract that matters — resolve when a
trust layer is reachable, REFUSE when it is not, and work when consumed from
another repo via PELTIER_HOME. It deliberately does not assert a statistical
outcome: a shared runner is too noisy for that, which is precisely why the
skill calibrates before it claims.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
2026-07-14 13:39:38 -04:00
co-authored by Claude Opus 4.8
parent 899d55f4b6
commit af3fc3dccd
8 changed files with 645 additions and 0 deletions
+57
View File
@@ -91,3 +91,60 @@ jobs:
$PERF record -e cpu-clock -o /tmp/p.data -- sh -c 'i=0; while [ $i -lt 2000000 ]; do i=$((i+1)); done' && $PERF report -i /tmp/p.data --stdio | head -5 || echo "cpu-clock sampling unavailable"
echo "--- hardware cycle counter:"
$PERF stat -e cycles true 2>&1 | tail -3 || true
# The peltier skill's contract: preflight must resolve the trust layer when
# one is reachable, and REFUSE when it is not. The refusal is the
# safety-critical path — a skill that silently degrades to a hand-rolled
# timing loop is worse than no skill — so it is asserted here, not assumed.
# Statistical outcomes are NOT asserted: a shared runner is too noisy for
# that, which is the whole reason the skill calibrates before it claims.
skill-preflight:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
- uses: Swatinem/rust-cache@v2
- name: Preflight resolves inside a peltier checkout
run: |
sh .claude/skills/peltier/scripts/preflight.sh | tee /tmp/pf.txt
grep -q '^STATUS=ok$' /tmp/pf.txt
grep -q "^PELTIER_HOME=$PWD$" /tmp/pf.txt
BR=$(sed -n 's/^BENCH_RUNNER=//p' /tmp/pf.txt); test -x "$BR"
- name: Preflight REFUSES with no trust layer reachable
run: |
# The skill must be copied OUT of the checkout: preflight also walks
# up from its own location, so running it in-place would find this
# very repo and (correctly) resolve — testing nothing.
cp -r .claude/skills/peltier /tmp/skillcopy
cd /tmp && mkdir -p not-a-repo && cd not-a-repo
set +e
env -u PELTIER_HOME sh /tmp/skillcopy/scripts/preflight.sh > out.txt 2>&1
rc=$?
set -e
cat out.txt
test "$rc" -ne 0 || { echo "FAIL: preflight did not refuse"; exit 1; }
grep -q '^STATUS=refuse$' out.txt
- name: Preflight works when consumed from another repo (PELTIER_HOME)
run: |
cd /tmp && mkdir -p other-project && cd other-project
PELTIER_HOME="$GITHUB_WORKSPACE" \
sh "$GITHUB_WORKSPACE/.claude/skills/peltier/scripts/preflight.sh" | tee pf.txt
grep -q '^STATUS=ok$' pf.txt
- name: The harness the skill drives actually runs (plumbing, not verdict)
run: |
BR=$(sed -n 's/^BENCH_RUNNER=//p' /tmp/pf.txt)
cp .claude/skills/peltier/assets/accept.template.toml /tmp/accept.toml
"$BR" --config /tmp/accept.toml compare \
--baseline 'sleep 0.02' --candidate 'sleep 0.02' | tee /tmp/cmp.txt
grep -q 'speedup (baseline/candidate): median .*95% CI \[' /tmp/cmp.txt
grep -q '^verdict: ' /tmp/cmp.txt
- name: Noise floor of this runner (informational — A/A must find nothing)
continue-on-error: true
run: |
BR=$(sed -n 's/^BENCH_RUNNER=//p' /tmp/pf.txt)
"$BR" --config /tmp/accept.toml aa --cmd 'sleep 0.02'