mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
skill: expose the trust layer as a consumable peltier skill
The reusable artifact here was never the vendored targets — it is the discipline: prove equivalence, prove significance, or claim nothing. This packages it as a Claude Code skill usable from any repo. Two modes. `verify` works anywhere, on any two shell commands in any language: equivalence before timing -> A/A calibration -> interleaved A/B with a bootstrap CI -> a verdict decided by the CI lower bound, never the median. `attempt` drives the full in-repo loop (gates, pristine-baseline verdict, ledger row, ROI report). It drives the real bench-runner binary and refuses rather than degrade: on an unsupported host, or with no trust layer reachable, preflight stops and says so instead of falling back to a hand-rolled timing loop. A second copy of the statistics is a second thing that can lie. A refusal ends the claim, not the work. An adversarial review of the skill caught six high-severity defects, all fixed here — several of them the very errors the skill exists to prevent: - verify mode had no risk classification at all. bench-runner emits only accepted/rejected-bench; needs-human-review is minted by the in-repo verdict path, which verify does not run. A concurrency change with byte-identical output would have sailed through as "accepted". The skill now makes the agent the classifier and says so explicitly. - the accept template exposed `pin_prefix`, which is inert for compare/aa/calibrate (only `verdict` reads it) — an agent would have set it and reported pinning that never happened. Now documents wrapping both sides in taskset instead. - the equivalence gate compared stdout only, so two commands that write to stderr and a dist/ tree would "GOLDEN PASS" on two empty files. Now hashes stdout + stderr + exit code + artifact tree, and an empty diff over empty output is a failed gate, not a passed one. - nothing told the agent to build two coexisting immutable artifacts, so a `cargo run` baseline could rebuild from the patched tree mid-bench — the exact trap phase1-verdict-smoke-001 recorded. - service mode was documented as generic HTTP. It is not: the server must take <port> <doc> <iters>, print READY, and close the connection. A stock Express app hangs the harness forever. Contract now stated. - the CI refusal test could not pass: preflight also walks up from its own location, so running it in-place found this very repo. The test now copies the skill out of the tree first. CI job `skill-preflight` asserts the contract that matters — resolve when a trust layer is reachable, REFUSE when it is not, and work when consumed from another repo via PELTIER_HOME. It deliberately does not assert a statistical outcome: a shared runner is too noisy for that, which is precisely why the skill calibrates before it claims. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
@@ -91,3 +91,60 @@ jobs:
|
||||
$PERF record -e cpu-clock -o /tmp/p.data -- sh -c 'i=0; while [ $i -lt 2000000 ]; do i=$((i+1)); done' && $PERF report -i /tmp/p.data --stdio | head -5 || echo "cpu-clock sampling unavailable"
|
||||
echo "--- hardware cycle counter:"
|
||||
$PERF stat -e cycles true 2>&1 | tail -3 || true
|
||||
|
||||
# The peltier skill's contract: preflight must resolve the trust layer when
|
||||
# one is reachable, and REFUSE when it is not. The refusal is the
|
||||
# safety-critical path — a skill that silently degrades to a hand-rolled
|
||||
# timing loop is worse than no skill — so it is asserted here, not assumed.
|
||||
# Statistical outcomes are NOT asserted: a shared runner is too noisy for
|
||||
# that, which is the whole reason the skill calibrates before it claims.
|
||||
skill-preflight:
|
||||
runs-on: ubuntu-latest
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- uses: dtolnay/rust-toolchain@stable
|
||||
- uses: Swatinem/rust-cache@v2
|
||||
|
||||
- name: Preflight resolves inside a peltier checkout
|
||||
run: |
|
||||
sh .claude/skills/peltier/scripts/preflight.sh | tee /tmp/pf.txt
|
||||
grep -q '^STATUS=ok$' /tmp/pf.txt
|
||||
grep -q "^PELTIER_HOME=$PWD$" /tmp/pf.txt
|
||||
BR=$(sed -n 's/^BENCH_RUNNER=//p' /tmp/pf.txt); test -x "$BR"
|
||||
|
||||
- name: Preflight REFUSES with no trust layer reachable
|
||||
run: |
|
||||
# The skill must be copied OUT of the checkout: preflight also walks
|
||||
# up from its own location, so running it in-place would find this
|
||||
# very repo and (correctly) resolve — testing nothing.
|
||||
cp -r .claude/skills/peltier /tmp/skillcopy
|
||||
cd /tmp && mkdir -p not-a-repo && cd not-a-repo
|
||||
set +e
|
||||
env -u PELTIER_HOME sh /tmp/skillcopy/scripts/preflight.sh > out.txt 2>&1
|
||||
rc=$?
|
||||
set -e
|
||||
cat out.txt
|
||||
test "$rc" -ne 0 || { echo "FAIL: preflight did not refuse"; exit 1; }
|
||||
grep -q '^STATUS=refuse$' out.txt
|
||||
|
||||
- name: Preflight works when consumed from another repo (PELTIER_HOME)
|
||||
run: |
|
||||
cd /tmp && mkdir -p other-project && cd other-project
|
||||
PELTIER_HOME="$GITHUB_WORKSPACE" \
|
||||
sh "$GITHUB_WORKSPACE/.claude/skills/peltier/scripts/preflight.sh" | tee pf.txt
|
||||
grep -q '^STATUS=ok$' pf.txt
|
||||
|
||||
- name: The harness the skill drives actually runs (plumbing, not verdict)
|
||||
run: |
|
||||
BR=$(sed -n 's/^BENCH_RUNNER=//p' /tmp/pf.txt)
|
||||
cp .claude/skills/peltier/assets/accept.template.toml /tmp/accept.toml
|
||||
"$BR" --config /tmp/accept.toml compare \
|
||||
--baseline 'sleep 0.02' --candidate 'sleep 0.02' | tee /tmp/cmp.txt
|
||||
grep -q 'speedup (baseline/candidate): median .*95% CI \[' /tmp/cmp.txt
|
||||
grep -q '^verdict: ' /tmp/cmp.txt
|
||||
|
||||
- name: Noise floor of this runner (informational — A/A must find nothing)
|
||||
continue-on-error: true
|
||||
run: |
|
||||
BR=$(sed -n 's/^BENCH_RUNNER=//p' /tmp/pf.txt)
|
||||
"$BR" --config /tmp/accept.toml aa --cmd 'sleep 0.02'
|
||||
|
||||
Reference in New Issue
Block a user