mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
The reusable artifact here was never the vendored targets — it is the discipline: prove equivalence, prove significance, or claim nothing. This packages it as a Claude Code skill usable from any repo. Two modes. `verify` works anywhere, on any two shell commands in any language: equivalence before timing -> A/A calibration -> interleaved A/B with a bootstrap CI -> a verdict decided by the CI lower bound, never the median. `attempt` drives the full in-repo loop (gates, pristine-baseline verdict, ledger row, ROI report). It drives the real bench-runner binary and refuses rather than degrade: on an unsupported host, or with no trust layer reachable, preflight stops and says so instead of falling back to a hand-rolled timing loop. A second copy of the statistics is a second thing that can lie. A refusal ends the claim, not the work. An adversarial review of the skill caught six high-severity defects, all fixed here — several of them the very errors the skill exists to prevent: - verify mode had no risk classification at all. bench-runner emits only accepted/rejected-bench; needs-human-review is minted by the in-repo verdict path, which verify does not run. A concurrency change with byte-identical output would have sailed through as "accepted". The skill now makes the agent the classifier and says so explicitly. - the accept template exposed `pin_prefix`, which is inert for compare/aa/calibrate (only `verdict` reads it) — an agent would have set it and reported pinning that never happened. Now documents wrapping both sides in taskset instead. - the equivalence gate compared stdout only, so two commands that write to stderr and a dist/ tree would "GOLDEN PASS" on two empty files. Now hashes stdout + stderr + exit code + artifact tree, and an empty diff over empty output is a failed gate, not a passed one. - nothing told the agent to build two coexisting immutable artifacts, so a `cargo run` baseline could rebuild from the patched tree mid-bench — the exact trap phase1-verdict-smoke-001 recorded. - service mode was documented as generic HTTP. It is not: the server must take <port> <doc> <iters>, print READY, and close the connection. A stock Express app hangs the harness forever. Contract now stated. - the CI refusal test could not pass: preflight also walks up from its own location, so running it in-place found this very repo. The test now copies the skill out of the tree first. CI job `skill-preflight` asserts the contract that matters — resolve when a trust layer is reachable, REFUSE when it is not, and work when consumed from another repo via PELTIER_HOME. It deliberately does not assert a statistical outcome: a shared runner is too noisy for that, which is precisely why the skill calibrates before it claims. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
151 lines
6.4 KiB
YAML
151 lines
6.4 KiB
YAML
name: ci
|
|
on:
|
|
pull_request:
|
|
push:
|
|
branches: [main]
|
|
|
|
jobs:
|
|
trust-layer:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- uses: dtolnay/rust-toolchain@stable
|
|
- uses: Swatinem/rust-cache@v2
|
|
- run: cargo test --workspace
|
|
- run: cargo clippy --workspace --all-targets -- -D warnings
|
|
- run: cargo fmt --all --check
|
|
|
|
# Phase 1: target gates run mechanically, and the bench workload
|
|
# executes inside a NO-NETWORK container (SPEC §3.1/§10). The runner
|
|
# is a shared VM — fine for gates; bench numbers for verdicts come
|
|
# from calibrated hardware, not this job.
|
|
target-gates-no-net:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- uses: dtolnay/rust-toolchain@stable
|
|
- uses: Swatinem/rust-cache@v2
|
|
- name: Toolchains for C/C++ targets
|
|
run: sudo apt-get update -qq && sudo apt-get install -y -qq clang cmake
|
|
- name: Fetch pinned targets
|
|
run: |
|
|
sh targets/fetch.sh comrak
|
|
sh targets/fetch.sh tokei
|
|
sh targets/fetch.sh cjson
|
|
- name: Generate + verify pinned corpora
|
|
run: |
|
|
sh corpora/comrak/gen-corpus.sh
|
|
sh corpora/tokei/gen-corpus.sh
|
|
sh corpora/cjson/gen-corpus.sh
|
|
- name: Build tokei baseline
|
|
run: cargo build --release --manifest-path targets/tokei/workspace/Cargo.toml
|
|
- name: Gates — tokei (Rust; corpus pin -> upstream tests -> golden replay)
|
|
run: cargo run -p diff-test -- tokei
|
|
- name: Gates — cjson (C; clang build -> Unity/CTest -> golden replay)
|
|
run: cargo run -p diff-test -- cjson
|
|
- name: Bench workload inside no-network container
|
|
run: |
|
|
docker run --rm --network=none \
|
|
-v "$PWD:/w" -w /w \
|
|
debian:bookworm-slim \
|
|
sh -c "targets/tokei/workspace/target/release/tokei --sort code corpora/tokei/input | sha256sum"
|
|
|
|
# Informational: measure THIS runner's noise floor with the A/A +
|
|
# injection harness and archive the evidence. A shared VM failing the
|
|
# bars is expected — the artifact is the point (trust layer judges
|
|
# environments empirically; verdict benches belong on bench-metal).
|
|
runner-calibration:
|
|
runs-on: ubuntu-latest
|
|
continue-on-error: true
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- uses: dtolnay/rust-toolchain@stable
|
|
- uses: Swatinem/rust-cache@v2
|
|
- name: Calibrate (reduced sessions, synthetic CPU-bound workload)
|
|
run: |
|
|
mkdir -p calib
|
|
cargo run -p bench-runner -- calibrate \
|
|
--cmd "head -c 3000000 /dev/urandom | sha256sum" \
|
|
--sessions 20 --out calib/gh-runner-calibration.json \
|
|
| tee calib/summary.txt
|
|
- uses: actions/upload-artifact@v4
|
|
if: always()
|
|
with:
|
|
name: runner-calibration
|
|
path: calib/
|
|
|
|
# Informational: does perf work here, and with which event sources?
|
|
perf-probe:
|
|
runs-on: ubuntu-latest
|
|
continue-on-error: true
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- name: Probe perf availability
|
|
run: |
|
|
sudo apt-get -qq update && sudo apt-get -qq install -y linux-tools-common linux-tools-generic || true
|
|
PERF=$(ls /usr/lib/linux-tools/*/perf 2>/dev/null | head -1 || command -v perf || true)
|
|
echo "perf binary: ${PERF:-none}"
|
|
[ -n "$PERF" ] || exit 0
|
|
sudo sysctl -w kernel.perf_event_paranoid=1 || true
|
|
echo "--- software clock sampling:"
|
|
$PERF record -e cpu-clock -o /tmp/p.data -- sh -c 'i=0; while [ $i -lt 2000000 ]; do i=$((i+1)); done' && $PERF report -i /tmp/p.data --stdio | head -5 || echo "cpu-clock sampling unavailable"
|
|
echo "--- hardware cycle counter:"
|
|
$PERF stat -e cycles true 2>&1 | tail -3 || true
|
|
|
|
# The peltier skill's contract: preflight must resolve the trust layer when
|
|
# one is reachable, and REFUSE when it is not. The refusal is the
|
|
# safety-critical path — a skill that silently degrades to a hand-rolled
|
|
# timing loop is worse than no skill — so it is asserted here, not assumed.
|
|
# Statistical outcomes are NOT asserted: a shared runner is too noisy for
|
|
# that, which is the whole reason the skill calibrates before it claims.
|
|
skill-preflight:
|
|
runs-on: ubuntu-latest
|
|
steps:
|
|
- uses: actions/checkout@v4
|
|
- uses: dtolnay/rust-toolchain@stable
|
|
- uses: Swatinem/rust-cache@v2
|
|
|
|
- name: Preflight resolves inside a peltier checkout
|
|
run: |
|
|
sh .claude/skills/peltier/scripts/preflight.sh | tee /tmp/pf.txt
|
|
grep -q '^STATUS=ok$' /tmp/pf.txt
|
|
grep -q "^PELTIER_HOME=$PWD$" /tmp/pf.txt
|
|
BR=$(sed -n 's/^BENCH_RUNNER=//p' /tmp/pf.txt); test -x "$BR"
|
|
|
|
- name: Preflight REFUSES with no trust layer reachable
|
|
run: |
|
|
# The skill must be copied OUT of the checkout: preflight also walks
|
|
# up from its own location, so running it in-place would find this
|
|
# very repo and (correctly) resolve — testing nothing.
|
|
cp -r .claude/skills/peltier /tmp/skillcopy
|
|
cd /tmp && mkdir -p not-a-repo && cd not-a-repo
|
|
set +e
|
|
env -u PELTIER_HOME sh /tmp/skillcopy/scripts/preflight.sh > out.txt 2>&1
|
|
rc=$?
|
|
set -e
|
|
cat out.txt
|
|
test "$rc" -ne 0 || { echo "FAIL: preflight did not refuse"; exit 1; }
|
|
grep -q '^STATUS=refuse$' out.txt
|
|
|
|
- name: Preflight works when consumed from another repo (PELTIER_HOME)
|
|
run: |
|
|
cd /tmp && mkdir -p other-project && cd other-project
|
|
PELTIER_HOME="$GITHUB_WORKSPACE" \
|
|
sh "$GITHUB_WORKSPACE/.claude/skills/peltier/scripts/preflight.sh" | tee pf.txt
|
|
grep -q '^STATUS=ok$' pf.txt
|
|
|
|
- name: The harness the skill drives actually runs (plumbing, not verdict)
|
|
run: |
|
|
BR=$(sed -n 's/^BENCH_RUNNER=//p' /tmp/pf.txt)
|
|
cp .claude/skills/peltier/assets/accept.template.toml /tmp/accept.toml
|
|
"$BR" --config /tmp/accept.toml compare \
|
|
--baseline 'sleep 0.02' --candidate 'sleep 0.02' | tee /tmp/cmp.txt
|
|
grep -q 'speedup (baseline/candidate): median .*95% CI \[' /tmp/cmp.txt
|
|
grep -q '^verdict: ' /tmp/cmp.txt
|
|
|
|
- name: Noise floor of this runner (informational — A/A must find nothing)
|
|
continue-on-error: true
|
|
run: |
|
|
BR=$(sed -n 's/^BENCH_RUNNER=//p' /tmp/pf.txt)
|
|
"$BR" --config /tmp/accept.toml aa --cmd 'sleep 0.02'
|