Files
benbrownandClaude Opus 4.8 db5caf0b5d skill: fix unloadable SKILL.md frontmatter, and test it in CI
The description was an unquoted YAML scalar containing "trust layer:
equivalence" — a colon-space inside a plain scalar makes YAML parse it as a
nested mapping, so the frontmatter raised "mapping values are not allowed in
this context" and the skill would not load at all.

Quoted the scalar and replaced the colon with an em dash.

The real defect is that CI asserted the preflight script's behavior but never
checked that the skill it ships is loadable. Added a step that parses the
frontmatter, requires name/description to be non-empty strings, and requires
name to match the directory — so an unloadable skill fails the build instead
of failing the user.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
2026-07-14 13:59:59 -04:00

168 lines
7.3 KiB
YAML

name: ci
on:
pull_request:
push:
branches: [main]
jobs:
trust-layer:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
- uses: Swatinem/rust-cache@v2
- run: cargo test --workspace
- run: cargo clippy --workspace --all-targets -- -D warnings
- run: cargo fmt --all --check
# Phase 1: target gates run mechanically, and the bench workload
# executes inside a NO-NETWORK container (SPEC §3.1/§10). The runner
# is a shared VM — fine for gates; bench numbers for verdicts come
# from calibrated hardware, not this job.
target-gates-no-net:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
- uses: Swatinem/rust-cache@v2
- name: Toolchains for C/C++ targets
run: sudo apt-get update -qq && sudo apt-get install -y -qq clang cmake
- name: Fetch pinned targets
run: |
sh targets/fetch.sh comrak
sh targets/fetch.sh tokei
sh targets/fetch.sh cjson
- name: Generate + verify pinned corpora
run: |
sh corpora/comrak/gen-corpus.sh
sh corpora/tokei/gen-corpus.sh
sh corpora/cjson/gen-corpus.sh
- name: Build tokei baseline
run: cargo build --release --manifest-path targets/tokei/workspace/Cargo.toml
- name: Gates — tokei (Rust; corpus pin -> upstream tests -> golden replay)
run: cargo run -p diff-test -- tokei
- name: Gates — cjson (C; clang build -> Unity/CTest -> golden replay)
run: cargo run -p diff-test -- cjson
- name: Bench workload inside no-network container
run: |
docker run --rm --network=none \
-v "$PWD:/w" -w /w \
debian:bookworm-slim \
sh -c "targets/tokei/workspace/target/release/tokei --sort code corpora/tokei/input | sha256sum"
# Informational: measure THIS runner's noise floor with the A/A +
# injection harness and archive the evidence. A shared VM failing the
# bars is expected — the artifact is the point (trust layer judges
# environments empirically; verdict benches belong on bench-metal).
runner-calibration:
runs-on: ubuntu-latest
continue-on-error: true
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
- uses: Swatinem/rust-cache@v2
- name: Calibrate (reduced sessions, synthetic CPU-bound workload)
run: |
mkdir -p calib
cargo run -p bench-runner -- calibrate \
--cmd "head -c 3000000 /dev/urandom | sha256sum" \
--sessions 20 --out calib/gh-runner-calibration.json \
| tee calib/summary.txt
- uses: actions/upload-artifact@v4
if: always()
with:
name: runner-calibration
path: calib/
# Informational: does perf work here, and with which event sources?
perf-probe:
runs-on: ubuntu-latest
continue-on-error: true
steps:
- uses: actions/checkout@v4
- name: Probe perf availability
run: |
sudo apt-get -qq update && sudo apt-get -qq install -y linux-tools-common linux-tools-generic || true
PERF=$(ls /usr/lib/linux-tools/*/perf 2>/dev/null | head -1 || command -v perf || true)
echo "perf binary: ${PERF:-none}"
[ -n "$PERF" ] || exit 0
sudo sysctl -w kernel.perf_event_paranoid=1 || true
echo "--- software clock sampling:"
$PERF record -e cpu-clock -o /tmp/p.data -- sh -c 'i=0; while [ $i -lt 2000000 ]; do i=$((i+1)); done' && $PERF report -i /tmp/p.data --stdio | head -5 || echo "cpu-clock sampling unavailable"
echo "--- hardware cycle counter:"
$PERF stat -e cycles true 2>&1 | tail -3 || true
# The peltier skill's contract: preflight must resolve the trust layer when
# one is reachable, and REFUSE when it is not. The refusal is the
# safety-critical path — a skill that silently degrades to a hand-rolled
# timing loop is worse than no skill — so it is asserted here, not assumed.
# Statistical outcomes are NOT asserted: a shared runner is too noisy for
# that, which is the whole reason the skill calibrates before it claims.
skill-preflight:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- uses: dtolnay/rust-toolchain@stable
- uses: Swatinem/rust-cache@v2
- name: SKILL.md frontmatter is loadable
run: |
python3 - <<'PY'
import sys, yaml, pathlib
p = pathlib.Path(".claude/skills/peltier/SKILL.md")
text = p.read_text(encoding="utf-8")
if not text.startswith("---\n"):
sys.exit("SKILL.md must open with a --- frontmatter block")
fm = yaml.safe_load(text.split("---", 2)[1]) # raises on malformed YAML
for key in ("name", "description"):
if not isinstance(fm.get(key), str) or not fm[key].strip():
sys.exit(f"frontmatter '{key}' must be a non-empty string")
if fm["name"] != p.parent.name:
sys.exit(f"frontmatter name '{fm['name']}' != directory '{p.parent.name}'")
print(f"frontmatter OK: name={fm['name']}, description={len(fm['description'])} chars")
PY
- name: Preflight resolves inside a peltier checkout
run: |
sh .claude/skills/peltier/scripts/preflight.sh | tee /tmp/pf.txt
grep -q '^STATUS=ok$' /tmp/pf.txt
grep -q "^PELTIER_HOME=$PWD$" /tmp/pf.txt
BR=$(sed -n 's/^BENCH_RUNNER=//p' /tmp/pf.txt); test -x "$BR"
- name: Preflight REFUSES with no trust layer reachable
run: |
# The skill must be copied OUT of the checkout: preflight also walks
# up from its own location, so running it in-place would find this
# very repo and (correctly) resolve — testing nothing.
cp -r .claude/skills/peltier /tmp/skillcopy
cd /tmp && mkdir -p not-a-repo && cd not-a-repo
set +e
env -u PELTIER_HOME sh /tmp/skillcopy/scripts/preflight.sh > out.txt 2>&1
rc=$?
set -e
cat out.txt
test "$rc" -ne 0 || { echo "FAIL: preflight did not refuse"; exit 1; }
grep -q '^STATUS=refuse$' out.txt
- name: Preflight works when consumed from another repo (PELTIER_HOME)
run: |
cd /tmp && mkdir -p other-project && cd other-project
PELTIER_HOME="$GITHUB_WORKSPACE" \
sh "$GITHUB_WORKSPACE/.claude/skills/peltier/scripts/preflight.sh" | tee pf.txt
grep -q '^STATUS=ok$' pf.txt
- name: The harness the skill drives actually runs (plumbing, not verdict)
run: |
BR=$(sed -n 's/^BENCH_RUNNER=//p' /tmp/pf.txt)
cp .claude/skills/peltier/assets/accept.template.toml /tmp/accept.toml
"$BR" --config /tmp/accept.toml compare \
--baseline 'sleep 0.02' --candidate 'sleep 0.02' | tee /tmp/cmp.txt
grep -q 'speedup (baseline/candidate): median .*95% CI \[' /tmp/cmp.txt
grep -q '^verdict: ' /tmp/cmp.txt
- name: Noise floor of this runner (informational — A/A must find nothing)
continue-on-error: true
run: |
BR=$(sed -n 's/^BENCH_RUNNER=//p' /tmp/pf.txt)
"$BR" --config /tmp/accept.toml aa --cmd 'sleep 0.02'