The dispatch rule covered sh-then-ps1 but left two edges implicit: a host
with NEITHER launcher (now explicitly a refusal to report, not a license
to improvise), and the script-less install on a Windows host — the
ZeroClaw-on-Windows combination — which needs the ps1-from-checkout
command spelled out, not derived.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
'Use whichever your host can launch' left the selection order implicit and
did not distinguish the two failure kinds. Now stated: dispatch on launcher
availability (not OS) — sh first, ps1 only when the sh launcher itself is
missing; and a STATUS=refuse from either entry point is final, never a cue
to try the other one for a different answer. The CI byte-diff exists
precisely so there is no different answer.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
GitHub's PowerShell step wrapper appends 'exit $LASTEXITCODE', so the
inner preflight's deliberate exit 1 — the exact outcome the step had just
asserted — leaked out as the step's own failure. Clear it with an explicit
exit 0 once the refusal is verified. The 5.1 log from the failing run shows
the ps1 behaving perfectly: STATUS=refuse, correctly composed em dash,
'unsupported host Windows' reason.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
On a Windows host the preflight's job is to deliver a structured refusal
(measurement is Linux/POSIX-only). But a Windows-native harness with no
POSIX shell never got that far: preflight.sh fails to LAUNCH, and an agent
staring at "sh: not found" is exactly the agent that hand-rolls a timing
loop. The refusal doctrine has to be as portable as the skill.
- scripts/preflight.ps1: faithful port of the preflight contract, runs on
Windows PowerShell 5.1 and pwsh 7+ on any OS. Host check first (so
Windows refuses before touching anything), same walk-up resolution, same
always-build rule, same key=value output, same exit codes. The source is
pure ASCII with the shared em dashes composed at runtime — Windows
PowerShell 5.1 misdecodes BOM-less UTF-8, and that exact bug class
already bit this machine once (the francis-watchdog parse failure).
- Two implementations of one contract is a drift risk, so it is gated like
every other claim in this repo: the skill-preflight CI job now runs BOTH
scripts on the same ubuntu host and byte-diffs their stdout on all three
paths (in-checkout resolve, no-checkout refusal, PELTIER_HOME from a
foreign repo). Disagreement is a red build, not a footnote.
- New skill-preflight-windows job (windows-latest): asserts the native
refusal end-to-end under Windows PowerShell 5.1 AND pwsh (STATUS=refuse,
nonzero exit, the 'unsupported host' reason), plus Git Bash sh for good
measure. Verified live on a real Windows 11 box before shipping: 5.1
parses the pure-ASCII source clean and refuses with exit 1.
- SKILL.md / references/verify.md: document the second entry point ("use
whichever your host can launch; never both, never neither"); book chapter
and CLAUDE.md updated to match. .agents mirror regenerated (identical).
The ZeroClaw variant is unaffected: rm -rf scripts/ drops the ps1 too,
and its audit still passes (7 files scanned).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The cross-harness skill shipped (PR #17/#19) with README/CLAUDE.md/AGENTS.md
coverage, but the mdBook — the public disclosure surface — had zero mention
of it. By this project's own standard that is a parity defect: the flagship
consumable feature was undocumented where it is disclosed.
- New chapter "Consume it: the peltier skill" (docs/src/skill.md): install
paths and the harness matrix, the ZeroClaw script-blocking variant, the
gate order verify mode enforces, the refusal doctrine, and exactly which
of these claims CI asserts. Numbers carry their CI and workload.
- SUMMARY.md, architecture.md (layout now lists the skill + mirror),
reproduce.md, introduction.md: cross-links to the chapter.
- docs.yml: pull requests now BUILD the book (never deploy) — a broken
chapter or dangling include is a red PR check, not a post-merge surprise.
Also closes an audit finding: the book rebuilds when results/** changes,
since case-study chapters {{#include}} evidence files from there.
Book build verified locally with mdbook 0.4.x (skill.html renders).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The .agents mirror staged preflight.sh as a new path, so it missed the
canonical copy's 755. diff -r compares content, not modes — index modes
must match too.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The SKILL.md Agent Skills standard (agentskills.io) is now read natively by
Codex, Copilot, VS Code, Cursor, Gemini CLI, opencode, Goose, Amp, Zed,
Factory, and ZeroClaw — so the portable move is not per-harness adapter
files (forked content that drifts) but one canonical skill placed where
each harness actually looks.
- Spec-compliant frontmatter: license, compatibility (the Linux-only
measurement story, stated up front), version/author/tags top-level
(ZeroClaw's runtime reads those; the spec ignores unknown fields).
reference/ renamed to references/ per the spec's conventional layout.
- .agents/skills/peltier/ mirror: the cross-tool convergence path (Codex,
Zed, Cursor, Gemini CLI, Copilot, opencode, Goose, Amp). Byte-identical
to the .claude canonical, enforced by CI — drift between the two copies
is a red build, not a footnote.
- AGENTS.md at the repo root for the ~20 harnesses that read it (Claude
Code does not; CLAUDE.md remains its file). Deliberately thin: it points
at CLAUDE.md and the skill instead of restating rules.
- scripts/install-skill.sh: stamps a consumer repo at both discovery paths
(--layout claude|agents|both), or emits a script-less --zeroclaw-variant.
ZeroClaw's skill security policy blocks script files by default
(skills.allow_scripts off) — its audit rejected the full skill and passes
the variant, verified against the real zeroclaw binary. The variant loses
nothing: preflight.sh still exists exactly once, in the peltier checkout
the skill already requires, and SKILL.md now documents running it from
there. Re-implementing preflight inline is named as a refusal condition.
- CI (skill-preflight): frontmatter upgraded to a full Agent Skills spec
conformance check on BOTH copies (name rules, dir match, description and
body limits, metadata string-map), plus mirror equality and a functional
installer test (both layouts identical to canonical, variant carries no
scripts/, nonexistent target refused).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The description was an unquoted YAML scalar containing "trust layer:
equivalence" — a colon-space inside a plain scalar makes YAML parse it as a
nested mapping, so the frontmatter raised "mapping values are not allowed in
this context" and the skill would not load at all.
Quoted the scalar and replaced the colon with an em dash.
The real defect is that CI asserted the preflight script's behavior but never
checked that the skill it ships is loadable. Added a step that parses the
frontmatter, requires name/description to be non-empty strings, and requires
name to match the directory — so an unloadable skill fails the build instead
of failing the user.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The reusable artifact here was never the vendored targets — it is the
discipline: prove equivalence, prove significance, or claim nothing. This
packages it as a Claude Code skill usable from any repo.
Two modes. `verify` works anywhere, on any two shell commands in any
language: equivalence before timing -> A/A calibration -> interleaved A/B
with a bootstrap CI -> a verdict decided by the CI lower bound, never the
median. `attempt` drives the full in-repo loop (gates, pristine-baseline
verdict, ledger row, ROI report).
It drives the real bench-runner binary and refuses rather than degrade: on
an unsupported host, or with no trust layer reachable, preflight stops and
says so instead of falling back to a hand-rolled timing loop. A second copy
of the statistics is a second thing that can lie. A refusal ends the claim,
not the work.
An adversarial review of the skill caught six high-severity defects, all
fixed here — several of them the very errors the skill exists to prevent:
- verify mode had no risk classification at all. bench-runner emits only
accepted/rejected-bench; needs-human-review is minted by the in-repo
verdict path, which verify does not run. A concurrency change with
byte-identical output would have sailed through as "accepted". The skill
now makes the agent the classifier and says so explicitly.
- the accept template exposed `pin_prefix`, which is inert for
compare/aa/calibrate (only `verdict` reads it) — an agent would have set
it and reported pinning that never happened. Now documents wrapping both
sides in taskset instead.
- the equivalence gate compared stdout only, so two commands that write to
stderr and a dist/ tree would "GOLDEN PASS" on two empty files. Now hashes
stdout + stderr + exit code + artifact tree, and an empty diff over empty
output is a failed gate, not a passed one.
- nothing told the agent to build two coexisting immutable artifacts, so a
`cargo run` baseline could rebuild from the patched tree mid-bench — the
exact trap phase1-verdict-smoke-001 recorded.
- service mode was documented as generic HTTP. It is not: the server must
take <port> <doc> <iters>, print READY, and close the connection. A stock
Express app hangs the harness forever. Contract now stated.
- the CI refusal test could not pass: preflight also walks up from its own
location, so running it in-place found this very repo. The test now copies
the skill out of the tree first.
CI job `skill-preflight` asserts the contract that matters — resolve when a
trust layer is reachable, REFUSE when it is not, and work when consumed from
another repo via PELTIER_HOME. It deliberately does not assert a statistical
outcome: a shared runner is too noisy for that, which is precisely why the
skill calibrates before it claims.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
An adversarial audit found the enforcement story ahead of the code: gates
that were documented as hard gates were advisory, manual, or absent, and
several human-facing claims outran the machine record. This makes the
claims true.
High findings
- Differential fuzz was documentation, not enforcement: verdict hardcoded
fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case
studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real
pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and
it is inherently baseline-vs-candidate, so it runs on the accept path and
reports Skipped in the standalone `just gates` flow. New hard rule: a
machine `accepted` verdict is impossible unless the fuzz gate actually
passed — otherwise it caps at needs-human-review.
- The agent's system prompt claimed FP/concurrency/unsafe patches were
"routed to human review automatically". No such mechanism existed. Added
a conservative lexical risk classifier (crates/verdict/src/risk.rs) that
forces needs-human-review on any would-be accept carrying those signals;
fp-tolerance equivalence always signals.
- The local verdict path ran agent-influenced code unsandboxed. It now runs
under scripts/no-net.sh (network namespace) and fails closed. Upstream
test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when
present (scripts/pin-testsuite.sh generates them).
Also fixed, found while fixing the above
- The pipeline script's `set -e` was inert (brace group as the left operand
of `&&`), so a failing candidate build did not abort the run and verdict
could bench a stale binary. Now a scoped subshell whose status is branched
on at top level.
- A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path)
stranded RUNNING.lock and left read_verdict polling forever. Failure
accounting now lives outside the wrapper.
- harnessd's patch allowlist ignored git rename/copy headers and --no-prefix
diffs; the rename-escape case is now a committed regression test.
- The workspace did not compile on Windows (ungated Unix APIs). cfg-gated;
71 portable tests now run on both platforms.
- pin-check regenerated the corpus manifest instead of verifying it; it is
now verify-only, with a deliberate `just pin-corpus` to re-pin.
- Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics
(compare --perf-stat), never part of the accept decision.
Docs now match the code, including what is NOT yet enforced: no target ships
a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not
wired to the fp-tolerance gate — each is stated as mechanized-but-inert
rather than done. Corrected the stale "33 ledger rows" (34), the playbook's
class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation
claim (five accepted rows carry sanitizers_clean=false), the unsupported
"40,000 requests" figure, and the blanket "historical rows record
fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
benbrownandClaude Fable 5
