The reusable artifact here was never the vendored targets — it is the
discipline: prove equivalence, prove significance, or claim nothing. This
packages it as a Claude Code skill usable from any repo.
Two modes. `verify` works anywhere, on any two shell commands in any
language: equivalence before timing -> A/A calibration -> interleaved A/B
with a bootstrap CI -> a verdict decided by the CI lower bound, never the
median. `attempt` drives the full in-repo loop (gates, pristine-baseline
verdict, ledger row, ROI report).
It drives the real bench-runner binary and refuses rather than degrade: on
an unsupported host, or with no trust layer reachable, preflight stops and
says so instead of falling back to a hand-rolled timing loop. A second copy
of the statistics is a second thing that can lie. A refusal ends the claim,
not the work.
An adversarial review of the skill caught six high-severity defects, all
fixed here — several of them the very errors the skill exists to prevent:
- verify mode had no risk classification at all. bench-runner emits only
accepted/rejected-bench; needs-human-review is minted by the in-repo
verdict path, which verify does not run. A concurrency change with
byte-identical output would have sailed through as "accepted". The skill
now makes the agent the classifier and says so explicitly.
- the accept template exposed `pin_prefix`, which is inert for
compare/aa/calibrate (only `verdict` reads it) — an agent would have set
it and reported pinning that never happened. Now documents wrapping both
sides in taskset instead.
- the equivalence gate compared stdout only, so two commands that write to
stderr and a dist/ tree would "GOLDEN PASS" on two empty files. Now hashes
stdout + stderr + exit code + artifact tree, and an empty diff over empty
output is a failed gate, not a passed one.
- nothing told the agent to build two coexisting immutable artifacts, so a
`cargo run` baseline could rebuild from the patched tree mid-bench — the
exact trap phase1-verdict-smoke-001 recorded.
- service mode was documented as generic HTTP. It is not: the server must
take <port> <doc> <iters>, print READY, and close the connection. A stock
Express app hangs the harness forever. Contract now stated.
- the CI refusal test could not pass: preflight also walks up from its own
location, so running it in-place found this very repo. The test now copies
the skill out of the tree first.
CI job `skill-preflight` asserts the contract that matters — resolve when a
trust layer is reachable, REFUSE when it is not, and work when consumed from
another repo via PELTIER_HOME. It deliberately does not assert a statistical
outcome: a shared runner is too noisy for that, which is precisely why the
skill calibrates before it claims.
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
- docs/: an mdBook documentation site — thesis, architecture, measurement
discipline, equivalence gates, agent loop, isolation/anti-reward-hacking,
playbook, ROI, reproduce, roadmap, and the five case studies (pulled from
their single source in results/ via {{#include}}). Builds clean (21
pages).
- .github/workflows/docs.yml: builds mdBook and deploys to GitHub Pages on
push to main (one-time setup: Settings → Pages → Source = GitHub Actions).
- README.md: rewritten as the project front page — the thesis, the verified
results table, layout, quick start, how it stays honest, links to the
book. Replaces the stale Phase-0-era text.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
diff-test standalone (just gates / CI target-gates) passed the now-
templated build.binary into golden replay verbatim — literal {out}.
It now builds the current tree to targets/<t>/gates-build and tests
that, with a --binary override. CI's target-gates-no-net job gains
clang/cmake, fetches+gates the cjson C target (clang build -> Unity/
CTest -> golden replay) alongside tokei.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
The 8-session probe passed cleanly (0 false positives, 8/8 detection,
A/A CIs ~±0.6%); run the same 20-session protocol used for the local
trust anchor so GH-runner evidence is directly comparable.
Closes the CI-reachable slice of the Phase 1 infra gaps:
- trust-layer job: cargo test + clippy -D warnings + fmt
- target-gates-no-net: fetches pinned targets (new targets/fetch.sh),
regenerates + verifies corpora, runs 'just gates tokei'
mechanically, and executes the bench workload inside a
--network=none container (SPEC 3.1/10 no-net requirement)
- runner-calibration (informational): A/A + injection harness measures
the shared runner's noise floor per run, evidence uploaded as an
artifact — environments are judged empirically, not assumed
- perf-probe (informational): reports which perf event sources work
docs/infra.md maps each remaining hardware requirement (pinned cores,
governor/SMT control, separate-uid trust layer) to the bench-metal
self-hosted runner setup a human can complete in ~15 min.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
benbrownandClaude Opus 4.8
copilot-swe-agent[bot]andGitHub