diff --git a/.agents/skills/peltier/SKILL.md b/.agents/skills/peltier/SKILL.md index 5b1ce6b..8a37aa3 100644 --- a/.agents/skills/peltier/SKILL.md +++ b/.agents/skills/peltier/SKILL.md @@ -30,6 +30,13 @@ plugin, or copied into another project): sh /scripts/preflight.sh + # no POSIX shell on this host (Windows-native harness)? same contract: + powershell -NoProfile -ExecutionPolicy Bypass -File /scripts/preflight.ps1 + +The two entry points implement one contract and are byte-diffed against each +other in CI — same checks, same output, same refusals. Use whichever your +host can launch; never both, never neither. + Some harnesses block script files inside skills, so your installation may carry no `scripts/` directory. The script still exists exactly once — in the peltier checkout this skill drives (which you need anyway; it is where diff --git a/.agents/skills/peltier/references/verify.md b/.agents/skills/peltier/references/verify.md index 190819d..ffc2173 100644 --- a/.agents/skills/peltier/references/verify.md +++ b/.agents/skills/peltier/references/verify.md @@ -6,7 +6,9 @@ faster. Find out. Works on any pair of shell commands, in any language. ## 0. Preflight sh /scripts/preflight.sh - # or, if this installation carries no scripts/ (script-blocking harness): + # no POSIX shell (Windows-native harness)? same contract, CI-diffed against the sh version: + powershell -NoProfile -ExecutionPolicy Bypass -File /scripts/preflight.ps1 + # installation carries no scripts/ (script-blocking harness)? run from the checkout: sh "$PELTIER_HOME/.claude/skills/peltier/scripts/preflight.sh" Read the `key=value` output; you need `BENCH_RUNNER` and `PIN_SUPPORTED`. A diff --git a/.agents/skills/peltier/scripts/preflight.ps1 b/.agents/skills/peltier/scripts/preflight.ps1 new file mode 100644 index 0000000..4eb66ed --- /dev/null +++ b/.agents/skills/peltier/scripts/preflight.ps1 @@ -0,0 +1,112 @@ +# peltier preflight - PowerShell entry point. Same contract as preflight.sh, +# for hosts where no POSIX shell exists (Windows-native harnesses). Works on +# Windows PowerShell 5.1 and pwsh 7+ on any OS. +# +# THE CONTRACT IS DEFINED ONCE: this script and preflight.sh must emit +# byte-identical output for the same environment. CI enforces that by running +# both on the same host and diffing stdout (skill-preflight job). If you +# change one, change the other, or the build goes red. +# +# STATUS=ok means the harness is present and runnable here. It does NOT mean +# this host can measure your change: only the A/A self-test decides that. +# A refusal (STATUS=refuse, exit 1) ends the *claim*, not the work. +# +# Source is deliberately pure ASCII: Windows PowerShell 5.1 misdecodes +# BOM-less UTF-8, so the em dashes in the shared refusal strings are composed +# at runtime instead of written literally. + +$ErrorActionPreference = 'Stop' +$EM = [string][char]0x2014 # em dash, byte-identical to preflight.sh output + +function Say([string]$s) { Write-Output $s } +function Refuse([string]$reason) { + Say 'STATUS=refuse' + Say "REASON=$reason" + exit 1 +} + +# --- host ------------------------------------------------------------- +# bench-runner shells every timed run through `sh -c`, and the in-repo +# pipeline is POSIX-only. Windows compiles the workspace but cannot run it. +# Host check comes FIRST, exactly as in preflight.sh. +$onLinux = (Test-Path variable:IsLinux) -and $IsLinux +$onMac = (Test-Path variable:IsMacOS) -and $IsMacOS +if ($onLinux) { $hostKind = 'linux' } +elseif ($onMac) { $hostKind = 'darwin' } +else { + Refuse ("unsupported host 'Windows' $EM peltier's harness is Linux/POSIX-only at runtime. " + + "Run it on Linux (or macOS for verify mode). Do not substitute another benchmark.") +} + +# --- trust layer ------------------------------------------------------ +function Test-Peltier([string]$dir) { + (Test-Path (Join-Path $dir 'crates/bench-runner/Cargo.toml') -PathType Leaf) -and + (Test-Path (Join-Path $dir 'config/accept.toml') -PathType Leaf) +} + +# Walk up from a starting dir looking for a checkout; parents terminate at +# the filesystem root, so this cannot spin. +function Find-Peltier([string]$start) { + $d = $start + while (-not [string]::IsNullOrEmpty($d)) { + if (Test-Peltier $d) { return $d } + $parent = Split-Path -Parent $d + if ($parent -eq $d) { break } + $d = $parent + } + return $null +} + +$peltierHome = $null +if ($env:PELTIER_HOME) { + $canon = $null + try { $canon = (Resolve-Path -LiteralPath $env:PELTIER_HOME).Path } catch {} + if (-not $canon) { Refuse "PELTIER_HOME='$($env:PELTIER_HOME)' does not exist" } + if (-not (Test-Peltier $canon)) { + Refuse "PELTIER_HOME='$($env:PELTIER_HOME)' is not a peltier checkout (no crates/bench-runner/Cargo.toml + config/accept.toml)" + } + $peltierHome = $canon +} else { + # cwd first (working inside a checkout), then this script's own location + # (the skill still living inside one). A skill copied into another project + # finds neither - which is why PELTIER_HOME exists. + $peltierHome = Find-Peltier (Get-Location).Path + if (-not $peltierHome) { $peltierHome = Find-Peltier $PSScriptRoot } +} +if (-not $peltierHome) { + Refuse ("no peltier checkout found $EM clone https://github.com/bdbrown4/peltier.io and set " + + "PELTIER_HOME=/path/to/peltier.io. The statistics live in bench-runner; do not reimplement them.") +} + +if (-not (Get-Command cargo -ErrorAction SilentlyContinue)) { + Refuse "cargo not on PATH $EM needed to build bench-runner from $peltierHome" +} + +# --- build the real harness ------------------------------------------- +# Always build. cargo is a no-op when current, and skipping this because a +# binary happens to exist would silently measure with a stale copy of the +# statistics after a `git pull`. +$br = Join-Path $peltierHome 'target/release/bench-runner' +Push-Location $peltierHome +try { & cargo build --release -q -p bench-runner; $buildOk = ($LASTEXITCODE -eq 0) } +finally { Pop-Location } +if (-not $buildOk) { + Refuse "bench-runner failed to build in $peltierHome $EM fix the build before claiming any number" +} +if (-not (Test-Path $br -PathType Leaf)) { + Refuse "bench-runner did not appear at $br after a successful build" +} + +# --- pinning ---------------------------------------------------------- +# NOTE: config/accept.toml's `pin_prefix` is read ONLY by the in-repo verdict +# pipeline. For compare/aa/calibrate you must wrap the commands yourself +# (`taskset -c N ` on both sides). PIN_SUPPORTED just says whether that +# tool is available here. +if ($onLinux -and (Get-Command taskset -ErrorAction SilentlyContinue)) { $pin = 'yes' } else { $pin = 'no' } + +Say 'STATUS=ok' +Say "PELTIER_HOME=$peltierHome" +Say "BENCH_RUNNER=$br" +Say "HOST=$hostKind" +Say "PIN_SUPPORTED=$pin" +Say 'NOTE=harness runnable here; the A/A self-test decides whether it can measure your change' diff --git a/.claude/skills/peltier/SKILL.md b/.claude/skills/peltier/SKILL.md index 5b1ce6b..8a37aa3 100644 --- a/.claude/skills/peltier/SKILL.md +++ b/.claude/skills/peltier/SKILL.md @@ -30,6 +30,13 @@ plugin, or copied into another project): sh /scripts/preflight.sh + # no POSIX shell on this host (Windows-native harness)? same contract: + powershell -NoProfile -ExecutionPolicy Bypass -File /scripts/preflight.ps1 + +The two entry points implement one contract and are byte-diffed against each +other in CI — same checks, same output, same refusals. Use whichever your +host can launch; never both, never neither. + Some harnesses block script files inside skills, so your installation may carry no `scripts/` directory. The script still exists exactly once — in the peltier checkout this skill drives (which you need anyway; it is where diff --git a/.claude/skills/peltier/references/verify.md b/.claude/skills/peltier/references/verify.md index 190819d..ffc2173 100644 --- a/.claude/skills/peltier/references/verify.md +++ b/.claude/skills/peltier/references/verify.md @@ -6,7 +6,9 @@ faster. Find out. Works on any pair of shell commands, in any language. ## 0. Preflight sh /scripts/preflight.sh - # or, if this installation carries no scripts/ (script-blocking harness): + # no POSIX shell (Windows-native harness)? same contract, CI-diffed against the sh version: + powershell -NoProfile -ExecutionPolicy Bypass -File /scripts/preflight.ps1 + # installation carries no scripts/ (script-blocking harness)? run from the checkout: sh "$PELTIER_HOME/.claude/skills/peltier/scripts/preflight.sh" Read the `key=value` output; you need `BENCH_RUNNER` and `PIN_SUPPORTED`. A diff --git a/.claude/skills/peltier/scripts/preflight.ps1 b/.claude/skills/peltier/scripts/preflight.ps1 new file mode 100644 index 0000000..4eb66ed --- /dev/null +++ b/.claude/skills/peltier/scripts/preflight.ps1 @@ -0,0 +1,112 @@ +# peltier preflight - PowerShell entry point. Same contract as preflight.sh, +# for hosts where no POSIX shell exists (Windows-native harnesses). Works on +# Windows PowerShell 5.1 and pwsh 7+ on any OS. +# +# THE CONTRACT IS DEFINED ONCE: this script and preflight.sh must emit +# byte-identical output for the same environment. CI enforces that by running +# both on the same host and diffing stdout (skill-preflight job). If you +# change one, change the other, or the build goes red. +# +# STATUS=ok means the harness is present and runnable here. It does NOT mean +# this host can measure your change: only the A/A self-test decides that. +# A refusal (STATUS=refuse, exit 1) ends the *claim*, not the work. +# +# Source is deliberately pure ASCII: Windows PowerShell 5.1 misdecodes +# BOM-less UTF-8, so the em dashes in the shared refusal strings are composed +# at runtime instead of written literally. + +$ErrorActionPreference = 'Stop' +$EM = [string][char]0x2014 # em dash, byte-identical to preflight.sh output + +function Say([string]$s) { Write-Output $s } +function Refuse([string]$reason) { + Say 'STATUS=refuse' + Say "REASON=$reason" + exit 1 +} + +# --- host ------------------------------------------------------------- +# bench-runner shells every timed run through `sh -c`, and the in-repo +# pipeline is POSIX-only. Windows compiles the workspace but cannot run it. +# Host check comes FIRST, exactly as in preflight.sh. +$onLinux = (Test-Path variable:IsLinux) -and $IsLinux +$onMac = (Test-Path variable:IsMacOS) -and $IsMacOS +if ($onLinux) { $hostKind = 'linux' } +elseif ($onMac) { $hostKind = 'darwin' } +else { + Refuse ("unsupported host 'Windows' $EM peltier's harness is Linux/POSIX-only at runtime. " + + "Run it on Linux (or macOS for verify mode). Do not substitute another benchmark.") +} + +# --- trust layer ------------------------------------------------------ +function Test-Peltier([string]$dir) { + (Test-Path (Join-Path $dir 'crates/bench-runner/Cargo.toml') -PathType Leaf) -and + (Test-Path (Join-Path $dir 'config/accept.toml') -PathType Leaf) +} + +# Walk up from a starting dir looking for a checkout; parents terminate at +# the filesystem root, so this cannot spin. +function Find-Peltier([string]$start) { + $d = $start + while (-not [string]::IsNullOrEmpty($d)) { + if (Test-Peltier $d) { return $d } + $parent = Split-Path -Parent $d + if ($parent -eq $d) { break } + $d = $parent + } + return $null +} + +$peltierHome = $null +if ($env:PELTIER_HOME) { + $canon = $null + try { $canon = (Resolve-Path -LiteralPath $env:PELTIER_HOME).Path } catch {} + if (-not $canon) { Refuse "PELTIER_HOME='$($env:PELTIER_HOME)' does not exist" } + if (-not (Test-Peltier $canon)) { + Refuse "PELTIER_HOME='$($env:PELTIER_HOME)' is not a peltier checkout (no crates/bench-runner/Cargo.toml + config/accept.toml)" + } + $peltierHome = $canon +} else { + # cwd first (working inside a checkout), then this script's own location + # (the skill still living inside one). A skill copied into another project + # finds neither - which is why PELTIER_HOME exists. + $peltierHome = Find-Peltier (Get-Location).Path + if (-not $peltierHome) { $peltierHome = Find-Peltier $PSScriptRoot } +} +if (-not $peltierHome) { + Refuse ("no peltier checkout found $EM clone https://github.com/bdbrown4/peltier.io and set " + + "PELTIER_HOME=/path/to/peltier.io. The statistics live in bench-runner; do not reimplement them.") +} + +if (-not (Get-Command cargo -ErrorAction SilentlyContinue)) { + Refuse "cargo not on PATH $EM needed to build bench-runner from $peltierHome" +} + +# --- build the real harness ------------------------------------------- +# Always build. cargo is a no-op when current, and skipping this because a +# binary happens to exist would silently measure with a stale copy of the +# statistics after a `git pull`. +$br = Join-Path $peltierHome 'target/release/bench-runner' +Push-Location $peltierHome +try { & cargo build --release -q -p bench-runner; $buildOk = ($LASTEXITCODE -eq 0) } +finally { Pop-Location } +if (-not $buildOk) { + Refuse "bench-runner failed to build in $peltierHome $EM fix the build before claiming any number" +} +if (-not (Test-Path $br -PathType Leaf)) { + Refuse "bench-runner did not appear at $br after a successful build" +} + +# --- pinning ---------------------------------------------------------- +# NOTE: config/accept.toml's `pin_prefix` is read ONLY by the in-repo verdict +# pipeline. For compare/aa/calibrate you must wrap the commands yourself +# (`taskset -c N ` on both sides). PIN_SUPPORTED just says whether that +# tool is available here. +if ($onLinux -and (Get-Command taskset -ErrorAction SilentlyContinue)) { $pin = 'yes' } else { $pin = 'no' } + +Say 'STATUS=ok' +Say "PELTIER_HOME=$peltierHome" +Say "BENCH_RUNNER=$br" +Say "HOST=$hostKind" +Say "PIN_SUPPORTED=$pin" +Say 'NOTE=harness runnable here; the A/A self-test decides whether it can measure your change' diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 0744fd3..98b4c55 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -186,6 +186,31 @@ jobs: sh "$GITHUB_WORKSPACE/.claude/skills/peltier/scripts/preflight.sh" | tee pf.txt grep -q '^STATUS=ok$' pf.txt + - name: sh and ps1 preflights are byte-equivalent (two entry points, one contract) + run: | + # preflight.ps1 exists for hosts with no POSIX shell. Two + # implementations of the same contract can drift, so both run here + # on the same host and their stdout is byte-diffed on every path. + # 1. in-checkout resolve + sh .claude/skills/peltier/scripts/preflight.sh > /tmp/eq-sh.txt + pwsh -NoProfile -File .claude/skills/peltier/scripts/preflight.ps1 > /tmp/eq-ps.txt + diff /tmp/eq-sh.txt /tmp/eq-ps.txt + # 2. refusal with no checkout reachable (copied out of the repo) + cp -r .claude/skills/peltier /tmp/eqskill + mkdir -p /tmp/eq-not-a-repo && cd /tmp/eq-not-a-repo + set +e + env -u PELTIER_HOME sh /tmp/eqskill/scripts/preflight.sh > sh.txt 2>&1; shrc=$? + env -u PELTIER_HOME pwsh -NoProfile -File /tmp/eqskill/scripts/preflight.ps1 > ps.txt 2>&1; psrc=$? + set -e + test "$shrc" -ne 0 && test "$psrc" -ne 0 + diff sh.txt ps.txt + # 3. resolve via PELTIER_HOME from a foreign repo + mkdir -p /tmp/eq-foreign && cd /tmp/eq-foreign + PELTIER_HOME="$GITHUB_WORKSPACE" sh /tmp/eqskill/scripts/preflight.sh > sh2.txt + PELTIER_HOME="$GITHUB_WORKSPACE" pwsh -NoProfile -File /tmp/eqskill/scripts/preflight.ps1 > ps2.txt + diff sh2.txt ps2.txt + echo "equivalence holds on all three paths" + - name: The harness the skill drives actually runs (plumbing, not verdict) run: | BR=$(sed -n 's/^BENCH_RUNNER=//p' /tmp/pf.txt) @@ -200,3 +225,40 @@ jobs: run: | BR=$(sed -n 's/^BENCH_RUNNER=//p' /tmp/pf.txt) "$BR" --config /tmp/accept.toml aa --cmd 'sleep 0.02' + + # The refusal doctrine must survive hosts with no POSIX shell: on a real + # Windows runner, BOTH preflight entry points must refuse with a structured + # STATUS=refuse — never a launch failure an agent could misread as license + # to hand-roll a timing loop. Windows PowerShell 5.1 is tested separately + # from pwsh because 5.1 misdecodes BOM-less UTF-8 (the ps1 source is pure + # ASCII for exactly that reason). + skill-preflight-windows: + runs-on: windows-latest + steps: + - uses: actions/checkout@v4 + - name: ps1 refuses natively (Windows PowerShell 5.1) + shell: powershell + run: | + $out = & powershell -NoProfile -ExecutionPolicy Bypass -File .claude\skills\peltier\scripts\preflight.ps1 + $out | Write-Output + if ($LASTEXITCODE -eq 0) { throw "preflight did not refuse on Windows" } + $joined = $out -join "`n" + if ($joined -notlike "*STATUS=refuse*") { throw "no STATUS=refuse in output" } + if ($joined -notlike "*unsupported host 'Windows'*") { throw "missing refusal reason" } + - name: ps1 refuses natively (pwsh) + shell: pwsh + run: | + $out = & pwsh -NoProfile -File .claude\skills\peltier\scripts\preflight.ps1 + $out | Write-Output + if ($LASTEXITCODE -eq 0) { throw "preflight did not refuse under pwsh" } + if (($out -join "`n") -notlike "*STATUS=refuse*") { throw "no STATUS=refuse in output" } + - name: sh entry point also refuses here (Git Bash) + shell: bash + run: | + set +e + sh .claude/skills/peltier/scripts/preflight.sh > out.txt 2>&1 + rc=$? + set -e + cat out.txt + test "$rc" -ne 0 + grep -q '^STATUS=refuse$' out.txt diff --git a/CLAUDE.md b/CLAUDE.md index 20eb0e5..12dce14 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -29,7 +29,7 @@ Full engineering spec: `SPEC.md`. Read it before starting any phase. - `targets/` — vendored OSS targets for case studies (permissive licenses only) - `playbook/` — optimization classes, ordered, with preconditions and known risks - `results/` — generated reports and flamegraphs -- `.claude/skills/peltier/` — the trust layer packaged as a consumable **Agent Skill** (agentskills.io SKILL.md standard, spec-compliant frontmatter): `verify` mode (equivalence → A/A calibration → interleaved A/B + bootstrap CI) works in any repo against any two shell commands; `attempt` mode drives the full in-repo loop. Drives the real `bench-runner` binary and refuses on unsupported hosts rather than falling back to a second, forkable copy of the statistics. Cross-harness: mirrored byte-identically at `.agents/skills/peltier/` (the Codex/Zed/Cursor/Gemini/Copilot convergence path — CI enforces the mirror), `AGENTS.md` at the repo root points non-Claude harnesses here, and `scripts/install-skill.sh` stamps consumer repos (plus a script-less `--zeroclaw-variant`, since ZeroClaw blocks script files in skills by default — its preflight runs from the checkout). Preflight refusal, spec conformance, mirror equality, and the installer are all asserted in CI (`skill-preflight`). +- `.claude/skills/peltier/` — the trust layer packaged as a consumable **Agent Skill** (agentskills.io SKILL.md standard, spec-compliant frontmatter): `verify` mode (equivalence → A/A calibration → interleaved A/B + bootstrap CI) works in any repo against any two shell commands; `attempt` mode drives the full in-repo loop. Drives the real `bench-runner` binary and refuses on unsupported hosts rather than falling back to a second, forkable copy of the statistics. Cross-harness: mirrored byte-identically at `.agents/skills/peltier/` (the Codex/Zed/Cursor/Gemini/Copilot convergence path — CI enforces the mirror), `AGENTS.md` at the repo root points non-Claude harnesses here, and `scripts/install-skill.sh` stamps consumer repos (plus a script-less `--zeroclaw-variant`, since ZeroClaw blocks script files in skills by default — its preflight runs from the checkout). Preflight ships as two entry points implementing one contract — `preflight.sh` (POSIX) and `preflight.ps1` (Windows PowerShell 5.1/pwsh; pure-ASCII source, em dashes composed at runtime) — CI byte-diffs their output on all three paths and a `windows-latest` job asserts the native refusal. Preflight refusal, sh↔ps1 equivalence, spec conformance, mirror equality, and the installer are all asserted in CI (`skill-preflight`, `skill-preflight-windows`). **Phase 4 COMPLETE — all SPEC §5 exit criteria met. Phases 0–4 are all done.** Service-mode latency benchmarking shipped: a coordinated-omission-correct, open-loop, fixed-rate load generator (`crates/bench-runner/src/service.rs`) measuring latency from each request's intended send time (queueing counted, not hidden), interleaved A/B, exact percentiles, bootstrap p50/p99 CIs. A real service target — `targets/cjson/service.c`, a minimal HTTP server wrapping the patched `cJSON.c` (trust-layer, outside the workspace allowlist). Service calibration PASS (0/10 A/A false-positive, 10/10 injected-5%-latency-regression detection; `results/calibration/cjson-service-aa.json`). **The verified batch win phase3-cjson-002 measured under a 150 rps-target replay (20 interleaved rounds, 0 drops — the `phase4-cjson-service` ledger row records sessions=20, rate_rps=150, drop_rate=0.0; total request count is not machine-recorded): p50 latency +6.2%, 95% CI [+5.8%, +7.2%] — accepted; p99 NOT claimed (CI [0.07, 4.97], single-worker loopback tail jitter) — harness correctly rejected it.** Mechanical ROI report generator (`crates/report` bin, `just report`, SPEC §9): reads a ledger row + service-latency JSON → throughput→cores→dollars (27.5 cores / $9,621-per-year CI lower bound on a 500-core fleet) *and* latency percentiles, every figure with CI + workload + methodology inline; flags any non-accepted row. Case study: `results/phase4/`. **The roadmap is complete — the profile→hypothesize→patch→gated-verdict→ROI loop runs end to end across Rust and C, batch and service, with zero shipped false accepts across all phases.** diff --git a/docs/src/skill.md b/docs/src/skill.md index 7c90dfc..67486a3 100644 --- a/docs/src/skill.md +++ b/docs/src/skill.md @@ -83,7 +83,13 @@ document. In `verify` mode, the gate order is the content: Measurement requires a Linux/POSIX host — the same requirement as the rest of the pipeline. On Windows (or with no checkout reachable) the skill refuses and -says why. That is asserted, not assumed: on every pull request, CI runs the +says why. The refusal itself is cross-OS: preflight ships as both `preflight.sh` +(POSIX) and `preflight.ps1` (Windows PowerShell 5.1 / pwsh), so a Windows-native +harness with no POSIX shell still gets a structured `STATUS=refuse` instead of a +launch error it might misread as license to hand-roll a timing loop. Two +implementations of one contract is a drift risk, so CI byte-diffs their output +on every path (resolve, refuse, `PELTIER_HOME`) on the same host, and a Windows +runner asserts the native refusal on every pull request. That is asserted, not assumed: on every pull request, CI runs the preflight's refusal path (skill copied outside any checkout) and its resolve paths (in-checkout and via `PELTIER_HOME` from a foreign repo), alongside a live plumbing run of the harness it drives.