Merge pull request #21 from bdbrown4/skill/preflight-ps1

skill: cross-OS preflight — ps1 entry point, one CI-enforced contract
This commit is contained in:
2026-07-16 08:44:49 -04:00
committed by GitHub
9 changed files with 336 additions and 4 deletions
+11
View File
@@ -30,6 +30,17 @@ plugin, or copied into another project):
sh <skill-dir>/scripts/preflight.sh sh <skill-dir>/scripts/preflight.sh
# no POSIX shell on this host (Windows-native harness)? same contract:
powershell -NoProfile -ExecutionPolicy Bypass -File <skill-dir>/scripts/preflight.ps1
The two entry points implement one contract and are byte-diffed against each
other in CI — same checks, same output, same refusals. Dispatch on launcher
availability, not OS: try `sh` first; if the **launcher itself** is missing
(`sh: not found` — a launch failure, not a refusal), run the PowerShell entry
point instead. A `STATUS=refuse` from either is **final**: never run the
second entry point hoping for a different answer — they are kept
byte-identical in CI precisely so there is no different answer.
Some harnesses block script files inside skills, so your installation may Some harnesses block script files inside skills, so your installation may
carry no `scripts/` directory. The script still exists exactly once — in the carry no `scripts/` directory. The script still exists exactly once — in the
peltier checkout this skill drives (which you need anyway; it is where peltier checkout this skill drives (which you need anyway; it is where
+7 -1
View File
@@ -6,9 +6,15 @@ faster. Find out. Works on any pair of shell commands, in any language.
## 0. Preflight ## 0. Preflight
sh <skill-dir>/scripts/preflight.sh sh <skill-dir>/scripts/preflight.sh
# or, if this installation carries no scripts/ (script-blocking harness): # `sh` launcher missing ("sh: not found" — a launch failure, NOT a refusal)?
# same contract, byte-diffed against the sh version in CI:
powershell -NoProfile -ExecutionPolicy Bypass -File <skill-dir>/scripts/preflight.ps1
# installation carries no scripts/ (script-blocking harness)? run from the checkout:
sh "$PELTIER_HOME/.claude/skills/peltier/scripts/preflight.sh" sh "$PELTIER_HOME/.claude/skills/peltier/scripts/preflight.sh"
A `STATUS=refuse` from either entry point is final — do not run the other one
looking for a different answer.
Read the `key=value` output; you need `BENCH_RUNNER` and `PIN_SUPPORTED`. A Read the `key=value` output; you need `BENCH_RUNNER` and `PIN_SUPPORTED`. A
refusal ends the **claim**, not the work — report the reason verbatim and do refusal ends the **claim**, not the work — report the reason verbatim and do
not substitute another timing tool. not substitute another timing tool.
@@ -0,0 +1,112 @@
# peltier preflight - PowerShell entry point. Same contract as preflight.sh,
# for hosts where no POSIX shell exists (Windows-native harnesses). Works on
# Windows PowerShell 5.1 and pwsh 7+ on any OS.
#
# THE CONTRACT IS DEFINED ONCE: this script and preflight.sh must emit
# byte-identical output for the same environment. CI enforces that by running
# both on the same host and diffing stdout (skill-preflight job). If you
# change one, change the other, or the build goes red.
#
# STATUS=ok means the harness is present and runnable here. It does NOT mean
# this host can measure your change: only the A/A self-test decides that.
# A refusal (STATUS=refuse, exit 1) ends the *claim*, not the work.
#
# Source is deliberately pure ASCII: Windows PowerShell 5.1 misdecodes
# BOM-less UTF-8, so the em dashes in the shared refusal strings are composed
# at runtime instead of written literally.
$ErrorActionPreference = 'Stop'
$EM = [string][char]0x2014 # em dash, byte-identical to preflight.sh output
function Say([string]$s) { Write-Output $s }
function Refuse([string]$reason) {
Say 'STATUS=refuse'
Say "REASON=$reason"
exit 1
}
# --- host -------------------------------------------------------------
# bench-runner shells every timed run through `sh -c`, and the in-repo
# pipeline is POSIX-only. Windows compiles the workspace but cannot run it.
# Host check comes FIRST, exactly as in preflight.sh.
$onLinux = (Test-Path variable:IsLinux) -and $IsLinux
$onMac = (Test-Path variable:IsMacOS) -and $IsMacOS
if ($onLinux) { $hostKind = 'linux' }
elseif ($onMac) { $hostKind = 'darwin' }
else {
Refuse ("unsupported host 'Windows' $EM peltier's harness is Linux/POSIX-only at runtime. " +
"Run it on Linux (or macOS for verify mode). Do not substitute another benchmark.")
}
# --- trust layer ------------------------------------------------------
function Test-Peltier([string]$dir) {
(Test-Path (Join-Path $dir 'crates/bench-runner/Cargo.toml') -PathType Leaf) -and
(Test-Path (Join-Path $dir 'config/accept.toml') -PathType Leaf)
}
# Walk up from a starting dir looking for a checkout; parents terminate at
# the filesystem root, so this cannot spin.
function Find-Peltier([string]$start) {
$d = $start
while (-not [string]::IsNullOrEmpty($d)) {
if (Test-Peltier $d) { return $d }
$parent = Split-Path -Parent $d
if ($parent -eq $d) { break }
$d = $parent
}
return $null
}
$peltierHome = $null
if ($env:PELTIER_HOME) {
$canon = $null
try { $canon = (Resolve-Path -LiteralPath $env:PELTIER_HOME).Path } catch {}
if (-not $canon) { Refuse "PELTIER_HOME='$($env:PELTIER_HOME)' does not exist" }
if (-not (Test-Peltier $canon)) {
Refuse "PELTIER_HOME='$($env:PELTIER_HOME)' is not a peltier checkout (no crates/bench-runner/Cargo.toml + config/accept.toml)"
}
$peltierHome = $canon
} else {
# cwd first (working inside a checkout), then this script's own location
# (the skill still living inside one). A skill copied into another project
# finds neither - which is why PELTIER_HOME exists.
$peltierHome = Find-Peltier (Get-Location).Path
if (-not $peltierHome) { $peltierHome = Find-Peltier $PSScriptRoot }
}
if (-not $peltierHome) {
Refuse ("no peltier checkout found $EM clone https://github.com/bdbrown4/peltier.io and set " +
"PELTIER_HOME=/path/to/peltier.io. The statistics live in bench-runner; do not reimplement them.")
}
if (-not (Get-Command cargo -ErrorAction SilentlyContinue)) {
Refuse "cargo not on PATH $EM needed to build bench-runner from $peltierHome"
}
# --- build the real harness -------------------------------------------
# Always build. cargo is a no-op when current, and skipping this because a
# binary happens to exist would silently measure with a stale copy of the
# statistics after a `git pull`.
$br = Join-Path $peltierHome 'target/release/bench-runner'
Push-Location $peltierHome
try { & cargo build --release -q -p bench-runner; $buildOk = ($LASTEXITCODE -eq 0) }
finally { Pop-Location }
if (-not $buildOk) {
Refuse "bench-runner failed to build in $peltierHome $EM fix the build before claiming any number"
}
if (-not (Test-Path $br -PathType Leaf)) {
Refuse "bench-runner did not appear at $br after a successful build"
}
# --- pinning ----------------------------------------------------------
# NOTE: config/accept.toml's `pin_prefix` is read ONLY by the in-repo verdict
# pipeline. For compare/aa/calibrate you must wrap the commands yourself
# (`taskset -c N <cmd>` on both sides). PIN_SUPPORTED just says whether that
# tool is available here.
if ($onLinux -and (Get-Command taskset -ErrorAction SilentlyContinue)) { $pin = 'yes' } else { $pin = 'no' }
Say 'STATUS=ok'
Say "PELTIER_HOME=$peltierHome"
Say "BENCH_RUNNER=$br"
Say "HOST=$hostKind"
Say "PIN_SUPPORTED=$pin"
Say 'NOTE=harness runnable here; the A/A self-test decides whether it can measure your change'
+11
View File
@@ -30,6 +30,17 @@ plugin, or copied into another project):
sh <skill-dir>/scripts/preflight.sh sh <skill-dir>/scripts/preflight.sh
# no POSIX shell on this host (Windows-native harness)? same contract:
powershell -NoProfile -ExecutionPolicy Bypass -File <skill-dir>/scripts/preflight.ps1
The two entry points implement one contract and are byte-diffed against each
other in CI — same checks, same output, same refusals. Dispatch on launcher
availability, not OS: try `sh` first; if the **launcher itself** is missing
(`sh: not found` — a launch failure, not a refusal), run the PowerShell entry
point instead. A `STATUS=refuse` from either is **final**: never run the
second entry point hoping for a different answer — they are kept
byte-identical in CI precisely so there is no different answer.
Some harnesses block script files inside skills, so your installation may Some harnesses block script files inside skills, so your installation may
carry no `scripts/` directory. The script still exists exactly once — in the carry no `scripts/` directory. The script still exists exactly once — in the
peltier checkout this skill drives (which you need anyway; it is where peltier checkout this skill drives (which you need anyway; it is where
+7 -1
View File
@@ -6,9 +6,15 @@ faster. Find out. Works on any pair of shell commands, in any language.
## 0. Preflight ## 0. Preflight
sh <skill-dir>/scripts/preflight.sh sh <skill-dir>/scripts/preflight.sh
# or, if this installation carries no scripts/ (script-blocking harness): # `sh` launcher missing ("sh: not found" — a launch failure, NOT a refusal)?
# same contract, byte-diffed against the sh version in CI:
powershell -NoProfile -ExecutionPolicy Bypass -File <skill-dir>/scripts/preflight.ps1
# installation carries no scripts/ (script-blocking harness)? run from the checkout:
sh "$PELTIER_HOME/.claude/skills/peltier/scripts/preflight.sh" sh "$PELTIER_HOME/.claude/skills/peltier/scripts/preflight.sh"
A `STATUS=refuse` from either entry point is final — do not run the other one
looking for a different answer.
Read the `key=value` output; you need `BENCH_RUNNER` and `PIN_SUPPORTED`. A Read the `key=value` output; you need `BENCH_RUNNER` and `PIN_SUPPORTED`. A
refusal ends the **claim**, not the work — report the reason verbatim and do refusal ends the **claim**, not the work — report the reason verbatim and do
not substitute another timing tool. not substitute another timing tool.
@@ -0,0 +1,112 @@
# peltier preflight - PowerShell entry point. Same contract as preflight.sh,
# for hosts where no POSIX shell exists (Windows-native harnesses). Works on
# Windows PowerShell 5.1 and pwsh 7+ on any OS.
#
# THE CONTRACT IS DEFINED ONCE: this script and preflight.sh must emit
# byte-identical output for the same environment. CI enforces that by running
# both on the same host and diffing stdout (skill-preflight job). If you
# change one, change the other, or the build goes red.
#
# STATUS=ok means the harness is present and runnable here. It does NOT mean
# this host can measure your change: only the A/A self-test decides that.
# A refusal (STATUS=refuse, exit 1) ends the *claim*, not the work.
#
# Source is deliberately pure ASCII: Windows PowerShell 5.1 misdecodes
# BOM-less UTF-8, so the em dashes in the shared refusal strings are composed
# at runtime instead of written literally.
$ErrorActionPreference = 'Stop'
$EM = [string][char]0x2014 # em dash, byte-identical to preflight.sh output
function Say([string]$s) { Write-Output $s }
function Refuse([string]$reason) {
Say 'STATUS=refuse'
Say "REASON=$reason"
exit 1
}
# --- host -------------------------------------------------------------
# bench-runner shells every timed run through `sh -c`, and the in-repo
# pipeline is POSIX-only. Windows compiles the workspace but cannot run it.
# Host check comes FIRST, exactly as in preflight.sh.
$onLinux = (Test-Path variable:IsLinux) -and $IsLinux
$onMac = (Test-Path variable:IsMacOS) -and $IsMacOS
if ($onLinux) { $hostKind = 'linux' }
elseif ($onMac) { $hostKind = 'darwin' }
else {
Refuse ("unsupported host 'Windows' $EM peltier's harness is Linux/POSIX-only at runtime. " +
"Run it on Linux (or macOS for verify mode). Do not substitute another benchmark.")
}
# --- trust layer ------------------------------------------------------
function Test-Peltier([string]$dir) {
(Test-Path (Join-Path $dir 'crates/bench-runner/Cargo.toml') -PathType Leaf) -and
(Test-Path (Join-Path $dir 'config/accept.toml') -PathType Leaf)
}
# Walk up from a starting dir looking for a checkout; parents terminate at
# the filesystem root, so this cannot spin.
function Find-Peltier([string]$start) {
$d = $start
while (-not [string]::IsNullOrEmpty($d)) {
if (Test-Peltier $d) { return $d }
$parent = Split-Path -Parent $d
if ($parent -eq $d) { break }
$d = $parent
}
return $null
}
$peltierHome = $null
if ($env:PELTIER_HOME) {
$canon = $null
try { $canon = (Resolve-Path -LiteralPath $env:PELTIER_HOME).Path } catch {}
if (-not $canon) { Refuse "PELTIER_HOME='$($env:PELTIER_HOME)' does not exist" }
if (-not (Test-Peltier $canon)) {
Refuse "PELTIER_HOME='$($env:PELTIER_HOME)' is not a peltier checkout (no crates/bench-runner/Cargo.toml + config/accept.toml)"
}
$peltierHome = $canon
} else {
# cwd first (working inside a checkout), then this script's own location
# (the skill still living inside one). A skill copied into another project
# finds neither - which is why PELTIER_HOME exists.
$peltierHome = Find-Peltier (Get-Location).Path
if (-not $peltierHome) { $peltierHome = Find-Peltier $PSScriptRoot }
}
if (-not $peltierHome) {
Refuse ("no peltier checkout found $EM clone https://github.com/bdbrown4/peltier.io and set " +
"PELTIER_HOME=/path/to/peltier.io. The statistics live in bench-runner; do not reimplement them.")
}
if (-not (Get-Command cargo -ErrorAction SilentlyContinue)) {
Refuse "cargo not on PATH $EM needed to build bench-runner from $peltierHome"
}
# --- build the real harness -------------------------------------------
# Always build. cargo is a no-op when current, and skipping this because a
# binary happens to exist would silently measure with a stale copy of the
# statistics after a `git pull`.
$br = Join-Path $peltierHome 'target/release/bench-runner'
Push-Location $peltierHome
try { & cargo build --release -q -p bench-runner; $buildOk = ($LASTEXITCODE -eq 0) }
finally { Pop-Location }
if (-not $buildOk) {
Refuse "bench-runner failed to build in $peltierHome $EM fix the build before claiming any number"
}
if (-not (Test-Path $br -PathType Leaf)) {
Refuse "bench-runner did not appear at $br after a successful build"
}
# --- pinning ----------------------------------------------------------
# NOTE: config/accept.toml's `pin_prefix` is read ONLY by the in-repo verdict
# pipeline. For compare/aa/calibrate you must wrap the commands yourself
# (`taskset -c N <cmd>` on both sides). PIN_SUPPORTED just says whether that
# tool is available here.
if ($onLinux -and (Get-Command taskset -ErrorAction SilentlyContinue)) { $pin = 'yes' } else { $pin = 'no' }
Say 'STATUS=ok'
Say "PELTIER_HOME=$peltierHome"
Say "BENCH_RUNNER=$br"
Say "HOST=$hostKind"
Say "PIN_SUPPORTED=$pin"
Say 'NOTE=harness runnable here; the A/A self-test decides whether it can measure your change'
+68
View File
@@ -186,6 +186,31 @@ jobs:
sh "$GITHUB_WORKSPACE/.claude/skills/peltier/scripts/preflight.sh" | tee pf.txt sh "$GITHUB_WORKSPACE/.claude/skills/peltier/scripts/preflight.sh" | tee pf.txt
grep -q '^STATUS=ok$' pf.txt grep -q '^STATUS=ok$' pf.txt
- name: sh and ps1 preflights are byte-equivalent (two entry points, one contract)
run: |
# preflight.ps1 exists for hosts with no POSIX shell. Two
# implementations of the same contract can drift, so both run here
# on the same host and their stdout is byte-diffed on every path.
# 1. in-checkout resolve
sh .claude/skills/peltier/scripts/preflight.sh > /tmp/eq-sh.txt
pwsh -NoProfile -File .claude/skills/peltier/scripts/preflight.ps1 > /tmp/eq-ps.txt
diff /tmp/eq-sh.txt /tmp/eq-ps.txt
# 2. refusal with no checkout reachable (copied out of the repo)
cp -r .claude/skills/peltier /tmp/eqskill
mkdir -p /tmp/eq-not-a-repo && cd /tmp/eq-not-a-repo
set +e
env -u PELTIER_HOME sh /tmp/eqskill/scripts/preflight.sh > sh.txt 2>&1; shrc=$?
env -u PELTIER_HOME pwsh -NoProfile -File /tmp/eqskill/scripts/preflight.ps1 > ps.txt 2>&1; psrc=$?
set -e
test "$shrc" -ne 0 && test "$psrc" -ne 0
diff sh.txt ps.txt
# 3. resolve via PELTIER_HOME from a foreign repo
mkdir -p /tmp/eq-foreign && cd /tmp/eq-foreign
PELTIER_HOME="$GITHUB_WORKSPACE" sh /tmp/eqskill/scripts/preflight.sh > sh2.txt
PELTIER_HOME="$GITHUB_WORKSPACE" pwsh -NoProfile -File /tmp/eqskill/scripts/preflight.ps1 > ps2.txt
diff sh2.txt ps2.txt
echo "equivalence holds on all three paths"
- name: The harness the skill drives actually runs (plumbing, not verdict) - name: The harness the skill drives actually runs (plumbing, not verdict)
run: | run: |
BR=$(sed -n 's/^BENCH_RUNNER=//p' /tmp/pf.txt) BR=$(sed -n 's/^BENCH_RUNNER=//p' /tmp/pf.txt)
@@ -200,3 +225,46 @@ jobs:
run: | run: |
BR=$(sed -n 's/^BENCH_RUNNER=//p' /tmp/pf.txt) BR=$(sed -n 's/^BENCH_RUNNER=//p' /tmp/pf.txt)
"$BR" --config /tmp/accept.toml aa --cmd 'sleep 0.02' "$BR" --config /tmp/accept.toml aa --cmd 'sleep 0.02'
# The refusal doctrine must survive hosts with no POSIX shell: on a real
# Windows runner, BOTH preflight entry points must refuse with a structured
# STATUS=refuse — never a launch failure an agent could misread as license
# to hand-roll a timing loop. Windows PowerShell 5.1 is tested separately
# from pwsh because 5.1 misdecodes BOM-less UTF-8 (the ps1 source is pure
# ASCII for exactly that reason).
skill-preflight-windows:
runs-on: windows-latest
steps:
- uses: actions/checkout@v4
- name: ps1 refuses natively (Windows PowerShell 5.1)
shell: powershell
run: |
$out = & powershell -NoProfile -ExecutionPolicy Bypass -File .claude\skills\peltier\scripts\preflight.ps1
$out | Write-Output
if ($LASTEXITCODE -eq 0) { throw "preflight did not refuse on Windows" }
$joined = $out -join "`n"
if ($joined -notlike "*STATUS=refuse*") { throw "no STATUS=refuse in output" }
if ($joined -notlike "*unsupported host 'Windows'*") { throw "missing refusal reason" }
Write-Output "native refusal verified (5.1)"
# the runner appends `exit $LASTEXITCODE`; the refusal's exit 1 is
# the EXPECTED outcome we just asserted, so clear it explicitly
exit 0
- name: ps1 refuses natively (pwsh)
shell: pwsh
run: |
$out = & pwsh -NoProfile -File .claude\skills\peltier\scripts\preflight.ps1
$out | Write-Output
if ($LASTEXITCODE -eq 0) { throw "preflight did not refuse under pwsh" }
if (($out -join "`n") -notlike "*STATUS=refuse*") { throw "no STATUS=refuse in output" }
Write-Output "native refusal verified (pwsh)"
exit 0 # clear the asserted refusal's exit 1 (runner appends exit $LASTEXITCODE)
- name: sh entry point also refuses here (Git Bash)
shell: bash
run: |
set +e
sh .claude/skills/peltier/scripts/preflight.sh > out.txt 2>&1
rc=$?
set -e
cat out.txt
test "$rc" -ne 0
grep -q '^STATUS=refuse$' out.txt
+1 -1
View File
@@ -29,7 +29,7 @@ Full engineering spec: `SPEC.md`. Read it before starting any phase.
- `targets/` — vendored OSS targets for case studies (permissive licenses only) - `targets/` — vendored OSS targets for case studies (permissive licenses only)
- `playbook/` — optimization classes, ordered, with preconditions and known risks - `playbook/` — optimization classes, ordered, with preconditions and known risks
- `results/` — generated reports and flamegraphs - `results/` — generated reports and flamegraphs
- `.claude/skills/peltier/` — the trust layer packaged as a consumable **Agent Skill** (agentskills.io SKILL.md standard, spec-compliant frontmatter): `verify` mode (equivalence → A/A calibration → interleaved A/B + bootstrap CI) works in any repo against any two shell commands; `attempt` mode drives the full in-repo loop. Drives the real `bench-runner` binary and refuses on unsupported hosts rather than falling back to a second, forkable copy of the statistics. Cross-harness: mirrored byte-identically at `.agents/skills/peltier/` (the Codex/Zed/Cursor/Gemini/Copilot convergence path — CI enforces the mirror), `AGENTS.md` at the repo root points non-Claude harnesses here, and `scripts/install-skill.sh` stamps consumer repos (plus a script-less `--zeroclaw-variant`, since ZeroClaw blocks script files in skills by default — its preflight runs from the checkout). Preflight refusal, spec conformance, mirror equality, and the installer are all asserted in CI (`skill-preflight`). - `.claude/skills/peltier/` — the trust layer packaged as a consumable **Agent Skill** (agentskills.io SKILL.md standard, spec-compliant frontmatter): `verify` mode (equivalence → A/A calibration → interleaved A/B + bootstrap CI) works in any repo against any two shell commands; `attempt` mode drives the full in-repo loop. Drives the real `bench-runner` binary and refuses on unsupported hosts rather than falling back to a second, forkable copy of the statistics. Cross-harness: mirrored byte-identically at `.agents/skills/peltier/` (the Codex/Zed/Cursor/Gemini/Copilot convergence path — CI enforces the mirror), `AGENTS.md` at the repo root points non-Claude harnesses here, and `scripts/install-skill.sh` stamps consumer repos (plus a script-less `--zeroclaw-variant`, since ZeroClaw blocks script files in skills by default — its preflight runs from the checkout). Preflight ships as two entry points implementing one contract — `preflight.sh` (POSIX) and `preflight.ps1` (Windows PowerShell 5.1/pwsh; pure-ASCII source, em dashes composed at runtime) — CI byte-diffs their output on all three paths and a `windows-latest` job asserts the native refusal. Preflight refusal, sh↔ps1 equivalence, spec conformance, mirror equality, and the installer are all asserted in CI (`skill-preflight`, `skill-preflight-windows`).
**Phase 4 COMPLETE — all SPEC §5 exit criteria met. Phases 0–4 are all done.** Service-mode latency benchmarking shipped: a coordinated-omission-correct, open-loop, fixed-rate load generator (`crates/bench-runner/src/service.rs`) measuring latency from each request's intended send time (queueing counted, not hidden), interleaved A/B, exact percentiles, bootstrap p50/p99 CIs. A real service target — `targets/cjson/service.c`, a minimal HTTP server wrapping the patched `cJSON.c` (trust-layer, outside the workspace allowlist). Service calibration PASS (0/10 A/A false-positive, 10/10 injected-5%-latency-regression detection; `results/calibration/cjson-service-aa.json`). **The verified batch win phase3-cjson-002 measured under a 150 rps-target replay (20 interleaved rounds, 0 drops — the `phase4-cjson-service` ledger row records sessions=20, rate_rps=150, drop_rate=0.0; total request count is not machine-recorded): p50 latency +6.2%, 95% CI [+5.8%, +7.2%] — accepted; p99 NOT claimed (CI [0.07, 4.97], single-worker loopback tail jitter) — harness correctly rejected it.** Mechanical ROI report generator (`crates/report` bin, `just report`, SPEC §9): reads a ledger row + service-latency JSON → throughput→cores→dollars (27.5 cores / $9,621-per-year CI lower bound on a 500-core fleet) *and* latency percentiles, every figure with CI + workload + methodology inline; flags any non-accepted row. Case study: `results/phase4/`. **The roadmap is complete — the profile→hypothesize→patch→gated-verdict→ROI loop runs end to end across Rust and C, batch and service, with zero shipped false accepts across all phases.** **Phase 4 COMPLETE — all SPEC §5 exit criteria met. Phases 0–4 are all done.** Service-mode latency benchmarking shipped: a coordinated-omission-correct, open-loop, fixed-rate load generator (`crates/bench-runner/src/service.rs`) measuring latency from each request's intended send time (queueing counted, not hidden), interleaved A/B, exact percentiles, bootstrap p50/p99 CIs. A real service target — `targets/cjson/service.c`, a minimal HTTP server wrapping the patched `cJSON.c` (trust-layer, outside the workspace allowlist). Service calibration PASS (0/10 A/A false-positive, 10/10 injected-5%-latency-regression detection; `results/calibration/cjson-service-aa.json`). **The verified batch win phase3-cjson-002 measured under a 150 rps-target replay (20 interleaved rounds, 0 drops — the `phase4-cjson-service` ledger row records sessions=20, rate_rps=150, drop_rate=0.0; total request count is not machine-recorded): p50 latency +6.2%, 95% CI [+5.8%, +7.2%] — accepted; p99 NOT claimed (CI [0.07, 4.97], single-worker loopback tail jitter) — harness correctly rejected it.** Mechanical ROI report generator (`crates/report` bin, `just report`, SPEC §9): reads a ledger row + service-latency JSON → throughput→cores→dollars (27.5 cores / $9,621-per-year CI lower bound on a 500-core fleet) *and* latency percentiles, every figure with CI + workload + methodology inline; flags any non-accepted row. Case study: `results/phase4/`. **The roadmap is complete — the profile→hypothesize→patch→gated-verdict→ROI loop runs end to end across Rust and C, batch and service, with zero shipped false accepts across all phases.**
+7 -1
View File
@@ -83,7 +83,13 @@ document. In `verify` mode, the gate order is the content:
Measurement requires a Linux/POSIX host — the same requirement as the rest of Measurement requires a Linux/POSIX host — the same requirement as the rest of
the pipeline. On Windows (or with no checkout reachable) the skill refuses and the pipeline. On Windows (or with no checkout reachable) the skill refuses and
says why. That is asserted, not assumed: on every pull request, CI runs the says why. The refusal itself is cross-OS: preflight ships as both `preflight.sh`
(POSIX) and `preflight.ps1` (Windows PowerShell 5.1 / pwsh), so a Windows-native
harness with no POSIX shell still gets a structured `STATUS=refuse` instead of a
launch error it might misread as license to hand-roll a timing loop. Two
implementations of one contract is a drift risk, so CI byte-diffs their output
on every path (resolve, refuse, `PELTIER_HOME`) on the same host, and a Windows
runner asserts the native refusal on every pull request. That is asserted, not assumed: on every pull request, CI runs the
preflight's refusal path (skill copied outside any checkout) and its resolve preflight's refusal path (skill copied outside any checkout) and its resolve
paths (in-checkout and via `PELTIER_HOME` from a foreign repo), alongside a paths (in-checkout and via `PELTIER_HOME` from a foreign repo), alongside a
live plumbing run of the harness it drives. live plumbing run of the harness it drives.