diff --git a/.claude/skills/peltier/SKILL.md b/.claude/skills/peltier/SKILL.md new file mode 100644 index 0000000..fa55131 --- /dev/null +++ b/.claude/skills/peltier/SKILL.md @@ -0,0 +1,124 @@ +--- +name: peltier +description: Prove a performance change is real — or refuse to claim it. Use when optimizing code for speed, verifying a claimed speedup ("is this actually faster?", "did my optimization work?"), checking for a performance regression ("did this PR make things slower?"), or comparing the performance of two implementations. Runs peltier's trust layer: equivalence gates, A/A environment calibration, interleaved A/B with bootstrap confidence intervals. Never reports a naked percentage and never claims a win that is not statistically significant. Do NOT use for correctness-only work. +--- + +# peltier — prove it, or claim nothing + +A performance change that has not passed equivalence gates **and** shown a +statistically significant improvement **does not exist**. You may discard it +or log it. You may not claim it. + +Most benchmarking is wrong in ways that flatter the author: run-to-run noise +read as a win, a faster binary that computes something subtly different, one +lucky run out of five, a percentage with no interval attached. This skill +exists to make those failures impossible rather than unlikely. + +## Preflight — never skip, never work around + +Run `scripts/preflight.sh` **relative to this skill's directory** (resolve its +absolute path first — the skill may be installed at `~/.claude/skills/`, as a +plugin, or copied into another project): + + sh /scripts/preflight.sh + +It locates the trust layer, builds the real `bench-runner`, and checks this +host can run it. `STATUS=ok` does **not** mean the host can measure your +change — only the A/A self-test decides that. + +**If it refuses, that ends the claim, not the work.** You may still write the +patch and explain why you think it is faster. You may not attach a number to +it. Do not fall back to `time`, `hyperfine`, or a stopwatch loop — a number +you cannot stand behind is worse than no number, because someone will act on +it. The statistics live in `bench-runner` and are used from there; a second +copy of the math is a second thing that can lie. + +## Two modes + +| Mode | Use when | Read | +|---|---|---| +| **verify** | You have a baseline and a candidate — any repo, any language. | [reference/verify.md](reference/verify.md) | +| **attempt** | You are inside a peltier checkout and want the full loop: profile → hypothesize → patch → gated verdict → ledger row. | [reference/attempt.md](reference/attempt.md) | + +Default to **verify**. It is the portable one, and it is what almost every +request actually wants. + +### No candidate yet? Profile first — do not guess. + +"Measure before optimizing" is not advice, it is a gate. If you have one +program and a hunch, get profile data *before* writing a patch: `perf record` +(native), `py-spy` / `cProfile` (Python), `node --prof` / `--cpu-prof` (Node), +`cargo flamegraph` (Rust), or `just coz` for causal profiling inside a peltier +checkout. Then form a hypothesis about a *named* hot path, and only then patch. +Guessing where the time goes is how people spend a week on 0.3% of the runtime. + +## The order of the gates is the point + +Each gate can end the job, and ending early is a success, not a failure. + +1. **Equivalence — before you time anything.** A faster program that does + something different is not an optimization, it is a bug with good + benchmarks. Prove baseline and candidate produce identical output on a fixed + workload. Comparing two *empty* outputs is a failed gate, not a passed one. +2. **Calibration — can this machine measure this change at all?** The A/A + self-test runs the same command against itself and must find **nothing**. A + harness that "discovers" a speedup between a binary and itself is measuring + noise, and every number it gives you afterwards is fiction. +3. **Measurement — interleaved A/B, bootstrap CI.** Not "run A five times, then + B five times" — that hands any thermal or cache drift to whichever side ran + last. +4. **Risk — what did the patch actually touch?** Concurrency primitives, + `unsafe`/raw memory, floating-point reassociation: `needs-human-review`, + **regardless of how good the numbers are**. In verify mode *you* are the + classifier — `bench-runner` only ever emits `accepted` or `rejected-bench`, + so it cannot make this call for you. +5. **Verdict — the CI lower bound decides.** Accept only if the bootstrap 95% + CI lower bound of the speedup clears the threshold (default 2%). The median + is not the claim. The lower bound is the claim. + +## Non-negotiables + +- **Cheapest wins first:** build flags → LTO → PGO → allocator swap → *then* + code changes. Do not hand-write SIMD before you have tried `-O3`. +- **Every attempt is recorded, including failures.** A rejected hypothesis is a + result — it stops the next person from re-running your dead end. +- **A failed gate is a complete, valid outcome.** Write it up and move on. Do + not iterate on a rejected patch more than twice without a new hypothesis. +- **The thing being measured never grades itself.** The patch may not touch the + harness, the corpus, or the tests. + +## Never + +- **Never a naked percentage.** Every number carries its confidence interval + and the workload it was measured on. "12% faster" is not a result. +- **Never re-run a benchmark until it passes.** That is p-hacking with extra + steps. The first calibrated run is the answer. +- **Never claim a win the CI lower bound does not support**, however good the + median looks and however "obviously" faster the change is. +- **Never present a measurement from a host that failed calibration.** + +## What good looks like + +**A claim, fully dressed** (`phase3-cjson-002`, in the ledger): + +> Accepted. Speedup 1.088 (candidate is 8.8% faster), 95% CI [1.074, 1.100] — +> so the *defensible* claim is ≥7.4%, not 8.8%. Workload: 5.6 MB synthetic +> JSON, 20k records, parsed + serialized 2×, single thread. Gates: upstream +> suite green, golden replay byte-identical, ASan + UBSan clean. + +Note what that write-up does **not** say. It does not claim a differential-fuzz +run, because that row records `fuzz_iters=0` — the fuzzing of that era ran +out-of-band and the machine never witnessed it. Fuzz is a pipeline gate now, and +new rows carry a real count. Say what the record supports, not what you +remember doing. + +**Declining to claim what the data won't support** (`phase4-cjson-service`): +p50 latency improved 6.2% [95% CI +5.8%, +7.2%] at 150 rps — claimed. The p99 +CI came back [0.07, 4.97] — uselessly wide, so **p99 was not claimed at all**, +even though its median looked like a win. + +**Good numbers on a bad change** (`phase2-comrak-010`): 1.107 speedup, 95% CI +[1.079, 1.121], on 11 MB of markdown. Tight interval, real improvement, +**accepted — and wrong.** LeakSanitizer caught it afterwards and the row was +overturned. It never shipped. The bench was never the problem; skipping a gate +was. This is why the order above is not negotiable. diff --git a/.claude/skills/peltier/assets/accept.template.toml b/.claude/skills/peltier/assets/accept.template.toml new file mode 100644 index 0000000..58f32ed --- /dev/null +++ b/.claude/skills/peltier/assets/accept.template.toml @@ -0,0 +1,41 @@ +# peltier accept thresholds — consumer template. +# +# Copy this next to the code you are measuring and pass it with +# `bench-runner --config `. The defaults are peltier's own +# published bar; loosening them to make a result pass is not measurement, +# it is negotiation. + +# The bootstrap CI *lower bound* of the speedup ratio must clear 1 + threshold. +# The median is not the claim. The lower bound is the claim. +threshold = 0.02 + +# Measured runs per side, interleaved ABAB (SPEC minimum: 30). +runs_per_side = 30 + +# Warm-up runs per side, discarded — they pay for cold caches and JIT. +warmup_runs = 3 + +# Bootstrap resamples for the ratio CI. +bootstrap_iters = 10000 + +# Confidence level for the ratio CI. +confidence = 0.95 + +# Fixed PRNG seed so a calibration run is reproducible by anyone. +bootstrap_seed = 1887343589 + +# CPU pinning. WARNING: this key is INERT for `compare`, `aa`, and `calibrate` +# — it is read only by the in-repo `verdict` pipeline. Setting it here and then +# reporting "pinned to core N" would be a claim about a run that never happened. +# +# To actually pin a compare/aa/calibrate run, wrap BOTH sides identically in +# the command itself: --baseline 'taskset -c 2 ' --candidate 'taskset -c 2 ' +# +# Pinning tightens the A/A confidence interval ~5x on peltier's calibration +# hardware. An unpinned host is not disqualified; a NOISY one is, and the A/A +# self-test decides that empirically. Leave this empty. +pin_prefix = "" + +# Differential-fuzz iterations requested per gate run. Only used by the +# in-repo `verdict` pipeline, which requires a target to declare [gates].fuzz. +# fuzz_iters = 10000 diff --git a/.claude/skills/peltier/reference/attempt.md b/.claude/skills/peltier/reference/attempt.md new file mode 100644 index 0000000..2b00b5d --- /dev/null +++ b/.claude/skills/peltier/reference/attempt.md @@ -0,0 +1,104 @@ +# attempt — the full loop, inside a peltier checkout + +Profile → hypothesize → patch → gated verdict → ledger row. This mode only +applies when you are working **inside a peltier checkout** on a target that has +a `targets//target.toml`. Everywhere else, use +[verify.md](verify.md). + +Requires Linux (harnessd uses Unix sockets and `setsid`; every gate shells +through `sh`). + +## Before you touch anything + +**Read the ledger first.** It records every attempt ever made on this target, +including the failures — which is exactly what stops you re-running a dead +end: + + cargo run -p policy # ranks playbook classes by their evidence + +`policy` reads the ledger and ranks optimization classes by the Wilson lower +bound of their shippable-win rate. It is **advisory** — a learned prior over +the fixed cheapest-first order. The gates still decide every verdict. + +**Then respect the order.** Cheapest wins first: build flags → LTO → PGO → +allocator swap → *then* code changes. Do not hand-write SIMD before trying +`-O3`. The playbook (`playbook/`) has the preconditions and known risks for +each class. + +## The boundary you may not cross + +The agent may modify **only** allowlisted paths under `targets//`. +`crates/`, `config/`, `corpora/`, and the upstream test suites are read-only: +the thing being measured does not get to grade itself. `harnessd` enforces +this on every proposed diff (path allowlist over every diff header, including +`rename`/`copy` — a patch that tries to escape is rejected, not sanitized). + +## Log the hypothesis *before* patching + +A hypothesis written after seeing the numbers is not a hypothesis. State what +you think is slow, why, and what you expect the change to buy — then patch. + +## Run the gates + + just gates + +Corpus pin → test-suite pin (when `corpora//TESTSUITE.sha256` exists) → +upstream tests → golden replay. Note what this **does not** do: differential +fuzz needs a pristine baseline to differ against, so it reports `Skipped` here +and runs for real on the accept path below. `just gates` is a fast equivalence +check, not a fuzz run — do not report it as one. + +## The verdict — the only thing that can accept a change + + just verdict "" "" \ + --patch-file + +This rebuilds a **pristine baseline** from the pinned upstream commit (not +whatever is lying around in your tree), runs the full gate set against the +candidate, benches interleaved A/B against that baseline, and writes one +append-only ledger row with the verdict and the complete evidence. + +It will refuse to hand you an `accepted` unless: + +- the upstream test suite is green **and** golden replay is byte-identical (or + within the target's declared FP tolerance); +- the **differential-fuzz gate actually passed** — a real run reporting zero + divergences. No fuzz run, no machine accept. Ever; +- ASan/UBSan are clean (and TSan, when the target configures it); +- the bootstrap 95% CI lower bound of the speedup clears the threshold in + `config/accept.toml` (default 2%); +- and a conservative lexical risk classifier finds **no** concurrency, unsafe, + or floating-point signals in the patch. If it does, the verdict is capped at + `needs-human-review` no matter how good the numbers are. + +Any of these failing produces a rejected row. **That is a complete, valid +outcome** — the row is the deliverable. Write it and move on. Do not iterate on +a rejected patch more than twice without a genuinely new hypothesis. + +## Report + + just report + +Mechanically generates the ROI from the ledger row: throughput → cores → +dollars, and/or latency percentiles — every figure with its CI, workload, and +methodology printed inline. It flags any row that was not accepted, so a +rejected attempt cannot be laundered into a dollar figure. + +## Unattended + + just agent-attempt + +Runs one full attempt behind the OS boundary: `harnessd` as the trusted uid on +a Unix socket, the agent loop as an unprivileged user, the verdict pipeline +wrapped in a network namespace (`scripts/no-net.sh`) that **fails closed** — if +namespaces are unavailable it refuses to run rather than quietly running with +the network up. `just isolation-check` verifies the boundary from the agent's +side. + +## When the numbers are good and the change is scary + +FP-ordering, concurrency primitives, anything UB-adjacent: `needs-human-review`, +always, regardless of the speedup. This is not a formality. The one false +accept in peltier's history (`phase2-comrak-010`) passed its bench and was +caught by a human audit afterwards — which is precisely why the sanitizer lane +is now machine-enforced and why this rule does not bend. diff --git a/.claude/skills/peltier/reference/verify.md b/.claude/skills/peltier/reference/verify.md new file mode 100644 index 0000000..df90ba8 --- /dev/null +++ b/.claude/skills/peltier/reference/verify.md @@ -0,0 +1,203 @@ +# verify — prove a speedup in any repo + +You have two things that should do the same work, one of which you think is +faster. Find out. Works on any pair of shell commands, in any language. + +## 0. Preflight + + sh /scripts/preflight.sh + +Read the `key=value` output; you need `BENCH_RUNNER` and `PIN_SUPPORTED`. A +refusal ends the **claim**, not the work — report the reason verbatim and do +not substitute another timing tool. + + BR= + +## 1. Config + + cp /assets/accept.template.toml ./peltier-accept.toml + CFG=./peltier-accept.toml + +Do **not** loosen `threshold`, `runs_per_side`, or `confidence` to make a +result pass. That is not measurement, it is negotiation. + +**Pinning is not done through this file.** `pin_prefix` is read only by the +in-repo `verdict` pipeline — `compare`, `aa`, and `calibrate` ignore it +entirely (bench-runner's own fingerprint says `"caller-provided (wrap cmd in +taskset)"`). If `PIN_SUPPORTED=yes` and you know a core is free, pin by +wrapping **both sides identically**: + + --baseline 'taskset -c 2 ' --candidate 'taskset -c 2 ' + +An unpinned host is not disqualified. A *noisy* one is, and step 3 is what +decides that. + +## 2. Two immutable artifacts — not two states of one tree + +The harness runs the two sides **alternately, 66 times**. That is impossible if +"baseline" and "candidate" are the same working tree in different states. +Before measuring: + +- Build each side into its **own** location — a `git worktree` per side, or two + separate build/output directories. Never `git stash` between runs. +- Pass the **built artifact paths** as the commands. Never pass a wrapper that + can rebuild: `cargo run`, `npm start`, `go run`, `mvn exec` will happily + rebuild from whatever is in the tree *mid-benchmark*, so you can end up + timing the same binary twice — and paying build cost inside the timed window. +- Confirm the two artifacts actually differ: `sha256sum `. + +Peltier learned this the hard way (`phase1-verdict-smoke-001`: "workspace paths +get silently rebuilt from the patched tree by test runs"), which is why the +in-repo pipeline rebuilds a pristine baseline from the pinned upstream commit +rather than trusting the tree. + +## 3. Equivalence — before you time anything + +Capture **everything** each side produces, on a fixed, representative workload: + + run_side() { # $1 = tag, $2 = command + rm -rf "out.$1"; mkdir -p "out.$1" + ( eval "$2" ) > "out.$1/stdout" 2> "out.$1/stderr"; echo "$?" > "out.$1/exit" + } + run_side base "" + run_side cand "" + diff -r out.base out.cand && echo "GOLDEN PASS" || echo "DIVERGENT" + +If the command's real output is a **file tree** (a bundler, a compiler, an +image pipeline), hash that too — stdout is not the artifact: + + ( cd dist && find . -type f | sort | xargs sha256sum ) > out./tree + +**An empty diff over empty output is a failed gate, not a passed one.** If both +sides wrote nothing to stdout, nothing to stderr, and you captured no artifact +tree, you have compared nothing — stop and find the real output before going +further. Then run the project's own test suite against the candidate; it must +pass. + +- **Identical output + tests green** → proceed. +- **Divergent** → stop. No speedup may be claimed from a change you cannot show + is behavior-preserving. The one exception — a change that *legitimately* + alters output, floating-point reassociation being the common case — is + `needs-human-review`, never an auto-accept. Quantify the divergence (how many + values, by how much) and hand it to a human. + +## 4. Calibration — can this host measure this change? + +The A/A self-test runs the **same command on both sides**. It must find nothing: + + $BR --config "$CFG" aa --cmd '' + +- stdout `A/A self-test passed (null verdict)` → the harness is not inventing + speedups. Proceed. +- **Nonzero exit**, with `Error: A/A self-test FAILED: harness claims a speedup + from identical binaries` on **stderr** → stop. This host is too noisy to + measure this change; every number it produces afterwards is fiction. Report + the noise floor, then quiet the machine (pin a core, disable turbo/SMT, close + everything else) or move to calibrated hardware. + +For a claim anyone will act on, run the full calibration — a false-positive +rate over many A/A sessions *plus* proof the harness can detect a **known +injected regression**: + + $BR --config "$CFG" calibrate --cmd '' \ + --sessions 20 --slowdown 0.05 --out calibration.json + +Required: **<5% A/A false-positive rate** and **≥95% detection** of the injected +5% slowdown. A harness that cannot see a real 5% regression cannot be trusted +to confirm a real 5% win. Keep `calibration.json` — it is the evidence that the +measurement itself was sound. + +### Budget the runtime before you start + +`compare` executes the workload **(30 + 3) × 2 = 66 times**. `calibrate +--sessions 20` executes it roughly **2,650 times**. Size the workload to +**~0.2–2 s** of wall time: shorter and you measure process startup, longer and +calibration takes hours. Run `calibrate` in the background — it will blow +through a foreground command timeout. + +## 5. Measure — interleaved A/B, bootstrap CI + + $BR --config "$CFG" compare --baseline '' --candidate '' + +Runs are **interleaved** (ABAB…), not blocked, so thermal drift and cache state +hit both sides equally. Output: + + speedup (baseline/candidate): median 1.0884, 95% CI [1.0745, 1.1001] + verdict: accepted + +The ratio is `baseline/candidate`, so **>1 means the candidate is faster**. + +`--perf-stat` (Linux) adds cycles/instructions/cache-miss medians — useful for +explaining *why* something got faster, **never** part of the accept decision, +and usually unavailable in VMs. + +## 6. Risk — you are the classifier here + +`bench-runner` emits only `accepted` or `rejected-bench`. It **cannot** route a +dangerous change to review, because in verify mode nothing is inspecting your +diff. So inspect it yourself, and override the tool: + +Read the patch. If it touches any of these, the outcome is **`needs-human-review` +regardless of what `compare` printed**: + +- **Concurrency** — threads, locks, atomics, memory ordering, `volatile`, work + queues, async pools, `rayon`/`pthread`/`std::sync`. +- **Unsafe / raw memory** — `unsafe`, pointer casts, `transmute`, FFI, + manual allocation, uninitialized memory. +- **Floating-point reassociation** — reordering accumulation, `-ffast-math`, + FP SIMD, changed summation order. + +These changes can be *correct on your workload and wrong in production*, and a +green bench says nothing about that. A byte-identical output does not clear +them: a data race produces identical output right up until it doesn't. + +## 7. Verdict — read it honestly + +| What you see | What it means | What you may say | +|---|---|---| +| `verdict: accepted` | CI lower bound clears the threshold. | Claim it — with the **CI lower bound**, the interval, and the workload. | +| `verdict: rejected-bench`, CI spans 1.0 (e.g. `[0.983, 1.001]`) | Indistinguishable from no change. | "No shippable win." A real, complete result. | +| `verdict: rejected-bench`, CI **entirely below** 1.0 (e.g. `[0.91, 0.96]`) | A measured **regression**. | Report it as one, with the interval. Do not call it "no win." | +| Any risk signal from step 6 | Out of the machine's hands. | `needs-human-review`. Never auto-accept. | + +Claim the **lower bound**, not the median: a median of 1.088 with CI +[1.074, 1.100] supports "≥7.4% faster", not "8.8% faster". + +If the result is a rejection, **that is the answer**. Re-running until it passes +is p-hacking. Two failed hypotheses on the same hot path means you need a new +hypothesis, not another run. + +## Services (latency, not throughput) + +For a server, wall-clock-per-run is the wrong instrument. Service mode is an +open-loop, fixed-rate load generator that measures each request from its +**intended** send time, so queueing delay is counted rather than hidden +(coordinated omission). It reports p50/p99 with bootstrap CIs. + +**It is not a generic HTTP load generator.** The server binary must: + +- take ` ` as positional argv; +- print a line beginning with `READY` on stdout, **flushed**, once listening; +- answer `GET /` and **close the connection** (`Connection: close`) — the client + reads until EOF with no timeout, so a keep-alive server hangs the harness + forever. + +Point it at a stock Express or Flask app and it will either fail the READY +handshake or block indefinitely. Peltier wrote a thin adapter binary +(`targets/cjson/service.c`) for exactly this reason; you will need one too. + + $BR --config "$CFG" service --baseline-bin '' --candidate-bin '' \ + --doc --rate 150 --sessions 20 + +Calibrate it the same way first (`service-calibrate`). Tail percentiles on a +single-worker loopback setup are jittery — **if the p99 CI is wide, do not claim +p99**, even when p50 is solid. Peltier's own shipped service result claims p50 +(+6.2%, CI [+5.8%, +7.2%]) and explicitly declines to claim p99 (CI [0.07, 4.97]). + +## Record the attempt — including the failures + +Outside peltier there is no ledger, so write the evidence where the next person +will find it (a results file, the PR body): hypothesis, what changed, +equivalence result, calibration evidence, the CI, the workload, and the verdict. +**Failures are the most valuable rows** — they stop someone from re-running your +dead end in six months. diff --git a/.claude/skills/peltier/scripts/preflight.sh b/.claude/skills/peltier/scripts/preflight.sh new file mode 100755 index 0000000..c30dbc9 --- /dev/null +++ b/.claude/skills/peltier/scripts/preflight.sh @@ -0,0 +1,91 @@ +#!/bin/sh +# peltier preflight — locate the trust layer, check this host can RUN it, and +# build the real bench-runner. +# +# STATUS=ok means the harness is present and runnable here. It does NOT mean +# this host can measure your change: only the A/A self-test decides that, and +# it decides it empirically. Do not read STATUS=ok as "calibrated". +# +# A refusal (STATUS=refuse, nonzero exit) is a complete, valid outcome. It +# ends the *claim*, not the work: you may still write and explain a patch, you +# just may not attach a number to it. Never work around a refusal with a +# hand-rolled timing loop. +set -eu + +say() { printf '%s\n' "$1"; } +refuse() { + say "STATUS=refuse" + say "REASON=$1" + exit 1 +} +abspath() { ( CDPATH= cd -- "$1" 2>/dev/null && pwd ); } + +# --- host ------------------------------------------------------------- +# bench-runner shells every timed run through `sh -c`, and the in-repo +# pipeline (harnessd, service mode, the sh-based gates) is POSIX-only. +# Windows compiles the workspace but cannot run it. +uname_s=$(uname -s 2>/dev/null || echo unknown) +case "$uname_s" in + Linux) host=linux ;; + Darwin) host=darwin ;; + *) refuse "unsupported host '$uname_s' — peltier's harness is Linux/POSIX-only at runtime. Run it on Linux (or macOS for verify mode). Do not substitute another benchmark." ;; +esac + +# --- trust layer ------------------------------------------------------ +is_peltier() { [ -f "$1/crates/bench-runner/Cargo.toml" ] && [ -f "$1/config/accept.toml" ]; } + +# Walk up from a starting dir looking for a checkout. Absolute POSIX paths +# always terminate at "/", so this cannot spin. +walk_up() { + d=$1 + while [ -n "$d" ] && [ "$d" != "/" ]; do + if is_peltier "$d"; then printf '%s' "$d"; return 0; fi + d=$(dirname "$d") + done + is_peltier / && printf '/' + return 0 +} + +home="" +if [ -n "${PELTIER_HOME:-}" ]; then + canon=$(abspath "$PELTIER_HOME" || true) + [ -n "$canon" ] || refuse "PELTIER_HOME='$PELTIER_HOME' does not exist" + is_peltier "$canon" || refuse "PELTIER_HOME='$PELTIER_HOME' is not a peltier checkout (no crates/bench-runner/Cargo.toml + config/accept.toml)" + home=$canon +else + # cwd first (working inside a checkout), then this script's own location + # (the skill still living inside one). A skill copied into another project + # finds neither — which is why PELTIER_HOME exists. + home=$(walk_up "$(pwd)") + [ -n "$home" ] || home=$(walk_up "$(abspath "$(dirname -- "$0")")") +fi +[ -n "$home" ] || refuse "no peltier checkout found — clone https://github.com/bdbrown4/peltier.io and set PELTIER_HOME=/path/to/peltier.io. The statistics live in bench-runner; do not reimplement them." + +command -v cargo >/dev/null 2>&1 || refuse "cargo not on PATH — needed to build bench-runner from $home" + +# --- build the real harness ------------------------------------------- +# Always build. cargo is a no-op when current, and skipping this because a +# binary happens to exist would silently measure with a stale copy of the +# statistics after a `git pull`. +br="$home/target/release/bench-runner" +( cd "$home" && cargo build --release -q -p bench-runner ) \ + || refuse "bench-runner failed to build in $home — fix the build before claiming any number" +{ [ -f "$br" ] && [ -x "$br" ]; } || refuse "bench-runner did not appear at $br after a successful build" + +# --- pinning ---------------------------------------------------------- +# NOTE: config/accept.toml's `pin_prefix` is read ONLY by the in-repo verdict +# pipeline. For compare/aa/calibrate you must wrap the commands yourself +# (`taskset -c N ` on both sides). PIN_SUPPORTED just says whether that +# tool is available here. +if [ "$host" = linux ] && command -v taskset >/dev/null 2>&1; then + pin=yes +else + pin=no +fi + +say "STATUS=ok" +say "PELTIER_HOME=$home" +say "BENCH_RUNNER=$br" +say "HOST=$host" +say "PIN_SUPPORTED=$pin" +say "NOTE=harness runnable here; the A/A self-test decides whether it can measure your change" diff --git a/.gitattributes b/.gitattributes new file mode 100644 index 0000000..fae6997 --- /dev/null +++ b/.gitattributes @@ -0,0 +1,4 @@ +# Hash manifests and shell scripts must stay LF even on Windows checkouts — +# sha256sum -c and POSIX sh both break on CRLF working-tree copies. +*.sha256 text eol=lf +*.sh text eol=lf diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 5229460..aec64d5 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -91,3 +91,60 @@ jobs: $PERF record -e cpu-clock -o /tmp/p.data -- sh -c 'i=0; while [ $i -lt 2000000 ]; do i=$((i+1)); done' && $PERF report -i /tmp/p.data --stdio | head -5 || echo "cpu-clock sampling unavailable" echo "--- hardware cycle counter:" $PERF stat -e cycles true 2>&1 | tail -3 || true + + # The peltier skill's contract: preflight must resolve the trust layer when + # one is reachable, and REFUSE when it is not. The refusal is the + # safety-critical path — a skill that silently degrades to a hand-rolled + # timing loop is worse than no skill — so it is asserted here, not assumed. + # Statistical outcomes are NOT asserted: a shared runner is too noisy for + # that, which is the whole reason the skill calibrates before it claims. + skill-preflight: + runs-on: ubuntu-latest + steps: + - uses: actions/checkout@v4 + - uses: dtolnay/rust-toolchain@stable + - uses: Swatinem/rust-cache@v2 + + - name: Preflight resolves inside a peltier checkout + run: | + sh .claude/skills/peltier/scripts/preflight.sh | tee /tmp/pf.txt + grep -q '^STATUS=ok$' /tmp/pf.txt + grep -q "^PELTIER_HOME=$PWD$" /tmp/pf.txt + BR=$(sed -n 's/^BENCH_RUNNER=//p' /tmp/pf.txt); test -x "$BR" + + - name: Preflight REFUSES with no trust layer reachable + run: | + # The skill must be copied OUT of the checkout: preflight also walks + # up from its own location, so running it in-place would find this + # very repo and (correctly) resolve — testing nothing. + cp -r .claude/skills/peltier /tmp/skillcopy + cd /tmp && mkdir -p not-a-repo && cd not-a-repo + set +e + env -u PELTIER_HOME sh /tmp/skillcopy/scripts/preflight.sh > out.txt 2>&1 + rc=$? + set -e + cat out.txt + test "$rc" -ne 0 || { echo "FAIL: preflight did not refuse"; exit 1; } + grep -q '^STATUS=refuse$' out.txt + + - name: Preflight works when consumed from another repo (PELTIER_HOME) + run: | + cd /tmp && mkdir -p other-project && cd other-project + PELTIER_HOME="$GITHUB_WORKSPACE" \ + sh "$GITHUB_WORKSPACE/.claude/skills/peltier/scripts/preflight.sh" | tee pf.txt + grep -q '^STATUS=ok$' pf.txt + + - name: The harness the skill drives actually runs (plumbing, not verdict) + run: | + BR=$(sed -n 's/^BENCH_RUNNER=//p' /tmp/pf.txt) + cp .claude/skills/peltier/assets/accept.template.toml /tmp/accept.toml + "$BR" --config /tmp/accept.toml compare \ + --baseline 'sleep 0.02' --candidate 'sleep 0.02' | tee /tmp/cmp.txt + grep -q 'speedup (baseline/candidate): median .*95% CI \[' /tmp/cmp.txt + grep -q '^verdict: ' /tmp/cmp.txt + + - name: Noise floor of this runner (informational — A/A must find nothing) + continue-on-error: true + run: | + BR=$(sed -n 's/^BENCH_RUNNER=//p' /tmp/pf.txt) + "$BR" --config /tmp/accept.toml aa --cmd 'sleep 0.02' diff --git a/.gitignore b/.gitignore index 6cf1a9e..06ff0d8 100644 --- a/.gitignore +++ b/.gitignore @@ -234,7 +234,9 @@ targets/comrak/asan*/ targets/tokei/candidate-*/ targets/tokei/asan/ !results/calibration/*.json +!results/phase4/*.json targets/*/baseline/ +targets/*/fuzz-work/ # Phase 2 agent artifacts (transcripts kept; pending logs are scratch) results/pending/ @@ -250,3 +252,4 @@ targets/cjson/asan*/ targets/cjson/svc-baseline/ targets/cjson/svc-candidate/ docs/book/ + diff --git a/CLAUDE.md b/CLAUDE.md index 372ded2..7b09c30 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -16,58 +16,70 @@ Full engineering spec: `SPEC.md`. Read it before starting any phase. 4. **Equivalence gates are hard gates.** Changes touching floating-point ordering, concurrency primitives, or anything UB-adjacent are NEVER auto-accepted — verdict is `needs-human-review`. 5. **Every attempt goes in the ledger**, including failures and rejections. Failures are future training data. 6. **The agent never modifies the trust layer.** `crates/`, `config/`, `corpora/`, and upstream test suites are read-only to the agent. Patches may touch only allowlisted paths under `targets//`. See SPEC.md §10. -7. **Sandbox everything.** Target code runs only inside no-network containers. Never on the host. +7. **Sandbox everything.** Target code must never run unisolated on the host. *As built:* the harnessd-launched verdict pipeline execs through `scripts/no-net.sh` (Linux network namespace; **fails closed** — exit 97 — when namespaces are unavailable, unless `HOTPATH_ALLOW_UNISOLATED=1`, which runs it **with** network and stamps that fact into the ledger's isolation note). CI runs the bench workload under `docker run --network=none`. Two honest limits: a human invoking `just verdict` directly is **not** wrapped, and full-container (seccomp-restricted) isolation of the whole pipeline is still an **open gap**. See SPEC §10. 8. **Statistical bar:** accept only if the bootstrap 95% CI lower bound of speedup ≥ threshold in `config/accept.toml` (default 2%), from interleaved A/B runs on the same pinned hardware. ## Architecture map -- `crates/bench-runner` — trust layer: containerized, CPU-pinned, interleaved A/B benchmarking with bootstrap confidence intervals and A/A self-tests -- `crates/diff-test` — behavioral equivalence: upstream test suite + golden I/O replay + differential fuzzing of changed functions + sanitizers +- `crates/bench-runner` — trust layer: CPU-pinned, interleaved A/B benchmarking with bootstrap confidence intervals and A/A self-tests (it does **not** containerize anything itself — isolation is the wrapper's job, non-negotiable 7) +- `crates/diff-test` — behavioral equivalence: corpus/test-suite pin checks + upstream test suite + golden I/O replay + differential fuzz (a baseline-vs-candidate gate; sanitizers run on the accept path in `crates/verdict`, and `diff-test` reports them as skipped-with-reason) - `crates/ledger` — append-only SQLite record of every attempt: hypothesis, patch, gate results, bench deltas, verdict, cost - `crates/report` — ROI generator: cores saved × $/core-hr, latency percentile deltas, CIs, workload caveats printed on every number - `agent/` — Claude Agent SDK (Python) loop, prompts, tool definitions - `targets/` — vendored OSS targets for case studies (permissive licenses only) - `playbook/` — optimization classes, ordered, with preconditions and known risks - `results/` — generated reports and flamegraphs +- `.claude/skills/peltier/` — the trust layer packaged as a consumable Claude Code skill: `verify` mode (equivalence → A/A calibration → interleaved A/B + bootstrap CI) works in any repo against any two shell commands; `attempt` mode drives the full in-repo loop. Drives the real `bench-runner` binary and refuses on unsupported hosts rather than falling back to a second, forkable copy of the statistics. Preflight refusal is asserted in CI (`skill-preflight`). -**Phase 4 COMPLETE — all SPEC §5 exit criteria met. Phases 0–4 are all done.** Service-mode latency benchmarking shipped: a coordinated-omission-correct, open-loop, fixed-rate load generator (`crates/bench-runner/src/service.rs`) measuring latency from each request's intended send time (queueing counted, not hidden), interleaved A/B, exact percentiles, bootstrap p50/p99 CIs. A real service target — `targets/cjson/service.c`, a minimal HTTP server wrapping the patched `cJSON.c` (trust-layer, outside the workspace allowlist). Service calibration PASS (0/10 A/A false-positive, 10/10 injected-5%-latency-regression detection; `results/calibration/cjson-service-aa.json`). **The verified batch win phase3-cjson-002 measured under a 150 rps replay (20 rounds, 40k requests, 0 drops): p50 latency +6.2%, 95% CI [+5.8%, +7.2%] — accepted (`phase4-cjson-service` ledger row); p99 NOT claimed (CI [0.07, 4.97], single-worker loopback tail jitter) — harness correctly rejected it.** Mechanical ROI report generator (`crates/report` bin, `just report`, SPEC §9): reads a ledger row + service-latency JSON → throughput→cores→dollars (27.5 cores / $9,621-per-year CI lower bound on a 500-core fleet) *and* latency percentiles, every figure with CI + workload + methodology inline; flags any non-accepted row. Case study: `results/phase4/`. **The roadmap is complete — the profile→hypothesize→patch→gated-verdict→ROI loop runs end to end across Rust and C, batch and service, with zero shipped false accepts across all phases.** +**Phase 4 COMPLETE — all SPEC §5 exit criteria met. Phases 0–4 are all done.** Service-mode latency benchmarking shipped: a coordinated-omission-correct, open-loop, fixed-rate load generator (`crates/bench-runner/src/service.rs`) measuring latency from each request's intended send time (queueing counted, not hidden), interleaved A/B, exact percentiles, bootstrap p50/p99 CIs. A real service target — `targets/cjson/service.c`, a minimal HTTP server wrapping the patched `cJSON.c` (trust-layer, outside the workspace allowlist). Service calibration PASS (0/10 A/A false-positive, 10/10 injected-5%-latency-regression detection; `results/calibration/cjson-service-aa.json`). **The verified batch win phase3-cjson-002 measured under a 150 rps-target replay (20 interleaved rounds, 0 drops — the `phase4-cjson-service` ledger row records sessions=20, rate_rps=150, drop_rate=0.0; total request count is not machine-recorded): p50 latency +6.2%, 95% CI [+5.8%, +7.2%] — accepted; p99 NOT claimed (CI [0.07, 4.97], single-worker loopback tail jitter) — harness correctly rejected it.** Mechanical ROI report generator (`crates/report` bin, `just report`, SPEC §9): reads a ledger row + service-latency JSON → throughput→cores→dollars (27.5 cores / $9,621-per-year CI lower bound on a 500-core fleet) *and* latency percentiles, every figure with CI + workload + methodology inline; flags any non-accepted row. Case study: `results/phase4/`. **The roadmap is complete — the profile→hypothesize→patch→gated-verdict→ROI loop runs end to end across Rust and C, batch and service, with zero shipped false accepts across all phases.** -**Phase 5 COMPLETE — the two SPEC §13 research forks, both built and both routed to human review.** (a) *Learned class-selection policy* (`crates/policy`, `cargo run -p policy`): reads the append-only ledger and ranks optimization classes by the Wilson lower bound of their shippable-win rate — a learned prior over the fixed cheapest-first ordering, advisory only (gates still decide). A "win" counts only machine-sanitizer-verified accepts; the overturned comrak-010 and tier-gated mimalloc are excluded/held. On the current 34-row ledger it recommends algorithmic first (Wilson lb 0.066). (b) *Kernel lane* (`targets/matmul/kernel.c`, `crates/diff-test` FP-tolerance policy + `fp-compare` bin, `scripts/kernel-lane-demo.sh`): a matmul optimization (transpose-B cache locality + eight-accumulator ILP) that **reorders FP accumulation**, so byte-identical golden replay is the wrong gate — 244,901/262,144 values differ. The FP-tolerance gate (`abs 1e-4 + rel 1e-3`) accepts it, still REJECTS a +0.5 perturbation, and the bench measures 3.23× [3.16, 3.26] with the same interleaved A/B + bootstrap CI. Ledger row `phase5-matmul-opt` = needs-human-review (using the tolerance tier is a §8 signal, same posture as mimalloc). No GPU in this environment, so the GPU fork is the identical shape shown on CPU — only the timer and hardware change. Case study: `results/phase5/`; docs chapter `docs/src/research-forks.md`. Open follow-up: the standing accept-scoped ruling on `phase0-comrak-002` (mimalloc) is resolved (`results/rulings/`).** +**Hardening pass (2026-07-13, `audit/resolve-findings`) — what is *enforced* vs. what is only *mechanized*.** Five gates were addressed in code. **Two are live on the verdict path (a, b); three are wired but currently inert, because no target has opted in (c, d, e).** The distinction is stated precisely because collapsing it is exactly the overclaim this project exists to prevent. + +- **(a) Differential fuzz — live, and now blocks accepts.** `diff-test` runs the target's declared `[gates].fuzz` command and grades it strictly from a `FUZZ-RESULT iters= divergences=` line (no line = Failed, not a pass by silence); the ledger records the iteration count **actually executed**. It is a **baseline-vs-candidate** gate, so it runs *only* on the accept path (`crates/verdict`, the one flow that rebuilds a pristine baseline). The standalone `just gates` flow has no baseline, so the gate reports **Skipped with that reason** — `just gates` is a fast equivalence check, **not** a fuzz run. **Hard rule:** a machine `accepted` verdict is **impossible** unless the DifferentialFuzz gate actually *Passed*; skipped or failed caps the verdict at `needs-human-review`. `cjson`/`comrak`/`tokei` declare `[gates].fuzz`; `matmul` does not. *Ledger history:* fuzz ran out-of-band via `scripts/diff-fuzz-*.py` through Phase 5, so **8 of the 10 accepted rows record `fuzz_iters=0`** — the two exceptions, `phase0-tokei-002` and `phase0-tokei-003`, record `fuzz_iters=4332`. +- **(b) Lexical risk classifier — live.** `crates/verdict/src/risk.rs` scans the patch's **changed lines** for concurrency / `unsafe` / floating-point tokens; any signal on a would-be accept forces `needs-human-review`, and using fp-tolerance equivalence mode is itself always a signal. It is **lexical and deliberately over-triggering, not semantic** — it detects the presence of risk markers; it does not understand the code and cannot prove their absence. `harnessd` always passes `--patch-file`, so every agent-proposed patch is classified. +- **(c) Test-suite pinning — mechanized, INERT.** `diff-test` verifies `corpora//TESTSUITE.sha256` **when present** (mismatch = hard refusal, same posture as the corpus pin) and prints an explicit "suite unpinned" warning when absent. **No target ships a pin today — there are zero `TESTSUITE.sha256` files in the repo** — so the gate constrains nothing yet. Pins must be generated per target, post-fetch, by a deliberate `scripts/pin-testsuite.sh` run. +- **(d) TSan lane — mechanized, INERT.** `crates/verdict` builds and runs a TSan lane on the accept path **only when a target declares `[build].tsan`**. **No target declares it today, so no TSan coverage exists.** (ASan/UBSan *is* live on the accept path: a flagged run — or a target with no sanitizer build at all — caps the accept at `needs-human-review`.) +- **(e) FP-tolerance equivalence — mechanized in the pipeline, target NOT wired.** The gate pipeline honors an fp-tolerance policy (`targets//equivalence.toml`; requires `[corpus].golden_reference`), and any fp-tolerance run auto-routes to `needs-human-review`. But the only FP target, `matmul`, has an `equivalence.toml` and **no `target.toml`** — so it is not a pipeline target at all, and the kernel lane stays script-driven (`scripts/kernel-lane-demo.sh`). + +**Phase 5 COMPLETE — the two SPEC §13 research forks, both built and both routed to human review.** (a) *Learned class-selection policy* (`crates/policy`, `cargo run -p policy`): reads the append-only ledger and ranks optimization classes by the Wilson lower bound of their shippable-win rate — a learned prior over the fixed cheapest-first ordering, advisory only (gates still decide). A "win" counts only machine-sanitizer-verified accepts; the overturned comrak-010 and tier-gated mimalloc are excluded/held. On the current 34-row ledger it recommends algorithmic first (Wilson lb 0.066). (b) *Kernel lane* (`targets/matmul/kernel.c`, `crates/diff-test` FP-tolerance policy + `fp-compare` bin, `scripts/kernel-lane-demo.sh`): a matmul optimization (transpose-B cache locality + eight-accumulator ILP) that **reorders FP accumulation**, so byte-identical golden replay is the wrong gate — 244,901/262,144 values differ. The FP-tolerance policy (`abs 1e-4 + rel 1e-3`) accepts it, still REJECTS a +0.5 perturbation, and the bench measures 3.23× [3.16, 3.26] with the same interleaved A/B + bootstrap CI. Ledger row `phase5-matmul-opt` = needs-human-review (using the tolerance tier is a §8 signal, same posture as mimalloc). **Honest scope:** the FP-tolerance *mechanism* lives in the gate pipeline, but `matmul` is **not wired into it** — it has an `equivalence.toml` and no `target.toml`, so `diff-test`/`verdict` cannot load it. The lane is demonstrated script-driven (`fp-compare` + `bench-runner compare`), and that ledger row was recorded outside the automated verdict path. Wiring `matmul` as a real pipeline target is open work. No GPU in this environment, so the GPU fork is the identical shape shown on CPU — only the timer and hardware change. Case study: `results/phase5/`; docs chapter `docs/src/research-forks.md`. Open follow-up: the standing accept-scoped ruling on `phase0-comrak-002` (mimalloc) is resolved (`results/rulings/`).** ## Commands -Fill these in as the harness is built; keep them current: - -``` -just profile # perf record + flamegraph for a target's benchmark workload -just bench # interleaved A/B: pristine baseline vs working tree -just gates # test suite + golden replay + diff-fuzz + sanitizers -just verdict # runs gates + bench, writes ledger row, prints verdict -just report # ROI report from ledger -just aa # A/A calibration run (must show null result) -``` - -Built so far (command-level, pre-target-integration): +The real recipe list (`justfile`); keep it current: ``` just build / test / lint # trust-layer workspace: cargo build/test/clippy+fmt just aa [cmd] # A/A self-test of bench-runner on a shell command just compare # interleaved A/B of two shell commands, bootstrap CI -just gates # corpus pin + upstream tests + golden replay -just pin-check # verify corpus MANIFEST.sha256 +just gates # corpus pin (+ suite pin if present) + upstream tests + golden replay. + # diff-fuzz SKIPS here (needs a baseline) — it runs in `just verdict` +just pin-check # verify corpus MANIFEST.sha256 (verify-only, never re-pins) +just pin-corpus # deliberate corpus re-pin (writes MANIFEST.sha256) just calibrate # automated A/A + regression-injection calibration just verdict ... # gates + bench vs pristine-rebuilt baseline + ledger row -cargo run -p harnessd # agent IPC daemon: six-tool JSON surface (SPEC §3.5) +just report # ROI report from a ledger row (CIs + methodology inline) +just agent-attempt # one unattended agent attempt behind the OS boundary +just isolation-check # verify the SPEC §10 OS boundary (19 checks, both modes) +just service # service-mode latency A/B, CO-correct, p50/p99 CIs +just service-calibrate ... # latency A/A + injected-regression calibration +just coz # causal profile of a C/C++ target +cargo run -p harnessd # agent IPC daemon: seven-op JSON surface (SPEC §3.5) +cargo run -p policy # learned class-ranking prior from the ledger (advisory) ``` +**Platform note:** the harness is Linux/POSIX-only at runtime — bench-runner +service mode, harnessd (Unix sockets, `setsid`), and every sh-based gate and +isolation script assume a Unix host. Windows compiles the workspace and runs +the portable unit tests, but cannot run the pipeline. + ## Definition of done — one optimization attempt - [ ] Hypothesis logged in ledger *before* patching - [ ] Patch touches only allowlisted paths under `targets//` - [ ] Upstream test suite green - [ ] Golden replay byte-identical (or within the target's explicit FP tolerance policy) -- [ ] Differential fuzz on changed functions: no divergence (60s or 10k iterations minimum) -- [ ] ASan + UBSan clean; TSan if the patch touches anything threaded +- [ ] Differential fuzz on changed functions: no divergence (60s or 10k iterations minimum). Requires `[gates].fuzz` in the target's `target.toml` **and** the accept path (`just verdict`) — an accept is impossible without a Passed fuzz gate +- [ ] ASan + UBSan clean (machine-enforced on accept). TSan if the patch touches anything threaded — note the lane only runs when the target declares `[build].tsan`, and **no target declares one today**, so a threaded patch needs that opt-in added first - [ ] Interleaved A/B bench passes the CI-lower-bound threshold - [ ] Ledger row written with verdict and full evidence - [ ] If FP-ordering / concurrency / UB-adjacent: verdict `needs-human-review`, never auto-accept diff --git a/README.md b/README.md index e1141b3..3750919 100644 --- a/README.md +++ b/README.md @@ -21,7 +21,7 @@ hostile review. ## Verified results Every number carries its 95% CI and its workload, and every one survived the -gates above. Across **five phases and 33 ledger rows, zero shipped false +gates above. Across **five phases and 34 ledger rows, zero shipped false accepts** — including two pipeline over-accepts the audit caught before anything shipped, each becoming a permanent new gate. @@ -58,14 +58,43 @@ docs/ The mdBook site (deployed to GitHub Pages) just build / test / lint # trust-layer workspace just aa # A/A self-test — must yield a null verdict just gates # corpus pin + upstream tests + golden replay -just verdict ... # gates + bench vs pristine baseline + ledger row + # (fuzz needs a baseline — it runs in `just verdict`) +just verdict ... # gates + fuzz + sanitizers + bench vs pristine baseline just report # ROI report from a ledger row just isolation-check # 19 OS-boundary checks (both modes) ``` +The pipeline is **Linux/POSIX-only** at runtime (Unix sockets, `setsid`, +sh-based gates). Windows builds the workspace and runs the portable unit +tests, but cannot run the pipeline. + Full command reference and reproduction steps: [Reproduce it yourself](https://bdbrown4.github.io/peltier.io/reproduce.html). +## Use it on your own code + +The trust layer is not specific to these targets — it will judge any two shell +commands. `.claude/skills/peltier/` packages it as a Claude Code skill: + +```sh +# from any repo, with a peltier checkout reachable +export PELTIER_HOME=/path/to/peltier.io +mkdir -p .claude/skills +cp -r /path/to/peltier.io/.claude/skills/peltier .claude/skills/ +``` + +Then ask Claude to verify a speedup. The skill enforces the order that makes a +performance claim mean something: **equivalence before timing** (a faster +program that computes something different is a bug, not an optimization) → +**A/A calibration** (a host that "finds" a speedup between a binary and itself +cannot measure yours) → **interleaved A/B with a bootstrap CI** → a verdict +decided by the CI *lower bound*, never the median. + +It drives the real `bench-runner` binary and **refuses rather than degrade**: +on an unsupported host, or with no trust layer reachable, it stops and says so +instead of falling back to a hand-rolled timing loop. A second copy of the +statistics is a second thing that can lie. + ## How it stays honest - The agent speaks to the trust layer through **seven read-only-plus-one- diff --git a/SPEC.md b/SPEC.md index d7d55f2..2267e8e 100644 --- a/SPEC.md +++ b/SPEC.md @@ -43,6 +43,8 @@ Requirements: - **A/A self-test mode**: run the same binary as both sides; must produce a null verdict. Scheduled automatically before any measurement session. - Regression-injection self-test: a synthetic 5% slowdown must be detected ≥95% of the time. - Metrics captured per run: wall time, cycles, instructions, cache misses, branch misses (via `perf stat`), max RSS, and RAPL energy where available. + + > **Amended 2026-07-13.** As built: **wall time is the only metric a verdict is decided on.** Max RSS is captured per timed run on Unix (`wait4` / `ru_maxrss`) and recorded as baseline/candidate medians in the env fingerprint — it is **context, never an accept metric**. PMU counters (cycles, instructions, cache misses, branch misses) are opt-in diagnostics via `bench-runner compare --perf-stat` — printed with the workload caveat, never part of the accept decision, and unavailable in most VMs. RAPL energy capture was never implemented. The **"no-network container"** named in the environment-control bullet above is *not* what ships — see the §10 amendment for the isolation actually built (a network namespace on the agent path only). - Modes: (a) whole-program via hyperfine-style CLI timing; (b) microbench via criterion harness hooks; (c) service mode (Phase 4): wrk2 / vegeta load replay with **coordinated-omission-aware** latency recording (HdrHistogram). Acceptance: passes A/A calibration at <5% false-positive rate and injected-regression detection at ≥95%, documented in `results/calibration/`. @@ -57,6 +59,12 @@ Layered, all must pass: Corpus and test files live outside agent-writable paths and are hash-pinned; `diff-test` refuses to run if hashes mismatch. +> **Amended 2026-07-13 — layer 3 (differential fuzz) as actually built.** Fuzz is a **baseline-vs-candidate** gate: it needs both binaries, so it runs *only* on the accept path (`verdict`, the one flow that rebuilds a pristine baseline). The standalone `diff-test` / `just gates` flow has no baseline and records the gate as **Skipped, with that reason** — it is never faked by comparing the candidate against itself. `diff-test` executes the target's declared `[gates].fuzz` command and grades it strictly from a `FUZZ-RESULT iters= divergences=` line; a run that never prints the line is a **Failed** gate, not a pass by silence. The ledger records the iteration count actually executed. **Hard rule:** a machine `accepted` verdict is impossible unless this gate *Passed* — skipped or failed caps the verdict at `needs-human-review`. `cjson`, `comrak`, and `tokei` declare `[gates].fuzz`; `matmul` does not. The per-changed-function scoping and the 60s/10k minimum remain aspirational: the shipped fuzzers are per-target whole-binary differential drivers (`scripts/diff-fuzz-*.py`) with a configurable iteration budget. + +> **Amended 2026-07-13 — layer 4 (sanitizers) as actually built.** ASan+UBSan is machine-enforced on the accept path: `verdict` builds the target's `[build].sanitizer` binary, runs the pinned workload under it, and caps a flagged run — or a target that declares **no** sanitizer build — at `needs-human-review`. The **TSan lane is opt-in per target** (`[build].tsan` + `[build].tsan_binary`) rather than diff-triggered, and **no target declares it today**, so the lane is wired but **inert and no TSan coverage currently exists**; a threaded patch requires adding that opt-in first. **MIRI was never implemented.** + +> **Amended 2026-07-13 — test-suite pinning: mechanized, not yet active.** As built, upstream test suites are vendored inside `targets//workspace/` — an agent-writable path — so "unmodified" is enforced by pinning, not placement: `corpora//TESTSUITE.sha256` (generated only by a deliberate human run of `scripts/pin-testsuite.sh`) is verified by `diff-test` before the suite runs; a hash mismatch is a hard refusal, same posture as the corpus pin. Targets without a pin file run with an explicit "suite unpinned" warning, and `targets/fetch.sh` verifies the pin after every fetch. **Status: no target ships a pin — there are zero `TESTSUITE.sha256` files in the repo — so this gate is currently inert and constrains nothing.** It becomes real only when pins are generated per target after a fetch. (The *corpus* pin, `MANIFEST.sha256`, ships for every target and **is** enforced today.) + ### 3.3 `ledger` Append-only SQLite. One row per attempt: @@ -70,7 +78,7 @@ Phase 0–2: `perf record` + flamegraphs (inferno) + `perf stat`; hotspot rankin ### 3.5 `agent/` -Claude Agent SDK (Python). Tools exposed to the model: `read_profile`, `read_ledger`, `read_playbook`, `propose_patch(diff, hypothesis)`, `run_verdict`, `read_target_source` (read-only). Deliberately **not** exposed: shell on the host, writes outside `targets//`, any access to `crates/`, `config/`, `corpora/`. +Claude Agent SDK (Python). Tools exposed to the model — seven: `read_profile`, `read_ledger`, `read_playbook`, `read_target_source` (read-only), `propose_patch(diff, hypothesis)`, `run_verdict`, `read_verdict`. Deliberately **not** exposed: shell on the host, writes outside `targets//`, any access to `crates/`, `config/`, `corpora/`. *(Amended 2026-07-13: `run_verdict` launches the pipeline detached and returns immediately; the pollable `read_verdict` — added in Phase 2 when the pipeline outlived the MCP transport timeout — reads the result, making seven operations total.)* Prompting spine: (1) state hypothesis before patching; (2) prefer the cheapest untried playbook class; (3) max two iterations per rejected hypothesis; (4) a rejection with a clean ledger row is a successful outcome. @@ -79,7 +87,7 @@ Prompting spine: (1) state hypothesis before patching; (2) prefer the cheapest u Consumes ledger rows, emits per-engagement ROI report: - Throughput jobs: cores saved = fleet_cores × (1 − 1/speedup); $ = cores × $/core-hr × hours/yr (rates in `config/pricing.toml`). - Services: p50/p95/p99 deltas under replayed traffic, with the business-value mapping left as a customer-supplied parameter and one worked example. -- Optional: joules saved (RAPL) for the sustainability line. +- Optional: joules saved (RAPL) for the sustainability line. *(Amended 2026-07-13: never implemented — RAPL energy is not captured anywhere in the harness, so no report emits a joules figure. See the §3.1 amendment.)* - Every figure carries CI + workload description + environment fingerprint. Caveats are printed on the report, not in an appendix. ## 4. Optimization playbook v0 (strict order) @@ -106,6 +114,8 @@ Each class ships as `playbook/NN-name.md`: preconditions (profile signature that Language order: Rust first (criterion/proptest/MIRI make Phases 1–2 cheaper), C/C++ second (the richer legacy hunting ground), services third. +> **Amended 2026-07-13.** The Phase 2 criterion as written — "zero false accepts across ≥20 audited attempts" — was violated once *in-pipeline*: `phase2-comrak-010` was falsely accepted by the pipeline (no machine sanitizer gate existed yet), caught by the 100% human audit, overturned, and never shipped. The criterion was met in the form **zero *shipped* false accepts**; the overturn drove the machine-enforced sanitizer gate and stands documented in the Phase 2 case study. + ## 6. Target selection criteria (Phase 0–3) CPU-bound (>70% user time on profile); meaningful test suite; existing benchmark or easily scripted workload; permissive license; active but not hyper-optimized (skip anything with a dedicated perf team); builds cleanly in container. Good hunting: codecs, parsers, serializers, image processing, compression, static-site generators, linters. @@ -123,6 +133,8 @@ CPU-bound (>70% user time on profile); meaningful test suite; existing benchmark - **needs-human-review** (never auto-accepted): FP reassociation/contraction or `-ffast-math`-class flags; any diff touching atomics, locks, memory ordering, or thread counts; unsafe blocks / raw pointer arithmetic introduced; anything the sanitizers flag even as warnings. - **Auto-rejected**: any gate failure, any divergence under differential fuzz, patch outside allowlist. +> **Amended 2026-07-13.** The needs-human-review tier is machine-routed, not honor-system: `verdict` runs a **conservative lexical risk classifier** over the patch's changed lines (substring token lists for concurrency, unsafe, and floating-point signals) and forces any would-be accept carrying a signal to `needs-human-review`; using **fp-tolerance equivalence mode** at all also auto-routes to review; the manual `--needs-human-review` flag remains. The classifier is lexical and deliberately over-triggering — it cheaply catches the presence of risk markers, it does not prove their absence. + ## 9. ROI methodology (the pitch artifact) The report answers one question: *what did the stopwatch say, and what does that cost or buy?* Throughput → cores → dollars via public cloud pricing; latency → percentile deltas under recorded production-shaped load, priced by the customer's own latency-value number; batch windows → wall-clock hours returned to the schedule. All three come with CIs and a workload statement. The methodology section ships *inside* every report so the number survives hostile review — that is the differentiation. @@ -136,6 +148,14 @@ The agent's incentive is "make the number go up"; the design must make cheating - Target code executes only in no-network containers (bench containers additionally seccomp-restricted). - Periodic human audit of accepted wins (100% in Phase 2, sampled thereafter); any false accept is a stop-the-line event. +> **Amended 2026-07-13 — isolation as actually built.** The bullet above ("target code executes only in no-network containers, seccomp-restricted") is **not** what ships. What ships: +> +> - **The `harnessd`-launched verdict pipeline** — the agent path — execs through `scripts/no-net.sh`, a **network-namespace** wrapper (`unshare --net --map-current-user`). `HOTPATH_VERDICT_WRAPPER` overrides the wrapper. +> - **It fails closed.** Where namespaces are unavailable the wrapper **exits 97 without running the pipeline**, rather than degrading silently. The only bypass is the explicit `HOTPATH_ALLOW_UNISOLATED=1` override — which runs the pipeline **with full network access**, warns loudly on stderr, and is recorded verbatim in the ledger's isolation note (`"no-net.sh (HOTPATH_ALLOW_UNISOLATED=1 — network NOT isolated)"`), so a run can never *claim* an isolation it did not have. +> - **In CI**, the bench workload runs under `docker run --network=none`. +> +> **Two gaps, stated plainly.** (1) **Only the harnessd (agent) path is wrapped.** A human invoking `just verdict` directly runs **unwrapped** on the host; such rows record `isolation: "unwrapped-host"`. (2) **Full-container isolation of the whole pipeline remains open** — the wrapper isolates the *network*, not the filesystem or the syscall surface; the seccomp-restricted bench container of this section is not built. The agent-side OS boundary (mount-namespace / unprivileged-uid isolation, `just isolation-check`) is a separate mechanism and *is* shipped. + ## 11. Risk register | Risk | Mitigation | @@ -149,10 +169,14 @@ The agent's incentive is "make the number go up"; the design must make cheating | License/IP hygiene | Permissive-license targets only for public case studies | | Wins too small to matter | Cheap-wins-first ordering keeps cost-per-attempt low; ledger prevents re-grinding dead ends | +> **Amended 2026-07-13 — which mitigations above are actually live.** The "Concurrency bugs introduced" and "UB introduced" rows overstate the machine coverage. Live today: **UBSan** (bundled with ASan on the accept path, machine-enforced) and the **human-review tier**, which is machine-routed by the lexical risk classifier of the §8 amendment — so a concurrency-token or `unsafe`-token patch *is* forced to `needs-human-review`, conservatively and lexically. **Not live: the TSan gate** (opt-in via `[build].tsan`; no target declares it) and **MIRI** (never implemented). The concurrency and UB rows therefore rest on the review tier and UBSan, not on TSan/MIRI. + ## 12. Tech stack Rust workspace (`bench-runner`, `diff-test`, `ledger`, `report`) · `just` · hyperfine-style timing + criterion hooks · perf, inferno flamegraphs, coz · proptest, cargo-fuzz, libFuzzer · ASan/UBSan/TSan, MIRI · podman/docker (no-net) · wrk2 + HdrHistogram, vegeta · SQLite · Python 3.11+ with Claude Agent SDK (`agent/`) — confirm current SDK package name and auth at https://docs.claude.com/en/docs/claude-code/overview. +> **Amended 2026-07-13 — the stack as actually used.** This list is the *planned* stack, not an inventory. In use: the Rust workspace (plus `verdict`, `harnessd`, `policy`), `just`, in-house interleaved timing and an in-house open-loop load generator (**not** hyperfine/criterion/wrk2/vegeta), valgrind-callgrind and coz for profiling (`perf` is often unavailable in the container), Python differential fuzzers (`scripts/diff-fuzz-*.py` — **not** proptest/cargo-fuzz/libFuzzer), ASan+UBSan, SQLite, and the Agent SDK. **Not used: MIRI (never implemented), TSan (wired but no target opts in), and podman/docker for the local verdict path** — local isolation is a network namespace (`scripts/no-net.sh`); Docker `--network=none` is used in CI. + ## 13. Later / research fork (explicitly out of scope now) - GPU kernel lane (where 2–10x gaps live): Triton/CUDA targets, correctness via reference-kernel differential testing, reward = measured kernel time. diff --git a/agent/hotpath_agent/loop.py b/agent/hotpath_agent/loop.py index a431a3d..7303dec 100644 --- a/agent/hotpath_agent/loop.py +++ b/agent/hotpath_agent/loop.py @@ -1,7 +1,7 @@ -"""Phase 2 agent loop (SPEC §3.5): Claude Agent SDK driving the six +"""Phase 2 agent loop (SPEC §3.5): Claude Agent SDK driving the seven harness tools over a standalone stdio MCP server. The agent gets NO built-in tools — no shell, no file writes; every capability is one of -the six IPC operations, and the trust layer's allowlist/gates remain +the seven IPC operations, and the trust layer's allowlist/gates remain the hard boundary (SPEC §10). Run one attempt: diff --git a/agent/hotpath_agent/mcp_server.py b/agent/hotpath_agent/mcp_server.py index dbb51b6..822d605 100644 --- a/agent/hotpath_agent/mcp_server.py +++ b/agent/hotpath_agent/mcp_server.py @@ -1,4 +1,4 @@ -"""Standalone MCP stdio server exposing the six harness tools (SPEC §3.5). +"""Standalone MCP stdio server exposing the seven harness tools (SPEC §3.5). Spawned as a subprocess by the agent loop; speaks MCP over stdio. Wraps the harnessd IPC client — the trust-layer allowlist and gates remain the diff --git a/agent/hotpath_agent/prompts.py b/agent/hotpath_agent/prompts.py index 7eb9254..e5ac1fd 100644 --- a/agent/hotpath_agent/prompts.py +++ b/agent/hotpath_agent/prompts.py @@ -24,8 +24,12 @@ Rules, in priority order: Never restructure a patch to influence measurement rather than to express the hypothesis. -5. Equivalence is not yours to relax. Floating-point flag changes, - concurrency-primitive changes, and new unsafe code are routed to - human review automatically; do not attempt to phrase a patch to - avoid that routing. +5. Equivalence is not yours to relax. The trust layer (crates/verdict) + runs a conservative lexical risk classifier over every patch's + changed lines: concurrency, unsafe, and floating-point signals force + any would-be accept to needs-human-review, and fp-tolerance + equivalence mode always routes to human review. The classifier is + lexical and deliberately over-triggering, not a semantic analysis; + a human --needs-human-review flag also exists. Do not attempt to + phrase a patch to slip past that routing. """ diff --git a/agent/hotpath_agent/tools.py b/agent/hotpath_agent/tools.py index 4ab4152..5cdc343 100644 --- a/agent/hotpath_agent/tools.py +++ b/agent/hotpath_agent/tools.py @@ -1,6 +1,6 @@ """Tool surface exposed to the model (SPEC §3.5). -Exactly six tools, all mediated by harnessd over line-delimited JSON — +Exactly seven tools, all mediated by harnessd over line-delimited JSON — the agent process has no direct filesystem or shell capability beyond what these return. Deliberately NOT exposed: shell on the host, writes outside targets//, any access to crates/, config/, corpora/. diff --git a/config/accept.toml b/config/accept.toml index 8497e82..e13a06c 100644 --- a/config/accept.toml +++ b/config/accept.toml @@ -23,3 +23,6 @@ bootstrap_seed = 1887343589 # Calibration evidence for this machine shows pinning tightens the A/A # CI ~5x (results/calibration/). Empty string disables. pin_prefix = "taskset -c 2" + +# Differential-fuzz iterations requested per gate run (SPEC §3.2). +# fuzz_iters = 10000 diff --git a/corpora/README.md b/corpora/README.md index 79dbb31..e0b78f2 100644 --- a/corpora/README.md +++ b/corpora/README.md @@ -8,8 +8,15 @@ Every corpus directory carries a `MANIFEST.sha256` (`sha256sum` format: before every run and refuses to run on any mismatch — a tampered corpus is a stop-the-line event. -Regenerate a manifest only as a deliberate human action: +`just pin-check ` regenerates the corpus and verifies it against +the manifest; it never rewrites the manifest. Regenerate a manifest only +as a deliberate human action: ```sh -cd corpora/ && find . -type f ! -name MANIFEST.sha256 -exec sha256sum {} + > MANIFEST.sha256 +just pin-corpus # gen-corpus.sh --pin ``` + +A directory may also carry a `TESTSUITE.sha256` pinning the upstream test +suite (repo-root-relative paths). `diff-test` verifies it before running +upstream tests and `targets/fetch.sh` verifies it after every fetch; +generate it with `scripts/pin-testsuite.sh ...`. diff --git a/corpora/cjson/gen-corpus.sh b/corpora/cjson/gen-corpus.sh index 3763094..a594b15 100644 --- a/corpora/cjson/gen-corpus.sh +++ b/corpora/cjson/gen-corpus.sh @@ -4,8 +4,16 @@ # with escapes and unicode, ints, floats (the strtod/sprintf hot path), # booleans, nulls. No randomness — byte-identical on every machine, so # the MANIFEST.sha256 pin is reproducible. Run from repo root: -# sh corpora/cjson/gen-corpus.sh +# sh corpora/cjson/gen-corpus.sh [--check|--pin] +# Default/--check: regenerate and verify against MANIFEST.sha256 (never +# rewrites it). --pin: deliberately rewrite the manifest. set -eu +mode="check" +case "${1:-}" in + "" | --check) ;; + --pin) mode="pin" ;; + *) echo "usage: gen-corpus.sh [--check|--pin]" >&2; exit 2 ;; +esac DIR="$(cd "$(dirname "$0")" && pwd)" OUT="$DIR/input/big.json" mkdir -p "$DIR/input" @@ -57,6 +65,14 @@ with open(sys.argv[1], "w") as f: PY echo "wrote $OUT ($(wc -c < "$OUT") bytes)" -# Pin the corpus and regenerate the manifest. -( cd "$DIR/input" && find . -type f -print0 | sort -z | xargs -0 sha256sum ) > "$DIR/MANIFEST.sha256" -echo "manifest: $DIR/MANIFEST.sha256" +if [ "$mode" = "pin" ]; then + ( cd "$DIR/input" && find . -type f -print0 | sort -z | xargs -0 sha256sum ) > "$DIR/MANIFEST.sha256" + echo "re-pinned $DIR/MANIFEST.sha256 — a deliberate human action (corpora/README.md);" + echo "commit the new manifest with justification." +else + if [ ! -f "$DIR/MANIFEST.sha256" ]; then + echo "no MANIFEST.sha256 — pin the corpus with: sh corpora/cjson/gen-corpus.sh --pin" >&2 + exit 1 + fi + ( cd "$DIR/input" && sha256sum -c ../MANIFEST.sha256 ) +fi diff --git a/corpora/comrak/gen-corpus.sh b/corpora/comrak/gen-corpus.sh index 91ab707..f1588ea 100755 --- a/corpora/comrak/gen-corpus.sh +++ b/corpora/comrak/gen-corpus.sh @@ -1,7 +1,21 @@ #!/bin/sh # Regenerate the comrak bench corpus from the pinned progit checkout # (targets/comrak/target.toml [source.submodules]), then verify the pin. +# Default/--check verifies MANIFEST.sha256 (never rewrites it); --pin +# deliberately rewrites it. set -eu +mode="check" +case "${1:-}" in + "" | --check) ;; + --pin) mode="pin" ;; + *) echo "usage: gen-corpus.sh [--check|--pin]" >&2; exit 2 ;; +esac cd "$(dirname "$0")" cat ../../targets/comrak/workspace/vendor/progit/*/*/*.markdown > progit.md -sha256sum -c MANIFEST.sha256 +if [ "$mode" = "pin" ]; then + sha256sum progit.md > MANIFEST.sha256 + echo "re-pinned MANIFEST.sha256 — a deliberate human action (corpora/README.md);" + echo "commit the new manifest with justification." +else + sha256sum -c MANIFEST.sha256 +fi diff --git a/corpora/tokei/gen-corpus.sh b/corpora/tokei/gen-corpus.sh index 1f54cff..b8a3f23 100755 --- a/corpora/tokei/gen-corpus.sh +++ b/corpora/tokei/gen-corpus.sh @@ -2,12 +2,26 @@ # Regenerate the tokei bench input: 8 replicas of two pinned trees # (progit @ 61833a5, comrak src/ @ 45c1995), .git removed. Verifies the # manifest for copy1; copies 2-8 are byte-identical replicas of copy1. +# Default/--check verifies MANIFEST.sha256 (never rewrites it); --pin +# deliberately rewrites it. set -eu +mode="check" +case "${1:-}" in + "" | --check) ;; + --pin) mode="pin" ;; + *) echo "usage: gen-corpus.sh [--check|--pin]" >&2; exit 2 ;; +esac cd "$(dirname "$0")" rm -rf input mkdir -p input/copy1 cp -r ../../targets/comrak/workspace/vendor/progit input/copy1/progit cp -r ../../targets/comrak/workspace/src input/copy1/src rm -rf input/copy1/progit/.git -(cd input/copy1 && sha256sum -c ../../MANIFEST.sha256 --quiet) || { echo "corpus pin FAILED"; exit 1; } +if [ "$mode" = "pin" ]; then + (cd input/copy1 && find . -type f -print0 | sort -z | xargs -0 sha256sum) > MANIFEST.sha256 + echo "re-pinned MANIFEST.sha256 — a deliberate human action (corpora/README.md);" + echo "commit the new manifest with justification." +else + (cd input/copy1 && sha256sum -c ../../MANIFEST.sha256 --quiet) || { echo "corpus pin FAILED"; exit 1; } +fi for i in 2 3 4 5 6 7 8; do cp -r input/copy1 "input/copy$i"; done diff --git a/crates/bench-runner/Cargo.toml b/crates/bench-runner/Cargo.toml index 2281413..d73a6f3 100644 --- a/crates/bench-runner/Cargo.toml +++ b/crates/bench-runner/Cargo.toml @@ -10,7 +10,9 @@ repository.workspace = true anyhow.workspace = true clap.workspace = true ledger = { path = "../ledger" } -libc = "0.2" serde.workspace = true serde_json.workspace = true toml.workspace = true + +[target.'cfg(unix)'.dependencies] +libc = "0.2" diff --git a/crates/bench-runner/src/config.rs b/crates/bench-runner/src/config.rs index 6f6d822..389d308 100644 --- a/crates/bench-runner/src/config.rs +++ b/crates/bench-runner/src/config.rs @@ -23,6 +23,13 @@ pub struct AcceptConfig { /// both sides identically. #[serde(default)] pub pin_prefix: String, + /// Differential-fuzz iterations requested per gate run (SPEC §3.2). + #[serde(default = "default_fuzz_iters")] + pub fuzz_iters: u64, +} + +fn default_fuzz_iters() -> u64 { + 10_000 } impl Default for AcceptConfig { @@ -35,6 +42,7 @@ impl Default for AcceptConfig { confidence: 0.95, bootstrap_seed: 0x707E_17E5, pin_prefix: String::new(), + fuzz_iters: default_fuzz_iters(), } } } @@ -52,3 +60,31 @@ impl AcceptConfig { Ok(cfg) } } + +#[cfg(test)] +mod tests { + use super::*; + + const MINIMAL: &str = "threshold = 0.02\nruns_per_side = 30\nwarmup_runs = 3\n\ + bootstrap_iters = 10000\nconfidence = 0.95\nbootstrap_seed = 7\n"; + + #[test] + fn fuzz_iters_defaults_when_absent() { + let cfg: AcceptConfig = toml::from_str(MINIMAL).unwrap(); + assert_eq!(cfg.fuzz_iters, 10_000); + assert_eq!(AcceptConfig::default().fuzz_iters, 10_000); + } + + #[test] + fn fuzz_iters_overridable() { + let raw = format!("{MINIMAL}fuzz_iters = 500\n"); + let cfg: AcceptConfig = toml::from_str(&raw).unwrap(); + assert_eq!(cfg.fuzz_iters, 500); + } + + #[test] + fn unknown_fields_still_rejected() { + let raw = format!("{MINIMAL}fuz_iters = 500\n"); + assert!(toml::from_str::(&raw).is_err()); + } +} diff --git a/crates/bench-runner/src/counters.rs b/crates/bench-runner/src/counters.rs new file mode 100644 index 0000000..1d8fd64 --- /dev/null +++ b/crates/bench-runner/src/counters.rs @@ -0,0 +1,218 @@ +//! Opt-in `perf stat` PMU counter capture. Diagnostics only — counters +//! never feed the accept decision (wall time does, SPEC §3.1). Each +//! timed run is wrapped `perf stat -x, -e -o -- sh -c +//! `; the CSV file is parsed after every run and per-side medians +//! are printed with the standard caveats. Unix-only at runtime; the +//! parser is portable and unit-tested. + +use crate::exec::MeasuredSamples; +use crate::schedule::{interleaved, Side}; +use crate::stats; +use anyhow::Context; +use std::path::Path; +use std::process::{Command, Stdio}; +use std::time::Instant; + +pub const PERF_EVENTS: [&str; 4] = ["cycles", "instructions", "cache-misses", "branch-misses"]; + +/// One side's counter samples: `samples[i]` holds one value per measured +/// run for `PERF_EVENTS[i]`; runs where perf reported `` or +/// `` contribute nothing. +#[derive(Debug, Clone, Default)] +pub struct PerfSide { + pub samples: [Vec; 4], +} + +#[derive(Debug, Clone, Default)] +pub struct PerfReport { + pub baseline: PerfSide, + pub candidate: PerfSide, +} + +/// Parse `perf stat -x,` CSV: field 0 is the counter value, field 2 the +/// event name (perf may append a modifier, e.g. `cycles:u`). Comment +/// lines, blank lines, and non-numeric values (``, +/// ``) are skipped. Returns (event, count) in file order. +pub fn parse_perf_csv(text: &str) -> Vec<(String, u64)> { + let mut out = Vec::new(); + for line in text.lines() { + let line = line.trim(); + if line.is_empty() || line.starts_with('#') { + continue; + } + let mut fields = line.split(','); + let value = fields.next().unwrap_or(""); + let _unit = fields.next(); + let Some(event) = fields.next() else { + continue; + }; + let Ok(v) = value.trim().parse::() else { + continue; + }; + out.push((event.trim().to_string(), v)); + } + out +} + +/// Index into PERF_EVENTS, tolerating a perf modifier suffix +/// (`cycles:u` when perf_event_paranoid restricts kernel counting). +fn event_index(event: &str) -> Option { + PERF_EVENTS.iter().position(|name| { + event == *name || (event.starts_with(name) && event[name.len()..].starts_with(':')) + }) +} + +impl PerfSide { + fn absorb(&mut self, parsed: &[(String, u64)]) { + for (event, value) in parsed { + if let Some(i) = event_index(event) { + self.samples[i].push(*value); + } + } + } +} + +impl PerfReport { + /// Per-side medians, each with its command (workload) and run count. + pub fn print(&self, baseline_cmd: &str, candidate_cmd: &str) { + println!("perf-stat per-side medians:"); + for (label, cmd, side) in [ + ("baseline ", baseline_cmd, &self.baseline), + ("candidate", candidate_cmd, &self.candidate), + ] { + let cols: Vec = PERF_EVENTS + .iter() + .zip(&side.samples) + .map(|(name, xs)| { + if xs.is_empty() { + format!("{name}=n/a") + } else { + let v: Vec = xs.iter().map(|&x| x as f64).collect(); + format!( + "{name}={:.0} (median of {} runs)", + stats::median(&v), + xs.len() + ) + } + }) + .collect(); + println!(" {label} `{cmd}`: {}", cols.join(", ")); + } + println!( + " note: PMU counters unavailable in most VMs — flag is opt-in diagnostics, \ + not part of the accept decision" + ); + } +} + +fn time_once_perf(shell_cmd: &str, out_file: &Path) -> anyhow::Result<(f64, Vec<(String, u64)>)> { + let start = Instant::now(); + let status = Command::new("perf") + .args(["stat", "-x,", "-e", &PERF_EVENTS.join(",")]) + .arg("-o") + .arg(out_file) + .args(["--", "sh", "-c", shell_cmd]) + .stdout(Stdio::null()) + .stderr(Stdio::null()) + .status() + .with_context(|| format!("failed to spawn perf stat for: {shell_cmd}"))?; + let elapsed = start.elapsed().as_secs_f64(); + anyhow::ensure!( + status.success(), + "perf stat run exited nonzero: {shell_cmd}" + ); + let text = std::fs::read_to_string(out_file) + .with_context(|| format!("perf stat output missing: {}", out_file.display()))?; + Ok((elapsed, parse_perf_csv(&text))) +} + +/// Interleaved A/B with each timed run wrapped in `perf stat`. The perf +/// wrapper overhead lands on both sides identically, but this path is +/// diagnostics only — verdicts always measure through the unwrapped +/// `exec::run_interleaved`. +pub fn run_interleaved_perf( + baseline_cmd: &str, + candidate_cmd: &str, + runs_per_side: usize, + warmup: usize, +) -> anyhow::Result<(MeasuredSamples, PerfReport)> { + anyhow::ensure!(cfg!(unix), "--perf-stat requires a Unix host with perf(1)"); + let tmp = std::env::temp_dir().join(format!("bench-runner-perf-{}.csv", std::process::id())); + let mut samples = MeasuredSamples { + baseline_s: Vec::with_capacity(runs_per_side), + candidate_s: Vec::with_capacity(runs_per_side), + baseline_max_rss_kib: Vec::new(), + candidate_max_rss_kib: Vec::new(), + }; + let mut report = PerfReport::default(); + for (side, measured) in interleaved(runs_per_side, warmup) { + let cmd = match side { + Side::Baseline => baseline_cmd, + Side::Candidate => candidate_cmd, + }; + let (elapsed, parsed) = time_once_perf(cmd, &tmp)?; + if measured { + match side { + Side::Baseline => { + samples.baseline_s.push(elapsed); + report.baseline.absorb(&parsed); + } + Side::Candidate => { + samples.candidate_s.push(elapsed); + report.candidate.absorb(&parsed); + } + } + } + } + let _ = std::fs::remove_file(&tmp); + Ok((samples, report)) +} + +#[cfg(test)] +mod tests { + use super::*; + + const CANNED: &str = "\ +# started on Mon Jul 13 12:00:00 2026 + +123456789,,cycles,401000000,100.00,, +234567890,,instructions:u,401000000,100.00,1.90,insn per cycle +,,cache-misses,0,100.00,, +,,branch-misses,0,0.00,, +garbage line without commas +999,,unrelated-event,1,100.00,, +"; + + #[test] + fn parses_values_and_skips_junk() { + let parsed = parse_perf_csv(CANNED); + assert_eq!( + parsed, + vec![ + ("cycles".to_string(), 123_456_789), + ("instructions:u".to_string(), 234_567_890), + ("unrelated-event".to_string(), 999), + ] + ); + } + + #[test] + fn event_index_tolerates_modifier_suffix_only() { + assert_eq!(event_index("cycles"), Some(0)); + assert_eq!(event_index("instructions:u"), Some(1)); + assert_eq!(event_index("cache-misses"), Some(2)); + assert_eq!(event_index("cycle"), None); + assert_eq!(event_index("cycles_total"), None); + assert_eq!(event_index("unrelated-event"), None); + } + + #[test] + fn absorb_buckets_by_event() { + let mut side = PerfSide::default(); + side.absorb(&parse_perf_csv(CANNED)); + assert_eq!(side.samples[0], vec![123_456_789]); + assert_eq!(side.samples[1], vec![234_567_890]); + assert!(side.samples[2].is_empty(), " is skipped"); + assert!(side.samples[3].is_empty(), " is skipped"); + } +} diff --git a/crates/bench-runner/src/exec.rs b/crates/bench-runner/src/exec.rs index a3d2100..1f84504 100644 --- a/crates/bench-runner/src/exec.rs +++ b/crates/bench-runner/src/exec.rs @@ -1,29 +1,87 @@ //! Command execution and timing. Whole-program mode (hyperfine-style): //! spawn the command, measure wall time, discard warm-ups per the -//! schedule. `perf stat` counters and RAPL energy are Phase 1 follow-ups -//! (tracked in README roadmap); wall time alone is enough to bring up -//! A/A calibration. +//! schedule. Max RSS is captured per run via `wait4` on Unix; opt-in +//! `perf stat` counters live in `counters` (diagnostics only). use crate::schedule::{interleaved, Side}; use anyhow::Context; -use std::process::{Command, Stdio}; +use std::process::{Child, Command, ExitStatus, Stdio}; use std::time::Instant; #[derive(Debug, Clone)] pub struct MeasuredSamples { pub baseline_s: Vec, pub candidate_s: Vec, + /// Per-run max resident set size, normalized to KiB from `wait4` + /// ru_maxrss (whose own unit is platform-defined — see + /// `maxrss_to_kib`). Empty on non-Unix hosts. + pub baseline_max_rss_kib: Vec, + pub candidate_max_rss_kib: Vec, } -fn time_once(shell_cmd: &str, inject_spin_s: f64) -> anyhow::Result { +/// Raw `ru_maxrss` → KiB. +/// +/// The unit of `ru_maxrss` is *platform-defined*, and the platforms +/// disagree: Linux and the BSDs report kilobytes, while Apple platforms +/// (macOS et al.) report **bytes**. Taking the raw value as KiB everywhere +/// — as this did — silently inflates every macOS reading by 1024×, and the +/// number is not cosmetic: verdict records its median as +/// `env_fingerprint.max_rss_kib` in the ledger, where an allocator-swap +/// attempt is judged partly on memory. Everything downstream is KiB, so the +/// raw value is normalized here, at the syscall edge. +/// +/// `raw_is_bytes` is the platform predicate (passed at the one call site, so +/// this stays a pure function testable on every host). Bytes round *up*: a +/// nonzero RSS must never record as 0 KiB. Negative values — a kernel that +/// reports nothing — clamp to 0 rather than wrapping into the exabytes. +#[cfg(any(unix, test))] +fn maxrss_to_kib(raw: i64, raw_is_bytes: bool) -> u64 { + let raw = raw.max(0) as u64; + if raw_is_bytes { + raw.div_ceil(1024) + } else { + raw + } +} + +/// Reap the child, returning its exit status and max RSS in KiB when the +/// platform reports it. +#[cfg(unix)] +fn wait_with_rusage(child: &mut Child) -> anyhow::Result<(ExitStatus, Option)> { + use std::os::unix::process::ExitStatusExt; + let pid = child.id() as libc::pid_t; + let mut status: libc::c_int = 0; + let mut usage: libc::rusage = unsafe { std::mem::zeroed() }; + loop { + let r = unsafe { libc::wait4(pid, &mut status, 0, &mut usage) }; + if r == pid { + break; + } + let err = std::io::Error::last_os_error(); + if err.raw_os_error() != Some(libc::EINTR) { + return Err(err).context("wait4 failed"); + } + } + // Apple's getrusage reports ru_maxrss in bytes; Linux and the BSDs in KiB. + let max_rss_kib = maxrss_to_kib(usage.ru_maxrss as i64, cfg!(target_vendor = "apple")); + Ok((ExitStatus::from_raw(status), Some(max_rss_kib))) +} + +#[cfg(not(unix))] +fn wait_with_rusage(child: &mut Child) -> anyhow::Result<(ExitStatus, Option)> { + Ok((child.wait()?, None)) +} + +fn time_once(shell_cmd: &str, inject_spin_s: f64) -> anyhow::Result<(f64, Option)> { let start = Instant::now(); - let status = Command::new("sh") + let mut child = Command::new("sh") .arg("-c") .arg(shell_cmd) .stdout(Stdio::null()) .stderr(Stdio::null()) - .status() + .spawn() .with_context(|| format!("failed to spawn: {shell_cmd}"))?; + let (status, max_rss_kib) = wait_with_rusage(&mut child)?; if inject_spin_s > 0.0 { // Calibration-only: a real busy-wait inside the timed window, so an // injected slowdown flows through the entire measurement pipeline @@ -35,7 +93,7 @@ fn time_once(shell_cmd: &str, inject_spin_s: f64) -> anyhow::Result { } let elapsed = start.elapsed().as_secs_f64(); anyhow::ensure!(status.success(), "command exited nonzero: {shell_cmd}"); - Ok(elapsed) + Ok((elapsed, max_rss_kib)) } /// Run baseline and candidate commands on the interleaved schedule and @@ -53,19 +111,56 @@ pub fn run_interleaved( let mut out = MeasuredSamples { baseline_s: Vec::with_capacity(runs_per_side), candidate_s: Vec::with_capacity(runs_per_side), + baseline_max_rss_kib: Vec::new(), + candidate_max_rss_kib: Vec::new(), }; for (side, measured) in interleaved(runs_per_side, warmup) { let (cmd, spin) = match side { Side::Baseline => (baseline_cmd, 0.0), Side::Candidate => (candidate_cmd, inject_candidate_spin_s), }; - let elapsed = time_once(cmd, spin)?; + let (elapsed, max_rss_kib) = time_once(cmd, spin)?; if measured { match side { - Side::Baseline => out.baseline_s.push(elapsed), - Side::Candidate => out.candidate_s.push(elapsed), + Side::Baseline => { + out.baseline_s.push(elapsed); + out.baseline_max_rss_kib.extend(max_rss_kib); + } + Side::Candidate => { + out.candidate_s.push(elapsed); + out.candidate_max_rss_kib.extend(max_rss_kib); + } } } } Ok(out) } + +#[cfg(test)] +mod tests { + use super::maxrss_to_kib; + + #[test] + fn maxrss_kib_passthrough_when_platform_reports_kib() { + // Linux / BSD: already KiB. + assert_eq!(maxrss_to_kib(2048, false), 2048); + assert_eq!(maxrss_to_kib(0, false), 0); + } + + #[test] + fn maxrss_bytes_are_converted_not_recorded_raw() { + // Apple: bytes. 2 MiB is 2048 KiB, not 2_097_152 — the 1024× + // inflation this guards against. + assert_eq!(maxrss_to_kib(2_097_152, true), 2048); + // Rounds up: a nonzero RSS never records as 0 KiB. + assert_eq!(maxrss_to_kib(1, true), 1); + assert_eq!(maxrss_to_kib(1025, true), 2); + assert_eq!(maxrss_to_kib(0, true), 0); + } + + #[test] + fn maxrss_negative_clamps_instead_of_wrapping() { + assert_eq!(maxrss_to_kib(-1, false), 0); + assert_eq!(maxrss_to_kib(-1, true), 0); + } +} diff --git a/crates/bench-runner/src/lib.rs b/crates/bench-runner/src/lib.rs index 55cfbf1..96b3ffe 100644 --- a/crates/bench-runner/src/lib.rs +++ b/crates/bench-runner/src/lib.rs @@ -3,9 +3,11 @@ //! The product is this crate: interleaved A/B measurement, bootstrap CIs //! over the ratio of medians, threshold-gated accept decisions, and A/A //! self-calibration. Pure statistics live here and are unit-tested; -//! process execution and `perf stat` integration live in `exec`. +//! process execution lives in `exec`; opt-in `perf stat` diagnostics +//! live in `counters`. pub mod config; +pub mod counters; pub mod exec; pub mod fingerprint; pub mod schedule; diff --git a/crates/bench-runner/src/main.rs b/crates/bench-runner/src/main.rs index 1c56ee2..e4e5824 100644 --- a/crates/bench-runner/src/main.rs +++ b/crates/bench-runner/src/main.rs @@ -1,5 +1,7 @@ use anyhow::Result; -use bench_runner::{config::AcceptConfig, decide, exec, fingerprint::EnvFingerprint, stats}; +use bench_runner::{ + config::AcceptConfig, counters, decide, exec, fingerprint::EnvFingerprint, stats, +}; use clap::{Parser, Subcommand}; use ledger::Verdict; use std::path::PathBuf; @@ -25,6 +27,11 @@ enum Cmd { baseline: String, #[arg(long)] candidate: String, + /// Wrap each timed run in `perf stat` and print per-side counter + /// medians (Unix only; opt-in diagnostics, never part of the + /// accept decision). + #[arg(long, default_value_t = false)] + perf_stat: bool, }, /// A/A self-test: same command both sides; must yield a null verdict. Aa { @@ -429,12 +436,13 @@ fn main() -> Result<()> { None => AcceptConfig::default(), }; - let (baseline, candidate, aa_mode) = match &cli.command { + let (baseline, candidate, aa_mode, perf_stat) = match &cli.command { Cmd::Compare { baseline, candidate, - } => (baseline.clone(), candidate.clone(), false), - Cmd::Aa { cmd } => (cmd.clone(), cmd.clone(), true), + perf_stat, + } => (baseline.clone(), candidate.clone(), false, *perf_stat), + Cmd::Aa { cmd } => (cmd.clone(), cmd.clone(), true, false), Cmd::Calibrate { cmd, sessions, @@ -502,13 +510,24 @@ fn main() -> Result<()> { cfg.runs_per_side, cfg.warmup_runs ); - let samples = exec::run_interleaved( - &baseline, - &candidate, - cfg.runs_per_side, - cfg.warmup_runs, - 0.0, - )?; + let (samples, perf_report) = if perf_stat { + let (samples, report) = counters::run_interleaved_perf( + &baseline, + &candidate, + cfg.runs_per_side, + cfg.warmup_runs, + )?; + (samples, Some(report)) + } else { + let samples = exec::run_interleaved( + &baseline, + &candidate, + cfg.runs_per_side, + cfg.warmup_runs, + 0.0, + )?; + (samples, None) + }; let ci = stats::bootstrap_ratio_ci( &samples.baseline_s, &samples.candidate_s, @@ -524,6 +543,9 @@ fn main() -> Result<()> { ci.lo, ci.hi ); + if let Some(report) = &perf_report { + report.print(&baseline, &candidate); + } if aa_mode { // Null verdict required: an "accept" here is a calibration failure. diff --git a/crates/bench-runner/src/service.rs b/crates/bench-runner/src/service.rs index 06ec4a9..d2bf670 100644 --- a/crates/bench-runner/src/service.rs +++ b/crates/bench-runner/src/service.rs @@ -147,6 +147,7 @@ fn spawn_server(cfg: &ServiceCfg, port: u16) -> Result { Ok(child) } +#[cfg(unix)] fn set_linger_zero(s: &TcpStream) { // SO_LINGER with a 0 timeout → RST on close instead of FIN, so the // tens of thousands of short client connections a load session opens @@ -170,6 +171,12 @@ fn set_linger_zero(s: &TcpStream) { } } +#[cfg(not(unix))] +fn set_linger_zero(_s: &TcpStream) { + // Service mode is POSIX-only at runtime (`sh`, taskset); this stub + // only keeps the crate compiling on non-Unix hosts. +} + fn one_request(port: u16) -> std::io::Result<()> { let mut s = TcpStream::connect((Ipv4Addr::LOCALHOST, port))?; s.set_nodelay(true)?; diff --git a/crates/diff-test/src/lib.rs b/crates/diff-test/src/lib.rs index a769a4e..b073630 100644 --- a/crates/diff-test/src/lib.rs +++ b/crates/diff-test/src/lib.rs @@ -43,6 +43,49 @@ pub fn all_passed(results: &[(GateLayer, GateOutcome)]) -> bool { .any(|(_, o)| matches!(o, GateOutcome::Passed)) } +/// Inputs to one gate run: the candidate binary under test, the pristine +/// baseline to differ it against, and the differential-fuzz iteration +/// budget (from AcceptConfig, default 10_000). +#[derive(Debug, Clone, Copy)] +pub struct GateInputs<'a> { + /// Patched binary (repo-root-relative): golden replay runs it, and it + /// is the `{candidate}` side of differential fuzz. + pub candidate_binary: &'a str, + /// Pristine baseline binary (repo-root-relative), the `{baseline}` side + /// of differential fuzz. `None` when the caller has no baseline to + /// differ against — the standalone `just gates` flow builds only the + /// working tree — and the fuzz gate is then skipped with that reason + /// rather than silently comparing the candidate to itself. The accept + /// path (`just verdict`) rebuilds a pristine baseline and passes it, + /// and only a fuzz gate that actually Passed can mint an accept. + pub baseline_binary: Option<&'a str>, + pub fuzz_iters: u64, +} + +/// Everything one gate run produced: the four layers in fixed order +/// (UpstreamTests, GoldenReplay, DifferentialFuzz, Sanitizers) plus the +/// fuzz counts and equivalence mode that verdict records in the ledger. +#[derive(Debug, Clone)] +pub struct GateReport { + pub gates: Vec<(GateLayer, GateOutcome)>, + /// Fuzz iterations actually executed (parsed from the FUZZ-RESULT + /// line); 0 when the fuzz gate was skipped or never reported. + pub fuzz_iters: u64, + pub fuzz_divergence: bool, + /// "byte-identical" | "fp-tolerance" + pub equivalence_mode: String, +} + +impl GateReport { + pub fn outcome(&self, layer: GateLayer) -> Option<&GateOutcome> { + self.gates.iter().find(|(l, _)| *l == layer).map(|(_, o)| o) + } + + pub fn all_passed(&self) -> bool { + all_passed(&self.gates) + } +} + #[cfg(test)] mod tests { use super::*; @@ -71,4 +114,32 @@ mod tests { )]; assert!(!all_passed(&results)); } + + #[test] + fn report_outcome_looks_up_by_layer() { + let report = GateReport { + gates: vec![ + (GateLayer::UpstreamTests, GateOutcome::Passed), + ( + GateLayer::GoldenReplay, + GateOutcome::Failed { + detail: "stdout diverged".into(), + }, + ), + ], + fuzz_iters: 0, + fuzz_divergence: false, + equivalence_mode: "byte-identical".into(), + }; + assert_eq!( + report.outcome(GateLayer::UpstreamTests), + Some(&GateOutcome::Passed) + ); + assert!(matches!( + report.outcome(GateLayer::GoldenReplay), + Some(GateOutcome::Failed { .. }) + )); + assert_eq!(report.outcome(GateLayer::Sanitizers), None); + assert!(!report.all_passed()); + } } diff --git a/crates/diff-test/src/main.rs b/crates/diff-test/src/main.rs index fb7b165..d8e887b 100644 --- a/crates/diff-test/src/main.rs +++ b/crates/diff-test/src/main.rs @@ -1,12 +1,18 @@ -//! Gate orchestration CLI: corpus pin → upstream tests → golden replay. -//! Differential fuzz and sanitizers remain per-attempt manual steps in -//! Phase 1 and are reported as skipped with a reason. +//! Gate orchestration CLI: corpus + test-suite pins → upstream tests → +//! policy-aware golden replay → differential fuzz. Sanitizers run on the +//! accept path in verdict and are reported as skipped with a reason. +//! +//! Differential fuzz is skipped here too, and for the same structural +//! reason: it differs a *pristine baseline* against the candidate, and this +//! flow builds only the working tree — there is nothing to differ against. +//! The accept path (`just verdict`) rebuilds a pristine baseline, passes it +//! in, and refuses to mint an accept unless the fuzz gate actually passed. use anyhow::Result; use clap::Parser; use diff_test::orchestrate::run_core_gates; use diff_test::target::{subst_out, TargetSpec}; -use diff_test::{all_passed, GateOutcome}; +use diff_test::{GateInputs, GateLayer, GateOutcome}; use std::path::PathBuf; #[derive(Parser)] @@ -21,6 +27,13 @@ struct Cli { /// tree is built to targets//gates-build and that is tested. #[arg(long)] binary: Option, + /// Differential-fuzz iteration budget, substituted for {iters} in the + /// target's [gates].fuzz command. Inert on this flow: with no pristine + /// baseline to differ against, the fuzz gate is skipped. The accept path + /// (`just verdict`) rebuilds one and takes its budget from + /// config/accept.toml. + #[arg(long, default_value_t = 10_000)] + fuzz_iters: u64, } fn main() -> Result<()> { @@ -47,15 +60,42 @@ fn main() -> Result<()> { subst_out(&spec.build.binary, &out) } }; - let results = run_core_gates(&root, &spec, &binary)?; - for (layer, outcome) in &results { + let report = run_core_gates( + &root, + &spec, + &GateInputs { + candidate_binary: &binary, + // No pristine baseline on this flow — see the module doc. The + // fuzz gate reports Skipped with that reason rather than + // comparing the candidate against itself. + baseline_binary: None, + fuzz_iters: cli.fuzz_iters, + }, + )?; + for (layer, outcome) in &report.gates { match outcome { + GateOutcome::Passed if *layer == GateLayer::DifferentialFuzz => { + println!("{layer:?}: PASS ({} iters)", report.fuzz_iters) + } GateOutcome::Passed => println!("{layer:?}: PASS"), GateOutcome::Failed { detail } => println!("{layer:?}: FAIL — {detail}"), GateOutcome::Skipped { reason } => println!("{layer:?}: skipped — {reason}"), } } - if all_passed(&results) { + if report.all_passed() { + // Inside all_passed(), the fuzz gate is Passed or Skipped — never + // Failed. Say so plainly: a PASS here must not read as a verified + // differential-fuzz run, which is exactly the confusion that let + // accepted rows carry fuzz_iters=0. + if !matches!( + report.outcome(GateLayer::DifferentialFuzz), + Some(GateOutcome::Passed) + ) { + println!( + "note: differential fuzz did not run — `just gates` is not an accept path; only \ + `just verdict` (pristine baseline + fuzz gate) can accept a change" + ); + } println!("gates: PASS ({})", cli.target); Ok(()) } else { diff --git a/crates/diff-test/src/orchestrate.rs b/crates/diff-test/src/orchestrate.rs index 5825cf9..77e6927 100644 --- a/crates/diff-test/src/orchestrate.rs +++ b/crates/diff-test/src/orchestrate.rs @@ -1,10 +1,20 @@ //! Core gate sequence, callable from the diff-test CLI and from verdict: -//! corpus pin (refuses on mismatch) → upstream tests → golden replay. +//! corpus + test-suite pins (refuse on mismatch) → upstream tests → +//! policy-aware golden replay → differential fuzz. Sanitizers run on the +//! accept path in verdict and are reported here as skipped with a reason. +//! +//! Differential fuzz is the one gate that is not purely a property of the +//! candidate: it differs the candidate against a *pristine baseline*, so it +//! needs both binaries. Callers that have only a candidate (the standalone +//! `just gates` flow) pass `baseline_binary: None` and the gate is skipped +//! with that reason recorded — it is never faked by comparing the candidate +//! against itself, and verdict refuses to mint an accept without it. +use crate::policy::EquivalencePolicy; use crate::target::{expected_golden_hash, TargetSpec}; -use crate::{pin, GateLayer, GateOutcome}; +use crate::{pin, GateInputs, GateLayer, GateOutcome, GateReport}; use sha2::{Digest, Sha256}; -use std::path::Path; +use std::path::{Path, PathBuf}; use std::process::{Command, Stdio}; fn sh(root: &Path, cmd: &str) -> anyhow::Result { @@ -17,14 +27,155 @@ fn sh(root: &Path, cmd: &str) -> anyhow::Result { .output()?) } -/// Run the mechanical gates for `target`, testing `binary` (repo-root- -/// relative) in the golden replay. Corpus-pin mismatch is an error, not -/// a gate failure: nothing runs on a tampered corpus. +/// TESTSUITE.sha256 lives beside the corpus MANIFEST.sha256 (both under +/// corpora//), so its path derives from the manifest's. +fn suite_manifest_path(corpus_manifest: &Path) -> PathBuf { + corpus_manifest + .parent() + .unwrap_or_else(|| Path::new("")) + .join("TESTSUITE.sha256") +} + +fn parse_u64_strict(s: &str) -> Option { + if s.is_empty() || !s.bytes().all(|b| b.is_ascii_digit()) { + return None; + } + s.parse().ok() +} + +/// Parse the last `FUZZ-RESULT iters= divergences=` stdout line +/// of a fuzz command (exact ASCII form, single spaces). Returns +/// `(iters, divergences)`, or None when no line matches. +pub fn parse_fuzz_result(stdout: &str) -> Option<(u64, u64)> { + stdout.lines().rev().find_map(|line| { + let line = line.strip_suffix('\r').unwrap_or(line); + let rest = line.strip_prefix("FUZZ-RESULT iters=")?; + let (iters, rest) = rest.split_once(' ')?; + let divergences = rest.strip_prefix("divergences=")?; + Some((parse_u64_strict(iters)?, parse_u64_strict(divergences)?)) + }) +} + +/// Recorded verbatim when the target declares no fuzz command at all. +const FUZZ_SKIP_NO_COMMAND: &str = "no fuzz command declared in target.toml [gates].fuzz"; + +/// Recorded verbatim when a fuzz command exists but the caller has no +/// pristine baseline: differential fuzz differs two binaries, and comparing +/// the candidate against itself would be a gate that cannot fail. +const FUZZ_SKIP_NO_BASELINE: &str = + "differential fuzz needs a pristine baseline binary; run the accept path (`just verdict`), \ + which rebuilds one"; + +/// Substitute the three `[gates].fuzz` template placeholders. The contract +/// with `targets//target.toml` and `scripts/diff-fuzz-*.py`: `{iters}` +/// is the iteration budget, `{baseline}` the pristine baseline binary and +/// `{candidate}` the patched one (both repo-root-relative paths, and the +/// command runs from the repo root). A fuzz script must never hardcode a +/// baseline path — the harness owns where the pristine build lands. +fn fuzz_command(tmpl: &str, iters: u64, baseline: &str, candidate: &str) -> String { + tmpl.replace("{iters}", &iters.to_string()) + .replace("{baseline}", baseline) + .replace("{candidate}", candidate) +} + +/// What the differential-fuzz gate will do, decided before any process runs. +#[derive(Debug, PartialEq, Eq)] +enum FuzzPlan { + Skip(&'static str), + Run(String), +} + +/// Differential fuzz is a baseline-vs-candidate gate: it needs both sides. +/// Missing either one is a Skip with the reason recorded, never a silent +/// self-comparison and never a hard error (the standalone gates flow has +/// no baseline by construction). +fn plan_fuzz(fuzz: Option<&str>, inputs: &GateInputs) -> FuzzPlan { + match (fuzz, inputs.baseline_binary) { + (None, _) => FuzzPlan::Skip(FUZZ_SKIP_NO_COMMAND), + (Some(_), None) => FuzzPlan::Skip(FUZZ_SKIP_NO_BASELINE), + (Some(tmpl), Some(baseline)) => FuzzPlan::Run(fuzz_command( + tmpl, + inputs.fuzz_iters, + baseline, + inputs.candidate_binary, + )), + } +} + +/// Grade one executed fuzz run. Passed iff the command exited 0, printed a +/// well-formed `FUZZ-RESULT` line, and reported zero divergences — a run +/// that never reports the line is a Failed gate, not a pass by silence. +/// Returns `(outcome, iters_actually_executed, divergence)`. +fn fuzz_gate_outcome( + stdout: &str, + success: bool, + exit_code: Option, +) -> (GateOutcome, u64, bool) { + match parse_fuzz_result(stdout) { + None => ( + GateOutcome::Failed { + detail: "fuzz command did not report FUZZ-RESULT".into(), + }, + 0, + false, + ), + Some((iters, divergences)) => ( + if success && divergences == 0 { + GateOutcome::Passed + } else { + GateOutcome::Failed { + detail: format!("exit={exit_code:?} iters={iters} divergences={divergences}"), + } + }, + iters, + divergences > 0, + ), + } +} + +/// The fp-tolerance policy fails closed without a committed reference: +/// there is no hash to fall back to that a tolerance could compare against. +fn fp_reference(golden_reference: Option<&PathBuf>) -> Result<&PathBuf, GateOutcome> { + golden_reference.ok_or_else(|| GateOutcome::Failed { + detail: "fp-tolerance policy requires corpus.golden_reference in target.toml".into(), + }) +} + +/// GoldenReplay verdict under an fp-tolerance policy: the replay stdout is +/// compared token-wise against the committed reference output. +fn fp_golden_outcome( + policy: &EquivalencePolicy, + reference: &str, + stdout: &str, + success: bool, + exit_code: Option, +) -> GateOutcome { + if !success { + return GateOutcome::Failed { + detail: format!("golden replay command failed (exit={exit_code:?})"), + }; + } + match policy.compare(reference, stdout) { + Ok(()) => GateOutcome::Passed, + Err(d) => GateOutcome::Failed { + detail: format!( + "diverged at token/offset {}: baseline={:?} candidate={:?} — {}", + d.token, d.baseline, d.candidate, d.reason + ), + }, + } +} + +/// Run the mechanical gates for `target`, testing `inputs.candidate_binary` +/// (repo-root-relative) in golden replay and differential fuzz. Differential +/// fuzz additionally needs `inputs.baseline_binary` to differ against and is +/// skipped-with-reason without one. Pin mismatches (corpus or test suite) +/// are errors, not gate failures: nothing runs on tampered inputs. pub fn run_core_gates( root: &Path, spec: &TargetSpec, - binary: &str, -) -> anyhow::Result> { + inputs: &GateInputs, +) -> anyhow::Result { let verified = pin::verify_manifest( &root.join(&spec.corpus.manifest), &root.join(&spec.corpus.root), @@ -34,6 +185,24 @@ pub fn run_core_gates( spec.corpus.manifest.display() ); + // Test-suite pin: same refuse-on-mismatch posture as the corpus pin, + // but the manifest is optional until a human deliberately generates it. + let suite_manifest = suite_manifest_path(&spec.corpus.manifest); + if root.join(&suite_manifest).exists() { + let verified = pin::verify_manifest(&root.join(&suite_manifest), root).map_err(|e| { + anyhow::anyhow!("upstream test suite hash mismatch — refusing to run: {e}") + })?; + eprintln!( + "test-suite pin: {verified} files verified against {}", + suite_manifest.display() + ); + } else { + eprintln!( + "test-suite pin: {} not found — suite unpinned (generate with scripts/pin-testsuite.sh)", + suite_manifest.display() + ); + } + let mut results = Vec::new(); let t = sh(root, &spec.gates.tests)?; @@ -51,34 +220,285 @@ pub fn run_core_gates( }, )); - let g = sh(root, &spec.gates.golden.replace("{binary}", binary))?; - let expected = expected_golden_hash(&root.join(&spec.corpus.golden_sha256))?; - let actual = format!("{:x}", Sha256::digest(&g.stdout)); - results.push(( - GateLayer::GoldenReplay, - if g.status.success() && actual == expected { - GateOutcome::Passed - } else { - GateOutcome::Failed { - detail: format!( - "exit={:?} expected={expected} actual={actual}", - g.status.code() - ), + let policy = EquivalencePolicy::load(&spec.target_dir(root))?; + let equivalence_mode = match &policy { + EquivalencePolicy::ByteIdentical => "byte-identical", + EquivalencePolicy::FpTolerance { .. } => "fp-tolerance", + }; + let golden_cmd = spec + .gates + .golden + .replace("{binary}", inputs.candidate_binary); + let golden = match &policy { + EquivalencePolicy::ByteIdentical => { + let g = sh(root, &golden_cmd)?; + let expected = expected_golden_hash(&root.join(&spec.corpus.golden_sha256))?; + let actual = format!("{:x}", Sha256::digest(&g.stdout)); + if g.status.success() && actual == expected { + GateOutcome::Passed + } else { + GateOutcome::Failed { + detail: format!( + "exit={:?} expected={expected} actual={actual}", + g.status.code() + ), + } } - }, - )); + } + EquivalencePolicy::FpTolerance { .. } => { + match fp_reference(spec.corpus.golden_reference.as_ref()) { + Err(outcome) => outcome, + Ok(reference_path) => { + let reference = + std::fs::read_to_string(root.join(reference_path)).map_err(|e| { + anyhow::anyhow!( + "cannot read corpus.golden_reference {}: {e}", + reference_path.display() + ) + })?; + let g = sh(root, &golden_cmd)?; + fp_golden_outcome( + &policy, + &reference, + &String::from_utf8_lossy(&g.stdout), + g.status.success(), + g.status.code(), + ) + } + } + } + }; + results.push((GateLayer::GoldenReplay, golden)); + + let (fuzz_outcome, fuzz_iters, fuzz_divergence) = + match plan_fuzz(spec.gates.fuzz.as_deref(), inputs) { + FuzzPlan::Skip(reason) => ( + GateOutcome::Skipped { + reason: reason.to_string(), + }, + 0, + false, + ), + FuzzPlan::Run(cmd) => { + let f = sh(root, &cmd)?; + fuzz_gate_outcome( + &String::from_utf8_lossy(&f.stdout), + f.status.success(), + f.status.code(), + ) + } + }; + results.push((GateLayer::DifferentialFuzz, fuzz_outcome)); - results.push(( - GateLayer::DifferentialFuzz, - GateOutcome::Skipped { - reason: "per-attempt manual step in Phase 1 (needs old/new pair)".into(), - }, - )); results.push(( GateLayer::Sanitizers, GateOutcome::Skipped { - reason: "per-attempt manual step in Phase 1 (nightly toolchain)".into(), + reason: + "sanitizers run on the accept path in verdict (ASan/UBSan, TSan when configured)" + .into(), }, )); - Ok(results) + + Ok(GateReport { + gates: results, + fuzz_iters, + fuzz_divergence, + equivalence_mode: equivalence_mode.to_string(), + }) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn suite_manifest_derives_from_corpus_manifest() { + assert_eq!( + suite_manifest_path(Path::new("corpora/cjson/MANIFEST.sha256")), + Path::new("corpora/cjson").join("TESTSUITE.sha256") + ); + assert_eq!( + suite_manifest_path(Path::new("MANIFEST.sha256")), + Path::new("TESTSUITE.sha256") + ); + } + + #[test] + fn fuzz_result_parses_last_exact_line() { + assert_eq!( + parse_fuzz_result("FUZZ-RESULT iters=10000 divergences=0\n"), + Some((10000, 0)) + ); + // Last matching line wins; trailing non-matching output is ignored. + let out = "FUZZ-RESULT iters=1 divergences=1\nprogress...\nFUZZ-RESULT iters=9999 divergences=2\ndone\n"; + assert_eq!(parse_fuzz_result(out), Some((9999, 2))); + // CRLF-terminated capture still parses. + assert_eq!( + parse_fuzz_result("FUZZ-RESULT iters=42 divergences=0\r\n"), + Some((42, 0)) + ); + } + + #[test] + fn fuzz_result_rejects_malformed_lines() { + for bad in [ + "", + "no result here", + " FUZZ-RESULT iters=1 divergences=0", // leading space + "FUZZ-RESULT iters=1 divergences=0", // double space + "FUZZ-RESULT iters=abc divergences=0", // non-numeric + "FUZZ-RESULT iters=1 divergences=-1", // sign not allowed + "FUZZ-RESULT iters=1 divergences=0 xx", // trailing junk + "FUZZ-RESULT divergences=0 iters=1", // wrong field order + "FUZZ-RESULT iters=1", // missing field + ] { + assert_eq!(parse_fuzz_result(bad), None, "should reject: {bad:?}"); + } + } + + fn inputs(baseline: Option<&'static str>) -> GateInputs<'static> { + GateInputs { + candidate_binary: "targets/cjson/cand/cjson-bench", + baseline_binary: baseline, + fuzz_iters: 10_000, + } + } + + const TMPL: &str = "python3 scripts/diff-fuzz-cjson.py targets/cjson/fuzz-work \ + {iters} {baseline} {candidate}"; + + #[test] + fn fuzz_command_substitutes_all_three_placeholders() { + assert_eq!( + fuzz_command(TMPL, 500, "base/bin", "cand/bin"), + "python3 scripts/diff-fuzz-cjson.py targets/cjson/fuzz-work 500 base/bin cand/bin" + ); + // Repeated placeholders all substitute; unknown braces are left alone. + assert_eq!( + fuzz_command("{candidate} {candidate} {iters} {other}", 2, "b", "c"), + "c c 2 {other}" + ); + } + + #[test] + fn fuzz_skipped_when_no_command_declared() { + assert_eq!( + plan_fuzz(None, &inputs(Some("targets/cjson/baseline/cjson-bench"))), + FuzzPlan::Skip("no fuzz command declared in target.toml [gates].fuzz") + ); + // Absent command wins even without a baseline: nothing to run either way. + assert_eq!( + plan_fuzz(None, &inputs(None)), + FuzzPlan::Skip(FUZZ_SKIP_NO_COMMAND) + ); + } + + /// The regression this branch exists for: `just gates` (and CI) build only + /// the working tree, so there is no pristine baseline. Before the fix the + /// fuzz script silently hardcoded targets//baseline/… — a path nothing + /// on that flow builds — and the gate hard-failed. Now it skips, loudly. + #[test] + fn fuzz_skipped_when_declared_but_no_baseline() { + assert_eq!( + plan_fuzz(Some(TMPL), &inputs(None)), + FuzzPlan::Skip( + "differential fuzz needs a pristine baseline binary; run the accept path \ + (`just verdict`), which rebuilds one" + ) + ); + } + + #[test] + fn fuzz_runs_with_both_sides() { + assert_eq!( + plan_fuzz( + Some(TMPL), + &inputs(Some("targets/cjson/baseline/cjson-bench")) + ), + FuzzPlan::Run( + "python3 scripts/diff-fuzz-cjson.py targets/cjson/fuzz-work 10000 \ + targets/cjson/baseline/cjson-bench targets/cjson/cand/cjson-bench" + .to_string() + ) + ); + } + + #[test] + fn fuzz_gate_grades_clean_run_as_pass() { + let (outcome, iters, diverged) = + fuzz_gate_outcome("FUZZ-RESULT iters=10000 divergences=0\n", true, Some(0)); + assert_eq!(outcome, GateOutcome::Passed); + // The ledger records iterations *actually executed*, not the budget. + assert_eq!(iters, 10_000); + assert!(!diverged); + } + + #[test] + fn fuzz_gate_fails_on_divergence_nonzero_exit_or_no_result_line() { + // Divergence reported: fail, and the count is still recorded. + let (outcome, iters, diverged) = + fuzz_gate_outcome("FUZZ-RESULT iters=900 divergences=3\n", false, Some(1)); + assert!( + matches!(outcome, GateOutcome::Failed { detail } if detail.contains("divergences=3")) + ); + assert_eq!(iters, 900); + assert!(diverged); + + // Zero divergences but a nonzero exit (crash/timeout) is NOT a pass. + let (outcome, ..) = + fuzz_gate_outcome("FUZZ-RESULT iters=10 divergences=0\n", false, Some(2)); + assert!( + matches!(outcome, GateOutcome::Failed { detail } if detail.contains("exit=Some(2)")) + ); + + // A run that never reports the contract line fails closed (e.g. the + // script died before printing — the old hardcoded-baseline breakage). + let (outcome, iters, diverged) = + fuzz_gate_outcome("Traceback (most recent call last):\n", false, Some(1)); + assert_eq!( + outcome, + GateOutcome::Failed { + detail: "fuzz command did not report FUZZ-RESULT".into() + } + ); + assert_eq!(iters, 0); + assert!(!diverged); + } + + #[test] + fn fp_tolerance_requires_golden_reference() { + let err = fp_reference(None).unwrap_err(); + assert_eq!( + err, + GateOutcome::Failed { + detail: "fp-tolerance policy requires corpus.golden_reference in target.toml" + .into() + } + ); + let path = PathBuf::from("corpora/matmul/GOLDEN.reference"); + assert_eq!(fp_reference(Some(&path)).unwrap(), &path); + } + + #[test] + fn fp_golden_outcome_applies_policy() { + let policy = EquivalencePolicy::FpTolerance { + abs: 1e-6, + rel: 1e-5, + }; + // Last-ULP reordering within tolerance passes. + assert_eq!( + fp_golden_outcome(&policy, "1.0000000 2.0", "1.0000001 2.0", true, Some(0)), + GateOutcome::Passed + ); + // A genuine wrong value fails with the divergence in the detail. + assert!(matches!( + fp_golden_outcome(&policy, "1.0 2.0", "1.0 2.5", true, Some(0)), + GateOutcome::Failed { detail } if detail.contains("token/offset 1") + )); + // A failing replay command fails regardless of output. + assert!(matches!( + fp_golden_outcome(&policy, "1.0", "1.0", false, Some(3)), + GateOutcome::Failed { detail } if detail.contains("exit=Some(3)") + )); + } } diff --git a/crates/diff-test/src/target.rs b/crates/diff-test/src/target.rs index 2596358..31b5ef4 100644 --- a/crates/diff-test/src/target.rs +++ b/crates/diff-test/src/target.rs @@ -12,6 +12,10 @@ pub struct TargetSpec { pub bench: Bench, pub gates: Gates, pub corpus: Corpus, + /// Target name (the directory under `targets/`), captured by `load`; + /// not part of target.toml itself. + #[serde(skip)] + name: String, } #[derive(Debug, Deserialize)] @@ -52,6 +56,14 @@ pub struct Build { /// Sanitizer binary path with `{out}` substituted. #[serde(default)] pub sanitizer_binary: Option, + /// TSan build command; `{out}` substituted. Run on the accept path + /// when configured — a patch touching concurrency without a TSan + /// lane cannot be machine-accepted. + #[serde(default)] + pub tsan: Option, + /// TSan binary path with `{out}` substituted. + #[serde(default)] + pub tsan_binary: Option, } /// Substitute the `{out}` isolation-dir placeholder in a build command or @@ -66,6 +78,11 @@ pub struct Gates { pub tests: String, /// Golden replay command; stdout is hashed. `{binary}` substituted. pub golden: String, + /// Differential-fuzz command; `{candidate}` and `{iters}` substituted. + /// Must print `FUZZ-RESULT iters= divergences=` as its final + /// stdout line. Absent → the fuzz gate is skipped with a reason. + #[serde(default)] + pub fuzz: Option, } #[derive(Debug, Deserialize)] @@ -77,6 +94,11 @@ pub struct Corpus { /// File whose last whitespace-delimited-first-field line is the /// expected sha256 of the golden command's stdout. pub golden_sha256: PathBuf, + /// Committed reference stdout of the golden command, repo-root- + /// relative. Required by fp-tolerance targets: tolerance comparison + /// needs the actual values, not a hash. + #[serde(default)] + pub golden_reference: Option, } impl TargetSpec { @@ -84,7 +106,15 @@ impl TargetSpec { let path = repo_root.join("targets").join(name).join("target.toml"); let raw = std::fs::read_to_string(&path) .map_err(|e| anyhow::anyhow!("cannot read {}: {e}", path.display()))?; - Ok(toml::from_str(&raw)?) + let mut spec: Self = toml::from_str(&raw)?; + spec.name = name.to_string(); + Ok(spec) + } + + /// This target's directory under the repo root (`targets/`), + /// where per-target policy files like equivalence.toml live. + pub fn target_dir(&self, root: &Path) -> PathBuf { + root.join("targets").join(&self.name) } } @@ -100,3 +130,48 @@ pub fn expected_golden_hash(path: &Path) -> anyhow::Result { .map(str::to_string) .ok_or_else(|| anyhow::anyhow!("no hash entry in {}", path.display())) } + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn load_captures_name_and_defaults_new_fields() { + let root = std::env::temp_dir().join("hotpath-target-load"); + let dir = root.join("targets").join("demo"); + std::fs::create_dir_all(&dir).unwrap(); + std::fs::write( + dir.join("target.toml"), + r#" +[source] +repo = "https://example.invalid/demo.git" +commit = "deadbeef" +license = "MIT" + +[build] +baseline = "make -C targets/demo OUT={out}" +binary = "{out}/demo" + +[bench] +command = "{binary} bench" +workload = "demo workload" + +[gates] +tests = "make -C targets/demo test" +golden = "{binary} replay" + +[corpus] +manifest = "corpora/demo/MANIFEST.sha256" +root = "corpora/demo" +golden_sha256 = "corpora/demo/GOLDEN.sha256" +"#, + ) + .unwrap(); + let spec = TargetSpec::load(&root, "demo").unwrap(); + assert_eq!(spec.target_dir(&root), root.join("targets").join("demo")); + assert!(spec.gates.fuzz.is_none()); + assert!(spec.corpus.golden_reference.is_none()); + assert!(spec.build.tsan.is_none()); + assert!(spec.build.tsan_binary.is_none()); + } +} diff --git a/crates/harnessd/Cargo.toml b/crates/harnessd/Cargo.toml index dce4568..bb080b8 100644 --- a/crates/harnessd/Cargo.toml +++ b/crates/harnessd/Cargo.toml @@ -1,6 +1,6 @@ [package] name = "harnessd" -description = "Agent-facing IPC daemon: the six-tool surface (SPEC §3.5) over line-delimited JSON on stdio; patches applied only after path-allowlist checks" +description = "Agent-facing IPC daemon: the seven-op JSON surface (SPEC §3.5) over line-delimited JSON on stdio; patches applied only after path-allowlist checks" version.workspace = true edition.workspace = true license.workspace = true diff --git a/crates/harnessd/src/allowlist.rs b/crates/harnessd/src/allowlist.rs new file mode 100644 index 0000000..fe154e0 --- /dev/null +++ b/crates/harnessd/src/allowlist.rs @@ -0,0 +1,309 @@ +//! Diff-header path vetting: the allowlist between agent-authored diffs +//! and `git apply` in the target workspace. Portable (no Unix +//! dependencies) so the escape-regression tests run on every development +//! platform, not just the daemon's POSIX hosts. + +use anyhow::{anyhow, ensure, Result}; +use std::path::{Component, Path}; + +/// Relative paths only; no parent components, no .git, no absolutes. +pub fn check_rel_path(p: &str) -> Result<()> { + let path = Path::new(p); + ensure!(!path.is_absolute(), "absolute paths forbidden"); + for c in path.components() { + match c { + Component::Normal(seg) => ensure!(seg != ".git", ".git access forbidden"), + Component::CurDir => {} + _ => anyhow::bail!("path escapes workspace: {p}"), + } + } + Ok(()) +} + +const RENAME_COPY_HEADERS: [&str; 4] = ["rename from ", "rename to ", "copy from ", "copy to "]; + +/// Appended to every `---`/`+++` rejection. That rule is the one place where a +/// hunk BODY line can be taken for a header — `-` and `+` are the body markers +/// themselves — and the bare error ("lacks the a/ or b/ prefix", "path escapes +/// workspace") is baffling if all you did was delete a comment. Say the cause +/// out loud. No other header rule can collide: a body line always carries a +/// `-`, `+` or space marker, so it can never begin with `diff --git ` or +/// `rename `. +const HEADER_COLLISION_NOTE: &str = "\ + note: this line was vetted as a `---`/`+++` file header. The scanner does not track hunk \ + boundaries — deliberate, and it stays that way: a real header must never slip past it as body \ + text — so a hunk body line reaches this rule too. REMOVING a line whose own text begins with \ + `-- ` renders at column 0 as `--- ...` (an unindented SQL/Lua `-- comment`), and ADDING one \ + that begins with `++ ` renders as `+++ ...`. If that is what this line is, the patch is being \ + rejected on a false positive. Workaround: leave such lines untouched as context and edit \ + around them. Rewording or indenting the line does NOT help — the removal itself is what \ + renders as `--- `. Mind the space: `--x`/`++x` (C decrement/increment) never collide."; + +/// Vet every path-bearing header line of a unified git diff: `diff --git`, +/// `---`, `+++`, and the rename/copy extended headers. The whole diff is +/// rejected if any extracted path is absolute, contains a `..` component, +/// is quoted (git's C-style quoting can encode `..` invisibly), or — for +/// `---`/`+++` lines — lacks the standard `a/`/`b/` prefix (`--no-prefix` +/// output). Every line is scanned without tracking hunk boundaries, so a +/// content line colliding with a header prefix is vetted too: over-rejection +/// is the safe direction, and `git apply` remains the backstop. +pub fn check_diff_paths(diff: &str) -> Result<()> { + for line in diff.lines() { + if let Some(rest) = line.strip_prefix("diff --git ") { + let (a, b) = split_git_header_paths(rest).ok_or_else(|| { + anyhow!( + "cannot parse diff header {line:?}: expected \ + `diff --git a/ b/` with plain unquoted paths" + ) + })?; + vet_extracted(a)?; + vet_extracted(b)?; + } else if line.starts_with("--- ") || line.starts_with("+++ ") { + vet_file_header(&line[4..]).map_err(|e| { + anyhow!("{e}\n offending line: {line:?}\n {HEADER_COLLISION_NOTE}") + })?; + } else if let Some(rest) = RENAME_COPY_HEADERS + .iter() + .find_map(|pre| line.strip_prefix(pre)) + { + vet_extracted(header_path(rest))?; + } + } + Ok(()) +} + +/// Vet the path on a `---`/`+++` file-header line; `rest` is the text after +/// the four-byte marker. Errors are wrapped by the caller with +/// `HEADER_COLLISION_NOTE`, since any line reaching here *might* be a hunk +/// body line rather than a header. +fn vet_file_header(rest: &str) -> Result<()> { + let p = header_path(rest); + if p == "/dev/null" { + return Ok(()); + } + ensure!( + !p.starts_with('"'), + "quoted diff paths forbidden: {p} (rewrite the patch with plain ASCII paths)" + ); + let stripped = p + .strip_prefix("a/") + .or_else(|| p.strip_prefix("b/")) + .ok_or_else(|| { + anyhow!( + "diff path {p:?} lacks the a/ or b/ prefix — regenerate with \ + standard `git diff` output (--no-prefix diffs are rejected)" + ) + })?; + vet_extracted(stripped) +} + +/// Header paths may carry a traditional-diff `\t` suffix. +fn header_path(rest: &str) -> &str { + rest.split('\t').next().unwrap_or(rest).trim_end() +} + +/// `diff --git a/ b/`: split at the LAST ` b/` so a ` b/` embedded +/// in cannot truncate ; splitting only ever removes the three +/// bytes ` b/`, so a `..` component in either real path always survives +/// intact into one of the two fragments. +fn split_git_header_paths(rest: &str) -> Option<(&str, &str)> { + let a_rest = rest.strip_prefix("a/")?; + let idx = a_rest.rfind(" b/")?; + Some((&a_rest[..idx], a_rest[idx + 3..].trim_end())) +} + +fn vet_extracted(p: &str) -> Result<()> { + if p == "/dev/null" { + return Ok(()); + } + ensure!(!p.is_empty(), "empty path in diff header"); + ensure!( + !p.starts_with('"'), + "quoted diff paths forbidden: {p} (rewrite the patch with plain ASCII paths)" + ); + ensure!( + !p.starts_with('/'), + "absolute diff paths forbidden (only /dev/null is allowed, for new/deleted files): {p}" + ); + check_rel_path(p) +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn rel_path_rules() { + assert!(check_rel_path("src/lib.rs").is_ok()); + assert!(check_rel_path("./src/lib.rs").is_ok()); + assert!(check_rel_path("../outside").is_err()); + assert!(check_rel_path("a/../b").is_err()); + assert!(check_rel_path("/etc/passwd").is_err()); + assert!(check_rel_path(".git/config").is_err()); + assert!(check_rel_path("a/.git/hooks").is_err()); + } + + #[test] + fn normal_diff_passes() { + let diff = "diff --git a/src/lib.rs b/src/lib.rs\n\ + index 1111111..2222222 100644\n\ + --- a/src/lib.rs\n\ + +++ b/src/lib.rs\n\ + @@ -1,2 +1,2 @@\n\ + -old\n\ + +new\n"; + assert!(check_diff_paths(diff).is_ok()); + } + + #[test] + fn new_and_deleted_file_dev_null_passes() { + let diff = "diff --git a/newfile.rs b/newfile.rs\n\ + new file mode 100644\n\ + --- /dev/null\n\ + +++ b/newfile.rs\n\ + @@ -0,0 +1 @@\n\ + +hello\n\ + diff --git a/gone.rs b/gone.rs\n\ + deleted file mode 100644\n\ + --- a/gone.rs\n\ + +++ /dev/null\n\ + @@ -1 +0,0 @@\n\ + -bye\n"; + assert!(check_diff_paths(diff).is_ok()); + } + + #[test] + fn in_workspace_rename_passes() { + let diff = "diff --git a/old.rs b/nested/new.rs\n\ + similarity index 100%\n\ + rename from old.rs\n\ + rename to nested/new.rs\n"; + assert!(check_diff_paths(diff).is_ok()); + } + + /// The exact fixture from the apply-escape audit: a rename whose + /// destination climbs out of the workspace. `---`/`+++` scanning alone + /// never sees it — the paths live only in the `diff --git` and + /// rename headers. + #[test] + fn rename_escape_fixture_rejected() { + let diff = "diff --git a/src.txt b/../outside/escaped.txt\n\ + similarity index 100%\n\ + rename from src.txt\n\ + rename to ../outside/escaped.txt\n"; + assert!(check_diff_paths(diff).is_err()); + } + + #[test] + fn rename_headers_rejected_without_git_header_line() { + let diff = "similarity index 100%\n\ + rename from src.txt\n\ + rename to ../outside/escaped.txt\n"; + assert!(check_diff_paths(diff).is_err()); + } + + #[test] + fn copy_escape_rejected() { + let diff = "diff --git a/src.txt b/src2.txt\n\ + copy from src.txt\n\ + copy to ../outside/copied.txt\n"; + assert!(check_diff_paths(diff).is_err()); + } + + #[test] + fn no_prefix_diff_rejected() { + let diff = "diff --git src/lib.rs src/lib.rs\n\ + index 1111111..2222222 100644\n\ + --- src/lib.rs\n\ + +++ src/lib.rs\n\ + @@ -1 +1 @@\n\ + -old\n\ + +new\n"; + let err = check_diff_paths(diff).unwrap_err().to_string(); + assert!( + err.contains("diff --git") || err.contains("a/ or b/ prefix"), + "{err}" + ); + } + + #[test] + fn absolute_path_diff_rejected() { + let diff = "diff --git a/x b/x\n\ + rename from x\n\ + rename to /etc/passwd\n"; + assert!(check_diff_paths(diff).is_err()); + let plus = "--- a/x\n\ + +++ /etc/passwd\n"; + assert!(check_diff_paths(plus).is_err()); + } + + #[test] + fn parent_component_in_file_headers_rejected() { + assert!(check_diff_paths("--- a/../escape.rs\n+++ b/escape.rs\n").is_err()); + assert!(check_diff_paths("--- a/ok.rs\n+++ b/nested/../../escape.rs\n").is_err()); + assert!(check_diff_paths("diff --git a/ok.rs b/../escape.rs\n").is_err()); + } + + #[test] + fn dot_git_path_rejected() { + let diff = "diff --git a/.git/hooks/pre-commit b/.git/hooks/pre-commit\n\ + --- a/.git/hooks/pre-commit\n\ + +++ b/.git/hooks/pre-commit\n"; + assert!(check_diff_paths(diff).is_err()); + } + + #[test] + fn quoted_paths_rejected() { + assert!(check_diff_paths("diff --git \"a/x\" \"b/y\"\n").is_err()); + assert!(check_diff_paths("--- \"a/x\"\n+++ b/y\n").is_err()); + assert!(check_diff_paths("rename to \"\\056\\056/evil\"\n").is_err()); + } + + #[test] + fn embedded_b_slash_cannot_hide_escape() { + // Whatever way ` b/` splits this header, the `..` lands in a fragment. + assert!(check_diff_paths("diff --git a/x b/y b/../evil\n").is_err()); + assert!(check_diff_paths("diff --git a/../evil b/y b/z\n").is_err()); + } + + /// A removed body line that renders as `--- ` at column 0 — deleting an + /// unindented SQL/Lua `-- comment` — is still REJECTED (fail-closed is the + /// point; tracking hunk boundaries here would risk desyncing from git's own + /// parser). But the error must name that cause instead of leaving an agent + /// staring at a complaint about a/ and b/ prefixes it never wrote. + #[test] + fn body_line_colliding_with_header_prefix_explains_itself() { + let diff = "diff --git a/q.sql b/q.sql\n\ + --- a/q.sql\n\ + +++ b/q.sql\n\ + @@ -1,2 +1 @@\n\ + --- drop this comment\n\ + \x20SELECT 1;\n"; + let err = check_diff_paths(diff).unwrap_err().to_string(); + assert!(err.contains("--- drop this comment"), "{err}"); + assert!(err.contains("hunk body line"), "{err}"); + assert!(err.contains("context"), "{err}"); + } + + /// The collision needs the SPACE. A removed `--x;` renders as `---x;` and an + /// added `++x;` as `+++x;` — neither is a `--- `/`+++ ` header, and both + /// must sail through. Guards against anyone "fixing" the note above by + /// widening the prefix match. + #[test] + fn body_line_without_the_space_does_not_collide() { + let diff = "diff --git a/x.c b/x.c\n\ + --- a/x.c\n\ + +++ b/x.c\n\ + @@ -1,2 +1,2 @@\n\ + ---x;\n\ + +++x;\n"; + assert!(check_diff_paths(diff).is_ok()); + } + + #[test] + fn timestamp_suffix_tolerated() { + let diff = "--- a/src/lib.rs\t2026-07-13 00:00:00\n\ + +++ b/src/lib.rs\t2026-07-13 00:00:00\n"; + assert!(check_diff_paths(diff).is_ok()); + } +} diff --git a/crates/harnessd/src/daemon.rs b/crates/harnessd/src/daemon.rs new file mode 100644 index 0000000..a5e3e7d --- /dev/null +++ b/crates/harnessd/src/daemon.rs @@ -0,0 +1,556 @@ +//! The Unix daemon proper: stdio/socket serving, op dispatch, and the +//! detached verdict pipeline (setsid + Unix sockets keep this POSIX-only; +//! path vetting lives in the portable `allowlist` module). + +use crate::allowlist; +use anyhow::{anyhow, ensure, Result}; +use serde_json::{json, Value}; +use sha2::{Digest, Sha256}; +use std::io::{BufRead, Write}; +use std::os::unix::net::UnixListener; +use std::path::Path; +use std::process::Command; +use std::sync::Mutex; + +/// Serializes mutating ops (git apply, pending writes) across socket +/// connections; reads don't need it but the cost is nil at one agent. +static HANDLE_LOCK: Mutex<()> = Mutex::new(()); + +pub fn run() -> Result<()> { + let root = std::env::current_dir()?.canonicalize()?; + let args: Vec = std::env::args().collect(); + if let Some(i) = args.iter().position(|a| a == "--socket") { + let path = args + .get(i + 1) + .ok_or_else(|| anyhow!("--socket requires a path"))?; + return serve_socket(&root, Path::new(path)); + } + let stdin = std::io::stdin(); + let mut stdout = std::io::stdout().lock(); + for line in stdin.lock().lines() { + let line = line?; + if line.trim().is_empty() { + continue; + } + let resp = respond(&root, &line); + writeln!(stdout, "{resp}")?; + stdout.flush()?; + } + Ok(()) +} + +/// SPEC §10 socket mode: harnessd runs as the trusted uid and listens on a +/// Unix socket; the agent process runs as an unprivileged user whose only +/// write path into the repo is this daemon. Socket permissions (owner/group/ +/// mode) are set by the supervisor script that starts us. +fn serve_socket(root: &Path, sock: &Path) -> Result<()> { + if let Some(dir) = sock.parent() { + std::fs::create_dir_all(dir)?; + } + let _ = std::fs::remove_file(sock); + let listener = UnixListener::bind(sock)?; + eprintln!("harnessd listening on {}", sock.display()); + for conn in listener.incoming() { + let stream = match conn { + Ok(s) => s, + Err(e) => { + eprintln!("accept error: {e}"); + continue; + } + }; + let root = root.to_path_buf(); + std::thread::spawn(move || { + let reader = match stream.try_clone() { + Ok(s) => std::io::BufReader::new(s), + Err(_) => return, + }; + let mut writer = stream; + for line in reader.lines() { + let Ok(line) = line else { break }; + if line.trim().is_empty() { + continue; + } + let resp = respond(&root, &line); + if writeln!(writer, "{resp}").is_err() { + break; + } + } + }); + } + Ok(()) +} + +fn respond(root: &Path, line: &str) -> Value { + // read_verdict long-polls the ledger (read-only) and must not hold + // the lock that serializes mutating ops while it sleeps. + let is_read_verdict = serde_json::from_str::(line) + .ok() + .and_then(|v| { + v.get("op") + .and_then(Value::as_str) + .map(|o| o == "read_verdict") + }) + .unwrap_or(false); + let _guard = if is_read_verdict { + None + } else { + Some(HANDLE_LOCK.lock().unwrap_or_else(|p| p.into_inner())) + }; + match handle(root, line) { + Ok(v) => json!({"ok": true, "result": v}), + Err(e) => json!({"ok": false, "error": e.to_string()}), + } +} + +/// Single-quote a string for safe embedding in a generated shell script. +fn shq(s: &str) -> String { + format!("'{}'", s.replace('\'', "'\\''")) +} + +/// The exact probe `scripts/no-net.sh` runs before it decides whether to +/// isolate: can this host open an unprivileged network namespace? Run it here +/// too — same uid, same environment, same host as the wrapper we are about to +/// spawn — so the isolation note we record describes the run that actually +/// happened instead of the one we hoped for. +fn netns_available() -> bool { + Command::new("unshare") + .args(["--net", "--map-current-user", "true"]) + .stdout(std::process::Stdio::null()) + .stderr(std::process::Stdio::null()) + .status() + .map(|s| s.success()) + .unwrap_or(false) +} + +fn field<'a>(req: &'a Value, key: &str) -> Result<&'a str> { + req.get(key) + .and_then(Value::as_str) + .ok_or_else(|| anyhow!("missing string field: {key}")) +} + +/// A target name must be a plain directory name — no separators, no dots. +fn check_target(name: &str) -> Result<()> { + ensure!( + !name.is_empty() + && name + .chars() + .all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_'), + "invalid target name" + ); + Ok(()) +} + +const RUNNING_LOCK: &str = "results/pending/RUNNING.lock"; + +/// The detached verdict pipeline benches the workspace tree it was launched +/// against; HANDLE_LOCK cannot cover it, so a lock file does. Refuse any op +/// that would swap that tree while a pipeline is in flight. +fn ensure_no_running_pipeline(root: &Path) -> Result<()> { + let lock = root.join(RUNNING_LOCK); + if lock.exists() { + let rid = std::fs::read_to_string(&lock).unwrap_or_default(); + let rid = rid.trim(); + let rid = if rid.is_empty() { "unknown" } else { rid }; + anyhow::bail!( + "verdict pipeline {rid} in flight ({RUNNING_LOCK} present — wait for \ + read_verdict to resolve it, or remove the lock if the pipeline is dead)" + ); + } + Ok(()) +} + +fn handle(root: &Path, line: &str) -> Result { + let req: Value = serde_json::from_str(line)?; + match field(&req, "op")? { + "read_profile" => { + let t = field(&req, "target")?; + check_target(t)?; + let text = std::fs::read_to_string(root.join(format!("results/{t}/hotspots.txt")))?; + Ok(json!({"hotspots_text": text})) + } + "read_ledger" => { + let t = field(&req, "target")?; + check_target(t)?; + // Read-only surface over the append-only DB. + let ledger = ledger::Ledger::open(&root.join("results/ledger.sqlite"))?; + let classes = ledger.attempted_classes(t)?; + let history = ledger.attempt_history(t)?; + Ok(json!({ + "attempted_playbook_classes": classes, + "attempts": history, + "total_attempts": ledger.count()?, + "note": "a class may be re-entered with a materially NEW hypothesis; \ + never duplicate a (hotspot, class, hypothesis) that has a verdict", + })) + } + "read_playbook" => { + let class = match req.get("class") { + Some(Value::Number(n)) => n.as_u64().ok_or_else(|| anyhow!("bad class"))? as u8, + Some(Value::String(s)) => s.parse()?, + _ => anyhow::bail!("missing class"), + }; + ensure!((1..=7).contains(&class), "playbook class must be 1-7"); + let dir = root.join("playbook"); + let entry = std::fs::read_dir(&dir)? + .filter_map(|e| e.ok()) + .find(|e| { + e.file_name() + .to_string_lossy() + .starts_with(&format!("{class:02}-")) + }) + .ok_or_else(|| anyhow!("no playbook file for class {class}"))?; + Ok(json!({"markdown": std::fs::read_to_string(entry.path())?})) + } + "read_target_source" => { + let t = field(&req, "target")?; + let p = field(&req, "path")?; + check_target(t)?; + allowlist::check_rel_path(p)?; + // Serve the PRISTINE (HEAD) content, not the working tree: + // propose_patch applies each proposal against a reset + // workspace, so reads must show the same base the diff needs + // — otherwise reads after a proposal show the patched tree + // and the next diff's context lines drift (phase2-comrak-005 + // burned turns rediscovering this). + let ws = root.join(format!("targets/{t}/workspace")); + let shown = Command::new("git") + .args(["-C", ws.to_str().unwrap(), "show", &format!("HEAD:{p}")]) + .output()?; + ensure!( + shown.status.success(), + "cannot read {p} at HEAD: {}", + String::from_utf8_lossy(&shown.stderr).trim() + ); + let text = String::from_utf8_lossy(&shown.stdout).into_owned(); + // Optional line window so large source files (100KB+) can be read in + // chunks instead of overflowing one response — otherwise the agent + // is tempted to reach for a shell it must not have. + let lines: Vec<&str> = text.lines().collect(); + let total = lines.len(); + let offset = req.get("offset").and_then(Value::as_u64).unwrap_or(0) as usize; + let limit = req + .get("limit") + .and_then(Value::as_u64) + .map(|v| v as usize) + .unwrap_or(usize::MAX); + let end = offset.saturating_add(limit).min(total); + let start = offset.min(total); + let slice = lines[start..end] + .iter() + .enumerate() + .map(|(i, l)| format!("{}\t{}", start + i + 1, l)) + .collect::>() + .join("\n"); + Ok(json!({ + "content": slice, + "total_lines": total, + "returned_lines": [start + 1, end], + "truncated": end < total, + })) + } + "propose_patch" => { + let t = field(&req, "target")?; + let diff = field(&req, "diff")?; + let hypothesis = field(&req, "hypothesis")?; + check_target(t)?; + ensure_no_running_pipeline(root)?; + // git rejects a diff without a final newline with an opaque + // "corrupt patch" — say it plainly (phase2-comrak-004 burned + // several turns rediscovering this). + ensure!( + diff.ends_with('\n'), + "diff must end with a trailing newline (git would report 'corrupt patch' at EOF)" + ); + // Allowlist: every path named by any diff header must be a safe + // relative path (the git -C below roots them in the target + // workspace; nothing outside it is reachable). + allowlist::check_diff_paths(diff)?; + let ws = root.join(format!("targets/{t}/workspace")); + // Every proposal stands alone: reset the workspace to HEAD + // first so probe/iteration patches cannot accumulate into a + // tested tree the ledger's patch field doesn't describe + // (phase2-comrak-004 audit finding). + ensure!( + Command::new("git") + .args(["-C", ws.to_str().unwrap(), "checkout", "--", "."]) + .status()? + .success(), + "workspace reset failed" + ); + ensure!( + Command::new("git") + .args(["-C", ws.to_str().unwrap(), "clean", "-fdq"]) + .status()? + .success(), + "workspace clean failed" + ); + let run_git_apply = |check: bool| -> Result { + let mut args = vec!["-C", ws.to_str().unwrap(), "apply"]; + if check { + args.push("--check"); + } + args.push("-"); + let mut child = Command::new("git") + .args(&args) + .stdin(std::process::Stdio::piped()) + .stderr(std::process::Stdio::piped()) + .spawn()?; + child.stdin.take().unwrap().write_all(diff.as_bytes())?; + Ok(child.wait_with_output()?) + }; + let checked = run_git_apply(true)?; + ensure!( + checked.status.success(), + "diff does not apply cleanly against the pristine workspace \ + (proposals are standalone; prior proposals were reset): {}", + String::from_utf8_lossy(&checked.stderr).trim() + ); + let applied = run_git_apply(false)?; + ensure!( + applied.status.success(), + "git apply failed: {}", + String::from_utf8_lossy(&applied.stderr).trim() + ); + let patch_id = format!("{:x}", Sha256::digest(diff.as_bytes()))[..12].to_string(); + let pending = root.join("results/pending"); + std::fs::create_dir_all(&pending)?; + std::fs::write( + pending.join(format!("{patch_id}.json")), + serde_json::to_string_pretty( + &json!({"target": t, "hypothesis": hypothesis, "diff": diff}), + )?, + )?; + Ok(json!({"patch_id": patch_id})) + } + "run_verdict" => { + let patch_id = field(&req, "patch_id")?; + ensure!( + patch_id.chars().all(|c| c.is_ascii_hexdigit()) && patch_id.len() == 12, + "bad patch id" + ); + let run_id = field(&req, "run_id")?; + ensure!( + run_id + .chars() + .all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_'), + "bad run id" + ); + let class = field(&req, "playbook_class")?; + let hotspot = field(&req, "hotspot")?; + ensure_no_running_pipeline(root)?; + let pending: Value = serde_json::from_str(&std::fs::read_to_string( + root.join(format!("results/pending/{patch_id}.json")), + )?)?; + let t = pending["target"].as_str().unwrap(); + let diff = pending["diff"].as_str().unwrap(); + // Defense in depth: the pending file sat on disk since + // propose_patch vetted it — vet again before applying. + allowlist::check_diff_paths(diff)?; + let spec = diff_test::target::TargetSpec::load(root, t)?; + let diff_path = root.join(format!("results/pending/{patch_id}.json.diff")); + std::fs::write(&diff_path, diff)?; + // Re-establish the exact tree the pipeline must test: proposals + // after this one may have reset/repatched the workspace. + let ws = root.join(format!("targets/{t}/workspace")); + let git_ws = |args: &[&str]| -> Result<()> { + let out = Command::new("git").arg("-C").arg(&ws).args(args).output()?; + ensure!( + out.status.success(), + "git {} failed in target workspace: {}", + args.join(" "), + String::from_utf8_lossy(&out.stderr).trim() + ); + Ok(()) + }; + git_ws(&["checkout", "--", "."])?; + git_ws(&["clean", "-fdq"])?; + let dp = diff_path.to_str().unwrap(); + git_ws(&["apply", "--check", dp]).map_err(|e| { + anyhow!( + "pending diff {patch_id} no longer applies to the pristine \ + workspace — re-propose it: {e}" + ) + })?; + git_ws(&["apply", dp])?; + // The build + verdict pipeline runs for minutes — far past the MCP + // transport's per-call cap. Launch it detached, writing progress to + // a log; the agent observes the result via the read_verdict op, + // which reads the append-only ledger row once written. + // {out} isolates the candidate build (cargo: CARGO_TARGET_DIR; + // make: build+copy into it) — language-agnostic, mirrors the + // baseline rebuild in verdict. + let cand_dir = root + .join(format!("targets/{t}/candidate-{patch_id}")) + .to_string_lossy() + .into_owned(); + let cand_build = diff_test::target::subst_out(&spec.build.baseline, &cand_dir); + let cand_bin = diff_test::target::subst_out(&spec.build.binary, &cand_dir); + let log = format!("results/pending/{run_id}.log"); + // Target code must not reach the network: the pipeline launches + // through a wrapper (default scripts/no-net.sh, a network-namespace + // exec); HOTPATH_VERDICT_WRAPPER overrides with a single executable + // path for hosts where unshare is unavailable. + let (wrapper, isolation_note) = match std::env::var("HOTPATH_VERDICT_WRAPPER") { + Ok(w) => { + let note = w.clone(); + (w, note) + } + Err(_) => { + let w = root + .join("scripts/no-net.sh") + .to_string_lossy() + .into_owned(); + // Honesty: this note is echoed to the agent AND stamped into + // the ledger's env fingerprint. With HOTPATH_ALLOW_UNISOLATED=1 + // set, no-net.sh execs the pipeline with FULL network access on + // any host where it cannot open a netns — recording a bare + // "no-net.sh" there would claim an isolation the run never had. + // Probe the same condition the wrapper will test, and say which + // of the two actually applies. + let allow_unisolated = std::env::var("HOTPATH_ALLOW_UNISOLATED") + .map(|v| v == "1") + .unwrap_or(false); + let note = if allow_unisolated && !netns_available() { + "no-net.sh (HOTPATH_ALLOW_UNISOLATED=1 — network NOT isolated)" + } else { + "no-net.sh" + }; + (w, note.to_string()) + } + }; + // Two scripts, because failure accounting has to survive a wrapper + // that never execs its argv. + // + // INNER — runs inside the isolation wrapper. The guarded section is + // a SUBSHELL, not the left operand of an `&&`: POSIX sh ignores + // `set -e` inside any compound command that is an AND-OR operand + // other than the last, so the previous + // { set -e; ; verdict; } && rm -f lock || { ...; } + // shape left errexit inert. A failing candidate build fell straight + // through to `cargo run -p verdict`, which then benched — and wrote + // a ledger row for — a STALE or missing candidate binary. That is + // the exact class of false evidence this project exists to prevent. + // A standalone subshell scopes errexit and makes it live again, so a + // failed build aborts before verdict can measure anything. The + // subshell is the script's last command, so its status is the + // script's status. + let inner = format!( + "printf %s {rid} > {lock}\n\ + ( set -e\n{build}\ncargo run -q -p verdict -- {tgt} \ + --rebuild-baseline --candidate-bin {cbin} --run-id {rid} --playbook-class {cls} \ + --hypothesis {hyp} --hotspot {hs} --patch-file {pf} --isolation-note {iso}\n\ + )\n", + rid = shq(run_id), + lock = shq(RUNNING_LOCK), + build = cand_build, + tgt = shq(t), + cbin = shq(&cand_bin), + cls = shq(class), + hyp = shq(pending["hypothesis"].as_str().unwrap()), + hs = shq(hotspot), + pf = shq(diff_path.to_str().unwrap()), + iso = shq(&isolation_note), + ); + // OUTER — runs OUTSIDE the wrapper: invoke the wrapper, then account + // for its exit status no matter what it did. scripts/no-net.sh exits + // 97 WITHOUT exec'ing its argv when unprivileged netns are + // unavailable, and a missing or non-executable HOTPATH_VERDICT_WRAPPER + // exits 127/126 — in all three cases the inner script never runs at + // all. While this accounting lived inside the wrapper, those cases + // removed no lock and wrote no marker: read_verdict polled "running" + // forever and every later propose_patch/run_verdict was refused with + // "verdict pipeline in flight" until a human deleted the lock by hand. + // Out here it always runs, so the run is reported failed and the + // daemon stays usable. The wrapper's own stderr (no-net.sh's refusal + // text) is already captured in the run log. + let script = format!( + "{wrapper} sh -c {inner}\n\ + rc=$?\n\ + rm -f {lock}\n\ + [ \"$rc\" -eq 0 ] || echo \"HOTPATH-PIPELINE-FAILED (exit $rc)\"\n", + wrapper = shq(&wrapper), + inner = shq(&inner), + lock = shq(RUNNING_LOCK), + ); + let logf = std::fs::File::create(root.join(&log))?; + let errf = logf.try_clone()?; + // Write the lock before spawning: the detached script's own + // write lands only after exec latency, and a propose_patch in + // that window would swap the tree under the pipeline. + let lock_path = root.join(RUNNING_LOCK); + std::fs::write(&lock_path, run_id)?; + // The wrapper is invoked BY the outer script (and is shq'd into it), + // not by setsid — that is what puts the accounting outside it. + let spawned = Command::new("setsid") + .arg("sh") + .arg("-c") + .arg(&script) + .current_dir(root) + .stdout(std::process::Stdio::from(logf)) + .stderr(std::process::Stdio::from(errf)) + .spawn(); + if let Err(e) = spawned { + let _ = std::fs::remove_file(&lock_path); + return Err(anyhow!("failed to launch verdict pipeline: {e}")); + } + Ok(json!({"status": "started", "run_id": run_id, "log": log, + "isolation": isolation_note, + "note": "pipeline runs detached; poll read_verdict with this run_id"})) + } + "read_verdict" => { + let run_id = field(&req, "run_id")?; + // Long-poll: the pipeline runs for minutes and the MCP transport + // caps a tool call at 60s. Waiting ~45s server-side per poll cuts + // the agent's turn burn ~15x versus instant "running" replies. + let deadline = std::time::Instant::now() + std::time::Duration::from_secs(45); + loop { + let ledger = ledger::Ledger::open(&root.join("results/ledger.sqlite"))?; + if let Some(v) = ledger.verdict_summary(run_id)? { + return Ok(v); + } + drop(ledger); + // A pipeline that died before the verdict binary ran never + // writes a row; the launcher's failure accounting — which runs + // outside the isolation wrapper, so it survives a wrapper that + // refuses to exec — marks the log instead. + if run_id + .chars() + .all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_') + { + let log = root.join(format!("results/pending/{run_id}.log")); + if let Ok(text) = std::fs::read_to_string(&log) { + if text.contains("HOTPATH-PIPELINE-FAILED") { + let tail: String = text + .lines() + .rev() + .take(12) + .collect::>() + .into_iter() + .rev() + .collect::>() + .join("\n"); + return Ok(json!({ + "status": "failed", + "note": "the pipeline died before producing a verdict — usually a \ + candidate build failure, otherwise a crash or the isolation \ + wrapper refusing to run (scripts/no-net.sh exits 97 when it \ + cannot open a network namespace). The marker line carries the \ + exit status. No ledger row was or will be written for this \ + run_id, and the run lock has been released. Log tail follows.", + "log_tail": tail, + })); + } + } + } + if std::time::Instant::now() >= deadline { + return Ok(json!({"status": "running", + "note": "no ledger row yet; the pipeline is still building/benching — poll again"})); + } + std::thread::sleep(std::time::Duration::from_secs(3)); + } + } + other => Err(anyhow!( + "unknown op: {other} (seven ops, SPEC §3.5 + async read_verdict)" + )), + } +} diff --git a/crates/harnessd/src/main.rs b/crates/harnessd/src/main.rs index d7638f8..4589639 100644 --- a/crates/harnessd/src/main.rs +++ b/crates/harnessd/src/main.rs @@ -1,438 +1,29 @@ //! harnessd — the only door between the agent and the trust layer //! (SPEC §3.5, §10). One JSON request per stdin line, one JSON response -//! per stdout line. The agent gets exactly six operations; everything -//! else (shell, trust-layer writes, patches outside the target -//! workspace) simply has no code path here. Phase 1 caveat, recorded: -//! same-uid filesystem read-only enforcement still requires the -//! separate-user/container setup (Phase 1 infra gap). +//! per stdout line. The agent gets exactly seven operations (the six +//! synchronous ops plus the async read_verdict poll); everything else +//! (shell, trust-layer writes, patches outside the target workspace) +//! simply has no code path here. Phase 1 caveat, recorded: same-uid +//! filesystem read-only enforcement still requires the separate-user/ +//! container setup (Phase 1 infra gap). +//! +//! The daemon itself is POSIX-only (Unix sockets, setsid); the diff +//! allowlist is a portable module so its escape-regression tests run on +//! every development platform. -use anyhow::{anyhow, ensure, Result}; -use serde_json::{json, Value}; -use sha2::{Digest, Sha256}; -use std::io::{BufRead, Write}; -use std::os::unix::net::UnixListener; -use std::path::{Component, Path}; -use std::process::Command; -use std::sync::Mutex; +// On non-Unix only the allowlist tests compile against this module. +#[cfg_attr(not(unix), allow(dead_code))] +mod allowlist; +#[cfg(unix)] +mod daemon; -/// Serializes mutating ops (git apply, pending writes) across socket -/// connections; reads don't need it but the cost is nil at one agent. -static HANDLE_LOCK: Mutex<()> = Mutex::new(()); - -fn main() -> Result<()> { - let root = std::env::current_dir()?.canonicalize()?; - let args: Vec = std::env::args().collect(); - if let Some(i) = args.iter().position(|a| a == "--socket") { - let path = args - .get(i + 1) - .ok_or_else(|| anyhow!("--socket requires a path"))?; - return serve_socket(&root, Path::new(path)); - } - let stdin = std::io::stdin(); - let mut stdout = std::io::stdout().lock(); - for line in stdin.lock().lines() { - let line = line?; - if line.trim().is_empty() { - continue; - } - let resp = respond(&root, &line); - writeln!(stdout, "{resp}")?; - stdout.flush()?; - } - Ok(()) +#[cfg(unix)] +fn main() -> anyhow::Result<()> { + daemon::run() } -/// SPEC §10 socket mode: harnessd runs as the trusted uid and listens on a -/// Unix socket; the agent process runs as an unprivileged user whose only -/// write path into the repo is this daemon. Socket permissions (owner/group/ -/// mode) are set by the supervisor script that starts us. -fn serve_socket(root: &Path, sock: &Path) -> Result<()> { - if let Some(dir) = sock.parent() { - std::fs::create_dir_all(dir)?; - } - let _ = std::fs::remove_file(sock); - let listener = UnixListener::bind(sock)?; - eprintln!("harnessd listening on {}", sock.display()); - for conn in listener.incoming() { - let stream = match conn { - Ok(s) => s, - Err(e) => { - eprintln!("accept error: {e}"); - continue; - } - }; - let root = root.to_path_buf(); - std::thread::spawn(move || { - let reader = match stream.try_clone() { - Ok(s) => std::io::BufReader::new(s), - Err(_) => return, - }; - let mut writer = stream; - for line in reader.lines() { - let Ok(line) = line else { break }; - if line.trim().is_empty() { - continue; - } - let resp = respond(&root, &line); - if writeln!(writer, "{resp}").is_err() { - break; - } - } - }); - } - Ok(()) -} - -fn respond(root: &Path, line: &str) -> Value { - // read_verdict long-polls the ledger (read-only) and must not hold - // the lock that serializes mutating ops while it sleeps. - let is_read_verdict = serde_json::from_str::(line) - .ok() - .and_then(|v| { - v.get("op") - .and_then(Value::as_str) - .map(|o| o == "read_verdict") - }) - .unwrap_or(false); - let _guard = if is_read_verdict { - None - } else { - Some(HANDLE_LOCK.lock().unwrap_or_else(|p| p.into_inner())) - }; - match handle(root, line) { - Ok(v) => json!({"ok": true, "result": v}), - Err(e) => json!({"ok": false, "error": e.to_string()}), - } -} - -/// Single-quote a string for safe embedding in a generated shell script. -fn shq(s: &str) -> String { - format!("'{}'", s.replace('\'', "'\\''")) -} - -fn field<'a>(req: &'a Value, key: &str) -> Result<&'a str> { - req.get(key) - .and_then(Value::as_str) - .ok_or_else(|| anyhow!("missing string field: {key}")) -} - -/// A target name must be a plain directory name — no separators, no dots. -fn check_target(name: &str) -> Result<()> { - ensure!( - !name.is_empty() - && name - .chars() - .all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_'), - "invalid target name" - ); - Ok(()) -} - -/// Relative paths only; no parent components, no .git, no absolutes. -fn check_rel_path(p: &str) -> Result<()> { - let path = Path::new(p); - ensure!(!path.is_absolute(), "absolute paths forbidden"); - for c in path.components() { - match c { - Component::Normal(seg) => ensure!(seg != ".git", ".git access forbidden"), - Component::CurDir => {} - _ => anyhow::bail!("path escapes workspace: {p}"), - } - } - Ok(()) -} - -fn handle(root: &Path, line: &str) -> Result { - let req: Value = serde_json::from_str(line)?; - match field(&req, "op")? { - "read_profile" => { - let t = field(&req, "target")?; - check_target(t)?; - let text = std::fs::read_to_string(root.join(format!("results/{t}/hotspots.txt")))?; - Ok(json!({"hotspots_text": text})) - } - "read_ledger" => { - let t = field(&req, "target")?; - check_target(t)?; - // Read-only surface over the append-only DB. - let ledger = ledger::Ledger::open(&root.join("results/ledger.sqlite"))?; - let classes = ledger.attempted_classes(t)?; - let history = ledger.attempt_history(t)?; - Ok(json!({ - "attempted_playbook_classes": classes, - "attempts": history, - "total_attempts": ledger.count()?, - "note": "a class may be re-entered with a materially NEW hypothesis; \ - never duplicate a (hotspot, class, hypothesis) that has a verdict", - })) - } - "read_playbook" => { - let class = match req.get("class") { - Some(Value::Number(n)) => n.as_u64().ok_or_else(|| anyhow!("bad class"))? as u8, - Some(Value::String(s)) => s.parse()?, - _ => anyhow::bail!("missing class"), - }; - ensure!((1..=7).contains(&class), "playbook class must be 1-7"); - let dir = root.join("playbook"); - let entry = std::fs::read_dir(&dir)? - .filter_map(|e| e.ok()) - .find(|e| { - e.file_name() - .to_string_lossy() - .starts_with(&format!("{class:02}-")) - }) - .ok_or_else(|| anyhow!("no playbook file for class {class}"))?; - Ok(json!({"markdown": std::fs::read_to_string(entry.path())?})) - } - "read_target_source" => { - let t = field(&req, "target")?; - let p = field(&req, "path")?; - check_target(t)?; - check_rel_path(p)?; - // Serve the PRISTINE (HEAD) content, not the working tree: - // propose_patch applies each proposal against a reset - // workspace, so reads must show the same base the diff needs - // — otherwise reads after a proposal show the patched tree - // and the next diff's context lines drift (phase2-comrak-005 - // burned turns rediscovering this). - let ws = root.join(format!("targets/{t}/workspace")); - let shown = Command::new("git") - .args(["-C", ws.to_str().unwrap(), "show", &format!("HEAD:{p}")]) - .output()?; - ensure!( - shown.status.success(), - "cannot read {p} at HEAD: {}", - String::from_utf8_lossy(&shown.stderr).trim() - ); - let text = String::from_utf8_lossy(&shown.stdout).into_owned(); - // Optional line window so large source files (100KB+) can be read in - // chunks instead of overflowing one response — otherwise the agent - // is tempted to reach for a shell it must not have. - let lines: Vec<&str> = text.lines().collect(); - let total = lines.len(); - let offset = req.get("offset").and_then(Value::as_u64).unwrap_or(0) as usize; - let limit = req - .get("limit") - .and_then(Value::as_u64) - .map(|v| v as usize) - .unwrap_or(usize::MAX); - let end = offset.saturating_add(limit).min(total); - let start = offset.min(total); - let slice = lines[start..end] - .iter() - .enumerate() - .map(|(i, l)| format!("{}\t{}", start + i + 1, l)) - .collect::>() - .join("\n"); - Ok(json!({ - "content": slice, - "total_lines": total, - "returned_lines": [start + 1, end], - "truncated": end < total, - })) - } - "propose_patch" => { - let t = field(&req, "target")?; - let diff = field(&req, "diff")?; - let hypothesis = field(&req, "hypothesis")?; - check_target(t)?; - // git rejects a diff without a final newline with an opaque - // "corrupt patch" — say it plainly (phase2-comrak-004 burned - // several turns rediscovering this). - ensure!( - diff.ends_with('\n'), - "diff must end with a trailing newline (git would report 'corrupt patch' at EOF)" - ); - // Allowlist: every path named by the diff must be a safe - // relative path (the git -C below roots them in the target - // workspace; nothing outside it is reachable). `/dev/null` - // is the one absolute path git legitimately emits, for - // new/deleted files. - for l in diff.lines() { - if let Some(p) = l - .strip_prefix("--- a/") - .or_else(|| l.strip_prefix("+++ b/")) - { - check_rel_path(p.trim())?; - } - ensure!( - (!l.starts_with("--- /") && !l.starts_with("+++ /")) - || l == "--- /dev/null" - || l == "+++ /dev/null", - "absolute diff paths forbidden (only /dev/null is allowed, for new/deleted files)" - ); - } - let ws = root.join(format!("targets/{t}/workspace")); - // Every proposal stands alone: reset the workspace to HEAD - // first so probe/iteration patches cannot accumulate into a - // tested tree the ledger's patch field doesn't describe - // (phase2-comrak-004 audit finding). - ensure!( - Command::new("git") - .args(["-C", ws.to_str().unwrap(), "checkout", "--", "."]) - .status()? - .success(), - "workspace reset failed" - ); - ensure!( - Command::new("git") - .args(["-C", ws.to_str().unwrap(), "clean", "-fdq"]) - .status()? - .success(), - "workspace clean failed" - ); - let run_git_apply = |check: bool| -> Result { - let mut args = vec!["-C", ws.to_str().unwrap(), "apply"]; - if check { - args.push("--check"); - } - args.push("-"); - let mut child = Command::new("git") - .args(&args) - .stdin(std::process::Stdio::piped()) - .stderr(std::process::Stdio::piped()) - .spawn()?; - child.stdin.take().unwrap().write_all(diff.as_bytes())?; - Ok(child.wait_with_output()?) - }; - let checked = run_git_apply(true)?; - ensure!( - checked.status.success(), - "diff does not apply cleanly against the pristine workspace \ - (proposals are standalone; prior proposals were reset): {}", - String::from_utf8_lossy(&checked.stderr).trim() - ); - let applied = run_git_apply(false)?; - ensure!( - applied.status.success(), - "git apply failed: {}", - String::from_utf8_lossy(&applied.stderr).trim() - ); - let patch_id = format!("{:x}", Sha256::digest(diff.as_bytes()))[..12].to_string(); - let pending = root.join("results/pending"); - std::fs::create_dir_all(&pending)?; - std::fs::write( - pending.join(format!("{patch_id}.json")), - serde_json::to_string_pretty( - &json!({"target": t, "hypothesis": hypothesis, "diff": diff}), - )?, - )?; - Ok(json!({"patch_id": patch_id})) - } - "run_verdict" => { - let patch_id = field(&req, "patch_id")?; - ensure!( - patch_id.chars().all(|c| c.is_ascii_hexdigit()) && patch_id.len() == 12, - "bad patch id" - ); - let run_id = field(&req, "run_id")?; - ensure!( - run_id - .chars() - .all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_'), - "bad run id" - ); - let class = field(&req, "playbook_class")?; - let hotspot = field(&req, "hotspot")?; - let pending: Value = serde_json::from_str(&std::fs::read_to_string( - root.join(format!("results/pending/{patch_id}.json")), - )?)?; - let t = pending["target"].as_str().unwrap(); - let spec = diff_test::target::TargetSpec::load(root, t)?; - let diff_path = root.join(format!("results/pending/{patch_id}.json.diff")); - std::fs::write(&diff_path, pending["diff"].as_str().unwrap())?; - // The build + verdict pipeline runs for minutes — far past the MCP - // transport's per-call cap. Launch it detached, writing progress to - // a log; the agent observes the result via the read_verdict op, - // which reads the append-only ledger row once written. - // {out} isolates the candidate build (cargo: CARGO_TARGET_DIR; - // make: build+copy into it) — language-agnostic, mirrors the - // baseline rebuild in verdict. - let cand_dir = root - .join(format!("targets/{t}/candidate-{patch_id}")) - .to_string_lossy() - .into_owned(); - let cand_build = diff_test::target::subst_out(&spec.build.baseline, &cand_dir); - let cand_bin = diff_test::target::subst_out(&spec.build.binary, &cand_dir); - let log = format!("results/pending/{run_id}.log"); - // On any failure (candidate build, gates crash) the trap line - // marks the log so read_verdict can report "failed" instead of - // leaving the agent polling a verdict that will never exist. - let script = format!( - "{{ set -e\n{build}\ncargo run -q -p verdict -- {tgt} \ - --rebuild-baseline --candidate-bin {cbin} --run-id {rid} --playbook-class {cls} \ - --hypothesis {hyp} --hotspot {hs} --patch-file {pf}\n}} || echo HOTPATH-PIPELINE-FAILED\n", - build = cand_build, - tgt = shq(t), - cbin = shq(&cand_bin), - rid = shq(run_id), - cls = shq(class), - hyp = shq(pending["hypothesis"].as_str().unwrap()), - hs = shq(hotspot), - pf = shq(diff_path.to_str().unwrap()), - ); - let logf = std::fs::File::create(root.join(&log))?; - let errf = logf.try_clone()?; - Command::new("setsid") - .arg("sh") - .arg("-c") - .arg(&script) - .current_dir(root) - .stdout(std::process::Stdio::from(logf)) - .stderr(std::process::Stdio::from(errf)) - .spawn()?; - Ok(json!({"status": "started", "run_id": run_id, "log": log, - "note": "pipeline runs detached; poll read_verdict with this run_id"})) - } - "read_verdict" => { - let run_id = field(&req, "run_id")?; - // Long-poll: the pipeline runs for minutes and the MCP transport - // caps a tool call at 60s. Waiting ~45s server-side per poll cuts - // the agent's turn burn ~15x versus instant "running" replies. - let deadline = std::time::Instant::now() + std::time::Duration::from_secs(45); - loop { - let ledger = ledger::Ledger::open(&root.join("results/ledger.sqlite"))?; - if let Some(v) = ledger.verdict_summary(run_id)? { - return Ok(v); - } - drop(ledger); - // A pipeline that died before the verdict binary ran never - // writes a row; its trap line marks the log instead. - if run_id - .chars() - .all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_') - { - let log = root.join(format!("results/pending/{run_id}.log")); - if let Ok(text) = std::fs::read_to_string(&log) { - if text.contains("HOTPATH-PIPELINE-FAILED") { - let tail: String = text - .lines() - .rev() - .take(12) - .collect::>() - .into_iter() - .rev() - .collect::>() - .join("\n"); - return Ok(json!({ - "status": "failed", - "note": "the pipeline died before producing a verdict (usually a \ - candidate build failure); no ledger row was or will be \ - written for this run_id. Failure log tail follows.", - "log_tail": tail, - })); - } - } - } - if std::time::Instant::now() >= deadline { - return Ok(json!({"status": "running", - "note": "no ledger row yet; the pipeline is still building/benching — poll again"})); - } - std::thread::sleep(std::time::Duration::from_secs(3)); - } - } - other => Err(anyhow!( - "unknown op: {other} (seven ops, SPEC §3.5 + async read_verdict)" - )), - } +#[cfg(not(unix))] +fn main() { + eprintln!("harnessd requires a Unix host (Unix sockets, setsid)"); + std::process::exit(1); } diff --git a/crates/ledger/src/lib.rs b/crates/ledger/src/lib.rs index 71459b6..98ab99d 100644 --- a/crates/ledger/src/lib.rs +++ b/crates/ledger/src/lib.rs @@ -36,7 +36,10 @@ impl Verdict { } } -/// Gate outcomes for one attempt (SPEC §3.2). +/// Gate outcomes for one attempt (SPEC §3.2). The trailing fields are +/// defaulted so historical rows (written before they existed) keep +/// deserializing; they live only in the `gates` JSON column — no schema +/// change, append-only doctrine untouched. #[derive(Debug, Clone, Default, Serialize, Deserialize)] pub struct GateResults { pub upstream_tests: bool, @@ -44,6 +47,15 @@ pub struct GateResults { pub fuzz_iters: u64, pub fuzz_divergence: bool, pub sanitizers_clean: bool, + /// None = no TSan lane configured/run for this attempt. + #[serde(default)] + pub tsan_clean: Option, + /// Lexical NHR-classifier hits (crates/verdict src/risk.rs). + #[serde(default)] + pub risk_signals: Vec, + /// "byte-identical" | "fp-tolerance"; None for historical rows. + #[serde(default)] + pub equivalence_mode: Option, } /// Benchmark evidence for one attempt. All times in seconds; CIs are @@ -334,6 +346,42 @@ mod tests { assert_eq!(ledger.attempted_classes("example").unwrap(), vec![2]); } + #[test] + fn historical_gates_json_still_deserializes() { + // Exact shape of the 34 pre-existing rows' gates column: no + // tsan_clean / risk_signals / equivalence_mode keys. + let old = r#"{"upstream_tests":true,"golden_replay":true,"fuzz_iters":4332, + "fuzz_divergence":false,"sanitizers_clean":true}"#; + let g: GateResults = serde_json::from_str(old).unwrap(); + assert!(g.upstream_tests && g.golden_replay && g.sanitizers_clean); + assert_eq!(g.fuzz_iters, 4332); + assert_eq!(g.tsan_clean, None); + assert!(g.risk_signals.is_empty()); + assert_eq!(g.equivalence_mode, None); + } + + #[test] + fn extended_gates_round_trip() { + let g = GateResults { + upstream_tests: true, + golden_replay: true, + fuzz_iters: 10_000, + fuzz_divergence: false, + sanitizers_clean: true, + tsan_clean: Some(false), + risk_signals: vec!["concurrency".into()], + equivalence_mode: Some("fp-tolerance".into()), + }; + let json = serde_json::to_string(&g).unwrap(); + let back: GateResults = serde_json::from_str(&json).unwrap(); + assert_eq!(back.tsan_clean, Some(false)); + assert_eq!(back.risk_signals, vec!["concurrency".to_string()]); + assert_eq!(back.equivalence_mode.as_deref(), Some("fp-tolerance")); + // The keys the defensive Value readers depend on must survive. + let v: serde_json::Value = serde_json::from_str(&json).unwrap(); + assert_eq!(v.get("sanitizers_clean"), Some(&serde_json::json!(true))); + } + #[test] fn append_only_is_enforced() { let ledger = Ledger::open_in_memory().unwrap(); diff --git a/crates/verdict/src/main.rs b/crates/verdict/src/main.rs index 7efec72..ff43226 100644 --- a/crates/verdict/src/main.rs +++ b/crates/verdict/src/main.rs @@ -1,15 +1,25 @@ //! `just verdict`: the whole attempt pipeline in one command. //! Gates on the candidate → interleaved A/B bench vs the baseline → //! verdict by the CI-lower-bound rule → append-only ledger row. -//! Any gate failure is rejected-gate and the bench is skipped; the -//! --needs-human-review flag caps the verdict for FP/concurrency/ -//! sanitizer-flagged attempts (SPEC §8) — it never upgrades one. +//! Any gate failure is rejected-gate and the bench is skipped. A verdict +//! is only ever capped, never upgraded: the lexical risk classifier +//! (src/risk.rs) and fp-tolerance equivalence route any would-be accept +//! to needs-human-review, and the --needs-human-review flag forces the +//! same cap by hand (SPEC §8). +//! +//! This is also the only flow that can run differential fuzz, because it is +//! the only one holding both sides of the comparison: it rebuilds a pristine +//! baseline and passes it to the gates alongside the candidate. Hard rule — +//! a machine `accepted` requires that fuzz gate to have actually Passed; a +//! skipped or failed one caps the verdict at needs-human-review. + +mod risk; use anyhow::Result; use bench_runner::{config::AcceptConfig, decide, exec, fingerprint::EnvFingerprint, stats}; use clap::Parser; use diff_test::{ - all_passed, orchestrate::run_core_gates, target::TargetSpec, GateLayer, GateOutcome, + orchestrate::run_core_gates, target::TargetSpec, GateInputs, GateLayer, GateOutcome, }; use ledger::{Attempt, BenchEvidence, GateResults, Ledger, Verdict}; use std::path::PathBuf; @@ -48,6 +58,10 @@ struct Cli { /// Force the needs-human-review verdict tier (FP/concurrency/UB-adjacent). #[arg(long)] needs_human_review: bool, + /// How this run was isolated, recorded verbatim in the env + /// fingerprint (e.g. "no-net.sh"); absent = "unwrapped-host". + #[arg(long)] + isolation_note: Option, #[arg(long, default_value = "results/ledger.sqlite")] db: PathBuf, #[arg(long, default_value = "config/accept.toml")] @@ -107,6 +121,78 @@ fn rebuild_pristine_baseline( Ok(baseline) } +/// A run is flagged on nonzero exit OR any of these stderr markers. +/// halt_on_error makes sanitizer hits exit nonzero; the markers catch +/// builds whose runtime still exits 0 (and never a bare "ERROR:" from +/// the target's own logging, the old false-flag risk). +const SANITIZER_MARKERS: &[&str] = &[ + "ERROR: AddressSanitizer", + "ERROR: LeakSanitizer", + "ERROR: ThreadSanitizer", + "WARNING: ThreadSanitizer", + "runtime error:", +]; + +/// One instrumented build+run lane: templates, isolation subdir, and the +/// halt-on-error runtime options for its sanitizer family. +struct SanLane<'a> { + label: &'a str, + out_subdir: &'a str, + build_tpl: &'a str, + bin_tpl: &'a str, + envs: &'a [(&'a str, &'a str)], +} + +/// Build one instrumented lane and run the pinned bench workload under +/// it. Returns Some(clean), or None when the instrumented binary cannot +/// be built — an infrastructure failure, not a code defect: the caller +/// caps the accept at needs-human-review instead of crashing a verdict +/// that already carries a measured bench. +fn run_san_lane( + root: &std::path::Path, + target: &str, + spec: &TargetSpec, + lane: &SanLane, +) -> Result> { + let label = lane.label; + let out_dir = root + .join(format!("targets/{target}/{}", lane.out_subdir)) + .to_string_lossy() + .into_owned(); + println!("{label}: instrumented build of the patched tree…"); + let status = std::process::Command::new("sh") + .arg("-c") + .arg(diff_test::target::subst_out(lane.build_tpl, &out_dir)) + .current_dir(root) + .status()?; + if !status.success() { + println!("{label}: BUILD FAILED — cannot verify; capping accept at needs-human-review"); + return Ok(None); + } + let bin = diff_test::target::subst_out(lane.bin_tpl, &out_dir); + let mut run = std::process::Command::new("sh"); + run.arg("-c") + .arg(spec.bench.command.replace("{binary}", &bin)) + .current_dir(root); + for (k, v) in lane.envs { + run.env(k, v); + } + let out = run.output()?; + let stderr = String::from_utf8_lossy(&out.stderr); + let clean = out.status.success() && !SANITIZER_MARKERS.iter().any(|m| stderr.contains(m)); + if clean { + println!("{label}: clean"); + } else { + let lines: Vec<&str> = stderr.lines().filter(|l| !l.is_empty()).collect(); + let start = lines.len().saturating_sub(8); + println!("{label}: FLAGGED (rc={:?})", out.status.code()); + for l in &lines[start..] { + println!(" {l}"); + } + } + Ok(Some(clean)) +} + /// SPEC §8: an accept is final only if ASan+LSan are clean on the patched /// tree over the pinned workload. Stop-the-line fix after phase2-comrak-010: /// the pipeline auto-accepted an LSan-flagged teardown patch (sanitizers @@ -126,49 +212,67 @@ fn sanitizer_check( println!("sanitizers: target declares no sanitizer build — cannot verify an accept"); return Ok(None); }; - let out_dir = root - .join(format!("targets/{target}/asan")) - .to_string_lossy() - .into_owned(); - println!("sanitizers: ASan build of the patched tree…"); - let status = std::process::Command::new("sh") - .arg("-c") - .arg(diff_test::target::subst_out(build_tpl, &out_dir)) - .current_dir(root) - .status()?; - if !status.success() { - // Infrastructure failure (missing runtime, toolchain), not a code - // defect: don't crash the verdict and lose a measured bench — - // treat "cannot build the sanitizer binary" like "no sanitizer - // declared" so the accept caps at needs-human-review (None), with - // the failure logged for the operator to fix and re-run. - println!("sanitizers: BUILD FAILED — cannot verify; capping accept at needs-human-review"); + run_san_lane( + root, + target, + spec, + &SanLane { + label: "sanitizers", + out_subdir: "asan", + build_tpl, + bin_tpl, + envs: &[ + ("ASAN_OPTIONS", "halt_on_error=1:abort_on_error=1"), + ("UBSAN_OPTIONS", "halt_on_error=1:print_stacktrace=1"), + ], + }, + ) +} + +/// TSan lane (SPEC §8), run on the accept path when the target declares +/// a TSan build — call only when spec.build.tsan is Some. Returns None +/// when the lane cannot run (missing tsan_binary, build failure); the +/// caller caps the accept, mirroring the ASan posture. +fn tsan_check(root: &std::path::Path, target: &str, spec: &TargetSpec) -> Result> { + let (Some(build_tpl), Some(bin_tpl)) = (&spec.build.tsan, &spec.build.tsan_binary) else { + println!("tsan: build.tsan declared without build.tsan_binary — cannot verify"); return Ok(None); + }; + run_san_lane( + root, + target, + spec, + &SanLane { + label: "tsan", + out_subdir: "tsan", + build_tpl, + bin_tpl, + envs: &[("TSAN_OPTIONS", "halt_on_error=1")], + }, + ) +} + +/// One word for the state of the DifferentialFuzz gate, for the cap message. +fn fuzz_state(outcome: Option<&GateOutcome>) -> &'static str { + match outcome { + Some(GateOutcome::Passed) => "passed", + Some(GateOutcome::Failed { .. }) => "failed", + Some(GateOutcome::Skipped { .. }) => "skipped", + None => "not run", } - let asan_bin = diff_test::target::subst_out(bin_tpl, &out_dir); - let cmd = spec.bench.command.replace("{binary}", &asan_bin); - let out = std::process::Command::new("sh") - .arg("-c") - .arg(&cmd) - // detect_leaks is a no-op for the C/C++ UBSan combo but harmless. - .env("ASAN_OPTIONS", "detect_leaks=1") - .env("UBSAN_OPTIONS", "halt_on_error=1:print_stacktrace=1") - .current_dir(root) - .output()?; - let stderr = String::from_utf8_lossy(&out.stderr); - let clean = - out.status.success() && !stderr.contains("ERROR:") && !stderr.contains("runtime error:"); - if clean { - println!("sanitizers: clean"); +} + +/// Median of per-run max-RSS samples (KiB); an even count averages the +/// two middle values. +fn median_u64(xs: &[u64]) -> u64 { + let mut v = xs.to_vec(); + v.sort_unstable(); + let n = v.len(); + if n % 2 == 1 { + v[n / 2] } else { - let lines: Vec<&str> = stderr.lines().filter(|l| !l.is_empty()).collect(); - let start = lines.len().saturating_sub(8); - println!("sanitizers: FLAGGED (rc={:?})", out.status.code()); - for l in &lines[start..] { - println!(" {l}"); - } + (v[n / 2 - 1] + v[n / 2]) / 2 } - Ok(Some(clean)) } fn now_utc() -> String { @@ -197,23 +301,58 @@ fn main() -> Result<()> { })? }; - // Gates run on the candidate binary. - let gates = run_core_gates(&root, &spec, &cli.candidate_bin)?; - for (layer, outcome) in &gates { + let patch = match &cli.patch_file { + Some(p) => std::fs::read_to_string(p)?, + None => "(no source patch)".into(), + }; + + // Gates run on the candidate binary — except differential fuzz, which + // differs it against the pristine baseline built just above. This is the + // only flow that has both sides, and therefore the only flow that can + // run the fuzz gate at all (hence the accept rule enforced below). + let report = run_core_gates( + &root, + &spec, + &GateInputs { + candidate_binary: &cli.candidate_bin, + baseline_binary: Some(baseline_bin.as_str()), + fuzz_iters: cfg.fuzz_iters, + }, + )?; + for (layer, outcome) in &report.gates { match outcome { GateOutcome::Passed => println!("{layer:?}: PASS"), GateOutcome::Failed { detail } => println!("{layer:?}: FAIL — {detail}"), GateOutcome::Skipped { reason } => println!("{layer:?}: skipped — {reason}"), } } + let passed = |layer| matches!(report.outcome(layer), Some(GateOutcome::Passed)); let mut gate_results = GateResults { - upstream_tests: matches!(gates[0].1, GateOutcome::Passed), - golden_replay: matches!(gates[1].1, GateOutcome::Passed), - fuzz_iters: 0, - fuzz_divergence: false, + upstream_tests: passed(GateLayer::UpstreamTests), + golden_replay: passed(GateLayer::GoldenReplay), + fuzz_iters: report.fuzz_iters, + fuzz_divergence: report.fuzz_divergence, sanitizers_clean: false, + tsan_clean: None, + risk_signals: risk::classify(&patch, report.equivalence_mode == "fp-tolerance"), + equivalence_mode: Some(report.equivalence_mode.clone()), }; - debug_assert!(matches!(gates[0].0, GateLayer::UpstreamTests)); + if !gate_results.risk_signals.is_empty() { + println!( + "risk: lexical signals [{}] — a would-be accept routes to needs-human-review (SPEC §8)", + gate_results.risk_signals.join(", ") + ); + } + // Announced up front, like the risk signals, so the rule is in the + // transcript even when another cap fires first below. + let fuzz_passed = passed(GateLayer::DifferentialFuzz); + if !fuzz_passed { + println!( + "fuzz: differential-fuzz gate {} — a would-be accept routes to needs-human-review; \ + no machine accept without a real differential-fuzz run (SPEC §8)", + fuzz_state(report.outcome(GateLayer::DifferentialFuzz)) + ); + } let pin = &cfg.pin_prefix; let wrap = |bin: &str| { @@ -225,7 +364,7 @@ fn main() -> Result<()> { } }; - let (verdict, bench) = if !all_passed(&gates) { + let (verdict, bench) = if !report.all_passed() { println!("verdict: rejected-gate (bench skipped)"); (Verdict::RejectedGate, None) } else { @@ -285,8 +424,62 @@ fn main() -> Result<()> { v = Verdict::NeedsHumanReview; } } + if spec.build.tsan.is_some() { + match tsan_check(&root, &cli.target, &spec)? { + Some(clean) => { + gate_results.tsan_clean = Some(clean); + if !clean { + println!( + "verdict: TSan flag caps the accept at needs-human-review (SPEC §8)" + ); + v = Verdict::NeedsHumanReview; + } + } + None => { + println!("verdict: TSan lane declared but not verifiable; capping accept at needs-human-review (SPEC §8)"); + v = Verdict::NeedsHumanReview; + } + } + } + if v == Verdict::Accepted && !gate_results.risk_signals.is_empty() { + println!( + "verdict: risk signals [{}] route the accept to needs-human-review (SPEC §8)", + gate_results.risk_signals.join(", ") + ); + v = Verdict::NeedsHumanReview; + } + // Hard rule (audit remediation): differential fuzz is the gate + // that catches behavioral divergence the fixed corpus misses, so + // an accept minted without it is an unverified claim. The audit + // found accepted rows carrying fuzz_iters=0 — fuzz had run + // out-of-band, or not at all. A skipped or failed fuzz gate can + // no longer produce a machine accept, only needs-human-review. + if v == Verdict::Accepted && !fuzz_passed { + println!( + "verdict: differential fuzz {} — no accept without a real differential-fuzz \ + run against the pristine baseline; capping at needs-human-review. The target \ + must declare [gates].fuzz in targets/{}/target.toml, and the run must report \ + FUZZ-RESULT with zero divergences (SPEC §8)", + fuzz_state(report.outcome(GateLayer::DifferentialFuzz)), + cli.target + ); + v = Verdict::NeedsHumanReview; + } } let fp = EnvFingerprint::collect(pin, "system-default"); + let mut env_fingerprint = serde_json::json!({ + "fingerprint": fp, + "workload": spec.bench.workload, + "target_commit": spec.source.commit, + "isolation": cli.isolation_note.as_deref().unwrap_or("unwrapped-host"), + "gates_detail": "diff-fuzz gated in-pipeline against the pristine baseline (an accept requires it to have passed); ASan+UBSan enforced on the accept path, TSan when configured", + }); + if !samples.baseline_max_rss_kib.is_empty() && !samples.candidate_max_rss_kib.is_empty() { + env_fingerprint["max_rss_kib"] = serde_json::json!({ + "baseline_median": median_u64(&samples.baseline_max_rss_kib), + "candidate_median": median_u64(&samples.candidate_max_rss_kib), + }); + } ( v, Some(BenchEvidence { @@ -296,20 +489,11 @@ fn main() -> Result<()> { candidate_ci: (clo, chi), speedup_median: ratio.median, speedup_ci: (ratio.lo, ratio.hi), - env_fingerprint: serde_json::json!({ - "fingerprint": fp, - "workload": spec.bench.workload, - "target_commit": spec.source.commit, - "gates_detail": "diff-fuzz per-attempt manual; ASan+LSan enforced on the accept path", - }), + env_fingerprint, }), ) }; - let patch = match &cli.patch_file { - Some(p) => std::fs::read_to_string(p)?, - None => "(no source patch)".into(), - }; // Derive the phase from the run_id namespace (phaseN-*) so the column // matches the run_id prefix; the historical hardcoded `1` left every // verdict-written row tagged phase 1 regardless of namespace (found in @@ -344,3 +528,36 @@ fn main() -> Result<()> { ); Ok(()) } + +#[cfg(test)] +mod tests { + use super::{fuzz_state, median_u64}; + use diff_test::GateOutcome; + + #[test] + fn median_u64_odd_and_even() { + assert_eq!(median_u64(&[5]), 5); + assert_eq!(median_u64(&[9, 1, 5]), 5); + assert_eq!(median_u64(&[4, 2, 8, 6]), 5); + } + + /// Only `Passed` clears the accept bar; every other state (including a + /// gate that never ran) is a cap to needs-human-review. + #[test] + fn fuzz_state_names_every_outcome() { + assert_eq!(fuzz_state(Some(&GateOutcome::Passed)), "passed"); + assert_eq!( + fuzz_state(Some(&GateOutcome::Failed { + detail: "divergences=3".into() + })), + "failed" + ); + assert_eq!( + fuzz_state(Some(&GateOutcome::Skipped { + reason: "no baseline".into() + })), + "skipped" + ); + assert_eq!(fuzz_state(None), "not run"); + } +} diff --git a/crates/verdict/src/risk.rs b/crates/verdict/src/risk.rs new file mode 100644 index 0000000..2a91a5e --- /dev/null +++ b/crates/verdict/src/risk.rs @@ -0,0 +1,168 @@ +//! Lexical NHR risk classifier (SPEC §8). Scans only the changed lines +//! of a unified diff for concurrency / unsafe / floating-point tokens +//! and routes any would-be accept that hits one to needs-human-review. +//! Deliberately lexical and conservative: over-triggering costs a human +//! a look, under-triggering ships an unreviewed risk — so plain token +//! presence decides, no semantic analysis. + +const CONCURRENCY: &[&str] = &[ + "pthread_", + "std::sync", + "std::thread", + "Atomic", + "atomic_", + "__atomic_", + "__sync_", + "Mutex", + "RwLock", + "Condvar", + "mpsc", + "rayon::", + "memory_order", + "volatile", +]; + +const UNSAFE: &[&str] = &[ + "unsafe ", + "transmute", + "from_raw", + "MaybeUninit", + "UnsafeCell", +]; + +/// The spaced tokens carry their own trailing boundary; `f32`/`f64` get +/// hand boundary checks below. +const FP_SPACED: &[&str] = &["float ", "double "]; +const FP_TYPES: &[&str] = &["f32", "f64"]; + +fn is_ident_letter(b: u8) -> bool { + b.is_ascii_alphabetic() || b == b'_' +} + +/// True when `line` contains `token` not glued to an identifier letter on +/// either side. Digits are allowed neighbors so literal suffixes like +/// `1.0f32` still trigger; letter neighbors (identifiers, most hex +/// strings, e.g. `buf32`, `deadf64b`) do not. +fn has_fp_token(line: &str, token: &str) -> bool { + let bytes = line.as_bytes(); + let mut start = 0; + while let Some(pos) = line[start..].find(token) { + let i = start + pos; + let before_ok = i == 0 || !is_ident_letter(bytes[i - 1]); + let after_ok = i + token.len() >= bytes.len() || !is_ident_letter(bytes[i + token.len()]); + if before_ok && after_ok { + return true; + } + start = i + 1; + } + false +} + +fn is_changed_line(line: &str) -> bool { + (line.starts_with('+') && !line.starts_with("+++")) + || (line.starts_with('-') && !line.starts_with("---")) +} + +/// Classify a unified diff into risk-signal names. `fp_tolerance` is true +/// when the target's equivalence mode is fp-tolerance, itself a §8 signal. +/// Returned names are unique; empty means no signal. +pub fn classify(diff: &str, fp_tolerance: bool) -> Vec { + let (mut concurrency, mut unsafe_sig, mut fp) = (false, false, false); + for line in diff.lines().filter(|l| is_changed_line(l)) { + concurrency = concurrency || CONCURRENCY.iter().any(|t| line.contains(t)); + unsafe_sig = unsafe_sig || UNSAFE.iter().any(|t| line.contains(t)); + fp = fp + || FP_SPACED.iter().any(|t| line.contains(t)) + || FP_TYPES.iter().any(|t| has_fp_token(line, t)); + } + let mut signals = Vec::new(); + if concurrency { + signals.push("concurrency".to_string()); + } + if unsafe_sig { + signals.push("unsafe".to_string()); + } + if fp { + signals.push("floating-point".to_string()); + } + if fp_tolerance { + signals.push("fp-tolerance-equivalence".to_string()); + } + signals +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn concurrency_tokens_trigger() { + let diff = "--- a/src/lib.rs\n+++ b/src/lib.rs\n+ let m = Mutex::new(0);\n"; + assert_eq!(classify(diff, false), vec!["concurrency"]); + let diff = "- pthread_create(&t, NULL, worker, NULL);\n"; + assert_eq!(classify(diff, false), vec!["concurrency"]); + } + + #[test] + fn unsafe_tokens_trigger() { + let diff = "+ unsafe { ptr.read() }\n"; + assert_eq!(classify(diff, false), vec!["unsafe"]); + let diff = "+ let v: Vec = Vec::from_raw_parts(p, n, n);\n"; + assert_eq!(classify(diff, false), vec!["unsafe"]); + } + + #[test] + fn floating_point_tokens_trigger_with_boundaries() { + assert_eq!( + classify("+ let x: f64 = 1.0;\n", false), + vec!["floating-point"] + ); + // Literal suffixes have a digit neighbor and must still trigger. + assert_eq!( + classify("+ sum += 1.0f32;\n", false), + vec!["floating-point"] + ); + assert_eq!( + classify("+ double acc = 0.0;\n", false), + vec!["floating-point"] + ); + // Glued to identifier letters: not an FP token. + assert!(classify("+ let buf32 = [0u8; 4];\n", false).is_empty()); + assert!(classify("+ hash = \"deadf64beef\";\n", false).is_empty()); + } + + #[test] + fn fp_tolerance_flag_is_its_own_signal() { + assert_eq!(classify("", true), vec!["fp-tolerance-equivalence"]); + assert_eq!( + classify("+ let x: f32 = 0.5;\n", true), + vec!["floating-point", "fp-tolerance-equivalence"] + ); + } + + #[test] + fn clean_diff_yields_no_signals() { + let diff = "--- a/src/lib.rs\n+++ b/src/lib.rs\n@@ -1,3 +1,3 @@\n\ + - let n = items.len();\n+ let n = count;\n let done = true;\n"; + assert!(classify(diff, false).is_empty()); + } + + #[test] + fn only_changed_lines_are_scanned() { + // Tokens on context lines and in file headers must not trigger. + let diff = "--- a/src/mutex_wrapper.rs\n+++ b/src/mutex_wrapper.rs\n\ + @@ -10,4 +10,4 @@\n let guard = lock.mutex();\n\ + + let n = count;\n"; + assert!(classify(diff, false).is_empty()); + } + + #[test] + fn signals_are_deduped() { + let diff = "+ let a = Mutex::new(0);\n+ let b = RwLock::new(0);\n\ + + unsafe { transmute::(x) };\n"; + assert_eq!( + classify(diff, false), + vec!["concurrency", "unsafe", "floating-point"] + ); + } +} diff --git a/docs/src/equivalence.md b/docs/src/equivalence.md index 9ab7ef6..5f12033 100644 --- a/docs/src/equivalence.md +++ b/docs/src/equivalence.md @@ -7,6 +7,22 @@ candidate must prove it does the same thing the baseline does. These are ## The layers +0. **Pin checks.** Before any gate runs, the corpus is verified against its + committed `MANIFEST.sha256` and the harness **refuses to run** on a + mismatch — nothing is measured on tampered inputs. This ships and is + enforced for every target. + + The *test suite* has the same machinery but is **not yet active**: + `diff-test` verifies `corpora//TESTSUITE.sha256` when the file is + present (mismatch = hard refusal) and prints an explicit "suite unpinned" + warning when it is absent. **No target ships a suite pin today**, so this + check currently constrains nothing. It matters because upstream suites + are vendored under `targets//workspace/` — an agent-writable path — so + "the agent cannot edit the tests" rests on a pin that still has to be + generated (`scripts/pin-testsuite.sh`, a deliberate human action, run + after a fetch). Until then, that specific guarantee is a mechanism, not + a fact. + 1. **Upstream test suite.** The target's own tests, run against the patched code. For cJSON that is the vendored Unity/CTest suite (19 tests: `parse_number`, `parse_string`, `print_*`, `misc_tests`, …); for the Rust @@ -20,15 +36,49 @@ candidate must prove it does the same thing the baseline does. These are mismatch. 3. **Differential fuzzing.** Pristine vs candidate on thousands of mutated - inputs, comparing output. Every accepted win cleared **10,000 mutated - inputs with 0 divergences** — the tokei wins after canonicalizing benign - parallel-walker output ordering (a real finding: differential gates must - compare *semantics*, not bytes, where the target is legitimately - nondeterministic in presentation), the cJSON win on quote/number/escape - edge cases. + inputs, comparing output. -4. **Sanitizers.** ASan/LSan for Rust, ASan+UBSan for C. Run on **every - would-be accept** — this is machine-enforced, not manual (see below). + *Where it runs.* Fuzz is the one gate that is **not** a property of the + candidate alone: it differs the candidate against a **pristine baseline**, + so it needs both binaries. That means it runs **only on the accept path** + (`just verdict`, the one flow that rebuilds a pristine baseline). The + standalone `just gates` flow has no baseline, so the gate reports + **Skipped, with that reason recorded** — it is never faked by comparing + the candidate against itself. `just gates` is a fast equivalence check, + **not** a fuzz run. The gate is graded strictly: `diff-test` runs the + target's declared `[gates].fuzz` command and parses a + `FUZZ-RESULT iters= divergences=` line; a run that never prints one + **fails**, rather than passing by silence. + + *The hard rule.* A machine `accepted` verdict is now **impossible** + without this gate having actually passed. A skipped or failed fuzz gate + caps the verdict at `needs-human-review` — because an accept minted + without differential fuzz is an unverified claim. + + *Honest history.* Through Phase 5 this ran as an **out-of-band + per-attempt script** (`scripts/diff-fuzz-.py`), not inside the + pipeline — so **8 of the 10 accepted ledger rows record `fuzz_iters=0`** + even where a 10,000-input audit run is documented in the case study. The + two exceptions are the Phase 0 tokei rows, which record 4,332. Those + out-of-band runs produced real findings — the tokei wins passed only + after canonicalizing benign parallel-walker output ordering (differential + gates must compare *semantics*, not bytes, where the target is + legitimately nondeterministic in presentation), and the cJSON win cleared + quote/number/escape edge cases with 0 divergences. + +4. **Sanitizers.** ASan+UBSan (ASan/LSan on the Rust targets) is built and + run against the pinned workload on **every would-be accept** — the + sanitizer gate is machine-enforced, not manual (see below). A flagged + run, *or* a target that declares no sanitizer build at all, caps the + accept at `needs-human-review`. + + A **TSan lane** exists on the same accept path, but it only runs when a + target declares `[build].tsan`. **No target declares it today, so there + is currently no TSan coverage** — the lane is wired and dormant, and a + patch touching threads needs that opt-in added before it can be checked. + (MIRI, named in the spec, was never implemented.) The concurrency safety + net that *is* live is the review tier below, which routes any + concurrency-token patch to a human. ## The tier rule (SPEC §8) @@ -36,7 +86,11 @@ Some changes can pass every gate and still be unsafe to auto-ship. A change that touches floating-point ordering, concurrency primitives, `unsafe`, or carries a sanitizer flag is **never** auto-accepted — its verdict is capped at `needs-human-review`. The tier is decided by the *nature* of the change, -never by the size of the number. The mimalloc allocator swap +never by the size of the number. The routing is machine-enforced: `verdict` +runs a conservative **lexical risk classifier** over the patch's changed +lines (concurrency / unsafe / floating-point token lists — deliberately +over-triggering, not a semantic analysis) and any signal, or the use of +fp-tolerance equivalence mode at all, forces a would-be accept to review. The mimalloc allocator swap ([case study](./case-studies/build-flags.md)) is the canonical example: a real +4.6% win, correctly held for a human ruling because a global allocator change's safety is not something one benchmark can clear. diff --git a/docs/src/introduction.md b/docs/src/introduction.md index 3126526..dd13b17 100644 --- a/docs/src/introduction.md +++ b/docs/src/introduction.md @@ -29,7 +29,7 @@ and every one survived the gates above. These are not projections. | cJSON HTTP service — p50 latency under replayed load | +6.2%, CI [+5.8%, +7.2%] | | comrak (Rust) — mimalloc allocator swap (class 2) | +4.6%, CI [+3.3%, +5.8%], human-ruled | -Across **five phases and 33 ledger rows**, with **zero shipped false +Across **five phases and 34 ledger rows**, with **zero shipped false accepts** — including two occasions where the pipeline itself over-accepted and the audit caught it before anything shipped, each becoming a permanent new gate. diff --git a/docs/src/isolation.md b/docs/src/isolation.md index 9fee1f4..b2e578b 100644 --- a/docs/src/isolation.md +++ b/docs/src/isolation.md @@ -34,6 +34,35 @@ The SDK's tool allow/deny list is defense-in-depth, not the boundary — path into the repo is the harnessd Unix socket; the trust layer is owned by another uid. This is the production shape (`docs/infra.md`). +## Network isolation on the verdict path + +Target code should not be able to phone home during a measurement. What +is actually enforced, stated honestly: + +- **The agent path is wrapped.** When `harnessd` launches the verdict + pipeline, it execs through `scripts/no-net.sh`, which runs it inside an + unprivileged **network namespace** (`unshare --net --map-current-user`). + `HOTPATH_VERDICT_WRAPPER` overrides the wrapper. +- **It fails closed.** Where namespaces are unavailable the wrapper **exits + 97 without running the pipeline** — it does not degrade silently. The one + bypass, `HOTPATH_ALLOW_UNISOLATED=1`, runs the pipeline **with full + network access**; it warns loudly on stderr *and* the resulting ledger row + records the isolation note `no-net.sh (HOTPATH_ALLOW_UNISOLATED=1 — + network NOT isolated)`. A run cannot claim an isolation it did not have. +- **In CI**, the bench workload runs under `docker run --network=none`. + +Two limits, stated plainly rather than buried: + +- **Only the harnessd (agent) path is wrapped.** A human who invokes + `just verdict` directly runs it **unwrapped on the host**; those rows are + stamped `isolation: "unwrapped-host"` in the ledger. The wrapper is a + property of the agent pipeline, not of the `verdict` binary. +- **Full-container isolation is still an open gap.** The namespace isolates + the *network* — not the filesystem, not the syscall surface. The + fully containerized, seccomp-restricted bench container of SPEC §10 is + **not built**. The agent-side OS boundary above is a separate mechanism + and *is* shipped. + ## The two times the pipeline was wrong Anti-reward-hacking is not "it never fails" — it is "when it fails, the diff --git a/docs/src/measurement.md b/docs/src/measurement.md index 52a9efa..f2f3282 100644 --- a/docs/src/measurement.md +++ b/docs/src/measurement.md @@ -27,6 +27,22 @@ reproduces). A change is accepted only if the **CI lower bound** clears the threshold in `config/accept.toml` (default 2%) — not the median. The number you commit to is the one that survives the interval. +## Beyond wall time: max RSS and PMU counters + +Wall time is the **only** metric a verdict is ever decided on. Two +supporting measurements exist alongside it: + +- **Max RSS** is captured for every timed run on Unix (`wait4`'s + `ru_maxrss`) and recorded in the run's environment fingerprint as + baseline/candidate medians — so a speedup that quietly pays for itself + in memory is visible next to the number it bought. +- **PMU counters** (cycles, instructions, cache misses, branch misses) + are opt-in diagnostics: `bench-runner compare --perf-stat` wraps each + timed run in `perf stat` and prints per-side medians with the workload + caveat attached. PMU access is unavailable in most VMs, and the + counters never participate in the accept decision — they explain a + wall-time delta, they do not replace it. + ## The hardware has to earn trust first Before any measurement session, the harness runs two self-tests, recorded diff --git a/docs/src/playbook.md b/docs/src/playbook.md index 28ef5b9..7230aa9 100644 --- a/docs/src/playbook.md +++ b/docs/src/playbook.md @@ -8,14 +8,16 @@ the ledger stops you from re-grinding a class that already has a verdict. |---|---|---|---| | 1 | Build config | LTO, codegen-units, target-cpu | tokei ✅ +10.4% | | 2 | Allocator | Global allocator swap (mimalloc) | comrak ✅ +4.6% (human-ruled) | -| 3 | Allocation churn | Remove per-item malloc/free | tokei ✅, cJSON ✅ (+ one overturned) | +| 3 | Allocation churn | Remove per-item malloc/free | tokei ✅; cJSON rejected-bench (+ one overturned) | | 4 | Data layout | Make the cache hierarchy fit the access pattern | comrak (attempted) | | 5 | Algorithmic | Eliminate redundant work | tokei ×3 ✅, cJSON ✅ | | 6 | SIMD | Autovectorization enablement | comrak, tokei (attempted) | | 7 | Concurrency | Parallelism — always `needs-human-review` | — | Six classes were exercised across the case studies, with verified wins on -classes 1, 2, 3, and 5 across **both a Rust and a C target**. +classes 1, 2, 3, and 5 — spanning **both a Rust and a C target** (the C +win is the class-5 cJSON rewrite; cJSON's class-3 attempt was an honest +`rejected-bench`, its CI lower bound short of the 2% bar). ## Why the ordering is load-bearing diff --git a/docs/src/reproduce.md b/docs/src/reproduce.md index 3306f26..b470b81 100644 --- a/docs/src/reproduce.md +++ b/docs/src/reproduce.md @@ -5,6 +5,10 @@ through the same commands. ## Prerequisites +- **A Linux/POSIX host.** The pipeline is Unix-only at runtime: `harnessd` + (Unix sockets, `setsid`), service mode, the isolation wrapper, and every + sh-based gate assume it. Windows compiles the workspace and runs the + portable unit tests, but **cannot run the pipeline**. - Rust (stable) + `just`; `clang`/`cmake` for the C target; `valgrind` for callgrind profiling. Nightly Rust for the sanitizer gate. - Targets are vendored fresh (their `workspace/` is gitignored); each @@ -21,6 +25,14 @@ just verdict ... # gates + bench vs pristine baseline + ledger row just report # ROI report from a ledger row ``` +> **`just gates` does not fuzz.** Differential fuzz differs the candidate +> against a *pristine baseline*, and only `just verdict` builds one — so on +> the `gates` flow the fuzz layer reports **Skipped**, with the reason +> recorded, rather than comparing the candidate against itself. Use `gates` +> as a fast equivalence check; the fuzz gate (and the sanitizer lanes) run +> on the accept path, where an accept is impossible without a passed fuzz +> gate. + Service mode (Phase 4): ``` @@ -42,7 +54,7 @@ Every accepted win can be re-audited independently: ``` python3 scripts/audit-attempt.py # mechanical re-check -python3 scripts/diff-fuzz-.py 10000 # 10k fuzz +python3 scripts/diff-fuzz-.py 10000 ``` The ledger itself is the record: `results/ledger.sqlite`, append-only, diff --git a/docs/src/research-forks.md b/docs/src/research-forks.md index a06b5b0..5dc4d22 100644 --- a/docs/src/research-forks.md +++ b/docs/src/research-forks.md @@ -52,6 +52,19 @@ tolerance is defensible for the workload, exactly as with the mimalloc allocator swap. The machine measures and gates; the human ratifies the tolerance. +> **Scope, stated honestly.** The FP-tolerance *mechanism* is in the gate +> pipeline — `diff-test` honors an `equivalence.toml` policy (it needs a +> committed `[corpus].golden_reference`, and fails closed without one), and +> `verdict` auto-routes any fp-tolerance run to `needs-human-review`. But +> **`matmul` is not wired into that pipeline**: it has an `equivalence.toml` +> and **no `target.toml`**, which is the file `diff-test`/`verdict` load a +> target from. So the kernel lane is demonstrated **script-driven** +> (`scripts/kernel-lane-demo.sh`, using the `fp-compare` binary and +> `bench-runner compare` directly), and the `phase5-matmul-opt` ledger row +> was recorded outside the automated verdict path. Promoting `matmul` to a +> real pipeline target is open work — the policy is closed; that target's +> wiring is not. + **This is the GPU lane in miniature.** The environment has no GPU, so the GPU fork is shown on the CPU as what it actually is — the same trust machinery with a different timer. A Triton or CUDA kernel versus a reference diff --git a/docs/src/roadmap.md b/docs/src/roadmap.md index 9ee2048..d4e9076 100644 --- a/docs/src/roadmap.md +++ b/docs/src/roadmap.md @@ -44,5 +44,23 @@ Full write-up: [Research forks](./research-forks.md). The end goal is not vendored OSS case studies — it is a **real production system**. The same gates that matter for a JSON parser matter far more for a live CMS: an equivalence failure there is a customer-facing bug, and the -no-network sandbox and append-only audit trail are exactly what make it -safe to point an optimization agent at code that a business depends on. +network-isolated verdict path and append-only audit trail are what would +make it safe to point an optimization agent at code that a business depends +on. + +## Known gaps (what would have to close first) + +Honesty about the remaining edges, since the product is trust: + +- **Full-container isolation.** The verdict pipeline is network-isolated on + the agent path (`scripts/no-net.sh`); the seccomp-restricted bench + container of SPEC §10 is **not built**, and a human running `just verdict` + directly is unwrapped. +- **Test-suite pins.** The verify-or-refuse machinery exists, but **no + target ships a `TESTSUITE.sha256` yet** — so "the agent cannot edit the + upstream tests" is currently a mechanism awaiting its pins. +- **TSan.** The lane runs on the accept path only when a target declares + `[build].tsan`; **none does today**, so there is no TSan coverage. Class-7 + (concurrency) work depends on that opt-in landing. +- **Kernel lane wiring.** `matmul` demonstrates FP-tolerance equivalence + script-driven; it is not yet a `target.toml` pipeline target. diff --git a/justfile b/justfile index 689e5b5..d65952a 100644 --- a/justfile +++ b/justfile @@ -23,10 +23,15 @@ compare baseline candidate: cargo run -p bench-runner -- --config config/accept.toml compare \ --baseline "{{baseline}}" --candidate "{{candidate}}" -# Verify a target corpus against its pinned manifest -# (wired through diff-test CLI once it exists; manual check for now) +# Verify a target corpus against its pinned manifest (never rewrites it) pin-check target: - cd corpora/{{target}} && sh gen-corpus.sh + cd corpora/{{target}} && sh gen-corpus.sh --check + +# Deliberately re-pin a target corpus manifest — a trust-layer change; +# review and commit the new MANIFEST.sha256 with justification +pin-corpus target: + @echo "WARNING: rewriting corpora/{{target}}/MANIFEST.sha256 — deliberate human action (corpora/README.md)" + cd corpora/{{target}} && sh gen-corpus.sh --pin # Automated calibration: N A/A sessions (<5% false-positive required) + # N injected-slowdown sessions (>=95% detection required), JSON evidence @@ -34,8 +39,10 @@ calibrate cmd out sessions="20": cargo run -p bench-runner -- --config config/accept.toml calibrate \ --cmd "{{cmd}}" --sessions {{sessions}} --out "{{out}}" -# Equivalence gates for a target: corpus pin (refuses on mismatch) -> -# upstream tests -> golden replay. Fuzz/sanitizers stay per-attempt. +# Equivalence gates for a target: corpus + test-suite pins (refuse on +# mismatch) -> upstream tests -> golden replay. Differential fuzz needs a +# pristine baseline to differ against, so it reports Skipped here and runs +# for real on the accept path; sanitizers likewise run in verdict. gates target: cargo run -p diff-test -- {{target}} diff --git a/results/adversarial-review.md b/results/adversarial-review.md index 2a510a6..e72546a 100644 --- a/results/adversarial-review.md +++ b/results/adversarial-review.md @@ -57,3 +57,19 @@ consistent, with the compound tokei figure stated as an approximation the case study and is now enforced by the report generator too. - `phase0-comrak-002` (mimalloc) is `needs-human-review`, awaiting a ruling (see `results/rulings/`). + +## Erratum (2026-07-13) + +Finding 1's separation claim above — "only the historical overturned row +is `accepted + false`, so the rule cleanly separates them" — was too +strong. **Five** accepted rows carry `sanitizers_clean: false`: +`phase1-verdict-smoke-002`, `phase1-ipc-smoke-001`, `phase2-tokei-001`, +`phase2-tokei-003`, and `phase2-comrak-010`. All five pre-date the +machine-enforced sanitizer lane — the Phase 1 rows are harness smoke +tests, and the two Phase 2 tokei wins were sanitizer-audited by hand +during the 100% Phase 2 audit (see the Phase 2 case study) — but the +ledger's machine record cannot distinguish "manually audited clean" from +"never run", so the report generator's not-shippable rule flags all five, +not just the overturned row. That is the correct conservative behavior; +only the description of *why* the rule separates cleanly was wrong. The +original text stands unedited above; this note is the correction. diff --git a/results/phase2/case-study-agent-loop.md b/results/phase2/case-study-agent-loop.md index 093b2b3..0ebeb88 100644 --- a/results/phase2/case-study-agent-loop.md +++ b/results/phase2/case-study-agent-loop.md @@ -230,3 +230,14 @@ remains the deployment shape. pipeline into machine-enforced sanitizer gating. - **OS-level tool-boundary isolation:** met — was the tracked hardening gap; shipped and verified this phase. + +--- + +> **Note (2026-07-13).** The differential-fuzz and ASan/LSan runs +> described above ran **out-of-band** — per-attempt scripts during the +> 100% human audit, exactly as this case study records ("fuzz and +> sanitizers are per-attempt manual in this phase") — so the Phase 2 +> ledger rows carry `fuzz_iters=0` in their machine record. The fuzz +> gate is now pipeline-integrated (`diff-test` runs the target's +> declared fuzz command); rows written after this date record the +> iteration count actually executed. diff --git a/results/phase3/01-tokei-algorithmic-rust.md b/results/phase3/01-tokei-algorithmic-rust.md index 0b369e0..75ae534 100644 --- a/results/phase3/01-tokei-algorithmic-rust.md +++ b/results/phase3/01-tokei-algorithmic-rust.md @@ -62,3 +62,12 @@ increments, all on the same 40%-of-runtime function — and one honest rejection in between (`phase2-tokei-009`, a real +5.7% median whose noisy CI lower bound of +0.2% could not clear the bar, so it does not count). + +--- + +> **Note (2026-07-13).** The 10,000-input differential fuzz cited above +> ran **out-of-band** (`scripts/diff-fuzz-tokei.py`, per-attempt), not +> inside the verdict pipeline — so these ledger rows record +> `fuzz_iters=0` in their machine record. The fuzz gate is now +> pipeline-integrated; rows written after this date record the +> iteration count actually executed. diff --git a/results/phase3/04-cjson-c-target.md b/results/phase3/04-cjson-c-target.md index 5b75564..13eca1a 100644 --- a/results/phase3/04-cjson-c-target.md +++ b/results/phase3/04-cjson-c-target.md @@ -84,3 +84,12 @@ about the trust story is Rust-specific. The cheapest safe lever (skip a per-number allocation) wasn't enough; the algorithmic lever (stop doing a redundant heavyweight parse) was — and the pipeline proved it the same way it proves everything else. + +--- + +> **Note (2026-07-13).** The 10,000-input differential fuzz cited above +> ran **out-of-band** (`scripts/diff-fuzz-cjson.py`, per-attempt), not +> inside the verdict pipeline — so the `phase3-cjson-002` ledger row +> records `fuzz_iters=0` in its machine record. The fuzz gate is now +> pipeline-integrated; rows written after this date record the +> iteration count actually executed. diff --git a/results/phase3/README.md b/results/phase3/README.md index 734250e..e7bcb78 100644 --- a/results/phase3/README.md +++ b/results/phase3/README.md @@ -66,3 +66,13 @@ fuzz, and sanitizers. Every rejection is a real mechanism that simply didn't clear the 2% CI-lower-bound bar — and it is in the ledger too, because a rejection is a complete, valid outcome. Nothing shipped that the pipeline couldn't prove. + +> **Note (2026-07-13).** "Every win survived … a 10k-input differential +> fuzz, and sanitizers" is true of the **evidence**, not of the machine +> record at the time: through Phase 5 the fuzz ran **out-of-band** +> (`scripts/diff-fuzz-.py`, per-attempt) and the Phase 3 rows +> therefore carry `fuzz_iters=0`. The fuzz gate is now integrated into the +> accept path, where a passed fuzz gate is a **precondition** for a machine +> `accepted` verdict; rows written after this date record the iteration +> count actually executed. See the per-case-study notes in +> `01-tokei-algorithmic-rust.md` and `04-cjson-c-target.md`. diff --git a/results/phase4/README.md b/results/phase4/README.md index 217012f..8c3ceda 100644 --- a/results/phase4/README.md +++ b/results/phase4/README.md @@ -36,9 +36,11 @@ numbers. ## The result (`phase4-cjson-service`, ledger) The accepted batch win `phase3-cjson-002` (+8.85% throughput) was -measured as a **service under a 150 rps coordinated-omission-correct +measured as a **service under a 150 rps-target coordinated-omission-correct replay**, baseline (pristine cJSON v1.7.19) vs candidate (the banked -win), 20 interleaved rounds, server pinned, 40,000 requests, **0 drops**: +win), 20 interleaved rounds, server pinned, **0 drops** — the ledger row +records `sessions=20`, `rate_rps=150`, `drop_rate=0.0`; the total request +count was **not machine-recorded** (see errata below): | metric | baseline | candidate | speedup | 95% CI | |---|---|---|---|---| @@ -63,7 +65,10 @@ project in one line. ## The ROI, generated mechanically `just report phase4-cjson-service --service-json results/phase4/cjson-service.json` -→ [`cjson-service-roi.md`](cjson-service-roi.md). On a 500-core fleet at +→ [`cjson-service-roi.md`](cjson-service-roi.md). *(Note, 2026-07-13: the +raw `cjson-service.json` was not preserved — see errata below — so this +command is not reproducible as written; the generated report and the +ledger row are the surviving evidence.)* On a 500-core fleet at $0.04/core-hour, the +6.2% p50 win returns **27.5 cores / $9,621 per year at the CI lower bound** (median 29.4 cores / $10,290) — and the report quotes the lower bound, because the point estimate is not a @@ -79,3 +84,20 @@ that produced it, so the number survives hostile review. ledger row and the calibrated service JSON with no hand-editing; throughput→cores→dollars and latency percentiles, CIs and methodology inline. + +## Errata (2026-07-13) + +- **"40,000 requests" retracted.** This report originally stated the + replay comprised 40,000 requests. The `phase4-cjson-service` ledger row + records only `sessions=20`, `rate_rps=150`, and `drop_rate=0.0` — the + per-run request count was never machine-recorded, so the 40,000 figure + (an arithmetic reconstruction, not evidence) is withdrawn. The prose + above now states the ledger-backed facts only. The claimed result is + unaffected: p50 latency +6.2%, 95% CI [+5.8%, +7.2%], under the 150 + rps-target CO-correct replay, 20 interleaved rounds, 0 drops. +- **`results/phase4/cjson-service.json` was not preserved.** The + `.gitignore` rule `results/**/*.json` swept the raw service-latency + JSON that the report command consumes, so it no longer exists in the + repository. A `!results/phase4/*.json` carve-out has been added so + future service-mode evidence JSONs are committed alongside the + calibration JSONs. diff --git a/results/phase5/README.md b/results/phase5/README.md index ba83e73..b4370c4 100644 --- a/results/phase5/README.md +++ b/results/phase5/README.md @@ -137,3 +137,31 @@ it generalize?" objection with the two hardest cases the spec named: Zero shipped false accepts, still. The product is trust, and trust transferred. + +--- + +## Erratum (2026-07-13) — how far the kernel lane is actually wired + +The write-up above describes the FP-tolerance gate as though `matmul` runs +through the standard pipeline. It does not, and the distinction matters: + +- **The policy is in the pipeline.** `EquivalencePolicy` is honored by + `diff-test`'s gate sequence (an fp-tolerance target needs a committed + `[corpus].golden_reference`; the gate fails closed without one), and + `crates/verdict` treats fp-tolerance equivalence as an automatic + `needs-human-review` signal. That machinery is real and unit-tested. +- **`matmul` is not a pipeline target.** It ships `targets/matmul/kernel.c` + and `targets/matmul/equivalence.toml` but **no `target.toml`** — the file + `diff-test`/`verdict` load a target from. So the lane cannot currently be + run via `just gates` or `just verdict` at all. +- **Therefore the demonstration is script-driven.** + `scripts/kernel-lane-demo.sh` invokes the `fp-compare` binary and + `bench-runner compare` directly, and the `phase5-matmul-opt` ledger row + was recorded **outside** the automated verdict path. + +Nothing in the measured result changes — the 3.23× [3.16, 3.26] speedup, the +244,901/262,144 differing values, the rejected +0.5 perturbation, and the +`needs-human-review` verdict all stand as printed. What was overstated is the +*integration*: the equivalence policy is closed; wiring this target into the +pipeline is open work. The original text stands unedited above; this note is +the correction. diff --git a/scripts/diff-fuzz-cjson.py b/scripts/diff-fuzz-cjson.py index 4a6b2c3..eee0522 100644 --- a/scripts/diff-fuzz-cjson.py +++ b/scripts/diff-fuzz-cjson.py @@ -4,7 +4,21 @@ produce byte-identical stdout (serialization + checksum) AND identical exit code on mutated JSON. cJSON is a deterministic single-threaded parser, so raw byte comparison needs no canonicalization. - python3 scripts/diff-fuzz-cjson.py + python3 scripts/diff-fuzz-cjson.py + +All four arguments are required and are supplied by the harness: diff-test +substitutes {iters}, {baseline} and {candidate} into the target's +[gates].fuzz template and runs it from the repo root. Both binary paths may +be repo-root-relative OR absolute (the accept path rebuilds the pristine +baseline to an absolute path) — relative ones resolve against the repo root. +Nothing here may hardcode a baseline location: the harness owns where the +pristine build lands, and a hardcoded path is a gate that always hard-fails. + +Final stdout line is the machine contract diff-test parses: +FUZZ-RESULT iters= divergences=. Exit 0 iff the run completed with +zero divergences, 1 on divergence (the line is still printed, with the real +count), 2 on a usage/setup error — which prints no FUZZ-RESULT and is +therefore graded a hard gate FAILURE, never a pass. Seeds: slices of the pinned corpus plus hand JSON fragments. Mutations target parser edges: structural tokens, string escapes, number forms @@ -15,13 +29,12 @@ import random import subprocess import sys from pathlib import Path +from typing import NoReturn -ROOT = Path("/home/user/peltier.io") -BASE = ROOT / "targets/cjson/baseline/cjson-bench" -CAND = ROOT / sys.argv[3] if len(sys.argv) > 3 else ROOT / "targets/cjson/baseline/cjson-bench" +ROOT = Path(__file__).resolve().parents[1] CORPUS = ROOT / "corpora/cjson/input/big.json" -WORK = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("/tmp/difffuzz-cjson") -ITERS = int(sys.argv[2]) if len(sys.argv) > 2 else 10000 + +USAGE = "usage: diff-fuzz-cjson.py " TOKENS = [ b"{", b"}", b"[", b"]", b'"', b":", b",", b"\\", b"\\n", b"\\u00e9", b"\\uD83D", @@ -38,6 +51,59 @@ SEEDS_STATIC = [ ] +def die(msg: str) -> NoReturn: + """Fatal setup error: exit 2 having printed NO FUZZ-RESULT line. diff-test + grades a fuzz command that never reports the line as a hard gate failure, + which is the right posture for a gate that could not run at all — it must + never be gradeable as a pass.""" + print(f"diff-fuzz-cjson: {msg}", file=sys.stderr) + raise SystemExit(2) + + +def parse_args(argv: list[str]) -> tuple[Path, int, Path, Path]: + """ , all positional.""" + if len(argv) != 5: + die(f"expected 4 arguments, got {len(argv) - 1}\n{USAGE}") + work, raw_iters, baseline, candidate = argv[1:5] + if not raw_iters.isdigit(): + die(f"iters must be a non-negative integer, got {raw_iters!r}\n{USAGE}") + iters = int(raw_iters) + if iters < 1: + # A zero-iteration run would print FUZZ-RESULT iters=0 divergences=0 + # and exit 0 — a fuzz gate that passes without fuzzing anything. + die(f"iters must be >= 1, got {iters}\n{USAGE}") + # `ROOT / p` keeps a repo-root-relative path anchored at the repo root and + # leaves an absolute path untouched (pathlib drops the left operand when + # the right side is absolute). The harness supplies either form. + return ROOT / work, iters, ROOT / baseline, ROOT / candidate + + +def need_binary(path: Path, label: str) -> Path: + if not path.is_file(): + die( + f"{label} binary not found: {path}\nthe harness substitutes " + "{baseline}/{candidate} in [gates].fuzz — check targets/cjson/target.toml" + ) + return path + + +def keep_dir(work: Path, tag: str) -> Path: + """First unused divergence- directory under `work`. + + The RNG is seeded, so re-running in the same work dir re-diverges at the + same iteration: the kept-divergence directory already exists. The old + plain `mkdir()` raised FileExistsError there and killed the run *before* + the FUZZ-RESULT line, so diff-test reported a genuine divergence — the + whole point of the gate — as "fuzz command did not report FUZZ-RESULT". + This never raises; it picks the next free name instead. + """ + path, n = work / f"divergence-{tag}", 2 + while path.exists(): + path = work / f"divergence-{tag}-{n}" + n += 1 + return path + + def mutate(data: bytes, rng: random.Random) -> bytes: b = bytearray(data) for _ in range(rng.randrange(1, 8)): @@ -65,36 +131,54 @@ def run(binary: Path, path: Path) -> tuple[bytes, int]: return r.stdout, r.returncode -def main() -> int: +def main(argv: list[str]) -> int: + work, iters, baseline, candidate = parse_args(argv) + need_binary(baseline, "baseline") + need_binary(candidate, "candidate") + if not CORPUS.is_file(): + die(f"corpus not found: {CORPUS} (generate it with corpora/cjson/gen-corpus.sh)") + rng = random.Random(0x1CE_B00C) corpus = CORPUS.read_bytes() seeds = list(SEEDS_STATIC) # windows of the real corpus (often invalid mid-slice — exercises the - # error path, which must diverge identically or not at all) - for o in range(0, len(corpus) - 4096, len(corpus) // 200): + # error path, which must diverge identically or not at all). The max(1, …) + # guards are no-ops on the pinned 5.6 MB corpus (identical seed set, so + # the fuzz stays reproducible) and stop a degenerate tiny corpus from + # raising a zero-step ValueError before FUZZ-RESULT is ever printed. + for o in range(0, max(1, len(corpus) - 4096), max(1, len(corpus) // 200)): for n in (64, 512, 4096): seeds.append(corpus[o : o + n]) - WORK.mkdir(parents=True, exist_ok=True) - case = WORK / "case.json" + work.mkdir(parents=True, exist_ok=True) + case = work / "case.json" divergences = 0 - for i in range(ITERS): + for i in range(iters): payload = mutate(rng.choice(seeds), rng) case.write_bytes(payload) - a_out, a_rc = run(BASE, case) - b_out, b_rc = run(CAND, case) + a_out, a_rc = run(baseline, case) + b_out, b_rc = run(candidate, case) if a_out != b_out or a_rc != b_rc: divergences += 1 - keep = WORK / f"divergence-{i}" - keep.mkdir(exist_ok=True) - (keep / "input.json").write_bytes(payload) - (keep / "baseline.out").write_bytes(a_out + f"\nrc={a_rc}".encode()) - (keep / "candidate.out").write_bytes(b_out + f"\nrc={b_rc}".encode()) - print(f"DIVERGENCE at iter {i} (rc {a_rc} vs {b_rc}); kept in {keep}") + # Keeping the reproducer is best-effort: an I/O failure here must + # never cost the FUZZ-RESULT line and turn a real divergence into + # an unparseable gate error (see keep_dir). + try: + keep = keep_dir(work, str(i)) + keep.mkdir(parents=True, exist_ok=True) + (keep / "input.json").write_bytes(payload) + (keep / "baseline.out").write_bytes(a_out + f"\nrc={a_rc}".encode()) + (keep / "candidate.out").write_bytes(b_out + f"\nrc={b_rc}".encode()) + print(f"DIVERGENCE at iter {i} (rc {a_rc} vs {b_rc}); kept in {keep}") + except OSError as e: + print(f"DIVERGENCE at iter {i} (rc {a_rc} vs {b_rc}); " + f"could not keep artifacts: {e}") if (i + 1) % 1000 == 0: - print(f"{i + 1}/{ITERS}, {divergences} divergences", flush=True) - print(f"done: {ITERS} inputs, {divergences} divergences") + print(f"{i + 1}/{iters}, {divergences} divergences", flush=True) + print(f"done: {iters} inputs, {divergences} divergences") + # Machine contract, always the final stdout line on any completing run. + print(f"FUZZ-RESULT iters={iters} divergences={divergences}", flush=True) return 1 if divergences else 0 if __name__ == "__main__": - sys.exit(main()) + sys.exit(main(sys.argv)) diff --git a/scripts/diff-fuzz-comrak.py b/scripts/diff-fuzz-comrak.py index fc154f8..4308242 100644 --- a/scripts/diff-fuzz-comrak.py +++ b/scripts/diff-fuzz-comrak.py @@ -4,7 +4,21 @@ produce byte-identical HTML (stdout), stderr, and exit code on mutated markdown. comrak is a single-threaded stdin->stdout filter, so raw byte comparison needs no canonicalization. - python3 scripts/diff-fuzz-comrak.py + python3 scripts/diff-fuzz-comrak.py + +All four arguments are required and are supplied by the harness: diff-test +substitutes {iters}, {baseline} and {candidate} into the target's +[gates].fuzz template and runs it from the repo root. Both binary paths may +be repo-root-relative OR absolute (the accept path rebuilds the pristine +baseline to an absolute path) — relative ones resolve against the repo root. +Nothing here may hardcode a baseline location: the harness owns where the +pristine build lands, and a hardcoded path is a gate that always hard-fails. + +Final stdout line is the machine contract diff-test parses: +FUZZ-RESULT iters= divergences=. Exit 0 iff the run completed with +zero divergences, 1 on divergence (the line is still printed, with the real +count), 2 on a usage/setup error — which prints no FUZZ-RESULT and is +therefore graded a hard gate FAILURE, never a pass. Seeds: windows of the pinned Pro Git corpus. Mutations focus on inline/ block structure the parser and teardown paths exercise: emphasis, code @@ -15,13 +29,12 @@ import random import subprocess import sys from pathlib import Path +from typing import NoReturn -ROOT = Path("/home/user/peltier.io") -BASE = ROOT / "targets/comrak/baseline/release/comrak" -CAND = ROOT / sys.argv[3] +ROOT = Path(__file__).resolve().parents[1] CORPUS = ROOT / "corpora/comrak/progit.md" -WORK = Path(sys.argv[1]) -ITERS = int(sys.argv[2]) + +USAGE = "usage: diff-fuzz-comrak.py " TOKENS = [ b"**", b"*", b"__", b"_", b"`", b"```", b"~~~", b"~~", b"[", b"](", b")", @@ -30,6 +43,59 @@ TOKENS = [ ] +def die(msg: str) -> NoReturn: + """Fatal setup error: exit 2 having printed NO FUZZ-RESULT line. diff-test + grades a fuzz command that never reports the line as a hard gate failure, + which is the right posture for a gate that could not run at all — it must + never be gradeable as a pass.""" + print(f"diff-fuzz-comrak: {msg}", file=sys.stderr) + raise SystemExit(2) + + +def parse_args(argv: list[str]) -> tuple[Path, int, Path, Path]: + """ , all positional.""" + if len(argv) != 5: + die(f"expected 4 arguments, got {len(argv) - 1}\n{USAGE}") + work, raw_iters, baseline, candidate = argv[1:5] + if not raw_iters.isdigit(): + die(f"iters must be a non-negative integer, got {raw_iters!r}\n{USAGE}") + iters = int(raw_iters) + if iters < 1: + # A zero-iteration run would print FUZZ-RESULT iters=0 divergences=0 + # and exit 0 — a fuzz gate that passes without fuzzing anything. + die(f"iters must be >= 1, got {iters}\n{USAGE}") + # `ROOT / p` keeps a repo-root-relative path anchored at the repo root and + # leaves an absolute path untouched (pathlib drops the left operand when + # the right side is absolute). The harness supplies either form. + return ROOT / work, iters, ROOT / baseline, ROOT / candidate + + +def need_binary(path: Path, label: str) -> Path: + if not path.is_file(): + die( + f"{label} binary not found: {path}\nthe harness substitutes " + "{baseline}/{candidate} in [gates].fuzz — check targets/comrak/target.toml" + ) + return path + + +def keep_dir(work: Path, tag: str) -> Path: + """First unused divergence- directory under `work`. + + The RNG is seeded, so re-running in the same work dir re-diverges at the + same iteration: the kept-divergence directory already exists. The old + plain `mkdir()` raised FileExistsError there and killed the run *before* + the FUZZ-RESULT line, so diff-test reported a genuine divergence — the + whole point of the gate — as "fuzz command did not report FUZZ-RESULT". + This never raises; it picks the next free name instead. + """ + path, n = work / f"divergence-{tag}", 2 + while path.exists(): + path = work / f"divergence-{tag}-{n}" + n += 1 + return path + + def mutate(data: bytes, rng: random.Random) -> bytes: b = bytearray(data) for _ in range(rng.randrange(1, 8)): @@ -57,34 +123,54 @@ def run(binary: Path, payload: bytes) -> tuple[bytes, bytes, int]: return r.stdout, r.stderr, r.returncode -def main() -> int: +def main(argv: list[str]) -> int: + work, iters, baseline, candidate = parse_args(argv) + need_binary(baseline, "baseline") + need_binary(candidate, "candidate") + if not CORPUS.is_file(): + die(f"corpus not found: {CORPUS} (generate it with corpora/comrak/gen-corpus.sh)") + rng = random.Random(0xC0FFEE) corpus = CORPUS.read_bytes() - # Seed windows across the whole book, varied sizes. + # Seed windows across the whole book, varied sizes. The max(1, …) guards + # are no-ops on the pinned 11 MB corpus (identical seed set, so the fuzz + # stays reproducible); they stop a degenerate tiny corpus from raising a + # zero-step ValueError, or yielding no seeds at all, before FUZZ-RESULT + # is ever printed. seeds = [ corpus[o : o + n] - for o in range(0, len(corpus) - 65536, len(corpus) // 300) + for o in range(0, max(1, len(corpus) - 65536), max(1, len(corpus) // 300)) for n in (512, 4096, 32768) ] - WORK.mkdir(parents=True, exist_ok=True) + if not seeds: + die(f"corpus is empty, no seeds to fuzz from: {CORPUS}") + work.mkdir(parents=True, exist_ok=True) divergences = 0 - for i in range(ITERS): + for i in range(iters): payload = mutate(rng.choice(seeds), rng) - a = run(BASE, payload) - b = run(CAND, payload) + a = run(baseline, payload) + b = run(candidate, payload) if a != b: divergences += 1 - keep = WORK / f"divergence-{i}" - keep.mkdir() - (keep / "input.md").write_bytes(payload) - (keep / "baseline.out").write_bytes(a[0] + b"\n--stderr--\n" + a[1] + f"\nrc={a[2]}".encode()) - (keep / "candidate.out").write_bytes(b[0] + b"\n--stderr--\n" + b[1] + f"\nrc={b[2]}".encode()) - print(f"DIVERGENCE at iter {i}; kept in {keep}") + # Keeping the reproducer is best-effort: an I/O failure here must + # never cost the FUZZ-RESULT line and turn a real divergence into + # an unparseable gate error (see keep_dir). + try: + keep = keep_dir(work, str(i)) + keep.mkdir(parents=True, exist_ok=True) + (keep / "input.md").write_bytes(payload) + (keep / "baseline.out").write_bytes(a[0] + b"\n--stderr--\n" + a[1] + f"\nrc={a[2]}".encode()) + (keep / "candidate.out").write_bytes(b[0] + b"\n--stderr--\n" + b[1] + f"\nrc={b[2]}".encode()) + print(f"DIVERGENCE at iter {i}; kept in {keep}") + except OSError as e: + print(f"DIVERGENCE at iter {i}; could not keep artifacts: {e}") if (i + 1) % 1000 == 0: - print(f"{i + 1}/{ITERS}, {divergences} divergences", flush=True) - print(f"done: {ITERS} inputs, {divergences} divergences") + print(f"{i + 1}/{iters}, {divergences} divergences", flush=True) + print(f"done: {iters} inputs, {divergences} divergences") + # Machine contract, always the final stdout line on any completing run. + print(f"FUZZ-RESULT iters={iters} divergences={divergences}", flush=True) return 1 if divergences else 0 if __name__ == "__main__": - sys.exit(main()) + sys.exit(main(sys.argv)) diff --git a/scripts/diff-fuzz-tokei.py b/scripts/diff-fuzz-tokei.py index fdba595..5e56ffe 100644 --- a/scripts/diff-fuzz-tokei.py +++ b/scripts/diff-fuzz-tokei.py @@ -1,8 +1,25 @@ #!/usr/bin/env python3 -"""Differential fuzz for phase2-tokei-001: pristine vs candidate tokei +"""Differential fuzz for tokei attempts: pristine vs candidate tokei must produce byte-identical output on mutated source inputs targeting the patched code path (quote/comment token scanning). + python3 scripts/diff-fuzz-tokei.py + +All four arguments are required and are supplied by the harness: diff-test +substitutes {iters}, {baseline} and {candidate} into the target's +[gates].fuzz template and runs it from the repo root. Both binary paths may +be repo-root-relative OR absolute (the accept path rebuilds the pristine +baseline to an absolute path) — relative ones resolve against the repo root. +Nothing here may hardcode a baseline location: the harness owns where the +pristine build lands, and a hardcoded path is a gate that always hard-fails. + +Final stdout line is the machine contract diff-test parses: +FUZZ-RESULT iters= divergences=. Exit 0 iff the run completed with +zero divergences, 1 on divergence (the line is still printed, with the real +count), 2 on a usage/setup error — which prints no FUZZ-RESULT and is +therefore graded a hard gate FAILURE, never a pass. `iters` counts mutated +files (tokei scans a directory, so files are batched per invocation). + Seeds: files from the pinned corpus (many languages). Mutations focus on quote/comment structure: inserting/deleting/duplicating quote chars, comment openers/closers, backslashes, and random bytes; plus boundary @@ -13,13 +30,12 @@ import random import subprocess import sys from pathlib import Path +from typing import NoReturn -ROOT = Path("/home/user/peltier.io") -BASE = ROOT / "targets/tokei/baseline/release/tokei" -CAND = ROOT / "targets/tokei/candidate-4e3da7e6bd1c/release/tokei" +ROOT = Path(__file__).resolve().parents[1] CORPUS = ROOT / "corpora/tokei/input/copy1" -WORK = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("/tmp/difffuzz") -ITERS = int(sys.argv[2]) if len(sys.argv) > 2 else 10000 + +USAGE = "usage: diff-fuzz-tokei.py " TOKENS = [ b'"', b"'", b'`', b'"""', b"'''", b"/*", b"*/", b"//", b"#", b"--", @@ -27,6 +43,60 @@ TOKENS = [ b"r#\"", b"\"#", b"=begin", b"=end", b"'''", b'f"""', b"%{", b"}", ] + +def die(msg: str) -> NoReturn: + """Fatal setup error: exit 2 having printed NO FUZZ-RESULT line. diff-test + grades a fuzz command that never reports the line as a hard gate failure, + which is the right posture for a gate that could not run at all — it must + never be gradeable as a pass.""" + print(f"diff-fuzz-tokei: {msg}", file=sys.stderr) + raise SystemExit(2) + + +def parse_args(argv: list[str]) -> tuple[Path, int, Path, Path]: + """ , all positional.""" + if len(argv) != 5: + die(f"expected 4 arguments, got {len(argv) - 1}\n{USAGE}") + work, raw_iters, baseline, candidate = argv[1:5] + if not raw_iters.isdigit(): + die(f"iters must be a non-negative integer, got {raw_iters!r}\n{USAGE}") + iters = int(raw_iters) + if iters < 1: + # A zero-iteration run would print FUZZ-RESULT iters=0 divergences=0 + # and exit 0 — a fuzz gate that passes without fuzzing anything. + die(f"iters must be >= 1, got {iters}\n{USAGE}") + # `ROOT / p` keeps a repo-root-relative path anchored at the repo root and + # leaves an absolute path untouched (pathlib drops the left operand when + # the right side is absolute). The harness supplies either form. + return ROOT / work, iters, ROOT / baseline, ROOT / candidate + + +def need_binary(path: Path, label: str) -> Path: + if not path.is_file(): + die( + f"{label} binary not found: {path}\nthe harness substitutes " + "{baseline}/{candidate} in [gates].fuzz — check targets/tokei/target.toml" + ) + return path + + +def keep_dir(work: Path, tag: str) -> Path: + """First unused divergence- directory under `work`. + + The RNG is seeded, so re-running in the same work dir re-diverges at the + same batch: the kept-divergence directory already exists. The old plain + `rename()` raised (the destination is a non-empty directory) and killed + the run *before* the FUZZ-RESULT line, so diff-test reported a genuine + divergence — the whole point of the gate — as "fuzz command did not + report FUZZ-RESULT". This never raises; it picks the next free name. + """ + path, n = work / f"divergence-{tag}", 2 + while path.exists(): + path = work / f"divergence-{tag}-{n}" + n += 1 + return path + + def mutate(data: bytes, rng: random.Random) -> bytes: ops = rng.randrange(1, 6) b = bytearray(data) @@ -51,6 +121,7 @@ def mutate(data: bytes, rng: random.Random) -> bytes: b[pos:pos] = b"\n" return bytes(b) + def canonicalize(payload: bytes) -> str: """Parse tokei JSON and sort every list so benign output-order nondeterminism (parallel walker arrival order) doesn't count as a @@ -78,43 +149,62 @@ def run(binary: Path, d: Path) -> tuple[str, bytes, int]: canon = f"UNPARSEABLE:{e}:{r.stdout[:200]!r}" return canon + f"||rc={r.returncode}", r.stdout, r.returncode -def main() -> int: + +def main(argv: list[str]) -> int: + work, iters, baseline, candidate = parse_args(argv) + need_binary(baseline, "baseline") + need_binary(candidate, "candidate") + if not CORPUS.is_dir(): + die(f"corpus not found: {CORPUS} (generate it with corpora/tokei/gen-corpus.sh)") + rng = random.Random(0xB16B00B5) seeds = sorted(CORPUS.rglob("*")) seeds = [p for p in seeds if p.is_file() and p.stat().st_size < 200_000][:400] if not seeds: - print("no seeds found", file=sys.stderr) - return 2 - WORK.mkdir(parents=True, exist_ok=True) - fdir = WORK / "case" + die(f"no seeds found under {CORPUS}") + work.mkdir(parents=True, exist_ok=True) + fdir = work / "case" divergences = 0 done = 0 + runs = 0 # Batch: mutate BATCH files per tokei invocation (tokei scans a dir), # keeping each seed's extension so language detection stays realistic. BATCH = 50 - while done < ITERS: + while done < iters: if fdir.exists(): for f in fdir.iterdir(): f.unlink() fdir.mkdir(exist_ok=True) - for j in range(min(BATCH, ITERS - done)): + batch = min(BATCH, iters - done) + for j in range(batch): seed = rng.choice(seeds) name = f"m{done + j}{seed.suffix or '.txt'}" (fdir / name).write_bytes(mutate(seed.read_bytes(), rng)) - a_canon, a_raw, a_rc = run(BASE, fdir) - b_canon, b_raw, b_rc = run(CAND, fdir) + a_canon, a_raw, a_rc = run(baseline, fdir) + b_canon, b_raw, b_rc = run(candidate, fdir) if a_canon != b_canon: divergences += 1 - keep = WORK / f"divergence-{done}" - fdir.rename(keep) - (keep / "_baseline.out").write_bytes(a_raw + f"\nrc={a_rc}".encode()) - (keep / "_candidate.out").write_bytes(b_raw + f"\nrc={b_rc}".encode()) - print(f"DIVERGENCE at batch starting {done}; inputs+outputs kept in {keep}") - done += BATCH + # Keeping the reproducer is best-effort: an I/O failure here must + # never cost the FUZZ-RESULT line and turn a real divergence into + # an unparseable gate error (see keep_dir). If the rename fails, + # fdir survives and the next batch cleans it out. + try: + keep = keep_dir(work, str(done)) + fdir.rename(keep) + (keep / "_baseline.out").write_bytes(a_raw + f"\nrc={a_rc}".encode()) + (keep / "_candidate.out").write_bytes(b_raw + f"\nrc={b_rc}".encode()) + print(f"DIVERGENCE at batch starting {done}; inputs+outputs kept in {keep}") + except OSError as e: + print(f"DIVERGENCE at batch starting {done}; could not keep artifacts: {e}") + done += batch + runs += 1 if done % 1000 == 0: - print(f"{done}/{ITERS} mutated files, {divergences} divergences", flush=True) - print(f"done: {done} mutated files across {done // BATCH} runs, {divergences} divergences") + print(f"{done}/{iters} mutated files, {divergences} divergences", flush=True) + print(f"done: {done} mutated files across {runs} runs, {divergences} divergences") + # Machine contract, always the final stdout line on any completing run. + print(f"FUZZ-RESULT iters={done} divergences={divergences}", flush=True) return 1 if divergences else 0 + if __name__ == "__main__": - sys.exit(main()) + sys.exit(main(sys.argv)) diff --git a/scripts/no-net.sh b/scripts/no-net.sh new file mode 100755 index 0000000..16eed94 --- /dev/null +++ b/scripts/no-net.sh @@ -0,0 +1,25 @@ +#!/bin/sh +# Network-isolation wrapper for the local verdict path (SPEC §10): exec the +# given command inside a fresh network namespace so target code built and +# run by the pipeline cannot reach the network. harnessd invokes it as +# `no-net.sh sh -c ` (HOTPATH_VERDICT_WRAPPER overrides). +# CI gets the same guarantee from `docker --network=none` instead. +set -eu +if [ $# -eq 0 ]; then + echo "usage: scripts/no-net.sh [args...]" >&2 + exit 2 +fi +if unshare --net --map-current-user true 2>/dev/null; then + exec unshare --net --map-current-user -- "$@" +fi +if [ "${HOTPATH_ALLOW_UNISOLATED:-}" = "1" ]; then + echo "no-net.sh: WARNING: unshare --net unavailable and HOTPATH_ALLOW_UNISOLATED=1 —" >&2 + echo "no-net.sh: WARNING: running WITHOUT network isolation. Target code can reach the network." >&2 + exec "$@" +fi +echo "no-net.sh: cannot create a network namespace (unshare --net --map-current-user failed)." >&2 +echo "no-net.sh: refusing to run target code unisolated. Options:" >&2 +echo "no-net.sh: - run on a host with unprivileged user namespaces enabled" >&2 +echo "no-net.sh: - in CI, run the whole pipeline under docker --network=none" >&2 +echo "no-net.sh: - set HOTPATH_ALLOW_UNISOLATED=1 to override (loud warning, not for accepts)" >&2 +exit 97 diff --git a/scripts/pin-testsuite.sh b/scripts/pin-testsuite.sh new file mode 100755 index 0000000..45ae67a --- /dev/null +++ b/scripts/pin-testsuite.sh @@ -0,0 +1,63 @@ +#!/bin/sh +# Pin an upstream test suite: hash every file under the given repo-root- +# relative paths (typically targets//workspace/tests/...) into +# corpora//TESTSUITE.sha256, sha256sum two-space format. diff-test +# verifies this pin before running upstream tests; targets/fetch.sh +# verifies it after every fetch. Run from repo root: +# scripts/pin-testsuite.sh ... +set -eu +if [ $# -lt 2 ]; then + echo "usage: scripts/pin-testsuite.sh ..." >&2 + exit 2 +fi +target="$1" +shift +out="corpora/$target/TESTSUITE.sha256" +if [ ! -d "corpora/$target" ]; then + echo "pin-testsuite: corpora/$target does not exist (run from repo root?)" >&2 + exit 1 +fi +for p in "$@"; do + if [ ! -e "$p" ]; then + echo "pin-testsuite: $p not found (paths are repo-root-relative)" >&2 + exit 1 + fi +done + +# Build the pin out-of-place and install it only on success: a run that fails +# anywhere below must not truncate, replace, or half-write an existing pin — +# that pin is what diff-test refuses to run against on mismatch. +list="$out.files.$$" +tmp="$out.new.$$" +trap 'rm -f "$list" "$tmp"' EXIT INT TERM + +# LC_ALL=C: byte-order sort, so the same tree pins identically on every +# machine. -print0/-z: filenames with spaces or newlines survive intact. +find "$@" -type f -print0 | LC_ALL=C sort -z > "$list" +count=$(tr -dc '\0' < "$list" | wc -c | tr -d ' ') +if [ "$count" -eq 0 ]; then + # Was: `find ... | xargs -0 sha256sum > "$out"` — with no matches, xargs + # still ran sha256sum with zero file arguments, so it hashed STDIN (the + # empty pipe) and wrote a bogus " -" entry, then reported + # SUCCESS. That is a pin of nothing that later "verifies" by reading + # stdin. Refuse instead, and leave any existing pin untouched. + echo "pin-testsuite: no regular files under: $* — refusing to write $out" >&2 + exit 1 +fi + +# --text is GNU's default on Linux (a no-op there); it is spelled out because +# sha256sum defaults to *binary* mode on some platforms, emitting " *path" +# — a one-space-plus-asterisk form that diff-test's pin verifier (which splits +# on the two-space separator) does not parse. Paths stay repo-root-relative: +# find was handed repo-root-relative paths and we run from the repo root. +xargs -0 sha256sum --text -- < "$list" > "$tmp" + +lines=$(wc -l < "$tmp" | tr -d ' ') +if [ "$lines" -ne "$count" ]; then + echo "pin-testsuite: hashed $lines of $count files — refusing to write a partial pin" >&2 + exit 1 +fi +mv "$tmp" "$out" +echo "wrote $out ($count files)" +echo "pinning a test suite is a deliberate human action (corpora/README.md):" +echo "commit the new pin with justification." diff --git a/targets/cjson/target.toml b/targets/cjson/target.toml index 2d8388a..d34e156 100644 --- a/targets/cjson/target.toml +++ b/targets/cjson/target.toml @@ -37,6 +37,12 @@ workload = "5.6 MB synthetic JSON (20k records: nested objects/arrays, escaped/u tests = "cmake -S targets/cjson/workspace -B targets/cjson/testbuild -DENABLE_CJSON_TEST=On -DCMAKE_BUILD_TYPE=Debug >/dev/null 2>&1 && cmake --build targets/cjson/testbuild -j >/dev/null 2>&1 && ctest --test-dir targets/cjson/testbuild --output-on-failure" # Golden: full re-serialization + checksum of the corpus, byte-hashed. golden = "{binary} corpora/cjson/input/big.json 1" +# Differential fuzz, pipeline-integrated: diff-test substitutes {iters}, +# {baseline} (the pristine binary the accept path rebuilds) and {candidate}, +# then parses the script's final FUZZ-RESULT line. The harness owns where the +# pristine build lands, so the script takes the baseline from argv and never +# hardcodes a path; paths may be repo-root-relative or absolute. +fuzz = "python3 scripts/diff-fuzz-cjson.py targets/cjson/fuzz-work {iters} {baseline} {candidate}" [corpus] manifest = "corpora/cjson/MANIFEST.sha256" diff --git a/targets/comrak/target.toml b/targets/comrak/target.toml index 94b8d67..526cfee 100644 --- a/targets/comrak/target.toml +++ b/targets/comrak/target.toml @@ -33,6 +33,12 @@ workload = "Pro Git book, all translations, 11.0 MB concatenated markdown, singl tests = "cargo test --release --manifest-path targets/comrak/workspace/Cargo.toml" # Golden replay: corpus through the binary, stdout must be byte-identical. golden = "cat corpora/comrak/progit.md | {binary}" +# Differential fuzz, pipeline-integrated: diff-test substitutes {iters}, +# {baseline} (the pristine binary the accept path rebuilds) and {candidate}, +# then parses the script's final FUZZ-RESULT line. The harness owns where the +# pristine build lands, so the script takes the baseline from argv and never +# hardcodes a path; paths may be repo-root-relative or absolute. +fuzz = "python3 scripts/diff-fuzz-comrak.py targets/comrak/fuzz-work {iters} {baseline} {candidate}" [corpus] manifest = "corpora/comrak/MANIFEST.sha256" diff --git a/targets/fetch.sh b/targets/fetch.sh index 0f73a15..4c4dbff 100755 --- a/targets/fetch.sh +++ b/targets/fetch.sh @@ -21,4 +21,16 @@ grep -A0 '^"vendor/' "$toml" | while IFS= read -r line; do git -C "$ws/$sub" fetch -q origin "$scommit" 2>/dev/null || true git -C "$ws/$sub" checkout -q "$scommit" done +# Upstream test-suite pin: verify against repo root when present; a +# mismatch means the fetched suite is not the one that was audited. +suite="corpora/$name/TESTSUITE.sha256" +if [ -f "$suite" ]; then + if ! sha256sum -c --quiet "$suite"; then + echo "fetch: upstream test-suite pin MISMATCH for $name ($suite) — refusing" >&2 + exit 1 + fi + echo "$name test suite verified against $suite" +else + echo "note: no $suite — pin the suite with: scripts/pin-testsuite.sh $name " +fi echo "$name pinned at $commit" diff --git a/targets/tokei/target.toml b/targets/tokei/target.toml index 2b84248..8133be8 100644 --- a/targets/tokei/target.toml +++ b/targets/tokei/target.toml @@ -23,6 +23,12 @@ workload = "8 replicas of pinned progit + comrak/src trees, 221 MB, 4360 files, [gates] tests = "cargo test --release --manifest-path targets/tokei/workspace/Cargo.toml" golden = "RAYON_NUM_THREADS=1 {binary} --sort code corpora/tokei/input" +# Differential fuzz, pipeline-integrated: diff-test substitutes {iters}, +# {baseline} (the pristine binary the accept path rebuilds) and {candidate}, +# then parses the script's final FUZZ-RESULT line. The harness owns where the +# pristine build lands, so the script takes the baseline from argv and never +# hardcodes a path; paths may be repo-root-relative or absolute. +fuzz = "python3 scripts/diff-fuzz-tokei.py targets/tokei/fuzz-work {iters} {baseline} {candidate}" [corpus] manifest = "corpora/tokei/MANIFEST.sha256"