benbrown/peltier.io
2
1
Fork 0
mirror of https://github.com/bdbrown4/peltier.io.git synced 2026-11-07 10:58:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

gates: close the audit findings — enforce the gates the docs claimed

Browse Source 
An adversarial audit found the enforcement story ahead of the code: gates
that were documented as hard gates were advisory, manual, or absent, and
several human-facing claims outran the machine record. This makes the
claims true.

High findings

- Differential fuzz was documentation, not enforcement: verdict hardcoded
  fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case
  studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real
  pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and
  it is inherently baseline-vs-candidate, so it runs on the accept path and
  reports Skipped in the standalone `just gates` flow. New hard rule: a
  machine `accepted` verdict is impossible unless the fuzz gate actually
  passed — otherwise it caps at needs-human-review.
- The agent's system prompt claimed FP/concurrency/unsafe patches were
  "routed to human review automatically". No such mechanism existed. Added
  a conservative lexical risk classifier (crates/verdict/src/risk.rs) that
  forces needs-human-review on any would-be accept carrying those signals;
  fp-tolerance equivalence always signals.
- The local verdict path ran agent-influenced code unsandboxed. It now runs
  under scripts/no-net.sh (network namespace) and fails closed. Upstream
  test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when
  present (scripts/pin-testsuite.sh generates them).

Also fixed, found while fixing the above

- The pipeline script's `set -e` was inert (brace group as the left operand
  of `&&`), so a failing candidate build did not abort the run and verdict
  could bench a stale binary. Now a scoped subshell whose status is branched
  on at top level.
- A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path)
  stranded RUNNING.lock and left read_verdict polling forever. Failure
  accounting now lives outside the wrapper.
- harnessd's patch allowlist ignored git rename/copy headers and --no-prefix
  diffs; the rename-escape case is now a committed regression test.
- The workspace did not compile on Windows (ungated Unix APIs). cfg-gated;
  71 portable tests now run on both platforms.
- pin-check regenerated the corpus manifest instead of verifying it; it is
  now verify-only, with a deliberate `just pin-corpus` to re-pin.
- Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics
  (compare --perf-stat), never part of the accept decision.

Docs now match the code, including what is NOT yet enforced: no target ships
a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not
wired to the fp-tolerance gate — each is stated as mechanized-but-inert
rather than done. Corrected the stale "33 ledger rows" (34), the playbook's
class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation
claim (five accepted rows carry sanitizers_clean=false), the unsupported
"40,000 requests" figure, and the blanket "historical rows record
fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
benbrown
2026-07-14 13:10:14 -04:00
co-authored by Claude Opus 4.8
parent 21352efe6c
commit 899d55f4b6
57 changed files with 3308 additions and 692 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.gitattributes
+4
View File
@@ -0,0 +1,4 @@
# Hash manifests and shell scripts must stay LF even on Windows checkouts —
# sha256sum -c and POSIX sh both break on CRLF working-tree copies.
*.sha256 text eol=lf
*.sh text eol=lf
.gitignore
+3
View File
@@ -234,7 +234,9 @@ targets/comrak/asan*/
targets/tokei/candidate-*/
targets/tokei/asan/
!results/calibration/*.json
!results/phase4/*.json
targets/*/baseline/
targets/*/fuzz-work/
# Phase 2 agent artifacts (transcripts kept; pending logs are scratch)
results/pending/
@@ -250,3 +252,4 @@ targets/cjson/asan*/
targets/cjson/svc-baseline/
targets/cjson/svc-candidate/
docs/book/
CLAUDE.md
+33 -22
View File
@@ -16,13 +16,13 @@ Full engineering spec: `SPEC.md`. Read it before starting any phase.
4. **Equivalence gates are hard gates.** Changes touching floating-point ordering, concurrency primitives, or anything UB-adjacent are NEVER auto-accepted — verdict is `needs-human-review`.
5. **Every attempt goes in the ledger**, including failures and rejections. Failures are future training data.
6. **The agent never modifies the trust layer.** `crates/`, `config/`, `corpora/`, and upstream test suites are read-only to the agent. Patches may touch only allowlisted paths under `targets/<name>/`. See SPEC.md §10.
7. **Sandbox everything.** Target code runs only inside no-network containers. Never on the host.
7. **Sandbox everything.** Target code must never run unisolated on the host. *As built:* the harnessd-launched verdict pipeline execs through `scripts/no-net.sh` (Linux network namespace; **fails closed** — exit 97 — when namespaces are unavailable, unless `HOTPATH_ALLOW_UNISOLATED=1`, which runs it **with** network and stamps that fact into the ledger's isolation note). CI runs the bench workload under `docker run --network=none`. Two honest limits: a human invoking `just verdict` directly is **not** wrapped, and full-container (seccomp-restricted) isolation of the whole pipeline is still an **open gap**. See SPEC §10.
8. **Statistical bar:** accept only if the bootstrap 95% CI lower bound of speedup ≥ threshold in `config/accept.toml` (default 2%), from interleaved A/B runs on the same pinned hardware.
## Architecture map
- `crates/bench-runner` — trust layer: containerized, CPU-pinned, interleaved A/B benchmarking with bootstrap confidence intervals and A/A self-tests
- `crates/diff-test` — behavioral equivalence: upstream test suite + golden I/O replay + differential fuzzing of changed functions + sanitizers
- `crates/bench-runner` — trust layer: CPU-pinned, interleaved A/B benchmarking with bootstrap confidence intervals and A/A self-tests (it does **not** containerize anything itself — isolation is the wrapper's job, non-negotiable 7)
- `crates/diff-test` — behavioral equivalence: corpus/test-suite pin checks + upstream test suite + golden I/O replay + differential fuzz (a baseline-vs-candidate gate; sanitizers run on the accept path in `crates/verdict`, and `diff-test` reports them as skipped-with-reason)
- `crates/ledger` — append-only SQLite record of every attempt: hypothesis, patch, gate results, bench deltas, verdict, cost
- `crates/report` — ROI generator: cores saved × $/core-hr, latency percentile deltas, CIs, workload caveats printed on every number
- `agent/` — Claude Agent SDK (Python) loop, prompts, tool definitions
@@ -30,44 +30,55 @@ Full engineering spec: `SPEC.md`. Read it before starting any phase.
- `playbook/` — optimization classes, ordered, with preconditions and known risks
- `results/` — generated reports and flamegraphs
**Phase 4 COMPLETE — all SPEC §5 exit criteria met. Phases 0–4 are all done.** Service-mode latency benchmarking shipped: a coordinated-omission-correct, open-loop, fixed-rate load generator (`crates/bench-runner/src/service.rs`) measuring latency from each request's intended send time (queueing counted, not hidden), interleaved A/B, exact percentiles, bootstrap p50/p99 CIs. A real service target — `targets/cjson/service.c`, a minimal HTTP server wrapping the patched `cJSON.c` (trust-layer, outside the workspace allowlist). Service calibration PASS (0/10 A/A false-positive, 10/10 injected-5%-latency-regression detection; `results/calibration/cjson-service-aa.json`). **The verified batch win phase3-cjson-002 measured under a 150 rps replay (20 rounds, 40k requests, 0 drops): p50 latency +6.2%, 95% CI [+5.8%, +7.2%] — accepted (`phase4-cjson-service` ledger row); p99 NOT claimed (CI [0.07, 4.97], single-worker loopback tail jitter) — harness correctly rejected it.** Mechanical ROI report generator (`crates/report` bin, `just report`, SPEC §9): reads a ledger row + service-latency JSON → throughput→cores→dollars (27.5 cores / $9,621-per-year CI lower bound on a 500-core fleet) *and* latency percentiles, every figure with CI + workload + methodology inline; flags any non-accepted row. Case study: `results/phase4/`. **The roadmap is complete — the profile→hypothesize→patch→gated-verdict→ROI loop runs end to end across Rust and C, batch and service, with zero shipped false accepts across all phases.**
**Phase 4 COMPLETE — all SPEC §5 exit criteria met. Phases 0–4 are all done.** Service-mode latency benchmarking shipped: a coordinated-omission-correct, open-loop, fixed-rate load generator (`crates/bench-runner/src/service.rs`) measuring latency from each request's intended send time (queueing counted, not hidden), interleaved A/B, exact percentiles, bootstrap p50/p99 CIs. A real service target — `targets/cjson/service.c`, a minimal HTTP server wrapping the patched `cJSON.c` (trust-layer, outside the workspace allowlist). Service calibration PASS (0/10 A/A false-positive, 10/10 injected-5%-latency-regression detection; `results/calibration/cjson-service-aa.json`). **The verified batch win phase3-cjson-002 measured under a 150 rps-target replay (20 interleaved rounds, 0 drops — the `phase4-cjson-service` ledger row records sessions=20, rate_rps=150, drop_rate=0.0; total request count is not machine-recorded): p50 latency +6.2%, 95% CI [+5.8%, +7.2%] — accepted; p99 NOT claimed (CI [0.07, 4.97], single-worker loopback tail jitter) — harness correctly rejected it.** Mechanical ROI report generator (`crates/report` bin, `just report`, SPEC §9): reads a ledger row + service-latency JSON → throughput→cores→dollars (27.5 cores / $9,621-per-year CI lower bound on a 500-core fleet) *and* latency percentiles, every figure with CI + workload + methodology inline; flags any non-accepted row. Case study: `results/phase4/`. **The roadmap is complete — the profile→hypothesize→patch→gated-verdict→ROI loop runs end to end across Rust and C, batch and service, with zero shipped false accepts across all phases.**
**Phase 5 COMPLETE — the two SPEC §13 research forks, both built and both routed to human review.** (a) *Learned class-selection policy* (`crates/policy`, `cargo run -p policy`): reads the append-only ledger and ranks optimization classes by the Wilson lower bound of their shippable-win rate — a learned prior over the fixed cheapest-first ordering, advisory only (gates still decide). A "win" counts only machine-sanitizer-verified accepts; the overturned comrak-010 and tier-gated mimalloc are excluded/held. On the current 34-row ledger it recommends algorithmic first (Wilson lb 0.066). (b) *Kernel lane* (`targets/matmul/kernel.c`, `crates/diff-test` FP-tolerance policy + `fp-compare` bin, `scripts/kernel-lane-demo.sh`): a matmul optimization (transpose-B cache locality + eight-accumulator ILP) that **reorders FP accumulation**, so byte-identical golden replay is the wrong gate — 244,901/262,144 values differ. The FP-tolerance gate (`abs 1e-4 + rel 1e-3`) accepts it, still REJECTS a +0.5 perturbation, and the bench measures 3.23× [3.16, 3.26] with the same interleaved A/B + bootstrap CI. Ledger row `phase5-matmul-opt` = needs-human-review (using the tolerance tier is a §8 signal, same posture as mimalloc). No GPU in this environment, so the GPU fork is the identical shape shown on CPU — only the timer and hardware change. Case study: `results/phase5/`; docs chapter `docs/src/research-forks.md`. Open follow-up: the standing accept-scoped ruling on `phase0-comrak-002` (mimalloc) is resolved (`results/rulings/`).**
**Hardening pass (2026-07-13, `audit/resolve-findings`) — what is *enforced* vs. what is only *mechanized*.** Five gates were addressed in code. **Two are live on the verdict path (a, b); three are wired but currently inert, because no target has opted in (c, d, e).** The distinction is stated precisely because collapsing it is exactly the overclaim this project exists to prevent.
- **(a) Differential fuzz — live, and now blocks accepts.** `diff-test` runs the target's declared `[gates].fuzz` command and grades it strictly from a `FUZZ-RESULT iters=<n> divergences=<m>` line (no line = Failed, not a pass by silence); the ledger records the iteration count **actually executed**. It is a **baseline-vs-candidate** gate, so it runs *only* on the accept path (`crates/verdict`, the one flow that rebuilds a pristine baseline). The standalone `just gates` flow has no baseline, so the gate reports **Skipped with that reason** — `just gates` is a fast equivalence check, **not** a fuzz run. **Hard rule:** a machine `accepted` verdict is **impossible** unless the DifferentialFuzz gate actually *Passed*; skipped or failed caps the verdict at `needs-human-review`. `cjson`/`comrak`/`tokei` declare `[gates].fuzz`; `matmul` does not. *Ledger history:* fuzz ran out-of-band via `scripts/diff-fuzz-*.py` through Phase 5, so **8 of the 10 accepted rows record `fuzz_iters=0`** — the two exceptions, `phase0-tokei-002` and `phase0-tokei-003`, record `fuzz_iters=4332`.
- **(b) Lexical risk classifier — live.** `crates/verdict/src/risk.rs` scans the patch's **changed lines** for concurrency / `unsafe` / floating-point tokens; any signal on a would-be accept forces `needs-human-review`, and using fp-tolerance equivalence mode is itself always a signal. It is **lexical and deliberately over-triggering, not semantic** — it detects the presence of risk markers; it does not understand the code and cannot prove their absence. `harnessd` always passes `--patch-file`, so every agent-proposed patch is classified.
- **(c) Test-suite pinning — mechanized, INERT.** `diff-test` verifies `corpora/<t>/TESTSUITE.sha256` **when present** (mismatch = hard refusal, same posture as the corpus pin) and prints an explicit "suite unpinned" warning when absent. **No target ships a pin today — there are zero `TESTSUITE.sha256` files in the repo** — so the gate constrains nothing yet. Pins must be generated per target, post-fetch, by a deliberate `scripts/pin-testsuite.sh` run.
- **(d) TSan lane — mechanized, INERT.** `crates/verdict` builds and runs a TSan lane on the accept path **only when a target declares `[build].tsan`**. **No target declares it today, so no TSan coverage exists.** (ASan/UBSan *is* live on the accept path: a flagged run — or a target with no sanitizer build at all — caps the accept at `needs-human-review`.)
- **(e) FP-tolerance equivalence — mechanized in the pipeline, target NOT wired.** The gate pipeline honors an fp-tolerance policy (`targets/<t>/equivalence.toml`; requires `[corpus].golden_reference`), and any fp-tolerance run auto-routes to `needs-human-review`. But the only FP target, `matmul`, has an `equivalence.toml` and **no `target.toml`** — so it is not a pipeline target at all, and the kernel lane stays script-driven (`scripts/kernel-lane-demo.sh`).
**Phase 5 COMPLETE — the two SPEC §13 research forks, both built and both routed to human review.** (a) *Learned class-selection policy* (`crates/policy`, `cargo run -p policy`): reads the append-only ledger and ranks optimization classes by the Wilson lower bound of their shippable-win rate — a learned prior over the fixed cheapest-first ordering, advisory only (gates still decide). A "win" counts only machine-sanitizer-verified accepts; the overturned comrak-010 and tier-gated mimalloc are excluded/held. On the current 34-row ledger it recommends algorithmic first (Wilson lb 0.066). (b) *Kernel lane* (`targets/matmul/kernel.c`, `crates/diff-test` FP-tolerance policy + `fp-compare` bin, `scripts/kernel-lane-demo.sh`): a matmul optimization (transpose-B cache locality + eight-accumulator ILP) that **reorders FP accumulation**, so byte-identical golden replay is the wrong gate — 244,901/262,144 values differ. The FP-tolerance policy (`abs 1e-4 + rel 1e-3`) accepts it, still REJECTS a +0.5 perturbation, and the bench measures 3.23× [3.16, 3.26] with the same interleaved A/B + bootstrap CI. Ledger row `phase5-matmul-opt` = needs-human-review (using the tolerance tier is a §8 signal, same posture as mimalloc). **Honest scope:** the FP-tolerance *mechanism* lives in the gate pipeline, but `matmul` is **not wired into it** — it has an `equivalence.toml` and no `target.toml`, so `diff-test`/`verdict` cannot load it. The lane is demonstrated script-driven (`fp-compare` + `bench-runner compare`), and that ledger row was recorded outside the automated verdict path. Wiring `matmul` as a real pipeline target is open work. No GPU in this environment, so the GPU fork is the identical shape shown on CPU — only the timer and hardware change. Case study: `results/phase5/`; docs chapter `docs/src/research-forks.md`. Open follow-up: the standing accept-scoped ruling on `phase0-comrak-002` (mimalloc) is resolved (`results/rulings/`).**
## Commands
Fill these in as the harness is built; keep them current:
```
just profile <target> # perf record + flamegraph for a target's benchmark workload
just bench <target> <ref> # interleaved A/B: pristine baseline vs working tree
just gates <target> # test suite + golden replay + diff-fuzz + sanitizers
just verdict <target> # runs gates + bench, writes ledger row, prints verdict
just report <run-id> # ROI report from ledger
just aa <target> # A/A calibration run (must show null result)
```
Built so far (command-level, pre-target-integration):
The real recipe list (`justfile`); keep it current:
```
just build / test / lint # trust-layer workspace: cargo build/test/clippy+fmt
just aa [cmd] # A/A self-test of bench-runner on a shell command
just compare <a> <b> # interleaved A/B of two shell commands, bootstrap CI
just gates <target> # corpus pin + upstream tests + golden replay
just pin-check <target> # verify corpus MANIFEST.sha256
just gates <target> # corpus pin (+ suite pin if present) + upstream tests + golden replay.
# diff-fuzz SKIPS here (needs a baseline) — it runs in `just verdict`
just pin-check <target> # verify corpus MANIFEST.sha256 (verify-only, never re-pins)
just pin-corpus <target> # deliberate corpus re-pin (writes MANIFEST.sha256)
just calibrate <cmd> <out> # automated A/A + regression-injection calibration
just verdict <t> <bin> ... # gates + bench vs pristine-rebuilt baseline + ledger row
cargo run -p harnessd # agent IPC daemon: six-tool JSON surface (SPEC §3.5)
just report <run-id> # ROI report from a ledger row (CIs + methodology inline)
just agent-attempt <t> <id> # one unattended agent attempt behind the OS boundary
just isolation-check # verify the SPEC §10 OS boundary (19 checks, both modes)
just service <b> <c> <doc> # service-mode latency A/B, CO-correct, p50/p99 CIs
just service-calibrate ... # latency A/A + injected-regression calibration
just coz <target> # causal profile of a C/C++ target
cargo run -p harnessd # agent IPC daemon: seven-op JSON surface (SPEC §3.5)
cargo run -p policy # learned class-ranking prior from the ledger (advisory)
```
**Platform note:** the harness is Linux/POSIX-only at runtime — bench-runner
service mode, harnessd (Unix sockets, `setsid`), and every sh-based gate and
isolation script assume a Unix host. Windows compiles the workspace and runs
the portable unit tests, but cannot run the pipeline.
## Definition of done — one optimization attempt
- [ ] Hypothesis logged in ledger *before* patching
- [ ] Patch touches only allowlisted paths under `targets/<name>/`
- [ ] Upstream test suite green
- [ ] Golden replay byte-identical (or within the target's explicit FP tolerance policy)
- [ ] Differential fuzz on changed functions: no divergence (60s or 10k iterations minimum)
- [ ] ASan + UBSan clean; TSan if the patch touches anything threaded
- [ ] Differential fuzz on changed functions: no divergence (60s or 10k iterations minimum). Requires `[gates].fuzz` in the target's `target.toml` **and** the accept path (`just verdict`) — an accept is impossible without a Passed fuzz gate
- [ ] ASan + UBSan clean (machine-enforced on accept). TSan if the patch touches anything threaded — note the lane only runs when the target declares `[build].tsan`, and **no target declares one today**, so a threaded patch needs that opt-in added first
- [ ] Interleaved A/B bench passes the CI-lower-bound threshold
- [ ] Ledger row written with verdict and full evidence
- [ ] If FP-ordering / concurrency / UB-adjacent: verdict `needs-human-review`, never auto-accept
README.md
+7 -2
View File
@@ -21,7 +21,7 @@ hostile review.
## Verified results
Every number carries its 95% CI and its workload, and every one survived the
gates above. Across **five phases and 33 ledger rows, zero shipped false
gates above. Across **five phases and 34 ledger rows, zero shipped false
accepts** — including two pipeline over-accepts the audit caught before
anything shipped, each becoming a permanent new gate.
@@ -58,11 +58,16 @@ docs/ The mdBook site (deployed to GitHub Pages)
just build / test / lint # trust-layer workspace
just aa # A/A self-test — must yield a null verdict
just gates <target> # corpus pin + upstream tests + golden replay
just verdict <t> <bin> ... # gates + bench vs pristine baseline + ledger row
# (fuzz needs a baseline — it runs in `just verdict`)
just verdict <t> <bin> ... # gates + fuzz + sanitizers + bench vs pristine baseline
just report <run-id> # ROI report from a ledger row
just isolation-check # 19 OS-boundary checks (both modes)
```
The pipeline is **Linux/POSIX-only** at runtime (Unix sockets, `setsid`,
sh-based gates). Windows builds the workspace and runs the portable unit
tests, but cannot run the pipeline.
Full command reference and reproduction steps:
[Reproduce it yourself](https://bdbrown4.github.io/peltier.io/reproduce.html).
SPEC.md
+26 -2
View File
@@ -43,6 +43,8 @@ Requirements:
- **A/A self-test mode**: run the same binary as both sides; must produce a null verdict. Scheduled automatically before any measurement session.
- Regression-injection self-test: a synthetic 5% slowdown must be detected ≥95% of the time.
- Metrics captured per run: wall time, cycles, instructions, cache misses, branch misses (via `perf stat`), max RSS, and RAPL energy where available.
> **Amended 2026-07-13.** As built: **wall time is the only metric a verdict is decided on.** Max RSS is captured per timed run on Unix (`wait4` / `ru_maxrss`) and recorded as baseline/candidate medians in the env fingerprint — it is **context, never an accept metric**. PMU counters (cycles, instructions, cache misses, branch misses) are opt-in diagnostics via `bench-runner compare --perf-stat` — printed with the workload caveat, never part of the accept decision, and unavailable in most VMs. RAPL energy capture was never implemented. The **"no-network container"** named in the environment-control bullet above is *not* what ships — see the §10 amendment for the isolation actually built (a network namespace on the agent path only).
- Modes: (a) whole-program via hyperfine-style CLI timing; (b) microbench via criterion harness hooks; (c) service mode (Phase 4): wrk2 / vegeta load replay with **coordinated-omission-aware** latency recording (HdrHistogram).
Acceptance: passes A/A calibration at <5% false-positive rate and injected-regression detection at ≥95%, documented in `results/calibration/`.
@@ -57,6 +59,12 @@ Layered, all must pass:
Corpus and test files live outside agent-writable paths and are hash-pinned; `diff-test` refuses to run if hashes mismatch.
> **Amended 2026-07-13 — layer 3 (differential fuzz) as actually built.** Fuzz is a **baseline-vs-candidate** gate: it needs both binaries, so it runs *only* on the accept path (`verdict`, the one flow that rebuilds a pristine baseline). The standalone `diff-test` / `just gates` flow has no baseline and records the gate as **Skipped, with that reason** — it is never faked by comparing the candidate against itself. `diff-test` executes the target's declared `[gates].fuzz` command and grades it strictly from a `FUZZ-RESULT iters=<n> divergences=<m>` line; a run that never prints the line is a **Failed** gate, not a pass by silence. The ledger records the iteration count actually executed. **Hard rule:** a machine `accepted` verdict is impossible unless this gate *Passed* — skipped or failed caps the verdict at `needs-human-review`. `cjson`, `comrak`, and `tokei` declare `[gates].fuzz`; `matmul` does not. The per-changed-function scoping and the 60s/10k minimum remain aspirational: the shipped fuzzers are per-target whole-binary differential drivers (`scripts/diff-fuzz-*.py`) with a configurable iteration budget.
> **Amended 2026-07-13 — layer 4 (sanitizers) as actually built.** ASan+UBSan is machine-enforced on the accept path: `verdict` builds the target's `[build].sanitizer` binary, runs the pinned workload under it, and caps a flagged run — or a target that declares **no** sanitizer build — at `needs-human-review`. The **TSan lane is opt-in per target** (`[build].tsan` + `[build].tsan_binary`) rather than diff-triggered, and **no target declares it today**, so the lane is wired but **inert and no TSan coverage currently exists**; a threaded patch requires adding that opt-in first. **MIRI was never implemented.**
> **Amended 2026-07-13 — test-suite pinning: mechanized, not yet active.** As built, upstream test suites are vendored inside `targets/<t>/workspace/` — an agent-writable path — so "unmodified" is enforced by pinning, not placement: `corpora/<t>/TESTSUITE.sha256` (generated only by a deliberate human run of `scripts/pin-testsuite.sh`) is verified by `diff-test` before the suite runs; a hash mismatch is a hard refusal, same posture as the corpus pin. Targets without a pin file run with an explicit "suite unpinned" warning, and `targets/fetch.sh` verifies the pin after every fetch. **Status: no target ships a pin — there are zero `TESTSUITE.sha256` files in the repo — so this gate is currently inert and constrains nothing.** It becomes real only when pins are generated per target after a fetch. (The *corpus* pin, `MANIFEST.sha256`, ships for every target and **is** enforced today.)
### 3.3 `ledger`
Append-only SQLite. One row per attempt:
@@ -70,7 +78,7 @@ Phase 0–2: `perf record` + flamegraphs (inferno) + `perf stat`; hotspot rankin
### 3.5 `agent/`
Claude Agent SDK (Python). Tools exposed to the model: `read_profile`, `read_ledger`, `read_playbook`, `propose_patch(diff, hypothesis)`, `run_verdict`, `read_target_source` (read-only). Deliberately **not** exposed: shell on the host, writes outside `targets/<name>/`, any access to `crates/`, `config/`, `corpora/`.
Claude Agent SDK (Python). Tools exposed to the model — seven: `read_profile`, `read_ledger`, `read_playbook`, `read_target_source` (read-only), `propose_patch(diff, hypothesis)`, `run_verdict`, `read_verdict`. Deliberately **not** exposed: shell on the host, writes outside `targets/<name>/`, any access to `crates/`, `config/`, `corpora/`. *(Amended 2026-07-13: `run_verdict` launches the pipeline detached and returns immediately; the pollable `read_verdict` — added in Phase 2 when the pipeline outlived the MCP transport timeout — reads the result, making seven operations total.)*
Prompting spine: (1) state hypothesis before patching; (2) prefer the cheapest untried playbook class; (3) max two iterations per rejected hypothesis; (4) a rejection with a clean ledger row is a successful outcome.
@@ -79,7 +87,7 @@ Prompting spine: (1) state hypothesis before patching; (2) prefer the cheapest u
Consumes ledger rows, emits per-engagement ROI report:
- Throughput jobs: cores saved = fleet_cores × (1 − 1/speedup); $ = cores × $/core-hr × hours/yr (rates in `config/pricing.toml`).
- Services: p50/p95/p99 deltas under replayed traffic, with the business-value mapping left as a customer-supplied parameter and one worked example.
- Optional: joules saved (RAPL) for the sustainability line.
- Optional: joules saved (RAPL) for the sustainability line. *(Amended 2026-07-13: never implemented — RAPL energy is not captured anywhere in the harness, so no report emits a joules figure. See the §3.1 amendment.)*
- Every figure carries CI + workload description + environment fingerprint. Caveats are printed on the report, not in an appendix.
## 4. Optimization playbook v0 (strict order)
@@ -106,6 +114,8 @@ Each class ships as `playbook/NN-name.md`: preconditions (profile signature that
Language order: Rust first (criterion/proptest/MIRI make Phases 1–2 cheaper), C/C++ second (the richer legacy hunting ground), services third.
> **Amended 2026-07-13.** The Phase 2 criterion as written — "zero false accepts across ≥20 audited attempts" — was violated once *in-pipeline*: `phase2-comrak-010` was falsely accepted by the pipeline (no machine sanitizer gate existed yet), caught by the 100% human audit, overturned, and never shipped. The criterion was met in the form **zero *shipped* false accepts**; the overturn drove the machine-enforced sanitizer gate and stands documented in the Phase 2 case study.
## 6. Target selection criteria (Phase 0–3)
CPU-bound (>70% user time on profile); meaningful test suite; existing benchmark or easily scripted workload; permissive license; active but not hyper-optimized (skip anything with a dedicated perf team); builds cleanly in container. Good hunting: codecs, parsers, serializers, image processing, compression, static-site generators, linters.
@@ -123,6 +133,8 @@ CPU-bound (>70% user time on profile); meaningful test suite; existing benchmark
- **needs-human-review** (never auto-accepted): FP reassociation/contraction or `-ffast-math`-class flags; any diff touching atomics, locks, memory ordering, or thread counts; unsafe blocks / raw pointer arithmetic introduced; anything the sanitizers flag even as warnings.
- **Auto-rejected**: any gate failure, any divergence under differential fuzz, patch outside allowlist.
> **Amended 2026-07-13.** The needs-human-review tier is machine-routed, not honor-system: `verdict` runs a **conservative lexical risk classifier** over the patch's changed lines (substring token lists for concurrency, unsafe, and floating-point signals) and forces any would-be accept carrying a signal to `needs-human-review`; using **fp-tolerance equivalence mode** at all also auto-routes to review; the manual `--needs-human-review` flag remains. The classifier is lexical and deliberately over-triggering — it cheaply catches the presence of risk markers, it does not prove their absence.
## 9. ROI methodology (the pitch artifact)
The report answers one question: *what did the stopwatch say, and what does that cost or buy?* Throughput → cores → dollars via public cloud pricing; latency → percentile deltas under recorded production-shaped load, priced by the customer's own latency-value number; batch windows → wall-clock hours returned to the schedule. All three come with CIs and a workload statement. The methodology section ships *inside* every report so the number survives hostile review — that is the differentiation.
@@ -136,6 +148,14 @@ The agent's incentive is "make the number go up"; the design must make cheating
- Target code executes only in no-network containers (bench containers additionally seccomp-restricted).
- Periodic human audit of accepted wins (100% in Phase 2, sampled thereafter); any false accept is a stop-the-line event.
> **Amended 2026-07-13 — isolation as actually built.** The bullet above ("target code executes only in no-network containers, seccomp-restricted") is **not** what ships. What ships:
>
> - **The `harnessd`-launched verdict pipeline** — the agent path — execs through `scripts/no-net.sh`, a **network-namespace** wrapper (`unshare --net --map-current-user`). `HOTPATH_VERDICT_WRAPPER` overrides the wrapper.
> - **It fails closed.** Where namespaces are unavailable the wrapper **exits 97 without running the pipeline**, rather than degrading silently. The only bypass is the explicit `HOTPATH_ALLOW_UNISOLATED=1` override — which runs the pipeline **with full network access**, warns loudly on stderr, and is recorded verbatim in the ledger's isolation note (`"no-net.sh (HOTPATH_ALLOW_UNISOLATED=1 — network NOT isolated)"`), so a run can never *claim* an isolation it did not have.
> - **In CI**, the bench workload runs under `docker run --network=none`.
>
> **Two gaps, stated plainly.** (1) **Only the harnessd (agent) path is wrapped.** A human invoking `just verdict` directly runs **unwrapped** on the host; such rows record `isolation: "unwrapped-host"`. (2) **Full-container isolation of the whole pipeline remains open** — the wrapper isolates the *network*, not the filesystem or the syscall surface; the seccomp-restricted bench container of this section is not built. The agent-side OS boundary (mount-namespace / unprivileged-uid isolation, `just isolation-check`) is a separate mechanism and *is* shipped.
## 11. Risk register
| Risk | Mitigation |
@@ -149,10 +169,14 @@ The agent's incentive is "make the number go up"; the design must make cheating
| License/IP hygiene | Permissive-license targets only for public case studies |
| Wins too small to matter | Cheap-wins-first ordering keeps cost-per-attempt low; ledger prevents re-grinding dead ends |
> **Amended 2026-07-13 — which mitigations above are actually live.** The "Concurrency bugs introduced" and "UB introduced" rows overstate the machine coverage. Live today: **UBSan** (bundled with ASan on the accept path, machine-enforced) and the **human-review tier**, which is machine-routed by the lexical risk classifier of the §8 amendment — so a concurrency-token or `unsafe`-token patch *is* forced to `needs-human-review`, conservatively and lexically. **Not live: the TSan gate** (opt-in via `[build].tsan`; no target declares it) and **MIRI** (never implemented). The concurrency and UB rows therefore rest on the review tier and UBSan, not on TSan/MIRI.
## 12. Tech stack
Rust workspace (`bench-runner`, `diff-test`, `ledger`, `report`) · `just` · hyperfine-style timing + criterion hooks · perf, inferno flamegraphs, coz · proptest, cargo-fuzz, libFuzzer · ASan/UBSan/TSan, MIRI · podman/docker (no-net) · wrk2 + HdrHistogram, vegeta · SQLite · Python 3.11+ with Claude Agent SDK (`agent/`) — confirm current SDK package name and auth at https://docs.claude.com/en/docs/claude-code/overview.
> **Amended 2026-07-13 — the stack as actually used.** This list is the *planned* stack, not an inventory. In use: the Rust workspace (plus `verdict`, `harnessd`, `policy`), `just`, in-house interleaved timing and an in-house open-loop load generator (**not** hyperfine/criterion/wrk2/vegeta), valgrind-callgrind and coz for profiling (`perf` is often unavailable in the container), Python differential fuzzers (`scripts/diff-fuzz-*.py` — **not** proptest/cargo-fuzz/libFuzzer), ASan+UBSan, SQLite, and the Agent SDK. **Not used: MIRI (never implemented), TSan (wired but no target opts in), and podman/docker for the local verdict path** — local isolation is a network namespace (`scripts/no-net.sh`); Docker `--network=none` is used in CI.
## 13. Later / research fork (explicitly out of scope now)
- GPU kernel lane (where 2–10x gaps live): Triton/CUDA targets, correctness via reference-kernel differential testing, reward = measured kernel time.
agent/hotpath_agent/loop.py
+2 -2
View File
@@ -1,7 +1,7 @@
"""Phase 2 agent loop (SPEC §3.5): Claude Agent SDK driving the six
"""Phase 2 agent loop (SPEC §3.5): Claude Agent SDK driving the seven
harness tools over a standalone stdio MCP server. The agent gets NO
built-in tools — no shell, no file writes; every capability is one of
the six IPC operations, and the trust layer's allowlist/gates remain
the seven IPC operations, and the trust layer's allowlist/gates remain
the hard boundary (SPEC §10).
Run one attempt:
agent/hotpath_agent/mcp_server.py
+1 -1
View File
@@ -1,4 +1,4 @@
"""Standalone MCP stdio server exposing the six harness tools (SPEC §3.5).
"""Standalone MCP stdio server exposing the seven harness tools (SPEC §3.5).
Spawned as a subprocess by the agent loop; speaks MCP over stdio. Wraps
the harnessd IPC client — the trust-layer allowlist and gates remain the
agent/hotpath_agent/prompts.py
+8 -4
View File
@@ -24,8 +24,12 @@ Rules, in priority order:
Never restructure a patch to influence measurement rather than to
express the hypothesis.
5. Equivalence is not yours to relax. Floating-point flag changes,
concurrency-primitive changes, and new unsafe code are routed to
human review automatically; do not attempt to phrase a patch to
avoid that routing.
5. Equivalence is not yours to relax. The trust layer (crates/verdict)
runs a conservative lexical risk classifier over every patch's
changed lines: concurrency, unsafe, and floating-point signals force
any would-be accept to needs-human-review, and fp-tolerance
equivalence mode always routes to human review. The classifier is
lexical and deliberately over-triggering, not a semantic analysis;
a human --needs-human-review flag also exists. Do not attempt to
phrase a patch to slip past that routing.
"""
agent/hotpath_agent/tools.py
+1 -1
View File
@@ -1,6 +1,6 @@
"""Tool surface exposed to the model (SPEC §3.5).
Exactly six tools, all mediated by harnessd over line-delimited JSON —
Exactly seven tools, all mediated by harnessd over line-delimited JSON —
the agent process has no direct filesystem or shell capability beyond
what these return. Deliberately NOT exposed: shell on the host, writes
outside targets/<name>/, any access to crates/, config/, corpora/.
config/accept.toml
+3
View File
@@ -23,3 +23,6 @@ bootstrap_seed = 1887343589
# Calibration evidence for this machine shows pinning tightens the A/A
# CI ~5x (results/calibration/). Empty string disables.
pin_prefix = "taskset -c 2"
# Differential-fuzz iterations requested per gate run (SPEC §3.2).
# fuzz_iters = 10000
corpora/README.md
+9 -2
View File
@@ -8,8 +8,15 @@ Every corpus directory carries a `MANIFEST.sha256` (`sha256sum` format:
before every run and refuses to run on any mismatch — a tampered corpus
is a stop-the-line event.
Regenerate a manifest only as a deliberate human action:
`just pin-check <target>` regenerates the corpus and verifies it against
the manifest; it never rewrites the manifest. Regenerate a manifest only
as a deliberate human action:
```sh
cd corpora/<target> && find . -type f ! -name MANIFEST.sha256 -exec sha256sum {} + > MANIFEST.sha256
just pin-corpus <target> # gen-corpus.sh --pin
```
A directory may also carry a `TESTSUITE.sha256` pinning the upstream test
suite (repo-root-relative paths). `diff-test` verifies it before running
upstream tests and `targets/fetch.sh` verifies it after every fetch;
generate it with `scripts/pin-testsuite.sh <target> <path>...`.
corpora/cjson/gen-corpus.sh
+19 -3
View File
@@ -4,8 +4,16 @@
# with escapes and unicode, ints, floats (the strtod/sprintf hot path),
# booleans, nulls. No randomness — byte-identical on every machine, so
# the MANIFEST.sha256 pin is reproducible. Run from repo root:
# sh corpora/cjson/gen-corpus.sh
# sh corpora/cjson/gen-corpus.sh [--check|--pin]
# Default/--check: regenerate and verify against MANIFEST.sha256 (never
# rewrites it). --pin: deliberately rewrite the manifest.
set -eu
mode="check"
case "${1:-}" in
"" | --check) ;;
--pin) mode="pin" ;;
*) echo "usage: gen-corpus.sh [--check|--pin]" >&2; exit 2 ;;
esac
DIR="$(cd "$(dirname "$0")" && pwd)"
OUT="$DIR/input/big.json"
mkdir -p "$DIR/input"
@@ -57,6 +65,14 @@ with open(sys.argv[1], "w") as f:
PY
echo "wrote $OUT ($(wc -c < "$OUT") bytes)"
# Pin the corpus and regenerate the manifest.
if [ "$mode" = "pin" ]; then
( cd "$DIR/input" && find . -type f -print0 | sort -z | xargs -0 sha256sum ) > "$DIR/MANIFEST.sha256"
echo "manifest: $DIR/MANIFEST.sha256"
echo "re-pinned $DIR/MANIFEST.sha256 — a deliberate human action (corpora/README.md);"
echo "commit the new manifest with justification."
else
if [ ! -f "$DIR/MANIFEST.sha256" ]; then
echo "no MANIFEST.sha256 — pin the corpus with: sh corpora/cjson/gen-corpus.sh --pin" >&2
exit 1
fi
( cd "$DIR/input" && sha256sum -c ../MANIFEST.sha256 )
fi
corpora/comrak/gen-corpus.sh
+14
View File
@@ -1,7 +1,21 @@
#!/bin/sh
# Regenerate the comrak bench corpus from the pinned progit checkout
# (targets/comrak/target.toml [source.submodules]), then verify the pin.
# Default/--check verifies MANIFEST.sha256 (never rewrites it); --pin
# deliberately rewrites it.
set -eu
mode="check"
case "${1:-}" in
"" | --check) ;;
--pin) mode="pin" ;;
*) echo "usage: gen-corpus.sh [--check|--pin]" >&2; exit 2 ;;
esac
cd "$(dirname "$0")"
cat ../../targets/comrak/workspace/vendor/progit/*/*/*.markdown > progit.md
if [ "$mode" = "pin" ]; then
sha256sum progit.md > MANIFEST.sha256
echo "re-pinned MANIFEST.sha256 — a deliberate human action (corpora/README.md);"
echo "commit the new manifest with justification."
else
sha256sum -c MANIFEST.sha256
fi
corpora/tokei/gen-corpus.sh
+14
View File
@@ -2,12 +2,26 @@
# Regenerate the tokei bench input: 8 replicas of two pinned trees
# (progit @ 61833a5, comrak src/ @ 45c1995), .git removed. Verifies the
# manifest for copy1; copies 2-8 are byte-identical replicas of copy1.
# Default/--check verifies MANIFEST.sha256 (never rewrites it); --pin
# deliberately rewrites it.
set -eu
mode="check"
case "${1:-}" in
"" | --check) ;;
--pin) mode="pin" ;;
*) echo "usage: gen-corpus.sh [--check|--pin]" >&2; exit 2 ;;
esac
cd "$(dirname "$0")"
rm -rf input
mkdir -p input/copy1
cp -r ../../targets/comrak/workspace/vendor/progit input/copy1/progit
cp -r ../../targets/comrak/workspace/src input/copy1/src
rm -rf input/copy1/progit/.git
if [ "$mode" = "pin" ]; then
(cd input/copy1 && find . -type f -print0 | sort -z | xargs -0 sha256sum) > MANIFEST.sha256
echo "re-pinned MANIFEST.sha256 — a deliberate human action (corpora/README.md);"
echo "commit the new manifest with justification."
else
(cd input/copy1 && sha256sum -c ../../MANIFEST.sha256 --quiet) || { echo "corpus pin FAILED"; exit 1; }
fi
for i in 2 3 4 5 6 7 8; do cp -r input/copy1 "input/copy$i"; done
crates/bench-runner/Cargo.toml
+3 -1
View File
@@ -10,7 +10,9 @@ repository.workspace = true
anyhow.workspace = true
clap.workspace = true
ledger = { path = "../ledger" }
libc = "0.2"
serde.workspace = true
serde_json.workspace = true
toml.workspace = true
[target.'cfg(unix)'.dependencies]
libc = "0.2"
crates/bench-runner/src/config.rs
+36
View File
@@ -23,6 +23,13 @@ pub struct AcceptConfig {
/// both sides identically.
#[serde(default)]
pub pin_prefix: String,
/// Differential-fuzz iterations requested per gate run (SPEC §3.2).
#[serde(default = "default_fuzz_iters")]
pub fuzz_iters: u64,
}
fn default_fuzz_iters() -> u64 {
10_000
}
impl Default for AcceptConfig {
@@ -35,6 +42,7 @@ impl Default for AcceptConfig {
confidence: 0.95,
bootstrap_seed: 0x707E_17E5,
pin_prefix: String::new(),
fuzz_iters: default_fuzz_iters(),
}
}
}
@@ -52,3 +60,31 @@ impl AcceptConfig {
Ok(cfg)
}
}
#[cfg(test)]
mod tests {
use super::*;
const MINIMAL: &str = "threshold = 0.02\nruns_per_side = 30\nwarmup_runs = 3\n\
bootstrap_iters = 10000\nconfidence = 0.95\nbootstrap_seed = 7\n";
#[test]
fn fuzz_iters_defaults_when_absent() {
let cfg: AcceptConfig = toml::from_str(MINIMAL).unwrap();
assert_eq!(cfg.fuzz_iters, 10_000);
assert_eq!(AcceptConfig::default().fuzz_iters, 10_000);
}
#[test]
fn fuzz_iters_overridable() {
let raw = format!("{MINIMAL}fuzz_iters = 500\n");
let cfg: AcceptConfig = toml::from_str(&raw).unwrap();
assert_eq!(cfg.fuzz_iters, 500);
}
#[test]
fn unknown_fields_still_rejected() {
let raw = format!("{MINIMAL}fuz_iters = 500\n");
assert!(toml::from_str::<AcceptConfig>(&raw).is_err());
}
}
crates/bench-runner/src/counters.rs
+218
View File
@@ -0,0 +1,218 @@
//! Opt-in `perf stat` PMU counter capture. Diagnostics only — counters
//! never feed the accept decision (wall time does, SPEC §3.1). Each
//! timed run is wrapped `perf stat -x, -e <events> -o <tmpfile> -- sh -c
//! <cmd>`; the CSV file is parsed after every run and per-side medians
//! are printed with the standard caveats. Unix-only at runtime; the
//! parser is portable and unit-tested.
use crate::exec::MeasuredSamples;
use crate::schedule::{interleaved, Side};
use crate::stats;
use anyhow::Context;
use std::path::Path;
use std::process::{Command, Stdio};
use std::time::Instant;
pub const PERF_EVENTS: [&str; 4] = ["cycles", "instructions", "cache-misses", "branch-misses"];
/// One side's counter samples: `samples[i]` holds one value per measured
/// run for `PERF_EVENTS[i]`; runs where perf reported `<not counted>` or
/// `<not supported>` contribute nothing.
#[derive(Debug, Clone, Default)]
pub struct PerfSide {
pub samples: [Vec<u64>; 4],
}
#[derive(Debug, Clone, Default)]
pub struct PerfReport {
pub baseline: PerfSide,
pub candidate: PerfSide,
}
/// Parse `perf stat -x,` CSV: field 0 is the counter value, field 2 the
/// event name (perf may append a modifier, e.g. `cycles:u`). Comment
/// lines, blank lines, and non-numeric values (`<not counted>`,
/// `<not supported>`) are skipped. Returns (event, count) in file order.
pub fn parse_perf_csv(text: &str) -> Vec<(String, u64)> {
let mut out = Vec::new();
for line in text.lines() {
let line = line.trim();
if line.is_empty() || line.starts_with('#') {
continue;
}
let mut fields = line.split(',');
let value = fields.next().unwrap_or("");
let _unit = fields.next();
let Some(event) = fields.next() else {
continue;
};
let Ok(v) = value.trim().parse::<u64>() else {
continue;
};
out.push((event.trim().to_string(), v));
}
out
}
/// Index into PERF_EVENTS, tolerating a perf modifier suffix
/// (`cycles:u` when perf_event_paranoid restricts kernel counting).
fn event_index(event: &str) -> Option<usize> {
PERF_EVENTS.iter().position(|name| {
event == *name || (event.starts_with(name) && event[name.len()..].starts_with(':'))
})
}
impl PerfSide {
fn absorb(&mut self, parsed: &[(String, u64)]) {
for (event, value) in parsed {
if let Some(i) = event_index(event) {
self.samples[i].push(*value);
}
}
}
}
impl PerfReport {
/// Per-side medians, each with its command (workload) and run count.
pub fn print(&self, baseline_cmd: &str, candidate_cmd: &str) {
println!("perf-stat per-side medians:");
for (label, cmd, side) in [
("baseline ", baseline_cmd, &self.baseline),
("candidate", candidate_cmd, &self.candidate),
] {
let cols: Vec<String> = PERF_EVENTS
.iter()
.zip(&side.samples)
.map(|(name, xs)| {
if xs.is_empty() {
format!("{name}=n/a")
} else {
let v: Vec<f64> = xs.iter().map(|&x| x as f64).collect();
format!(
"{name}={:.0} (median of {} runs)",
stats::median(&v),
xs.len()
)
}
})
.collect();
println!(" {label} `{cmd}`: {}", cols.join(", "));
}
println!(
" note: PMU counters unavailable in most VMs — flag is opt-in diagnostics, \
not part of the accept decision"
);
}
}
fn time_once_perf(shell_cmd: &str, out_file: &Path) -> anyhow::Result<(f64, Vec<(String, u64)>)> {
let start = Instant::now();
let status = Command::new("perf")
.args(["stat", "-x,", "-e", &PERF_EVENTS.join(",")])
.arg("-o")
.arg(out_file)
.args(["--", "sh", "-c", shell_cmd])
.stdout(Stdio::null())
.stderr(Stdio::null())
.status()
.with_context(|| format!("failed to spawn perf stat for: {shell_cmd}"))?;
let elapsed = start.elapsed().as_secs_f64();
anyhow::ensure!(
status.success(),
"perf stat run exited nonzero: {shell_cmd}"
);
let text = std::fs::read_to_string(out_file)
.with_context(|| format!("perf stat output missing: {}", out_file.display()))?;
Ok((elapsed, parse_perf_csv(&text)))
}
/// Interleaved A/B with each timed run wrapped in `perf stat`. The perf
/// wrapper overhead lands on both sides identically, but this path is
/// diagnostics only — verdicts always measure through the unwrapped
/// `exec::run_interleaved`.
pub fn run_interleaved_perf(
baseline_cmd: &str,
candidate_cmd: &str,
runs_per_side: usize,
warmup: usize,
) -> anyhow::Result<(MeasuredSamples, PerfReport)> {
anyhow::ensure!(cfg!(unix), "--perf-stat requires a Unix host with perf(1)");
let tmp = std::env::temp_dir().join(format!("bench-runner-perf-{}.csv", std::process::id()));
let mut samples = MeasuredSamples {
baseline_s: Vec::with_capacity(runs_per_side),
candidate_s: Vec::with_capacity(runs_per_side),
baseline_max_rss_kib: Vec::new(),
candidate_max_rss_kib: Vec::new(),
};
let mut report = PerfReport::default();
for (side, measured) in interleaved(runs_per_side, warmup) {
let cmd = match side {
Side::Baseline => baseline_cmd,
Side::Candidate => candidate_cmd,
};
let (elapsed, parsed) = time_once_perf(cmd, &tmp)?;
if measured {
match side {
Side::Baseline => {
samples.baseline_s.push(elapsed);
report.baseline.absorb(&parsed);
}
Side::Candidate => {
samples.candidate_s.push(elapsed);
report.candidate.absorb(&parsed);
}
}
}
}
let _ = std::fs::remove_file(&tmp);
Ok((samples, report))
}
#[cfg(test)]
mod tests {
use super::*;
const CANNED: &str = "\
# started on Mon Jul 13 12:00:00 2026
123456789,,cycles,401000000,100.00,,
234567890,,instructions:u,401000000,100.00,1.90,insn per cycle
<not supported>,,cache-misses,0,100.00,,
<not counted>,,branch-misses,0,0.00,,
garbage line without commas
999,,unrelated-event,1,100.00,,
";
#[test]
fn parses_values_and_skips_junk() {
let parsed = parse_perf_csv(CANNED);
assert_eq!(
parsed,
vec![
("cycles".to_string(), 123_456_789),
("instructions:u".to_string(), 234_567_890),
("unrelated-event".to_string(), 999),
]
);
}
#[test]
fn event_index_tolerates_modifier_suffix_only() {
assert_eq!(event_index("cycles"), Some(0));
assert_eq!(event_index("instructions:u"), Some(1));
assert_eq!(event_index("cache-misses"), Some(2));
assert_eq!(event_index("cycle"), None);
assert_eq!(event_index("cycles_total"), None);
assert_eq!(event_index("unrelated-event"), None);
}
#[test]
fn absorb_buckets_by_event() {
let mut side = PerfSide::default();
side.absorb(&parse_perf_csv(CANNED));
assert_eq!(side.samples[0], vec![123_456_789]);
assert_eq!(side.samples[1], vec![234_567_890]);
assert!(side.samples[2].is_empty(), "<not supported> is skipped");
assert!(side.samples[3].is_empty(), "<not counted> is skipped");
}
}
crates/bench-runner/src/exec.rs
+106 -11
View File
@@ -1,29 +1,87 @@
//! Command execution and timing. Whole-program mode (hyperfine-style):
//! spawn the command, measure wall time, discard warm-ups per the
//! schedule. `perf stat` counters and RAPL energy are Phase 1 follow-ups
//! (tracked in README roadmap); wall time alone is enough to bring up
//! A/A calibration.
//! schedule. Max RSS is captured per run via `wait4` on Unix; opt-in
//! `perf stat` counters live in `counters` (diagnostics only).
use crate::schedule::{interleaved, Side};
use anyhow::Context;
use std::process::{Command, Stdio};
use std::process::{Child, Command, ExitStatus, Stdio};
use std::time::Instant;
#[derive(Debug, Clone)]
pub struct MeasuredSamples {
pub baseline_s: Vec<f64>,
pub candidate_s: Vec<f64>,
/// Per-run max resident set size, normalized to KiB from `wait4`
/// ru_maxrss (whose own unit is platform-defined — see
/// `maxrss_to_kib`). Empty on non-Unix hosts.
pub baseline_max_rss_kib: Vec<u64>,
pub candidate_max_rss_kib: Vec<u64>,
}
fn time_once(shell_cmd: &str, inject_spin_s: f64) -> anyhow::Result<f64> {
/// Raw `ru_maxrss` → KiB.
///
/// The unit of `ru_maxrss` is *platform-defined*, and the platforms
/// disagree: Linux and the BSDs report kilobytes, while Apple platforms
/// (macOS et al.) report **bytes**. Taking the raw value as KiB everywhere
/// — as this did — silently inflates every macOS reading by 1024×, and the
/// number is not cosmetic: verdict records its median as
/// `env_fingerprint.max_rss_kib` in the ledger, where an allocator-swap
/// attempt is judged partly on memory. Everything downstream is KiB, so the
/// raw value is normalized here, at the syscall edge.
///
/// `raw_is_bytes` is the platform predicate (passed at the one call site, so
/// this stays a pure function testable on every host). Bytes round *up*: a
/// nonzero RSS must never record as 0 KiB. Negative values — a kernel that
/// reports nothing — clamp to 0 rather than wrapping into the exabytes.
#[cfg(any(unix, test))]
fn maxrss_to_kib(raw: i64, raw_is_bytes: bool) -> u64 {
let raw = raw.max(0) as u64;
if raw_is_bytes {
raw.div_ceil(1024)
} else {
raw
}
}
/// Reap the child, returning its exit status and max RSS in KiB when the
/// platform reports it.
#[cfg(unix)]
fn wait_with_rusage(child: &mut Child) -> anyhow::Result<(ExitStatus, Option<u64>)> {
use std::os::unix::process::ExitStatusExt;
let pid = child.id() as libc::pid_t;
let mut status: libc::c_int = 0;
let mut usage: libc::rusage = unsafe { std::mem::zeroed() };
loop {
let r = unsafe { libc::wait4(pid, &mut status, 0, &mut usage) };
if r == pid {
break;
}
let err = std::io::Error::last_os_error();
if err.raw_os_error() != Some(libc::EINTR) {
return Err(err).context("wait4 failed");
}
}
// Apple's getrusage reports ru_maxrss in bytes; Linux and the BSDs in KiB.
let max_rss_kib = maxrss_to_kib(usage.ru_maxrss as i64, cfg!(target_vendor = "apple"));
Ok((ExitStatus::from_raw(status), Some(max_rss_kib)))
}
#[cfg(not(unix))]
fn wait_with_rusage(child: &mut Child) -> anyhow::Result<(ExitStatus, Option<u64>)> {
Ok((child.wait()?, None))
}
fn time_once(shell_cmd: &str, inject_spin_s: f64) -> anyhow::Result<(f64, Option<u64>)> {
let start = Instant::now();
let status = Command::new("sh")
let mut child = Command::new("sh")
.arg("-c")
.arg(shell_cmd)
.stdout(Stdio::null())
.stderr(Stdio::null())
.status()
.spawn()
.with_context(|| format!("failed to spawn: {shell_cmd}"))?;
let (status, max_rss_kib) = wait_with_rusage(&mut child)?;
if inject_spin_s > 0.0 {
// Calibration-only: a real busy-wait inside the timed window, so an
// injected slowdown flows through the entire measurement pipeline
@@ -35,7 +93,7 @@ fn time_once(shell_cmd: &str, inject_spin_s: f64) -> anyhow::Result<f64> {
}
let elapsed = start.elapsed().as_secs_f64();
anyhow::ensure!(status.success(), "command exited nonzero: {shell_cmd}");
Ok(elapsed)
Ok((elapsed, max_rss_kib))
}
/// Run baseline and candidate commands on the interleaved schedule and
@@ -53,19 +111,56 @@ pub fn run_interleaved(
let mut out = MeasuredSamples {
baseline_s: Vec::with_capacity(runs_per_side),
candidate_s: Vec::with_capacity(runs_per_side),
baseline_max_rss_kib: Vec::new(),
candidate_max_rss_kib: Vec::new(),
};
for (side, measured) in interleaved(runs_per_side, warmup) {
let (cmd, spin) = match side {
Side::Baseline => (baseline_cmd, 0.0),
Side::Candidate => (candidate_cmd, inject_candidate_spin_s),
};
let elapsed = time_once(cmd, spin)?;
let (elapsed, max_rss_kib) = time_once(cmd, spin)?;
if measured {
match side {
Side::Baseline => out.baseline_s.push(elapsed),
Side::Candidate => out.candidate_s.push(elapsed),
Side::Baseline => {
out.baseline_s.push(elapsed);
out.baseline_max_rss_kib.extend(max_rss_kib);
}
Side::Candidate => {
out.candidate_s.push(elapsed);
out.candidate_max_rss_kib.extend(max_rss_kib);
}
}
}
}
Ok(out)
}
#[cfg(test)]
mod tests {
use super::maxrss_to_kib;
#[test]
fn maxrss_kib_passthrough_when_platform_reports_kib() {
// Linux / BSD: already KiB.
assert_eq!(maxrss_to_kib(2048, false), 2048);
assert_eq!(maxrss_to_kib(0, false), 0);
}
#[test]
fn maxrss_bytes_are_converted_not_recorded_raw() {
// Apple: bytes. 2 MiB is 2048 KiB, not 2_097_152 — the 1024×
// inflation this guards against.
assert_eq!(maxrss_to_kib(2_097_152, true), 2048);
// Rounds up: a nonzero RSS never records as 0 KiB.
assert_eq!(maxrss_to_kib(1, true), 1);
assert_eq!(maxrss_to_kib(1025, true), 2);
assert_eq!(maxrss_to_kib(0, true), 0);
}
#[test]
fn maxrss_negative_clamps_instead_of_wrapping() {
assert_eq!(maxrss_to_kib(-1, false), 0);
assert_eq!(maxrss_to_kib(-1, true), 0);
}
}
crates/bench-runner/src/lib.rs
+3 -1
View File
@@ -3,9 +3,11 @@
//! The product is this crate: interleaved A/B measurement, bootstrap CIs
//! over the ratio of medians, threshold-gated accept decisions, and A/A
//! self-calibration. Pure statistics live here and are unit-tested;
//! process execution and `perf stat` integration live in `exec`.
//! process execution lives in `exec`; opt-in `perf stat` diagnostics
//! live in `counters`.
pub mod config;
pub mod counters;
pub mod exec;
pub mod fingerprint;
pub mod schedule;
crates/bench-runner/src/main.rs
+26 -4
View File
@@ -1,5 +1,7 @@
use anyhow::Result;
use bench_runner::{config::AcceptConfig, decide, exec, fingerprint::EnvFingerprint, stats};
use bench_runner::{
config::AcceptConfig, counters, decide, exec, fingerprint::EnvFingerprint, stats,
};
use clap::{Parser, Subcommand};
use ledger::Verdict;
use std::path::PathBuf;
@@ -25,6 +27,11 @@ enum Cmd {
baseline: String,
#[arg(long)]
candidate: String,
/// Wrap each timed run in `perf stat` and print per-side counter
/// medians (Unix only; opt-in diagnostics, never part of the
/// accept decision).
#[arg(long, default_value_t = false)]
perf_stat: bool,
},
/// A/A self-test: same command both sides; must yield a null verdict.
Aa {
@@ -429,12 +436,13 @@ fn main() -> Result<()> {
None => AcceptConfig::default(),
};
let (baseline, candidate, aa_mode) = match &cli.command {
let (baseline, candidate, aa_mode, perf_stat) = match &cli.command {
Cmd::Compare {
baseline,
candidate,
} => (baseline.clone(), candidate.clone(), false),
Cmd::Aa { cmd } => (cmd.clone(), cmd.clone(), true),
perf_stat,
} => (baseline.clone(), candidate.clone(), false, *perf_stat),
Cmd::Aa { cmd } => (cmd.clone(), cmd.clone(), true, false),
Cmd::Calibrate {
cmd,
sessions,
@@ -502,6 +510,15 @@ fn main() -> Result<()> {
cfg.runs_per_side, cfg.warmup_runs
);
let (samples, perf_report) = if perf_stat {
let (samples, report) = counters::run_interleaved_perf(
&baseline,
&candidate,
cfg.runs_per_side,
cfg.warmup_runs,
)?;
(samples, Some(report))
} else {
let samples = exec::run_interleaved(
&baseline,
&candidate,
@@ -509,6 +526,8 @@ fn main() -> Result<()> {
cfg.warmup_runs,
0.0,
)?;
(samples, None)
};
let ci = stats::bootstrap_ratio_ci(
&samples.baseline_s,
&samples.candidate_s,
@@ -524,6 +543,9 @@ fn main() -> Result<()> {
ci.lo,
ci.hi
);
if let Some(report) = &perf_report {
report.print(&baseline, &candidate);
}
if aa_mode {
// Null verdict required: an "accept" here is a calibration failure.
crates/bench-runner/src/service.rs
+7
View File
@@ -147,6 +147,7 @@ fn spawn_server(cfg: &ServiceCfg, port: u16) -> Result<Child> {
Ok(child)
}
#[cfg(unix)]
fn set_linger_zero(s: &TcpStream) {
// SO_LINGER with a 0 timeout → RST on close instead of FIN, so the
// tens of thousands of short client connections a load session opens
@@ -170,6 +171,12 @@ fn set_linger_zero(s: &TcpStream) {
}
}
#[cfg(not(unix))]
fn set_linger_zero(_s: &TcpStream) {
// Service mode is POSIX-only at runtime (`sh`, taskset); this stub
// only keeps the crate compiling on non-Unix hosts.
}
fn one_request(port: u16) -> std::io::Result<()> {
let mut s = TcpStream::connect((Ipv4Addr::LOCALHOST, port))?;
s.set_nodelay(true)?;
crates/diff-test/src/lib.rs
+71
View File
@@ -43,6 +43,49 @@ pub fn all_passed(results: &[(GateLayer, GateOutcome)]) -> bool {
.any(|(_, o)| matches!(o, GateOutcome::Passed))
}
/// Inputs to one gate run: the candidate binary under test, the pristine
/// baseline to differ it against, and the differential-fuzz iteration
/// budget (from AcceptConfig, default 10_000).
#[derive(Debug, Clone, Copy)]
pub struct GateInputs<'a> {
/// Patched binary (repo-root-relative): golden replay runs it, and it
/// is the `{candidate}` side of differential fuzz.
pub candidate_binary: &'a str,
/// Pristine baseline binary (repo-root-relative), the `{baseline}` side
/// of differential fuzz. `None` when the caller has no baseline to
/// differ against — the standalone `just gates` flow builds only the
/// working tree — and the fuzz gate is then skipped with that reason
/// rather than silently comparing the candidate to itself. The accept
/// path (`just verdict`) rebuilds a pristine baseline and passes it,
/// and only a fuzz gate that actually Passed can mint an accept.
pub baseline_binary: Option<&'a str>,
pub fuzz_iters: u64,
}
/// Everything one gate run produced: the four layers in fixed order
/// (UpstreamTests, GoldenReplay, DifferentialFuzz, Sanitizers) plus the
/// fuzz counts and equivalence mode that verdict records in the ledger.
#[derive(Debug, Clone)]
pub struct GateReport {
pub gates: Vec<(GateLayer, GateOutcome)>,
/// Fuzz iterations actually executed (parsed from the FUZZ-RESULT
/// line); 0 when the fuzz gate was skipped or never reported.
pub fuzz_iters: u64,
pub fuzz_divergence: bool,
/// "byte-identical" | "fp-tolerance"
pub equivalence_mode: String,
}
impl GateReport {
pub fn outcome(&self, layer: GateLayer) -> Option<&GateOutcome> {
self.gates.iter().find(|(l, _)| *l == layer).map(|(_, o)| o)
}
pub fn all_passed(&self) -> bool {
all_passed(&self.gates)
}
}
#[cfg(test)]
mod tests {
use super::*;
@@ -71,4 +114,32 @@ mod tests {
)];
assert!(!all_passed(&results));
}
#[test]
fn report_outcome_looks_up_by_layer() {
let report = GateReport {
gates: vec![
(GateLayer::UpstreamTests, GateOutcome::Passed),
(
GateLayer::GoldenReplay,
GateOutcome::Failed {
detail: "stdout diverged".into(),
},
),
],
fuzz_iters: 0,
fuzz_divergence: false,
equivalence_mode: "byte-identical".into(),
};
assert_eq!(
report.outcome(GateLayer::UpstreamTests),
Some(&GateOutcome::Passed)
);
assert!(matches!(
report.outcome(GateLayer::GoldenReplay),
Some(GateOutcome::Failed { .. })
));
assert_eq!(report.outcome(GateLayer::Sanitizers), None);
assert!(!report.all_passed());
}
}
crates/diff-test/src/main.rs
+47 -7
View File
@@ -1,12 +1,18 @@
//! Gate orchestration CLI: corpus pin → upstream tests → golden replay.
//! Differential fuzz and sanitizers remain per-attempt manual steps in
//! Phase 1 and are reported as skipped with a reason.
//! Gate orchestration CLI: corpus + test-suite pins → upstream tests →
//! policy-aware golden replay → differential fuzz. Sanitizers run on the
//! accept path in verdict and are reported as skipped with a reason.
//!
//! Differential fuzz is skipped here too, and for the same structural
//! reason: it differs a *pristine baseline* against the candidate, and this
//! flow builds only the working tree — there is nothing to differ against.
//! The accept path (`just verdict`) rebuilds a pristine baseline, passes it
//! in, and refuses to mint an accept unless the fuzz gate actually passed.
use anyhow::Result;
use clap::Parser;
use diff_test::orchestrate::run_core_gates;
use diff_test::target::{subst_out, TargetSpec};
use diff_test::{all_passed, GateOutcome};
use diff_test::{GateInputs, GateLayer, GateOutcome};
use std::path::PathBuf;
#[derive(Parser)]
@@ -21,6 +27,13 @@ struct Cli {
/// tree is built to targets/<name>/gates-build and that is tested.
#[arg(long)]
binary: Option<String>,
/// Differential-fuzz iteration budget, substituted for {iters} in the
/// target's [gates].fuzz command. Inert on this flow: with no pristine
/// baseline to differ against, the fuzz gate is skipped. The accept path
/// (`just verdict`) rebuilds one and takes its budget from
/// config/accept.toml.
#[arg(long, default_value_t = 10_000)]
fuzz_iters: u64,
}
fn main() -> Result<()> {
@@ -47,15 +60,42 @@ fn main() -> Result<()> {
subst_out(&spec.build.binary, &out)
}
};
let results = run_core_gates(&root, &spec, &binary)?;
for (layer, outcome) in &results {
let report = run_core_gates(
&root,
&spec,
&GateInputs {
candidate_binary: &binary,
// No pristine baseline on this flow — see the module doc. The
// fuzz gate reports Skipped with that reason rather than
// comparing the candidate against itself.
baseline_binary: None,
fuzz_iters: cli.fuzz_iters,
},
)?;
for (layer, outcome) in &report.gates {
match outcome {
GateOutcome::Passed if *layer == GateLayer::DifferentialFuzz => {
println!("{layer:?}: PASS ({} iters)", report.fuzz_iters)
}
GateOutcome::Passed => println!("{layer:?}: PASS"),
GateOutcome::Failed { detail } => println!("{layer:?}: FAIL — {detail}"),
GateOutcome::Skipped { reason } => println!("{layer:?}: skipped — {reason}"),
}
}
if all_passed(&results) {
if report.all_passed() {
// Inside all_passed(), the fuzz gate is Passed or Skipped — never
// Failed. Say so plainly: a PASS here must not read as a verified
// differential-fuzz run, which is exactly the confusion that let
// accepted rows carry fuzz_iters=0.
if !matches!(
report.outcome(GateLayer::DifferentialFuzz),
Some(GateOutcome::Passed)
) {
println!(
"note: differential fuzz did not run — `just gates` is not an accept path; only \
`just verdict` (pristine baseline + fuzz gate) can accept a change"
);
}
println!("gates: PASS ({})", cli.target);
Ok(())
} else {
crates/diff-test/src/orchestrate.rs
+439 -19
View File
@@ -1,10 +1,20 @@
//! Core gate sequence, callable from the diff-test CLI and from verdict:
//! corpus pin (refuses on mismatch) → upstream tests → golden replay.
//! corpus + test-suite pins (refuse on mismatch) → upstream tests →
//! policy-aware golden replay → differential fuzz. Sanitizers run on the
//! accept path in verdict and are reported here as skipped with a reason.
//!
//! Differential fuzz is the one gate that is not purely a property of the
//! candidate: it differs the candidate against a *pristine baseline*, so it
//! needs both binaries. Callers that have only a candidate (the standalone
//! `just gates` flow) pass `baseline_binary: None` and the gate is skipped
//! with that reason recorded — it is never faked by comparing the candidate
//! against itself, and verdict refuses to mint an accept without it.
use crate::policy::EquivalencePolicy;
use crate::target::{expected_golden_hash, TargetSpec};
use crate::{pin, GateLayer, GateOutcome};
use crate::{pin, GateInputs, GateLayer, GateOutcome, GateReport};
use sha2::{Digest, Sha256};
use std::path::Path;
use std::path::{Path, PathBuf};
use std::process::{Command, Stdio};
fn sh(root: &Path, cmd: &str) -> anyhow::Result<std::process::Output> {
@@ -17,14 +27,155 @@ fn sh(root: &Path, cmd: &str) -> anyhow::Result<std::process::Output> {
.output()?)
}
/// Run the mechanical gates for `target`, testing `binary` (repo-root-
/// relative) in the golden replay. Corpus-pin mismatch is an error, not
/// a gate failure: nothing runs on a tampered corpus.
/// TESTSUITE.sha256 lives beside the corpus MANIFEST.sha256 (both under
/// corpora/<target>/), so its path derives from the manifest's.
fn suite_manifest_path(corpus_manifest: &Path) -> PathBuf {
corpus_manifest
.parent()
.unwrap_or_else(|| Path::new(""))
.join("TESTSUITE.sha256")
}
fn parse_u64_strict(s: &str) -> Option<u64> {
if s.is_empty() || !s.bytes().all(|b| b.is_ascii_digit()) {
return None;
}
s.parse().ok()
}
/// Parse the last `FUZZ-RESULT iters=<u64> divergences=<u64>` stdout line
/// of a fuzz command (exact ASCII form, single spaces). Returns
/// `(iters, divergences)`, or None when no line matches.
pub fn parse_fuzz_result(stdout: &str) -> Option<(u64, u64)> {
stdout.lines().rev().find_map(|line| {
let line = line.strip_suffix('\r').unwrap_or(line);
let rest = line.strip_prefix("FUZZ-RESULT iters=")?;
let (iters, rest) = rest.split_once(' ')?;
let divergences = rest.strip_prefix("divergences=")?;
Some((parse_u64_strict(iters)?, parse_u64_strict(divergences)?))
})
}
/// Recorded verbatim when the target declares no fuzz command at all.
const FUZZ_SKIP_NO_COMMAND: &str = "no fuzz command declared in target.toml [gates].fuzz";
/// Recorded verbatim when a fuzz command exists but the caller has no
/// pristine baseline: differential fuzz differs two binaries, and comparing
/// the candidate against itself would be a gate that cannot fail.
const FUZZ_SKIP_NO_BASELINE: &str =
"differential fuzz needs a pristine baseline binary; run the accept path (`just verdict`), \
which rebuilds one";
/// Substitute the three `[gates].fuzz` template placeholders. The contract
/// with `targets/<t>/target.toml` and `scripts/diff-fuzz-*.py`: `{iters}`
/// is the iteration budget, `{baseline}` the pristine baseline binary and
/// `{candidate}` the patched one (both repo-root-relative paths, and the
/// command runs from the repo root). A fuzz script must never hardcode a
/// baseline path — the harness owns where the pristine build lands.
fn fuzz_command(tmpl: &str, iters: u64, baseline: &str, candidate: &str) -> String {
tmpl.replace("{iters}", &iters.to_string())
.replace("{baseline}", baseline)
.replace("{candidate}", candidate)
}
/// What the differential-fuzz gate will do, decided before any process runs.
#[derive(Debug, PartialEq, Eq)]
enum FuzzPlan {
Skip(&'static str),
Run(String),
}
/// Differential fuzz is a baseline-vs-candidate gate: it needs both sides.
/// Missing either one is a Skip with the reason recorded, never a silent
/// self-comparison and never a hard error (the standalone gates flow has
/// no baseline by construction).
fn plan_fuzz(fuzz: Option<&str>, inputs: &GateInputs) -> FuzzPlan {
match (fuzz, inputs.baseline_binary) {
(None, _) => FuzzPlan::Skip(FUZZ_SKIP_NO_COMMAND),
(Some(_), None) => FuzzPlan::Skip(FUZZ_SKIP_NO_BASELINE),
(Some(tmpl), Some(baseline)) => FuzzPlan::Run(fuzz_command(
tmpl,
inputs.fuzz_iters,
baseline,
inputs.candidate_binary,
)),
}
}
/// Grade one executed fuzz run. Passed iff the command exited 0, printed a
/// well-formed `FUZZ-RESULT` line, and reported zero divergences — a run
/// that never reports the line is a Failed gate, not a pass by silence.
/// Returns `(outcome, iters_actually_executed, divergence)`.
fn fuzz_gate_outcome(
stdout: &str,
success: bool,
exit_code: Option<i32>,
) -> (GateOutcome, u64, bool) {
match parse_fuzz_result(stdout) {
None => (
GateOutcome::Failed {
detail: "fuzz command did not report FUZZ-RESULT".into(),
},
0,
false,
),
Some((iters, divergences)) => (
if success && divergences == 0 {
GateOutcome::Passed
} else {
GateOutcome::Failed {
detail: format!("exit={exit_code:?} iters={iters} divergences={divergences}"),
}
},
iters,
divergences > 0,
),
}
}
/// The fp-tolerance policy fails closed without a committed reference:
/// there is no hash to fall back to that a tolerance could compare against.
fn fp_reference(golden_reference: Option<&PathBuf>) -> Result<&PathBuf, GateOutcome> {
golden_reference.ok_or_else(|| GateOutcome::Failed {
detail: "fp-tolerance policy requires corpus.golden_reference in target.toml".into(),
})
}
/// GoldenReplay verdict under an fp-tolerance policy: the replay stdout is
/// compared token-wise against the committed reference output.
fn fp_golden_outcome(
policy: &EquivalencePolicy,
reference: &str,
stdout: &str,
success: bool,
exit_code: Option<i32>,
) -> GateOutcome {
if !success {
return GateOutcome::Failed {
detail: format!("golden replay command failed (exit={exit_code:?})"),
};
}
match policy.compare(reference, stdout) {
Ok(()) => GateOutcome::Passed,
Err(d) => GateOutcome::Failed {
detail: format!(
"diverged at token/offset {}: baseline={:?} candidate={:?} — {}",
d.token, d.baseline, d.candidate, d.reason
),
},
}
}
/// Run the mechanical gates for `target`, testing `inputs.candidate_binary`
/// (repo-root-relative) in golden replay and differential fuzz. Differential
/// fuzz additionally needs `inputs.baseline_binary` to differ against and is
/// skipped-with-reason without one. Pin mismatches (corpus or test suite)
/// are errors, not gate failures: nothing runs on tampered inputs.
pub fn run_core_gates(
root: &Path,
spec: &TargetSpec,
binary: &str,
) -> anyhow::Result<Vec<(GateLayer, GateOutcome)>> {
inputs: &GateInputs,
) -> anyhow::Result<GateReport> {
let verified = pin::verify_manifest(
&root.join(&spec.corpus.manifest),
&root.join(&spec.corpus.root),
@@ -34,6 +185,24 @@ pub fn run_core_gates(
spec.corpus.manifest.display()
);
// Test-suite pin: same refuse-on-mismatch posture as the corpus pin,
// but the manifest is optional until a human deliberately generates it.
let suite_manifest = suite_manifest_path(&spec.corpus.manifest);
if root.join(&suite_manifest).exists() {
let verified = pin::verify_manifest(&root.join(&suite_manifest), root).map_err(|e| {
anyhow::anyhow!("upstream test suite hash mismatch — refusing to run: {e}")
})?;
eprintln!(
"test-suite pin: {verified} files verified against {}",
suite_manifest.display()
);
} else {
eprintln!(
"test-suite pin: {} not found — suite unpinned (generate with scripts/pin-testsuite.sh)",
suite_manifest.display()
);
}
let mut results = Vec::new();
let t = sh(root, &spec.gates.tests)?;
@@ -51,11 +220,20 @@ pub fn run_core_gates(
},
));
let g = sh(root, &spec.gates.golden.replace("{binary}", binary))?;
let policy = EquivalencePolicy::load(&spec.target_dir(root))?;
let equivalence_mode = match &policy {
EquivalencePolicy::ByteIdentical => "byte-identical",
EquivalencePolicy::FpTolerance { .. } => "fp-tolerance",
};
let golden_cmd = spec
.gates
.golden
.replace("{binary}", inputs.candidate_binary);
let golden = match &policy {
EquivalencePolicy::ByteIdentical => {
let g = sh(root, &golden_cmd)?;
let expected = expected_golden_hash(&root.join(&spec.corpus.golden_sha256))?;
let actual = format!("{:x}", Sha256::digest(&g.stdout));
results.push((
GateLayer::GoldenReplay,
if g.status.success() && actual == expected {
GateOutcome::Passed
} else {
@@ -65,20 +243,262 @@ pub fn run_core_gates(
g.status.code()
),
}
},
));
}
}
EquivalencePolicy::FpTolerance { .. } => {
match fp_reference(spec.corpus.golden_reference.as_ref()) {
Err(outcome) => outcome,
Ok(reference_path) => {
let reference =
std::fs::read_to_string(root.join(reference_path)).map_err(|e| {
anyhow::anyhow!(
"cannot read corpus.golden_reference {}: {e}",
reference_path.display()
)
})?;
let g = sh(root, &golden_cmd)?;
fp_golden_outcome(
&policy,
&reference,
&String::from_utf8_lossy(&g.stdout),
g.status.success(),
g.status.code(),
)
}
}
}
};
results.push((GateLayer::GoldenReplay, golden));
results.push((
GateLayer::DifferentialFuzz,
let (fuzz_outcome, fuzz_iters, fuzz_divergence) =
match plan_fuzz(spec.gates.fuzz.as_deref(), inputs) {
FuzzPlan::Skip(reason) => (
GateOutcome::Skipped {
reason: "per-attempt manual step in Phase 1 (needs old/new pair)".into(),
reason: reason.to_string(),
},
));
0,
false,
),
FuzzPlan::Run(cmd) => {
let f = sh(root, &cmd)?;
fuzz_gate_outcome(
&String::from_utf8_lossy(&f.stdout),
f.status.success(),
f.status.code(),
)
}
};
results.push((GateLayer::DifferentialFuzz, fuzz_outcome));
results.push((
GateLayer::Sanitizers,
GateOutcome::Skipped {
reason: "per-attempt manual step in Phase 1 (nightly toolchain)".into(),
reason:
"sanitizers run on the accept path in verdict (ASan/UBSan, TSan when configured)"
.into(),
},
));
Ok(results)
Ok(GateReport {
gates: results,
fuzz_iters,
fuzz_divergence,
equivalence_mode: equivalence_mode.to_string(),
})
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn suite_manifest_derives_from_corpus_manifest() {
assert_eq!(
suite_manifest_path(Path::new("corpora/cjson/MANIFEST.sha256")),
Path::new("corpora/cjson").join("TESTSUITE.sha256")
);
assert_eq!(
suite_manifest_path(Path::new("MANIFEST.sha256")),
Path::new("TESTSUITE.sha256")
);
}
#[test]
fn fuzz_result_parses_last_exact_line() {
assert_eq!(
parse_fuzz_result("FUZZ-RESULT iters=10000 divergences=0\n"),
Some((10000, 0))
);
// Last matching line wins; trailing non-matching output is ignored.
let out = "FUZZ-RESULT iters=1 divergences=1\nprogress...\nFUZZ-RESULT iters=9999 divergences=2\ndone\n";
assert_eq!(parse_fuzz_result(out), Some((9999, 2)));
// CRLF-terminated capture still parses.
assert_eq!(
parse_fuzz_result("FUZZ-RESULT iters=42 divergences=0\r\n"),
Some((42, 0))
);
}
#[test]
fn fuzz_result_rejects_malformed_lines() {
for bad in [
"",
"no result here",
" FUZZ-RESULT iters=1 divergences=0", // leading space
"FUZZ-RESULT iters=1 divergences=0", // double space
"FUZZ-RESULT iters=abc divergences=0", // non-numeric
"FUZZ-RESULT iters=1 divergences=-1", // sign not allowed
"FUZZ-RESULT iters=1 divergences=0 xx", // trailing junk
"FUZZ-RESULT divergences=0 iters=1", // wrong field order
"FUZZ-RESULT iters=1", // missing field
] {
assert_eq!(parse_fuzz_result(bad), None, "should reject: {bad:?}");
}
}
fn inputs(baseline: Option<&'static str>) -> GateInputs<'static> {
GateInputs {
candidate_binary: "targets/cjson/cand/cjson-bench",
baseline_binary: baseline,
fuzz_iters: 10_000,
}
}
const TMPL: &str = "python3 scripts/diff-fuzz-cjson.py targets/cjson/fuzz-work \
{iters} {baseline} {candidate}";
#[test]
fn fuzz_command_substitutes_all_three_placeholders() {
assert_eq!(
fuzz_command(TMPL, 500, "base/bin", "cand/bin"),
"python3 scripts/diff-fuzz-cjson.py targets/cjson/fuzz-work 500 base/bin cand/bin"
);
// Repeated placeholders all substitute; unknown braces are left alone.
assert_eq!(
fuzz_command("{candidate} {candidate} {iters} {other}", 2, "b", "c"),
"c c 2 {other}"
);
}
#[test]
fn fuzz_skipped_when_no_command_declared() {
assert_eq!(
plan_fuzz(None, &inputs(Some("targets/cjson/baseline/cjson-bench"))),
FuzzPlan::Skip("no fuzz command declared in target.toml [gates].fuzz")
);
// Absent command wins even without a baseline: nothing to run either way.
assert_eq!(
plan_fuzz(None, &inputs(None)),
FuzzPlan::Skip(FUZZ_SKIP_NO_COMMAND)
);
}
/// The regression this branch exists for: `just gates` (and CI) build only
/// the working tree, so there is no pristine baseline. Before the fix the
/// fuzz script silently hardcoded targets/<t>/baseline/… — a path nothing
/// on that flow builds — and the gate hard-failed. Now it skips, loudly.
#[test]
fn fuzz_skipped_when_declared_but_no_baseline() {
assert_eq!(
plan_fuzz(Some(TMPL), &inputs(None)),
FuzzPlan::Skip(
"differential fuzz needs a pristine baseline binary; run the accept path \
(`just verdict`), which rebuilds one"
)
);
}
#[test]
fn fuzz_runs_with_both_sides() {
assert_eq!(
plan_fuzz(
Some(TMPL),
&inputs(Some("targets/cjson/baseline/cjson-bench"))
),
FuzzPlan::Run(
"python3 scripts/diff-fuzz-cjson.py targets/cjson/fuzz-work 10000 \
targets/cjson/baseline/cjson-bench targets/cjson/cand/cjson-bench"
.to_string()
)
);
}
#[test]
fn fuzz_gate_grades_clean_run_as_pass() {
let (outcome, iters, diverged) =
fuzz_gate_outcome("FUZZ-RESULT iters=10000 divergences=0\n", true, Some(0));
assert_eq!(outcome, GateOutcome::Passed);
// The ledger records iterations *actually executed*, not the budget.
assert_eq!(iters, 10_000);
assert!(!diverged);
}
#[test]
fn fuzz_gate_fails_on_divergence_nonzero_exit_or_no_result_line() {
// Divergence reported: fail, and the count is still recorded.
let (outcome, iters, diverged) =
fuzz_gate_outcome("FUZZ-RESULT iters=900 divergences=3\n", false, Some(1));
assert!(
matches!(outcome, GateOutcome::Failed { detail } if detail.contains("divergences=3"))
);
assert_eq!(iters, 900);
assert!(diverged);
// Zero divergences but a nonzero exit (crash/timeout) is NOT a pass.
let (outcome, ..) =
fuzz_gate_outcome("FUZZ-RESULT iters=10 divergences=0\n", false, Some(2));
assert!(
matches!(outcome, GateOutcome::Failed { detail } if detail.contains("exit=Some(2)"))
);
// A run that never reports the contract line fails closed (e.g. the
// script died before printing — the old hardcoded-baseline breakage).
let (outcome, iters, diverged) =
fuzz_gate_outcome("Traceback (most recent call last):\n", false, Some(1));
assert_eq!(
outcome,
GateOutcome::Failed {
detail: "fuzz command did not report FUZZ-RESULT".into()
}
);
assert_eq!(iters, 0);
assert!(!diverged);
}
#[test]
fn fp_tolerance_requires_golden_reference() {
let err = fp_reference(None).unwrap_err();
assert_eq!(
err,
GateOutcome::Failed {
detail: "fp-tolerance policy requires corpus.golden_reference in target.toml"
.into()
}
);
let path = PathBuf::from("corpora/matmul/GOLDEN.reference");
assert_eq!(fp_reference(Some(&path)).unwrap(), &path);
}
#[test]
fn fp_golden_outcome_applies_policy() {
let policy = EquivalencePolicy::FpTolerance {
abs: 1e-6,
rel: 1e-5,
};
// Last-ULP reordering within tolerance passes.
assert_eq!(
fp_golden_outcome(&policy, "1.0000000 2.0", "1.0000001 2.0", true, Some(0)),
GateOutcome::Passed
);
// A genuine wrong value fails with the divergence in the detail.
assert!(matches!(
fp_golden_outcome(&policy, "1.0 2.0", "1.0 2.5", true, Some(0)),
GateOutcome::Failed { detail } if detail.contains("token/offset 1")
));
// A failing replay command fails regardless of output.
assert!(matches!(
fp_golden_outcome(&policy, "1.0", "1.0", false, Some(3)),
GateOutcome::Failed { detail } if detail.contains("exit=Some(3)")
));
}
}
crates/diff-test/src/target.rs
+76 -1
View File
@@ -12,6 +12,10 @@ pub struct TargetSpec {
pub bench: Bench,
pub gates: Gates,
pub corpus: Corpus,
/// Target name (the directory under `targets/`), captured by `load`;
/// not part of target.toml itself.
#[serde(skip)]
name: String,
}
#[derive(Debug, Deserialize)]
@@ -52,6 +56,14 @@ pub struct Build {
/// Sanitizer binary path with `{out}` substituted.
#[serde(default)]
pub sanitizer_binary: Option<String>,
/// TSan build command; `{out}` substituted. Run on the accept path
/// when configured — a patch touching concurrency without a TSan
/// lane cannot be machine-accepted.
#[serde(default)]
pub tsan: Option<String>,
/// TSan binary path with `{out}` substituted.
#[serde(default)]
pub tsan_binary: Option<String>,
}
/// Substitute the `{out}` isolation-dir placeholder in a build command or
@@ -66,6 +78,11 @@ pub struct Gates {
pub tests: String,
/// Golden replay command; stdout is hashed. `{binary}` substituted.
pub golden: String,
/// Differential-fuzz command; `{candidate}` and `{iters}` substituted.
/// Must print `FUZZ-RESULT iters=<n> divergences=<m>` as its final
/// stdout line. Absent → the fuzz gate is skipped with a reason.
#[serde(default)]
pub fuzz: Option<String>,
}
#[derive(Debug, Deserialize)]
@@ -77,6 +94,11 @@ pub struct Corpus {
/// File whose last whitespace-delimited-first-field line is the
/// expected sha256 of the golden command's stdout.
pub golden_sha256: PathBuf,
/// Committed reference stdout of the golden command, repo-root-
/// relative. Required by fp-tolerance targets: tolerance comparison
/// needs the actual values, not a hash.
#[serde(default)]
pub golden_reference: Option<PathBuf>,
}
impl TargetSpec {
@@ -84,7 +106,15 @@ impl TargetSpec {
let path = repo_root.join("targets").join(name).join("target.toml");
let raw = std::fs::read_to_string(&path)
.map_err(|e| anyhow::anyhow!("cannot read {}: {e}", path.display()))?;
Ok(toml::from_str(&raw)?)
let mut spec: Self = toml::from_str(&raw)?;
spec.name = name.to_string();
Ok(spec)
}
/// This target's directory under the repo root (`targets/<name>`),
/// where per-target policy files like equivalence.toml live.
pub fn target_dir(&self, root: &Path) -> PathBuf {
root.join("targets").join(&self.name)
}
}
@@ -100,3 +130,48 @@ pub fn expected_golden_hash(path: &Path) -> anyhow::Result<String> {
.map(str::to_string)
.ok_or_else(|| anyhow::anyhow!("no hash entry in {}", path.display()))
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn load_captures_name_and_defaults_new_fields() {
let root = std::env::temp_dir().join("hotpath-target-load");
let dir = root.join("targets").join("demo");
std::fs::create_dir_all(&dir).unwrap();
std::fs::write(
dir.join("target.toml"),
r#"
[source]
repo = "https://example.invalid/demo.git"
commit = "deadbeef"
license = "MIT"
[build]
baseline = "make -C targets/demo OUT={out}"
binary = "{out}/demo"
[bench]
command = "{binary} bench"
workload = "demo workload"
[gates]
tests = "make -C targets/demo test"
golden = "{binary} replay"
[corpus]
manifest = "corpora/demo/MANIFEST.sha256"
root = "corpora/demo"
golden_sha256 = "corpora/demo/GOLDEN.sha256"
"#,
)
.unwrap();
let spec = TargetSpec::load(&root, "demo").unwrap();
assert_eq!(spec.target_dir(&root), root.join("targets").join("demo"));
assert!(spec.gates.fuzz.is_none());
assert!(spec.corpus.golden_reference.is_none());
assert!(spec.build.tsan.is_none());
assert!(spec.build.tsan_binary.is_none());
}
}
crates/harnessd/Cargo.toml
+1 -1
View File
@@ -1,6 +1,6 @@
[package]
name = "harnessd"
description = "Agent-facing IPC daemon: the six-tool surface (SPEC §3.5) over line-delimited JSON on stdio; patches applied only after path-allowlist checks"
description = "Agent-facing IPC daemon: the seven-op JSON surface (SPEC §3.5) over line-delimited JSON on stdio; patches applied only after path-allowlist checks"
version.workspace = true
edition.workspace = true
license.workspace = true
crates/harnessd/src/allowlist.rs
+309
View File
@@ -0,0 +1,309 @@
//! Diff-header path vetting: the allowlist between agent-authored diffs
//! and `git apply` in the target workspace. Portable (no Unix
//! dependencies) so the escape-regression tests run on every development
//! platform, not just the daemon's POSIX hosts.
use anyhow::{anyhow, ensure, Result};
use std::path::{Component, Path};
/// Relative paths only; no parent components, no .git, no absolutes.
pub fn check_rel_path(p: &str) -> Result<()> {
let path = Path::new(p);
ensure!(!path.is_absolute(), "absolute paths forbidden");
for c in path.components() {
match c {
Component::Normal(seg) => ensure!(seg != ".git", ".git access forbidden"),
Component::CurDir => {}
_ => anyhow::bail!("path escapes workspace: {p}"),
}
}
Ok(())
}
const RENAME_COPY_HEADERS: [&str; 4] = ["rename from ", "rename to ", "copy from ", "copy to "];
/// Appended to every `---`/`+++` rejection. That rule is the one place where a
/// hunk BODY line can be taken for a header — `-` and `+` are the body markers
/// themselves — and the bare error ("lacks the a/ or b/ prefix", "path escapes
/// workspace") is baffling if all you did was delete a comment. Say the cause
/// out loud. No other header rule can collide: a body line always carries a
/// `-`, `+` or space marker, so it can never begin with `diff --git ` or
/// `rename `.
const HEADER_COLLISION_NOTE: &str = "\
note: this line was vetted as a `---`/`+++` file header. The scanner does not track hunk \
boundaries — deliberate, and it stays that way: a real header must never slip past it as body \
text — so a hunk body line reaches this rule too. REMOVING a line whose own text begins with \
`-- ` renders at column 0 as `--- ...` (an unindented SQL/Lua `-- comment`), and ADDING one \
that begins with `++ ` renders as `+++ ...`. If that is what this line is, the patch is being \
rejected on a false positive. Workaround: leave such lines untouched as context and edit \
around them. Rewording or indenting the line does NOT help — the removal itself is what \
renders as `--- `. Mind the space: `--x`/`++x` (C decrement/increment) never collide.";
/// Vet every path-bearing header line of a unified git diff: `diff --git`,
/// `---`, `+++`, and the rename/copy extended headers. The whole diff is
/// rejected if any extracted path is absolute, contains a `..` component,
/// is quoted (git's C-style quoting can encode `..` invisibly), or — for
/// `---`/`+++` lines — lacks the standard `a/`/`b/` prefix (`--no-prefix`
/// output). Every line is scanned without tracking hunk boundaries, so a
/// content line colliding with a header prefix is vetted too: over-rejection
/// is the safe direction, and `git apply` remains the backstop.
pub fn check_diff_paths(diff: &str) -> Result<()> {
for line in diff.lines() {
if let Some(rest) = line.strip_prefix("diff --git ") {
let (a, b) = split_git_header_paths(rest).ok_or_else(|| {
anyhow!(
"cannot parse diff header {line:?}: expected \
`diff --git a/<path> b/<path>` with plain unquoted paths"
)
})?;
vet_extracted(a)?;
vet_extracted(b)?;
} else if line.starts_with("--- ") || line.starts_with("+++ ") {
vet_file_header(&line[4..]).map_err(|e| {
anyhow!("{e}\n offending line: {line:?}\n {HEADER_COLLISION_NOTE}")
})?;
} else if let Some(rest) = RENAME_COPY_HEADERS
.iter()
.find_map(|pre| line.strip_prefix(pre))
{
vet_extracted(header_path(rest))?;
}
}
Ok(())
}
/// Vet the path on a `---`/`+++` file-header line; `rest` is the text after
/// the four-byte marker. Errors are wrapped by the caller with
/// `HEADER_COLLISION_NOTE`, since any line reaching here *might* be a hunk
/// body line rather than a header.
fn vet_file_header(rest: &str) -> Result<()> {
let p = header_path(rest);
if p == "/dev/null" {
return Ok(());
}
ensure!(
!p.starts_with('"'),
"quoted diff paths forbidden: {p} (rewrite the patch with plain ASCII paths)"
);
let stripped = p
.strip_prefix("a/")
.or_else(|| p.strip_prefix("b/"))
.ok_or_else(|| {
anyhow!(
"diff path {p:?} lacks the a/ or b/ prefix — regenerate with \
standard `git diff` output (--no-prefix diffs are rejected)"
)
})?;
vet_extracted(stripped)
}
/// Header paths may carry a traditional-diff `\t<timestamp>` suffix.
fn header_path(rest: &str) -> &str {
rest.split('\t').next().unwrap_or(rest).trim_end()
}
/// `diff --git a/<x> b/<y>`: split at the LAST ` b/` so a ` b/` embedded
/// in <x> cannot truncate <y>; splitting only ever removes the three
/// bytes ` b/`, so a `..` component in either real path always survives
/// intact into one of the two fragments.
fn split_git_header_paths(rest: &str) -> Option<(&str, &str)> {
let a_rest = rest.strip_prefix("a/")?;
let idx = a_rest.rfind(" b/")?;
Some((&a_rest[..idx], a_rest[idx + 3..].trim_end()))
}
fn vet_extracted(p: &str) -> Result<()> {
if p == "/dev/null" {
return Ok(());
}
ensure!(!p.is_empty(), "empty path in diff header");
ensure!(
!p.starts_with('"'),
"quoted diff paths forbidden: {p} (rewrite the patch with plain ASCII paths)"
);
ensure!(
!p.starts_with('/'),
"absolute diff paths forbidden (only /dev/null is allowed, for new/deleted files): {p}"
);
check_rel_path(p)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn rel_path_rules() {
assert!(check_rel_path("src/lib.rs").is_ok());
assert!(check_rel_path("./src/lib.rs").is_ok());
assert!(check_rel_path("../outside").is_err());
assert!(check_rel_path("a/../b").is_err());
assert!(check_rel_path("/etc/passwd").is_err());
assert!(check_rel_path(".git/config").is_err());
assert!(check_rel_path("a/.git/hooks").is_err());
}
#[test]
fn normal_diff_passes() {
let diff = "diff --git a/src/lib.rs b/src/lib.rs\n\
index 1111111..2222222 100644\n\
--- a/src/lib.rs\n\
+++ b/src/lib.rs\n\
@@ -1,2 +1,2 @@\n\
-old\n\
+new\n";
assert!(check_diff_paths(diff).is_ok());
}
#[test]
fn new_and_deleted_file_dev_null_passes() {
let diff = "diff --git a/newfile.rs b/newfile.rs\n\
new file mode 100644\n\
--- /dev/null\n\
+++ b/newfile.rs\n\
@@ -0,0 +1 @@\n\
+hello\n\
diff --git a/gone.rs b/gone.rs\n\
deleted file mode 100644\n\
--- a/gone.rs\n\
+++ /dev/null\n\
@@ -1 +0,0 @@\n\
-bye\n";
assert!(check_diff_paths(diff).is_ok());
}
#[test]
fn in_workspace_rename_passes() {
let diff = "diff --git a/old.rs b/nested/new.rs\n\
similarity index 100%\n\
rename from old.rs\n\
rename to nested/new.rs\n";
assert!(check_diff_paths(diff).is_ok());
}
/// The exact fixture from the apply-escape audit: a rename whose
/// destination climbs out of the workspace. `---`/`+++` scanning alone
/// never sees it — the paths live only in the `diff --git` and
/// rename headers.
#[test]
fn rename_escape_fixture_rejected() {
let diff = "diff --git a/src.txt b/../outside/escaped.txt\n\
similarity index 100%\n\
rename from src.txt\n\
rename to ../outside/escaped.txt\n";
assert!(check_diff_paths(diff).is_err());
}
#[test]
fn rename_headers_rejected_without_git_header_line() {
let diff = "similarity index 100%\n\
rename from src.txt\n\
rename to ../outside/escaped.txt\n";
assert!(check_diff_paths(diff).is_err());
}
#[test]
fn copy_escape_rejected() {
let diff = "diff --git a/src.txt b/src2.txt\n\
copy from src.txt\n\
copy to ../outside/copied.txt\n";
assert!(check_diff_paths(diff).is_err());
}
#[test]
fn no_prefix_diff_rejected() {
let diff = "diff --git src/lib.rs src/lib.rs\n\
index 1111111..2222222 100644\n\
--- src/lib.rs\n\
+++ src/lib.rs\n\
@@ -1 +1 @@\n\
-old\n\
+new\n";
let err = check_diff_paths(diff).unwrap_err().to_string();
assert!(
err.contains("diff --git") || err.contains("a/ or b/ prefix"),
"{err}"
);
}
#[test]
fn absolute_path_diff_rejected() {
let diff = "diff --git a/x b/x\n\
rename from x\n\
rename to /etc/passwd\n";
assert!(check_diff_paths(diff).is_err());
let plus = "--- a/x\n\
+++ /etc/passwd\n";
assert!(check_diff_paths(plus).is_err());
}
#[test]
fn parent_component_in_file_headers_rejected() {
assert!(check_diff_paths("--- a/../escape.rs\n+++ b/escape.rs\n").is_err());
assert!(check_diff_paths("--- a/ok.rs\n+++ b/nested/../../escape.rs\n").is_err());
assert!(check_diff_paths("diff --git a/ok.rs b/../escape.rs\n").is_err());
}
#[test]
fn dot_git_path_rejected() {
let diff = "diff --git a/.git/hooks/pre-commit b/.git/hooks/pre-commit\n\
--- a/.git/hooks/pre-commit\n\
+++ b/.git/hooks/pre-commit\n";
assert!(check_diff_paths(diff).is_err());
}
#[test]
fn quoted_paths_rejected() {
assert!(check_diff_paths("diff --git \"a/x\" \"b/y\"\n").is_err());
assert!(check_diff_paths("--- \"a/x\"\n+++ b/y\n").is_err());
assert!(check_diff_paths("rename to \"\\056\\056/evil\"\n").is_err());
}
#[test]
fn embedded_b_slash_cannot_hide_escape() {
// Whatever way ` b/` splits this header, the `..` lands in a fragment.
assert!(check_diff_paths("diff --git a/x b/y b/../evil\n").is_err());
assert!(check_diff_paths("diff --git a/../evil b/y b/z\n").is_err());
}
/// A removed body line that renders as `--- ` at column 0 — deleting an
/// unindented SQL/Lua `-- comment` — is still REJECTED (fail-closed is the
/// point; tracking hunk boundaries here would risk desyncing from git's own
/// parser). But the error must name that cause instead of leaving an agent
/// staring at a complaint about a/ and b/ prefixes it never wrote.
#[test]
fn body_line_colliding_with_header_prefix_explains_itself() {
let diff = "diff --git a/q.sql b/q.sql\n\
--- a/q.sql\n\
+++ b/q.sql\n\
@@ -1,2 +1 @@\n\
--- drop this comment\n\
\x20SELECT 1;\n";
let err = check_diff_paths(diff).unwrap_err().to_string();
assert!(err.contains("--- drop this comment"), "{err}");
assert!(err.contains("hunk body line"), "{err}");
assert!(err.contains("context"), "{err}");
}
/// The collision needs the SPACE. A removed `--x;` renders as `---x;` and an
/// added `++x;` as `+++x;` — neither is a `--- `/`+++ ` header, and both
/// must sail through. Guards against anyone "fixing" the note above by
/// widening the prefix match.
#[test]
fn body_line_without_the_space_does_not_collide() {
let diff = "diff --git a/x.c b/x.c\n\
--- a/x.c\n\
+++ b/x.c\n\
@@ -1,2 +1,2 @@\n\
---x;\n\
+++x;\n";
assert!(check_diff_paths(diff).is_ok());
}
#[test]
fn timestamp_suffix_tolerated() {
let diff = "--- a/src/lib.rs\t2026-07-13 00:00:00\n\
+++ b/src/lib.rs\t2026-07-13 00:00:00\n";
assert!(check_diff_paths(diff).is_ok());
}
}
crates/harnessd/src/daemon.rs
+556
View File
@@ -0,0 +1,556 @@
//! The Unix daemon proper: stdio/socket serving, op dispatch, and the
//! detached verdict pipeline (setsid + Unix sockets keep this POSIX-only;
//! path vetting lives in the portable `allowlist` module).
use crate::allowlist;
use anyhow::{anyhow, ensure, Result};
use serde_json::{json, Value};
use sha2::{Digest, Sha256};
use std::io::{BufRead, Write};
use std::os::unix::net::UnixListener;
use std::path::Path;
use std::process::Command;
use std::sync::Mutex;
/// Serializes mutating ops (git apply, pending writes) across socket
/// connections; reads don't need it but the cost is nil at one agent.
static HANDLE_LOCK: Mutex<()> = Mutex::new(());
pub fn run() -> Result<()> {
let root = std::env::current_dir()?.canonicalize()?;
let args: Vec<String> = std::env::args().collect();
if let Some(i) = args.iter().position(|a| a == "--socket") {
let path = args
.get(i + 1)
.ok_or_else(|| anyhow!("--socket requires a path"))?;
return serve_socket(&root, Path::new(path));
}
let stdin = std::io::stdin();
let mut stdout = std::io::stdout().lock();
for line in stdin.lock().lines() {
let line = line?;
if line.trim().is_empty() {
continue;
}
let resp = respond(&root, &line);
writeln!(stdout, "{resp}")?;
stdout.flush()?;
}
Ok(())
}
/// SPEC §10 socket mode: harnessd runs as the trusted uid and listens on a
/// Unix socket; the agent process runs as an unprivileged user whose only
/// write path into the repo is this daemon. Socket permissions (owner/group/
/// mode) are set by the supervisor script that starts us.
fn serve_socket(root: &Path, sock: &Path) -> Result<()> {
if let Some(dir) = sock.parent() {
std::fs::create_dir_all(dir)?;
}
let _ = std::fs::remove_file(sock);
let listener = UnixListener::bind(sock)?;
eprintln!("harnessd listening on {}", sock.display());
for conn in listener.incoming() {
let stream = match conn {
Ok(s) => s,
Err(e) => {
eprintln!("accept error: {e}");
continue;
}
};
let root = root.to_path_buf();
std::thread::spawn(move || {
let reader = match stream.try_clone() {
Ok(s) => std::io::BufReader::new(s),
Err(_) => return,
};
let mut writer = stream;
for line in reader.lines() {
let Ok(line) = line else { break };
if line.trim().is_empty() {
continue;
}
let resp = respond(&root, &line);
if writeln!(writer, "{resp}").is_err() {
break;
}
}
});
}
Ok(())
}
fn respond(root: &Path, line: &str) -> Value {
// read_verdict long-polls the ledger (read-only) and must not hold
// the lock that serializes mutating ops while it sleeps.
let is_read_verdict = serde_json::from_str::<Value>(line)
.ok()
.and_then(|v| {
v.get("op")
.and_then(Value::as_str)
.map(|o| o == "read_verdict")
})
.unwrap_or(false);
let _guard = if is_read_verdict {
None
} else {
Some(HANDLE_LOCK.lock().unwrap_or_else(|p| p.into_inner()))
};
match handle(root, line) {
Ok(v) => json!({"ok": true, "result": v}),
Err(e) => json!({"ok": false, "error": e.to_string()}),
}
}
/// Single-quote a string for safe embedding in a generated shell script.
fn shq(s: &str) -> String {
format!("'{}'", s.replace('\'', "'\\''"))
}
/// The exact probe `scripts/no-net.sh` runs before it decides whether to
/// isolate: can this host open an unprivileged network namespace? Run it here
/// too — same uid, same environment, same host as the wrapper we are about to
/// spawn — so the isolation note we record describes the run that actually
/// happened instead of the one we hoped for.
fn netns_available() -> bool {
Command::new("unshare")
.args(["--net", "--map-current-user", "true"])
.stdout(std::process::Stdio::null())
.stderr(std::process::Stdio::null())
.status()
.map(|s| s.success())
.unwrap_or(false)
}
fn field<'a>(req: &'a Value, key: &str) -> Result<&'a str> {
req.get(key)
.and_then(Value::as_str)
.ok_or_else(|| anyhow!("missing string field: {key}"))
}
/// A target name must be a plain directory name — no separators, no dots.
fn check_target(name: &str) -> Result<()> {
ensure!(
!name.is_empty()
&& name
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_'),
"invalid target name"
);
Ok(())
}
const RUNNING_LOCK: &str = "results/pending/RUNNING.lock";
/// The detached verdict pipeline benches the workspace tree it was launched
/// against; HANDLE_LOCK cannot cover it, so a lock file does. Refuse any op
/// that would swap that tree while a pipeline is in flight.
fn ensure_no_running_pipeline(root: &Path) -> Result<()> {
let lock = root.join(RUNNING_LOCK);
if lock.exists() {
let rid = std::fs::read_to_string(&lock).unwrap_or_default();
let rid = rid.trim();
let rid = if rid.is_empty() { "unknown" } else { rid };
anyhow::bail!(
"verdict pipeline {rid} in flight ({RUNNING_LOCK} present — wait for \
read_verdict to resolve it, or remove the lock if the pipeline is dead)"
);
}
Ok(())
}
fn handle(root: &Path, line: &str) -> Result<Value> {
let req: Value = serde_json::from_str(line)?;
match field(&req, "op")? {
"read_profile" => {
let t = field(&req, "target")?;
check_target(t)?;
let text = std::fs::read_to_string(root.join(format!("results/{t}/hotspots.txt")))?;
Ok(json!({"hotspots_text": text}))
}
"read_ledger" => {
let t = field(&req, "target")?;
check_target(t)?;
// Read-only surface over the append-only DB.
let ledger = ledger::Ledger::open(&root.join("results/ledger.sqlite"))?;
let classes = ledger.attempted_classes(t)?;
let history = ledger.attempt_history(t)?;
Ok(json!({
"attempted_playbook_classes": classes,
"attempts": history,
"total_attempts": ledger.count()?,
"note": "a class may be re-entered with a materially NEW hypothesis; \
never duplicate a (hotspot, class, hypothesis) that has a verdict",
}))
}
"read_playbook" => {
let class = match req.get("class") {
Some(Value::Number(n)) => n.as_u64().ok_or_else(|| anyhow!("bad class"))? as u8,
Some(Value::String(s)) => s.parse()?,
_ => anyhow::bail!("missing class"),
};
ensure!((1..=7).contains(&class), "playbook class must be 1-7");
let dir = root.join("playbook");
let entry = std::fs::read_dir(&dir)?
.filter_map(|e| e.ok())
.find(|e| {
e.file_name()
.to_string_lossy()
.starts_with(&format!("{class:02}-"))
})
.ok_or_else(|| anyhow!("no playbook file for class {class}"))?;
Ok(json!({"markdown": std::fs::read_to_string(entry.path())?}))
}
"read_target_source" => {
let t = field(&req, "target")?;
let p = field(&req, "path")?;
check_target(t)?;
allowlist::check_rel_path(p)?;
// Serve the PRISTINE (HEAD) content, not the working tree:
// propose_patch applies each proposal against a reset
// workspace, so reads must show the same base the diff needs
// — otherwise reads after a proposal show the patched tree
// and the next diff's context lines drift (phase2-comrak-005
// burned turns rediscovering this).
let ws = root.join(format!("targets/{t}/workspace"));
let shown = Command::new("git")
.args(["-C", ws.to_str().unwrap(), "show", &format!("HEAD:{p}")])
.output()?;
ensure!(
shown.status.success(),
"cannot read {p} at HEAD: {}",
String::from_utf8_lossy(&shown.stderr).trim()
);
let text = String::from_utf8_lossy(&shown.stdout).into_owned();
// Optional line window so large source files (100KB+) can be read in
// chunks instead of overflowing one response — otherwise the agent
// is tempted to reach for a shell it must not have.
let lines: Vec<&str> = text.lines().collect();
let total = lines.len();
let offset = req.get("offset").and_then(Value::as_u64).unwrap_or(0) as usize;
let limit = req
.get("limit")
.and_then(Value::as_u64)
.map(|v| v as usize)
.unwrap_or(usize::MAX);
let end = offset.saturating_add(limit).min(total);
let start = offset.min(total);
let slice = lines[start..end]
.iter()
.enumerate()
.map(|(i, l)| format!("{}\t{}", start + i + 1, l))
.collect::<Vec<_>>()
.join("\n");
Ok(json!({
"content": slice,
"total_lines": total,
"returned_lines": [start + 1, end],
"truncated": end < total,
}))
}
"propose_patch" => {
let t = field(&req, "target")?;
let diff = field(&req, "diff")?;
let hypothesis = field(&req, "hypothesis")?;
check_target(t)?;
ensure_no_running_pipeline(root)?;
// git rejects a diff without a final newline with an opaque
// "corrupt patch" — say it plainly (phase2-comrak-004 burned
// several turns rediscovering this).
ensure!(
diff.ends_with('\n'),
"diff must end with a trailing newline (git would report 'corrupt patch' at EOF)"
);
// Allowlist: every path named by any diff header must be a safe
// relative path (the git -C below roots them in the target
// workspace; nothing outside it is reachable).
allowlist::check_diff_paths(diff)?;
let ws = root.join(format!("targets/{t}/workspace"));
// Every proposal stands alone: reset the workspace to HEAD
// first so probe/iteration patches cannot accumulate into a
// tested tree the ledger's patch field doesn't describe
// (phase2-comrak-004 audit finding).
ensure!(
Command::new("git")
.args(["-C", ws.to_str().unwrap(), "checkout", "--", "."])
.status()?
.success(),
"workspace reset failed"
);
ensure!(
Command::new("git")
.args(["-C", ws.to_str().unwrap(), "clean", "-fdq"])
.status()?
.success(),
"workspace clean failed"
);
let run_git_apply = |check: bool| -> Result<std::process::Output> {
let mut args = vec!["-C", ws.to_str().unwrap(), "apply"];
if check {
args.push("--check");
}
args.push("-");
let mut child = Command::new("git")
.args(&args)
.stdin(std::process::Stdio::piped())
.stderr(std::process::Stdio::piped())
.spawn()?;
child.stdin.take().unwrap().write_all(diff.as_bytes())?;
Ok(child.wait_with_output()?)
};
let checked = run_git_apply(true)?;
ensure!(
checked.status.success(),
"diff does not apply cleanly against the pristine workspace \
(proposals are standalone; prior proposals were reset): {}",
String::from_utf8_lossy(&checked.stderr).trim()
);
let applied = run_git_apply(false)?;
ensure!(
applied.status.success(),
"git apply failed: {}",
String::from_utf8_lossy(&applied.stderr).trim()
);
let patch_id = format!("{:x}", Sha256::digest(diff.as_bytes()))[..12].to_string();
let pending = root.join("results/pending");
std::fs::create_dir_all(&pending)?;
std::fs::write(
pending.join(format!("{patch_id}.json")),
serde_json::to_string_pretty(
&json!({"target": t, "hypothesis": hypothesis, "diff": diff}),
)?,
)?;
Ok(json!({"patch_id": patch_id}))
}
"run_verdict" => {
let patch_id = field(&req, "patch_id")?;
ensure!(
patch_id.chars().all(|c| c.is_ascii_hexdigit()) && patch_id.len() == 12,
"bad patch id"
);
let run_id = field(&req, "run_id")?;
ensure!(
run_id
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_'),
"bad run id"
);
let class = field(&req, "playbook_class")?;
let hotspot = field(&req, "hotspot")?;
ensure_no_running_pipeline(root)?;
let pending: Value = serde_json::from_str(&std::fs::read_to_string(
root.join(format!("results/pending/{patch_id}.json")),
)?)?;
let t = pending["target"].as_str().unwrap();
let diff = pending["diff"].as_str().unwrap();
// Defense in depth: the pending file sat on disk since
// propose_patch vetted it — vet again before applying.
allowlist::check_diff_paths(diff)?;
let spec = diff_test::target::TargetSpec::load(root, t)?;
let diff_path = root.join(format!("results/pending/{patch_id}.json.diff"));
std::fs::write(&diff_path, diff)?;
// Re-establish the exact tree the pipeline must test: proposals
// after this one may have reset/repatched the workspace.
let ws = root.join(format!("targets/{t}/workspace"));
let git_ws = |args: &[&str]| -> Result<()> {
let out = Command::new("git").arg("-C").arg(&ws).args(args).output()?;
ensure!(
out.status.success(),
"git {} failed in target workspace: {}",
args.join(" "),
String::from_utf8_lossy(&out.stderr).trim()
);
Ok(())
};
git_ws(&["checkout", "--", "."])?;
git_ws(&["clean", "-fdq"])?;
let dp = diff_path.to_str().unwrap();
git_ws(&["apply", "--check", dp]).map_err(|e| {
anyhow!(
"pending diff {patch_id} no longer applies to the pristine \
workspace — re-propose it: {e}"
)
})?;
git_ws(&["apply", dp])?;
// The build + verdict pipeline runs for minutes — far past the MCP
// transport's per-call cap. Launch it detached, writing progress to
// a log; the agent observes the result via the read_verdict op,
// which reads the append-only ledger row once written.
// {out} isolates the candidate build (cargo: CARGO_TARGET_DIR;
// make: build+copy into it) — language-agnostic, mirrors the
// baseline rebuild in verdict.
let cand_dir = root
.join(format!("targets/{t}/candidate-{patch_id}"))
.to_string_lossy()
.into_owned();
let cand_build = diff_test::target::subst_out(&spec.build.baseline, &cand_dir);
let cand_bin = diff_test::target::subst_out(&spec.build.binary, &cand_dir);
let log = format!("results/pending/{run_id}.log");
// Target code must not reach the network: the pipeline launches
// through a wrapper (default scripts/no-net.sh, a network-namespace
// exec); HOTPATH_VERDICT_WRAPPER overrides with a single executable
// path for hosts where unshare is unavailable.
let (wrapper, isolation_note) = match std::env::var("HOTPATH_VERDICT_WRAPPER") {
Ok(w) => {
let note = w.clone();
(w, note)
}
Err(_) => {
let w = root
.join("scripts/no-net.sh")
.to_string_lossy()
.into_owned();
// Honesty: this note is echoed to the agent AND stamped into
// the ledger's env fingerprint. With HOTPATH_ALLOW_UNISOLATED=1
// set, no-net.sh execs the pipeline with FULL network access on
// any host where it cannot open a netns — recording a bare
// "no-net.sh" there would claim an isolation the run never had.
// Probe the same condition the wrapper will test, and say which
// of the two actually applies.
let allow_unisolated = std::env::var("HOTPATH_ALLOW_UNISOLATED")
.map(|v| v == "1")
.unwrap_or(false);
let note = if allow_unisolated && !netns_available() {
"no-net.sh (HOTPATH_ALLOW_UNISOLATED=1 — network NOT isolated)"
} else {
"no-net.sh"
};
(w, note.to_string())
}
};
// Two scripts, because failure accounting has to survive a wrapper
// that never execs its argv.
//
// INNER — runs inside the isolation wrapper. The guarded section is
// a SUBSHELL, not the left operand of an `&&`: POSIX sh ignores
// `set -e` inside any compound command that is an AND-OR operand
// other than the last, so the previous
// { set -e; <build>; verdict; } && rm -f lock || { ...; }
// shape left errexit inert. A failing candidate build fell straight
// through to `cargo run -p verdict`, which then benched — and wrote
// a ledger row for — a STALE or missing candidate binary. That is
// the exact class of false evidence this project exists to prevent.
// A standalone subshell scopes errexit and makes it live again, so a
// failed build aborts before verdict can measure anything. The
// subshell is the script's last command, so its status is the
// script's status.
let inner = format!(
"printf %s {rid} > {lock}\n\
( set -e\n{build}\ncargo run -q -p verdict -- {tgt} \
--rebuild-baseline --candidate-bin {cbin} --run-id {rid} --playbook-class {cls} \
--hypothesis {hyp} --hotspot {hs} --patch-file {pf} --isolation-note {iso}\n\
)\n",
rid = shq(run_id),
lock = shq(RUNNING_LOCK),
build = cand_build,
tgt = shq(t),
cbin = shq(&cand_bin),
cls = shq(class),
hyp = shq(pending["hypothesis"].as_str().unwrap()),
hs = shq(hotspot),
pf = shq(diff_path.to_str().unwrap()),
iso = shq(&isolation_note),
);
// OUTER — runs OUTSIDE the wrapper: invoke the wrapper, then account
// for its exit status no matter what it did. scripts/no-net.sh exits
// 97 WITHOUT exec'ing its argv when unprivileged netns are
// unavailable, and a missing or non-executable HOTPATH_VERDICT_WRAPPER
// exits 127/126 — in all three cases the inner script never runs at
// all. While this accounting lived inside the wrapper, those cases
// removed no lock and wrote no marker: read_verdict polled "running"
// forever and every later propose_patch/run_verdict was refused with
// "verdict pipeline in flight" until a human deleted the lock by hand.
// Out here it always runs, so the run is reported failed and the
// daemon stays usable. The wrapper's own stderr (no-net.sh's refusal
// text) is already captured in the run log.
let script = format!(
"{wrapper} sh -c {inner}\n\
rc=$?\n\
rm -f {lock}\n\
[ \"$rc\" -eq 0 ] || echo \"HOTPATH-PIPELINE-FAILED (exit $rc)\"\n",
wrapper = shq(&wrapper),
inner = shq(&inner),
lock = shq(RUNNING_LOCK),
);
let logf = std::fs::File::create(root.join(&log))?;
let errf = logf.try_clone()?;
// Write the lock before spawning: the detached script's own
// write lands only after exec latency, and a propose_patch in
// that window would swap the tree under the pipeline.
let lock_path = root.join(RUNNING_LOCK);
std::fs::write(&lock_path, run_id)?;
// The wrapper is invoked BY the outer script (and is shq'd into it),
// not by setsid — that is what puts the accounting outside it.
let spawned = Command::new("setsid")
.arg("sh")
.arg("-c")
.arg(&script)
.current_dir(root)
.stdout(std::process::Stdio::from(logf))
.stderr(std::process::Stdio::from(errf))
.spawn();
if let Err(e) = spawned {
let _ = std::fs::remove_file(&lock_path);
return Err(anyhow!("failed to launch verdict pipeline: {e}"));
}
Ok(json!({"status": "started", "run_id": run_id, "log": log,
"isolation": isolation_note,
"note": "pipeline runs detached; poll read_verdict with this run_id"}))
}
"read_verdict" => {
let run_id = field(&req, "run_id")?;
// Long-poll: the pipeline runs for minutes and the MCP transport
// caps a tool call at 60s. Waiting ~45s server-side per poll cuts
// the agent's turn burn ~15x versus instant "running" replies.
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(45);
loop {
let ledger = ledger::Ledger::open(&root.join("results/ledger.sqlite"))?;
if let Some(v) = ledger.verdict_summary(run_id)? {
return Ok(v);
}
drop(ledger);
// A pipeline that died before the verdict binary ran never
// writes a row; the launcher's failure accounting — which runs
// outside the isolation wrapper, so it survives a wrapper that
// refuses to exec — marks the log instead.
if run_id
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_')
{
let log = root.join(format!("results/pending/{run_id}.log"));
if let Ok(text) = std::fs::read_to_string(&log) {
if text.contains("HOTPATH-PIPELINE-FAILED") {
let tail: String = text
.lines()
.rev()
.take(12)
.collect::<Vec<_>>()
.into_iter()
.rev()
.collect::<Vec<_>>()
.join("\n");
return Ok(json!({
"status": "failed",
"note": "the pipeline died before producing a verdict — usually a \
candidate build failure, otherwise a crash or the isolation \
wrapper refusing to run (scripts/no-net.sh exits 97 when it \
cannot open a network namespace). The marker line carries the \
exit status. No ledger row was or will be written for this \
run_id, and the run lock has been released. Log tail follows.",
"log_tail": tail,
}));
}
}
}
if std::time::Instant::now() >= deadline {
return Ok(json!({"status": "running",
"note": "no ledger row yet; the pipeline is still building/benching — poll again"}));
}
std::thread::sleep(std::time::Duration::from_secs(3));
}
}
other => Err(anyhow!(
"unknown op: {other} (seven ops, SPEC §3.5 + async read_verdict)"
)),
}
}
crates/harnessd/src/main.rs
+22 -431
View File
@@ -1,438 +1,29 @@
//! harnessd — the only door between the agent and the trust layer
//! (SPEC §3.5, §10). One JSON request per stdin line, one JSON response
//! per stdout line. The agent gets exactly six operations; everything
//! else (shell, trust-layer writes, patches outside the target
//! workspace) simply has no code path here. Phase 1 caveat, recorded:
//! same-uid filesystem read-only enforcement still requires the
//! separate-user/container setup (Phase 1 infra gap).
//! per stdout line. The agent gets exactly seven operations (the six
//! synchronous ops plus the async read_verdict poll); everything else
//! (shell, trust-layer writes, patches outside the target workspace)
//! simply has no code path here. Phase 1 caveat, recorded: same-uid
//! filesystem read-only enforcement still requires the separate-user/
//! container setup (Phase 1 infra gap).
//!
//! The daemon itself is POSIX-only (Unix sockets, setsid); the diff
//! allowlist is a portable module so its escape-regression tests run on
//! every development platform.
use anyhow::{anyhow, ensure, Result};
use serde_json::{json, Value};
use sha2::{Digest, Sha256};
use std::io::{BufRead, Write};
use std::os::unix::net::UnixListener;
use std::path::{Component, Path};
use std::process::Command;
use std::sync::Mutex;
// On non-Unix only the allowlist tests compile against this module.
#[cfg_attr(not(unix), allow(dead_code))]
mod allowlist;
#[cfg(unix)]
mod daemon;
/// Serializes mutating ops (git apply, pending writes) across socket
/// connections; reads don't need it but the cost is nil at one agent.
static HANDLE_LOCK: Mutex<()> = Mutex::new(());
fn main() -> Result<()> {
let root = std::env::current_dir()?.canonicalize()?;
let args: Vec<String> = std::env::args().collect();
if let Some(i) = args.iter().position(|a| a == "--socket") {
let path = args
.get(i + 1)
.ok_or_else(|| anyhow!("--socket requires a path"))?;
return serve_socket(&root, Path::new(path));
}
let stdin = std::io::stdin();
let mut stdout = std::io::stdout().lock();
for line in stdin.lock().lines() {
let line = line?;
if line.trim().is_empty() {
continue;
}
let resp = respond(&root, &line);
writeln!(stdout, "{resp}")?;
stdout.flush()?;
}
Ok(())
#[cfg(unix)]
fn main() -> anyhow::Result<()> {
daemon::run()
}
/// SPEC §10 socket mode: harnessd runs as the trusted uid and listens on a
/// Unix socket; the agent process runs as an unprivileged user whose only
/// write path into the repo is this daemon. Socket permissions (owner/group/
/// mode) are set by the supervisor script that starts us.
fn serve_socket(root: &Path, sock: &Path) -> Result<()> {
if let Some(dir) = sock.parent() {
std::fs::create_dir_all(dir)?;
}
let _ = std::fs::remove_file(sock);
let listener = UnixListener::bind(sock)?;
eprintln!("harnessd listening on {}", sock.display());
for conn in listener.incoming() {
let stream = match conn {
Ok(s) => s,
Err(e) => {
eprintln!("accept error: {e}");
continue;
}
};
let root = root.to_path_buf();
std::thread::spawn(move || {
let reader = match stream.try_clone() {
Ok(s) => std::io::BufReader::new(s),
Err(_) => return,
};
let mut writer = stream;
for line in reader.lines() {
let Ok(line) = line else { break };
if line.trim().is_empty() {
continue;
}
let resp = respond(&root, &line);
if writeln!(writer, "{resp}").is_err() {
break;
}
}
});
}
Ok(())
}
fn respond(root: &Path, line: &str) -> Value {
// read_verdict long-polls the ledger (read-only) and must not hold
// the lock that serializes mutating ops while it sleeps.
let is_read_verdict = serde_json::from_str::<Value>(line)
.ok()
.and_then(|v| {
v.get("op")
.and_then(Value::as_str)
.map(|o| o == "read_verdict")
})
.unwrap_or(false);
let _guard = if is_read_verdict {
None
} else {
Some(HANDLE_LOCK.lock().unwrap_or_else(|p| p.into_inner()))
};
match handle(root, line) {
Ok(v) => json!({"ok": true, "result": v}),
Err(e) => json!({"ok": false, "error": e.to_string()}),
}
}
/// Single-quote a string for safe embedding in a generated shell script.
fn shq(s: &str) -> String {
format!("'{}'", s.replace('\'', "'\\''"))
}
fn field<'a>(req: &'a Value, key: &str) -> Result<&'a str> {
req.get(key)
.and_then(Value::as_str)
.ok_or_else(|| anyhow!("missing string field: {key}"))
}
/// A target name must be a plain directory name — no separators, no dots.
fn check_target(name: &str) -> Result<()> {
ensure!(
!name.is_empty()
&& name
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_'),
"invalid target name"
);
Ok(())
}
/// Relative paths only; no parent components, no .git, no absolutes.
fn check_rel_path(p: &str) -> Result<()> {
let path = Path::new(p);
ensure!(!path.is_absolute(), "absolute paths forbidden");
for c in path.components() {
match c {
Component::Normal(seg) => ensure!(seg != ".git", ".git access forbidden"),
Component::CurDir => {}
_ => anyhow::bail!("path escapes workspace: {p}"),
}
}
Ok(())
}
fn handle(root: &Path, line: &str) -> Result<Value> {
let req: Value = serde_json::from_str(line)?;
match field(&req, "op")? {
"read_profile" => {
let t = field(&req, "target")?;
check_target(t)?;
let text = std::fs::read_to_string(root.join(format!("results/{t}/hotspots.txt")))?;
Ok(json!({"hotspots_text": text}))
}
"read_ledger" => {
let t = field(&req, "target")?;
check_target(t)?;
// Read-only surface over the append-only DB.
let ledger = ledger::Ledger::open(&root.join("results/ledger.sqlite"))?;
let classes = ledger.attempted_classes(t)?;
let history = ledger.attempt_history(t)?;
Ok(json!({
"attempted_playbook_classes": classes,
"attempts": history,
"total_attempts": ledger.count()?,
"note": "a class may be re-entered with a materially NEW hypothesis; \
never duplicate a (hotspot, class, hypothesis) that has a verdict",
}))
}
"read_playbook" => {
let class = match req.get("class") {
Some(Value::Number(n)) => n.as_u64().ok_or_else(|| anyhow!("bad class"))? as u8,
Some(Value::String(s)) => s.parse()?,
_ => anyhow::bail!("missing class"),
};
ensure!((1..=7).contains(&class), "playbook class must be 1-7");
let dir = root.join("playbook");
let entry = std::fs::read_dir(&dir)?
.filter_map(|e| e.ok())
.find(|e| {
e.file_name()
.to_string_lossy()
.starts_with(&format!("{class:02}-"))
})
.ok_or_else(|| anyhow!("no playbook file for class {class}"))?;
Ok(json!({"markdown": std::fs::read_to_string(entry.path())?}))
}
"read_target_source" => {
let t = field(&req, "target")?;
let p = field(&req, "path")?;
check_target(t)?;
check_rel_path(p)?;
// Serve the PRISTINE (HEAD) content, not the working tree:
// propose_patch applies each proposal against a reset
// workspace, so reads must show the same base the diff needs
// — otherwise reads after a proposal show the patched tree
// and the next diff's context lines drift (phase2-comrak-005
// burned turns rediscovering this).
let ws = root.join(format!("targets/{t}/workspace"));
let shown = Command::new("git")
.args(["-C", ws.to_str().unwrap(), "show", &format!("HEAD:{p}")])
.output()?;
ensure!(
shown.status.success(),
"cannot read {p} at HEAD: {}",
String::from_utf8_lossy(&shown.stderr).trim()
);
let text = String::from_utf8_lossy(&shown.stdout).into_owned();
// Optional line window so large source files (100KB+) can be read in
// chunks instead of overflowing one response — otherwise the agent
// is tempted to reach for a shell it must not have.
let lines: Vec<&str> = text.lines().collect();
let total = lines.len();
let offset = req.get("offset").and_then(Value::as_u64).unwrap_or(0) as usize;
let limit = req
.get("limit")
.and_then(Value::as_u64)
.map(|v| v as usize)
.unwrap_or(usize::MAX);
let end = offset.saturating_add(limit).min(total);
let start = offset.min(total);
let slice = lines[start..end]
.iter()
.enumerate()
.map(|(i, l)| format!("{}\t{}", start + i + 1, l))
.collect::<Vec<_>>()
.join("\n");
Ok(json!({
"content": slice,
"total_lines": total,
"returned_lines": [start + 1, end],
"truncated": end < total,
}))
}
"propose_patch" => {
let t = field(&req, "target")?;
let diff = field(&req, "diff")?;
let hypothesis = field(&req, "hypothesis")?;
check_target(t)?;
// git rejects a diff without a final newline with an opaque
// "corrupt patch" — say it plainly (phase2-comrak-004 burned
// several turns rediscovering this).
ensure!(
diff.ends_with('\n'),
"diff must end with a trailing newline (git would report 'corrupt patch' at EOF)"
);
// Allowlist: every path named by the diff must be a safe
// relative path (the git -C below roots them in the target
// workspace; nothing outside it is reachable). `/dev/null`
// is the one absolute path git legitimately emits, for
// new/deleted files.
for l in diff.lines() {
if let Some(p) = l
.strip_prefix("--- a/")
.or_else(|| l.strip_prefix("+++ b/"))
{
check_rel_path(p.trim())?;
}
ensure!(
(!l.starts_with("--- /") && !l.starts_with("+++ /"))
|| l == "--- /dev/null"
|| l == "+++ /dev/null",
"absolute diff paths forbidden (only /dev/null is allowed, for new/deleted files)"
);
}
let ws = root.join(format!("targets/{t}/workspace"));
// Every proposal stands alone: reset the workspace to HEAD
// first so probe/iteration patches cannot accumulate into a
// tested tree the ledger's patch field doesn't describe
// (phase2-comrak-004 audit finding).
ensure!(
Command::new("git")
.args(["-C", ws.to_str().unwrap(), "checkout", "--", "."])
.status()?
.success(),
"workspace reset failed"
);
ensure!(
Command::new("git")
.args(["-C", ws.to_str().unwrap(), "clean", "-fdq"])
.status()?
.success(),
"workspace clean failed"
);
let run_git_apply = |check: bool| -> Result<std::process::Output> {
let mut args = vec!["-C", ws.to_str().unwrap(), "apply"];
if check {
args.push("--check");
}
args.push("-");
let mut child = Command::new("git")
.args(&args)
.stdin(std::process::Stdio::piped())
.stderr(std::process::Stdio::piped())
.spawn()?;
child.stdin.take().unwrap().write_all(diff.as_bytes())?;
Ok(child.wait_with_output()?)
};
let checked = run_git_apply(true)?;
ensure!(
checked.status.success(),
"diff does not apply cleanly against the pristine workspace \
(proposals are standalone; prior proposals were reset): {}",
String::from_utf8_lossy(&checked.stderr).trim()
);
let applied = run_git_apply(false)?;
ensure!(
applied.status.success(),
"git apply failed: {}",
String::from_utf8_lossy(&applied.stderr).trim()
);
let patch_id = format!("{:x}", Sha256::digest(diff.as_bytes()))[..12].to_string();
let pending = root.join("results/pending");
std::fs::create_dir_all(&pending)?;
std::fs::write(
pending.join(format!("{patch_id}.json")),
serde_json::to_string_pretty(
&json!({"target": t, "hypothesis": hypothesis, "diff": diff}),
)?,
)?;
Ok(json!({"patch_id": patch_id}))
}
"run_verdict" => {
let patch_id = field(&req, "patch_id")?;
ensure!(
patch_id.chars().all(|c| c.is_ascii_hexdigit()) && patch_id.len() == 12,
"bad patch id"
);
let run_id = field(&req, "run_id")?;
ensure!(
run_id
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_'),
"bad run id"
);
let class = field(&req, "playbook_class")?;
let hotspot = field(&req, "hotspot")?;
let pending: Value = serde_json::from_str(&std::fs::read_to_string(
root.join(format!("results/pending/{patch_id}.json")),
)?)?;
let t = pending["target"].as_str().unwrap();
let spec = diff_test::target::TargetSpec::load(root, t)?;
let diff_path = root.join(format!("results/pending/{patch_id}.json.diff"));
std::fs::write(&diff_path, pending["diff"].as_str().unwrap())?;
// The build + verdict pipeline runs for minutes — far past the MCP
// transport's per-call cap. Launch it detached, writing progress to
// a log; the agent observes the result via the read_verdict op,
// which reads the append-only ledger row once written.
// {out} isolates the candidate build (cargo: CARGO_TARGET_DIR;
// make: build+copy into it) — language-agnostic, mirrors the
// baseline rebuild in verdict.
let cand_dir = root
.join(format!("targets/{t}/candidate-{patch_id}"))
.to_string_lossy()
.into_owned();
let cand_build = diff_test::target::subst_out(&spec.build.baseline, &cand_dir);
let cand_bin = diff_test::target::subst_out(&spec.build.binary, &cand_dir);
let log = format!("results/pending/{run_id}.log");
// On any failure (candidate build, gates crash) the trap line
// marks the log so read_verdict can report "failed" instead of
// leaving the agent polling a verdict that will never exist.
let script = format!(
"{{ set -e\n{build}\ncargo run -q -p verdict -- {tgt} \
--rebuild-baseline --candidate-bin {cbin} --run-id {rid} --playbook-class {cls} \
--hypothesis {hyp} --hotspot {hs} --patch-file {pf}\n}} || echo HOTPATH-PIPELINE-FAILED\n",
build = cand_build,
tgt = shq(t),
cbin = shq(&cand_bin),
rid = shq(run_id),
cls = shq(class),
hyp = shq(pending["hypothesis"].as_str().unwrap()),
hs = shq(hotspot),
pf = shq(diff_path.to_str().unwrap()),
);
let logf = std::fs::File::create(root.join(&log))?;
let errf = logf.try_clone()?;
Command::new("setsid")
.arg("sh")
.arg("-c")
.arg(&script)
.current_dir(root)
.stdout(std::process::Stdio::from(logf))
.stderr(std::process::Stdio::from(errf))
.spawn()?;
Ok(json!({"status": "started", "run_id": run_id, "log": log,
"note": "pipeline runs detached; poll read_verdict with this run_id"}))
}
"read_verdict" => {
let run_id = field(&req, "run_id")?;
// Long-poll: the pipeline runs for minutes and the MCP transport
// caps a tool call at 60s. Waiting ~45s server-side per poll cuts
// the agent's turn burn ~15x versus instant "running" replies.
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(45);
loop {
let ledger = ledger::Ledger::open(&root.join("results/ledger.sqlite"))?;
if let Some(v) = ledger.verdict_summary(run_id)? {
return Ok(v);
}
drop(ledger);
// A pipeline that died before the verdict binary ran never
// writes a row; its trap line marks the log instead.
if run_id
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_')
{
let log = root.join(format!("results/pending/{run_id}.log"));
if let Ok(text) = std::fs::read_to_string(&log) {
if text.contains("HOTPATH-PIPELINE-FAILED") {
let tail: String = text
.lines()
.rev()
.take(12)
.collect::<Vec<_>>()
.into_iter()
.rev()
.collect::<Vec<_>>()
.join("\n");
return Ok(json!({
"status": "failed",
"note": "the pipeline died before producing a verdict (usually a \
candidate build failure); no ledger row was or will be \
written for this run_id. Failure log tail follows.",
"log_tail": tail,
}));
}
}
}
if std::time::Instant::now() >= deadline {
return Ok(json!({"status": "running",
"note": "no ledger row yet; the pipeline is still building/benching — poll again"}));
}
std::thread::sleep(std::time::Duration::from_secs(3));
}
}
other => Err(anyhow!(
"unknown op: {other} (seven ops, SPEC §3.5 + async read_verdict)"
)),
}
#[cfg(not(unix))]
fn main() {
eprintln!("harnessd requires a Unix host (Unix sockets, setsid)");
std::process::exit(1);
}
crates/ledger/src/lib.rs
+49 -1
View File
@@ -36,7 +36,10 @@ impl Verdict {
}
}
/// Gate outcomes for one attempt (SPEC §3.2).
/// Gate outcomes for one attempt (SPEC §3.2). The trailing fields are
/// defaulted so historical rows (written before they existed) keep
/// deserializing; they live only in the `gates` JSON column — no schema
/// change, append-only doctrine untouched.
#[derive(Debug, Clone, Default, Serialize, Deserialize)]
pub struct GateResults {
pub upstream_tests: bool,
@@ -44,6 +47,15 @@ pub struct GateResults {
pub fuzz_iters: u64,
pub fuzz_divergence: bool,
pub sanitizers_clean: bool,
/// None = no TSan lane configured/run for this attempt.
#[serde(default)]
pub tsan_clean: Option<bool>,
/// Lexical NHR-classifier hits (crates/verdict src/risk.rs).
#[serde(default)]
pub risk_signals: Vec<String>,
/// "byte-identical" | "fp-tolerance"; None for historical rows.
#[serde(default)]
pub equivalence_mode: Option<String>,
}
/// Benchmark evidence for one attempt. All times in seconds; CIs are
@@ -334,6 +346,42 @@ mod tests {
assert_eq!(ledger.attempted_classes("example").unwrap(), vec![2]);
}
#[test]
fn historical_gates_json_still_deserializes() {
// Exact shape of the 34 pre-existing rows' gates column: no
// tsan_clean / risk_signals / equivalence_mode keys.
let old = r#"{"upstream_tests":true,"golden_replay":true,"fuzz_iters":4332,
"fuzz_divergence":false,"sanitizers_clean":true}"#;
let g: GateResults = serde_json::from_str(old).unwrap();
assert!(g.upstream_tests && g.golden_replay && g.sanitizers_clean);
assert_eq!(g.fuzz_iters, 4332);
assert_eq!(g.tsan_clean, None);
assert!(g.risk_signals.is_empty());
assert_eq!(g.equivalence_mode, None);
}
#[test]
fn extended_gates_round_trip() {
let g = GateResults {
upstream_tests: true,
golden_replay: true,
fuzz_iters: 10_000,
fuzz_divergence: false,
sanitizers_clean: true,
tsan_clean: Some(false),
risk_signals: vec!["concurrency".into()],
equivalence_mode: Some("fp-tolerance".into()),
};
let json = serde_json::to_string(&g).unwrap();
let back: GateResults = serde_json::from_str(&json).unwrap();
assert_eq!(back.tsan_clean, Some(false));
assert_eq!(back.risk_signals, vec!["concurrency".to_string()]);
assert_eq!(back.equivalence_mode.as_deref(), Some("fp-tolerance"));
// The keys the defensive Value readers depend on must survive.
let v: serde_json::Value = serde_json::from_str(&json).unwrap();
assert_eq!(v.get("sanitizers_clean"), Some(&serde_json::json!(true)));
}
#[test]
fn append_only_is_enforced() {
let ledger = Ledger::open_in_memory().unwrap();
crates/verdict/src/main.rs
+279 -62
View File
@@ -1,15 +1,25 @@
//! `just verdict`: the whole attempt pipeline in one command.
//! Gates on the candidate → interleaved A/B bench vs the baseline →
//! verdict by the CI-lower-bound rule → append-only ledger row.
//! Any gate failure is rejected-gate and the bench is skipped; the
//! --needs-human-review flag caps the verdict for FP/concurrency/
//! sanitizer-flagged attempts (SPEC §8) — it never upgrades one.
//! Any gate failure is rejected-gate and the bench is skipped. A verdict
//! is only ever capped, never upgraded: the lexical risk classifier
//! (src/risk.rs) and fp-tolerance equivalence route any would-be accept
//! to needs-human-review, and the --needs-human-review flag forces the
//! same cap by hand (SPEC §8).
//!
//! This is also the only flow that can run differential fuzz, because it is
//! the only one holding both sides of the comparison: it rebuilds a pristine
//! baseline and passes it to the gates alongside the candidate. Hard rule —
//! a machine `accepted` requires that fuzz gate to have actually Passed; a
//! skipped or failed one caps the verdict at needs-human-review.
mod risk;
use anyhow::Result;
use bench_runner::{config::AcceptConfig, decide, exec, fingerprint::EnvFingerprint, stats};
use clap::Parser;
use diff_test::{
all_passed, orchestrate::run_core_gates, target::TargetSpec, GateLayer, GateOutcome,
orchestrate::run_core_gates, target::TargetSpec, GateInputs, GateLayer, GateOutcome,
};
use ledger::{Attempt, BenchEvidence, GateResults, Ledger, Verdict};
use std::path::PathBuf;
@@ -48,6 +58,10 @@ struct Cli {
/// Force the needs-human-review verdict tier (FP/concurrency/UB-adjacent).
#[arg(long)]
needs_human_review: bool,
/// How this run was isolated, recorded verbatim in the env
/// fingerprint (e.g. "no-net.sh"); absent = "unwrapped-host".
#[arg(long)]
isolation_note: Option<String>,
#[arg(long, default_value = "results/ledger.sqlite")]
db: PathBuf,
#[arg(long, default_value = "config/accept.toml")]
@@ -107,6 +121,78 @@ fn rebuild_pristine_baseline(
Ok(baseline)
}
/// A run is flagged on nonzero exit OR any of these stderr markers.
/// halt_on_error makes sanitizer hits exit nonzero; the markers catch
/// builds whose runtime still exits 0 (and never a bare "ERROR:" from
/// the target's own logging, the old false-flag risk).
const SANITIZER_MARKERS: &[&str] = &[
"ERROR: AddressSanitizer",
"ERROR: LeakSanitizer",
"ERROR: ThreadSanitizer",
"WARNING: ThreadSanitizer",
"runtime error:",
];
/// One instrumented build+run lane: templates, isolation subdir, and the
/// halt-on-error runtime options for its sanitizer family.
struct SanLane<'a> {
label: &'a str,
out_subdir: &'a str,
build_tpl: &'a str,
bin_tpl: &'a str,
envs: &'a [(&'a str, &'a str)],
}
/// Build one instrumented lane and run the pinned bench workload under
/// it. Returns Some(clean), or None when the instrumented binary cannot
/// be built — an infrastructure failure, not a code defect: the caller
/// caps the accept at needs-human-review instead of crashing a verdict
/// that already carries a measured bench.
fn run_san_lane(
root: &std::path::Path,
target: &str,
spec: &TargetSpec,
lane: &SanLane,
) -> Result<Option<bool>> {
let label = lane.label;
let out_dir = root
.join(format!("targets/{target}/{}", lane.out_subdir))
.to_string_lossy()
.into_owned();
println!("{label}: instrumented build of the patched tree…");
let status = std::process::Command::new("sh")
.arg("-c")
.arg(diff_test::target::subst_out(lane.build_tpl, &out_dir))
.current_dir(root)
.status()?;
if !status.success() {
println!("{label}: BUILD FAILED — cannot verify; capping accept at needs-human-review");
return Ok(None);
}
let bin = diff_test::target::subst_out(lane.bin_tpl, &out_dir);
let mut run = std::process::Command::new("sh");
run.arg("-c")
.arg(spec.bench.command.replace("{binary}", &bin))
.current_dir(root);
for (k, v) in lane.envs {
run.env(k, v);
}
let out = run.output()?;
let stderr = String::from_utf8_lossy(&out.stderr);
let clean = out.status.success() && !SANITIZER_MARKERS.iter().any(|m| stderr.contains(m));
if clean {
println!("{label}: clean");
} else {
let lines: Vec<&str> = stderr.lines().filter(|l| !l.is_empty()).collect();
let start = lines.len().saturating_sub(8);
println!("{label}: FLAGGED (rc={:?})", out.status.code());
for l in &lines[start..] {
println!(" {l}");
}
}
Ok(Some(clean))
}
/// SPEC §8: an accept is final only if ASan+LSan are clean on the patched
/// tree over the pinned workload. Stop-the-line fix after phase2-comrak-010:
/// the pipeline auto-accepted an LSan-flagged teardown patch (sanitizers
@@ -126,50 +212,68 @@ fn sanitizer_check(
println!("sanitizers: target declares no sanitizer build — cannot verify an accept");
return Ok(None);
};
let out_dir = root
.join(format!("targets/{target}/asan"))
.to_string_lossy()
.into_owned();
println!("sanitizers: ASan build of the patched tree…");
let status = std::process::Command::new("sh")
.arg("-c")
.arg(diff_test::target::subst_out(build_tpl, &out_dir))
.current_dir(root)
.status()?;
if !status.success() {
// Infrastructure failure (missing runtime, toolchain), not a code
// defect: don't crash the verdict and lose a measured bench —
// treat "cannot build the sanitizer binary" like "no sanitizer
// declared" so the accept caps at needs-human-review (None), with
// the failure logged for the operator to fix and re-run.
println!("sanitizers: BUILD FAILED — cannot verify; capping accept at needs-human-review");
run_san_lane(
root,
target,
spec,
&SanLane {
label: "sanitizers",
out_subdir: "asan",
build_tpl,
bin_tpl,
envs: &[
("ASAN_OPTIONS", "halt_on_error=1:abort_on_error=1"),
("UBSAN_OPTIONS", "halt_on_error=1:print_stacktrace=1"),
],
},
)
}
/// TSan lane (SPEC §8), run on the accept path when the target declares
/// a TSan build — call only when spec.build.tsan is Some. Returns None
/// when the lane cannot run (missing tsan_binary, build failure); the
/// caller caps the accept, mirroring the ASan posture.
fn tsan_check(root: &std::path::Path, target: &str, spec: &TargetSpec) -> Result<Option<bool>> {
let (Some(build_tpl), Some(bin_tpl)) = (&spec.build.tsan, &spec.build.tsan_binary) else {
println!("tsan: build.tsan declared without build.tsan_binary — cannot verify");
return Ok(None);
};
run_san_lane(
root,
target,
spec,
&SanLane {
label: "tsan",
out_subdir: "tsan",
build_tpl,
bin_tpl,
envs: &[("TSAN_OPTIONS", "halt_on_error=1")],
},
)
}
let asan_bin = diff_test::target::subst_out(bin_tpl, &out_dir);
let cmd = spec.bench.command.replace("{binary}", &asan_bin);
let out = std::process::Command::new("sh")
.arg("-c")
.arg(&cmd)
// detect_leaks is a no-op for the C/C++ UBSan combo but harmless.
.env("ASAN_OPTIONS", "detect_leaks=1")
.env("UBSAN_OPTIONS", "halt_on_error=1:print_stacktrace=1")
.current_dir(root)
.output()?;
let stderr = String::from_utf8_lossy(&out.stderr);
let clean =
out.status.success() && !stderr.contains("ERROR:") && !stderr.contains("runtime error:");
if clean {
println!("sanitizers: clean");
/// One word for the state of the DifferentialFuzz gate, for the cap message.
fn fuzz_state(outcome: Option<&GateOutcome>) -> &'static str {
match outcome {
Some(GateOutcome::Passed) => "passed",
Some(GateOutcome::Failed { .. }) => "failed",
Some(GateOutcome::Skipped { .. }) => "skipped",
None => "not run",
}
}
/// Median of per-run max-RSS samples (KiB); an even count averages the
/// two middle values.
fn median_u64(xs: &[u64]) -> u64 {
let mut v = xs.to_vec();
v.sort_unstable();
let n = v.len();
if n % 2 == 1 {
v[n / 2]
} else {
let lines: Vec<&str> = stderr.lines().filter(|l| !l.is_empty()).collect();
let start = lines.len().saturating_sub(8);
println!("sanitizers: FLAGGED (rc={:?})", out.status.code());
for l in &lines[start..] {
println!(" {l}");
(v[n / 2 - 1] + v[n / 2]) / 2
}
}
Ok(Some(clean))
}
fn now_utc() -> String {
let out = std::process::Command::new("date")
@@ -197,23 +301,58 @@ fn main() -> Result<()> {
})?
};
// Gates run on the candidate binary.
let gates = run_core_gates(&root, &spec, &cli.candidate_bin)?;
for (layer, outcome) in &gates {
let patch = match &cli.patch_file {
Some(p) => std::fs::read_to_string(p)?,
None => "(no source patch)".into(),
};
// Gates run on the candidate binary — except differential fuzz, which
// differs it against the pristine baseline built just above. This is the
// only flow that has both sides, and therefore the only flow that can
// run the fuzz gate at all (hence the accept rule enforced below).
let report = run_core_gates(
&root,
&spec,
&GateInputs {
candidate_binary: &cli.candidate_bin,
baseline_binary: Some(baseline_bin.as_str()),
fuzz_iters: cfg.fuzz_iters,
},
)?;
for (layer, outcome) in &report.gates {
match outcome {
GateOutcome::Passed => println!("{layer:?}: PASS"),
GateOutcome::Failed { detail } => println!("{layer:?}: FAIL — {detail}"),
GateOutcome::Skipped { reason } => println!("{layer:?}: skipped — {reason}"),
}
}
let passed = |layer| matches!(report.outcome(layer), Some(GateOutcome::Passed));
let mut gate_results = GateResults {
upstream_tests: matches!(gates[0].1, GateOutcome::Passed),
golden_replay: matches!(gates[1].1, GateOutcome::Passed),
fuzz_iters: 0,
fuzz_divergence: false,
upstream_tests: passed(GateLayer::UpstreamTests),
golden_replay: passed(GateLayer::GoldenReplay),
fuzz_iters: report.fuzz_iters,
fuzz_divergence: report.fuzz_divergence,
sanitizers_clean: false,
tsan_clean: None,
risk_signals: risk::classify(&patch, report.equivalence_mode == "fp-tolerance"),
equivalence_mode: Some(report.equivalence_mode.clone()),
};
debug_assert!(matches!(gates[0].0, GateLayer::UpstreamTests));
if !gate_results.risk_signals.is_empty() {
println!(
"risk: lexical signals [{}] — a would-be accept routes to needs-human-review (SPEC §8)",
gate_results.risk_signals.join(", ")
);
}
// Announced up front, like the risk signals, so the rule is in the
// transcript even when another cap fires first below.
let fuzz_passed = passed(GateLayer::DifferentialFuzz);
if !fuzz_passed {
println!(
"fuzz: differential-fuzz gate {} — a would-be accept routes to needs-human-review; \
no machine accept without a real differential-fuzz run (SPEC §8)",
fuzz_state(report.outcome(GateLayer::DifferentialFuzz))
);
}
let pin = &cfg.pin_prefix;
let wrap = |bin: &str| {
@@ -225,7 +364,7 @@ fn main() -> Result<()> {
}
};
let (verdict, bench) = if !all_passed(&gates) {
let (verdict, bench) = if !report.all_passed() {
println!("verdict: rejected-gate (bench skipped)");
(Verdict::RejectedGate, None)
} else {
@@ -285,8 +424,62 @@ fn main() -> Result<()> {
v = Verdict::NeedsHumanReview;
}
}
if spec.build.tsan.is_some() {
match tsan_check(&root, &cli.target, &spec)? {
Some(clean) => {
gate_results.tsan_clean = Some(clean);
if !clean {
println!(
"verdict: TSan flag caps the accept at needs-human-review (SPEC §8)"
);
v = Verdict::NeedsHumanReview;
}
}
None => {
println!("verdict: TSan lane declared but not verifiable; capping accept at needs-human-review (SPEC §8)");
v = Verdict::NeedsHumanReview;
}
}
}
if v == Verdict::Accepted && !gate_results.risk_signals.is_empty() {
println!(
"verdict: risk signals [{}] route the accept to needs-human-review (SPEC §8)",
gate_results.risk_signals.join(", ")
);
v = Verdict::NeedsHumanReview;
}
// Hard rule (audit remediation): differential fuzz is the gate
// that catches behavioral divergence the fixed corpus misses, so
// an accept minted without it is an unverified claim. The audit
// found accepted rows carrying fuzz_iters=0 — fuzz had run
// out-of-band, or not at all. A skipped or failed fuzz gate can
// no longer produce a machine accept, only needs-human-review.
if v == Verdict::Accepted && !fuzz_passed {
println!(
"verdict: differential fuzz {} — no accept without a real differential-fuzz \
run against the pristine baseline; capping at needs-human-review. The target \
must declare [gates].fuzz in targets/{}/target.toml, and the run must report \
FUZZ-RESULT with zero divergences (SPEC §8)",
fuzz_state(report.outcome(GateLayer::DifferentialFuzz)),
cli.target
);
v = Verdict::NeedsHumanReview;
}
}
let fp = EnvFingerprint::collect(pin, "system-default");
let mut env_fingerprint = serde_json::json!({
"fingerprint": fp,
"workload": spec.bench.workload,
"target_commit": spec.source.commit,
"isolation": cli.isolation_note.as_deref().unwrap_or("unwrapped-host"),
"gates_detail": "diff-fuzz gated in-pipeline against the pristine baseline (an accept requires it to have passed); ASan+UBSan enforced on the accept path, TSan when configured",
});
if !samples.baseline_max_rss_kib.is_empty() && !samples.candidate_max_rss_kib.is_empty() {
env_fingerprint["max_rss_kib"] = serde_json::json!({
"baseline_median": median_u64(&samples.baseline_max_rss_kib),
"candidate_median": median_u64(&samples.candidate_max_rss_kib),
});
}
(
v,
Some(BenchEvidence {
@@ -296,20 +489,11 @@ fn main() -> Result<()> {
candidate_ci: (clo, chi),
speedup_median: ratio.median,
speedup_ci: (ratio.lo, ratio.hi),
env_fingerprint: serde_json::json!({
"fingerprint": fp,
"workload": spec.bench.workload,
"target_commit": spec.source.commit,
"gates_detail": "diff-fuzz per-attempt manual; ASan+LSan enforced on the accept path",
}),
env_fingerprint,
}),
)
};
let patch = match &cli.patch_file {
Some(p) => std::fs::read_to_string(p)?,
None => "(no source patch)".into(),
};
// Derive the phase from the run_id namespace (phaseN-*) so the column
// matches the run_id prefix; the historical hardcoded `1` left every
// verdict-written row tagged phase 1 regardless of namespace (found in
@@ -344,3 +528,36 @@ fn main() -> Result<()> {
);
Ok(())
}
#[cfg(test)]
mod tests {
use super::{fuzz_state, median_u64};
use diff_test::GateOutcome;
#[test]
fn median_u64_odd_and_even() {
assert_eq!(median_u64(&[5]), 5);
assert_eq!(median_u64(&[9, 1, 5]), 5);
assert_eq!(median_u64(&[4, 2, 8, 6]), 5);
}
/// Only `Passed` clears the accept bar; every other state (including a
/// gate that never ran) is a cap to needs-human-review.
#[test]
fn fuzz_state_names_every_outcome() {
assert_eq!(fuzz_state(Some(&GateOutcome::Passed)), "passed");
assert_eq!(
fuzz_state(Some(&GateOutcome::Failed {
detail: "divergences=3".into()
})),
"failed"
);
assert_eq!(
fuzz_state(Some(&GateOutcome::Skipped {
reason: "no baseline".into()
})),
"skipped"
);
assert_eq!(fuzz_state(None), "not run");
}
}
crates/verdict/src/risk.rs
+168
View File
@@ -0,0 +1,168 @@
//! Lexical NHR risk classifier (SPEC §8). Scans only the changed lines
//! of a unified diff for concurrency / unsafe / floating-point tokens
//! and routes any would-be accept that hits one to needs-human-review.
//! Deliberately lexical and conservative: over-triggering costs a human
//! a look, under-triggering ships an unreviewed risk — so plain token
//! presence decides, no semantic analysis.
const CONCURRENCY: &[&str] = &[
"pthread_",
"std::sync",
"std::thread",
"Atomic",
"atomic_",
"__atomic_",
"__sync_",
"Mutex",
"RwLock",
"Condvar",
"mpsc",
"rayon::",
"memory_order",
"volatile",
];
const UNSAFE: &[&str] = &[
"unsafe ",
"transmute",
"from_raw",
"MaybeUninit",
"UnsafeCell",
];
/// The spaced tokens carry their own trailing boundary; `f32`/`f64` get
/// hand boundary checks below.
const FP_SPACED: &[&str] = &["float ", "double "];
const FP_TYPES: &[&str] = &["f32", "f64"];
fn is_ident_letter(b: u8) -> bool {
b.is_ascii_alphabetic() || b == b'_'
}
/// True when `line` contains `token` not glued to an identifier letter on
/// either side. Digits are allowed neighbors so literal suffixes like
/// `1.0f32` still trigger; letter neighbors (identifiers, most hex
/// strings, e.g. `buf32`, `deadf64b`) do not.
fn has_fp_token(line: &str, token: &str) -> bool {
let bytes = line.as_bytes();
let mut start = 0;
while let Some(pos) = line[start..].find(token) {
let i = start + pos;
let before_ok = i == 0 || !is_ident_letter(bytes[i - 1]);
let after_ok = i + token.len() >= bytes.len() || !is_ident_letter(bytes[i + token.len()]);
if before_ok && after_ok {
return true;
}
start = i + 1;
}
false
}
fn is_changed_line(line: &str) -> bool {
(line.starts_with('+') && !line.starts_with("+++"))
|| (line.starts_with('-') && !line.starts_with("---"))
}
/// Classify a unified diff into risk-signal names. `fp_tolerance` is true
/// when the target's equivalence mode is fp-tolerance, itself a §8 signal.
/// Returned names are unique; empty means no signal.
pub fn classify(diff: &str, fp_tolerance: bool) -> Vec<String> {
let (mut concurrency, mut unsafe_sig, mut fp) = (false, false, false);
for line in diff.lines().filter(|l| is_changed_line(l)) {
concurrency = concurrency || CONCURRENCY.iter().any(|t| line.contains(t));
unsafe_sig = unsafe_sig || UNSAFE.iter().any(|t| line.contains(t));
fp = fp
|| FP_SPACED.iter().any(|t| line.contains(t))
|| FP_TYPES.iter().any(|t| has_fp_token(line, t));
}
let mut signals = Vec::new();
if concurrency {
signals.push("concurrency".to_string());
}
if unsafe_sig {
signals.push("unsafe".to_string());
}
if fp {
signals.push("floating-point".to_string());
}
if fp_tolerance {
signals.push("fp-tolerance-equivalence".to_string());
}
signals
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn concurrency_tokens_trigger() {
let diff = "--- a/src/lib.rs\n+++ b/src/lib.rs\n+ let m = Mutex::new(0);\n";
assert_eq!(classify(diff, false), vec!["concurrency"]);
let diff = "- pthread_create(&t, NULL, worker, NULL);\n";
assert_eq!(classify(diff, false), vec!["concurrency"]);
}
#[test]
fn unsafe_tokens_trigger() {
let diff = "+ unsafe { ptr.read() }\n";
assert_eq!(classify(diff, false), vec!["unsafe"]);
let diff = "+ let v: Vec<u8> = Vec::from_raw_parts(p, n, n);\n";
assert_eq!(classify(diff, false), vec!["unsafe"]);
}
#[test]
fn floating_point_tokens_trigger_with_boundaries() {
assert_eq!(
classify("+ let x: f64 = 1.0;\n", false),
vec!["floating-point"]
);
// Literal suffixes have a digit neighbor and must still trigger.
assert_eq!(
classify("+ sum += 1.0f32;\n", false),
vec!["floating-point"]
);
assert_eq!(
classify("+ double acc = 0.0;\n", false),
vec!["floating-point"]
);
// Glued to identifier letters: not an FP token.
assert!(classify("+ let buf32 = [0u8; 4];\n", false).is_empty());
assert!(classify("+ hash = \"deadf64beef\";\n", false).is_empty());
}
#[test]
fn fp_tolerance_flag_is_its_own_signal() {
assert_eq!(classify("", true), vec!["fp-tolerance-equivalence"]);
assert_eq!(
classify("+ let x: f32 = 0.5;\n", true),
vec!["floating-point", "fp-tolerance-equivalence"]
);
}
#[test]
fn clean_diff_yields_no_signals() {
let diff = "--- a/src/lib.rs\n+++ b/src/lib.rs\n@@ -1,3 +1,3 @@\n\
- let n = items.len();\n+ let n = count;\n let done = true;\n";
assert!(classify(diff, false).is_empty());
}
#[test]
fn only_changed_lines_are_scanned() {
// Tokens on context lines and in file headers must not trigger.
let diff = "--- a/src/mutex_wrapper.rs\n+++ b/src/mutex_wrapper.rs\n\
@@ -10,4 +10,4 @@\n let guard = lock.mutex();\n\
+ let n = count;\n";
assert!(classify(diff, false).is_empty());
}
#[test]
fn signals_are_deduped() {
let diff = "+ let a = Mutex::new(0);\n+ let b = RwLock::new(0);\n\
+ unsafe { transmute::<u32, f32>(x) };\n";
assert_eq!(
classify(diff, false),
vec!["concurrency", "unsafe", "floating-point"]
);
}
}
docs/src/equivalence.md
+63 -9
View File
@@ -7,6 +7,22 @@ candidate must prove it does the same thing the baseline does. These are
## The layers
0. **Pin checks.** Before any gate runs, the corpus is verified against its
committed `MANIFEST.sha256` and the harness **refuses to run** on a
mismatch — nothing is measured on tampered inputs. This ships and is
enforced for every target.
The *test suite* has the same machinery but is **not yet active**:
`diff-test` verifies `corpora/<t>/TESTSUITE.sha256` when the file is
present (mismatch = hard refusal) and prints an explicit "suite unpinned"
warning when it is absent. **No target ships a suite pin today**, so this
check currently constrains nothing. It matters because upstream suites
are vendored under `targets/<t>/workspace/` — an agent-writable path — so
"the agent cannot edit the tests" rests on a pin that still has to be
generated (`scripts/pin-testsuite.sh`, a deliberate human action, run
after a fetch). Until then, that specific guarantee is a mechanism, not
a fact.
1. **Upstream test suite.** The target's own tests, run against the patched
code. For cJSON that is the vendored Unity/CTest suite (19 tests:
`parse_number`, `parse_string`, `print_*`, `misc_tests`, …); for the Rust
@@ -20,15 +36,49 @@ candidate must prove it does the same thing the baseline does. These are
mismatch.
3. **Differential fuzzing.** Pristine vs candidate on thousands of mutated
inputs, comparing output. Every accepted win cleared **10,000 mutated
inputs with 0 divergences** — the tokei wins after canonicalizing benign
parallel-walker output ordering (a real finding: differential gates must
compare *semantics*, not bytes, where the target is legitimately
nondeterministic in presentation), the cJSON win on quote/number/escape
edge cases.
inputs, comparing output.
4. **Sanitizers.** ASan/LSan for Rust, ASan+UBSan for C. Run on **every
would-be accept** — this is machine-enforced, not manual (see below).
*Where it runs.* Fuzz is the one gate that is **not** a property of the
candidate alone: it differs the candidate against a **pristine baseline**,
so it needs both binaries. That means it runs **only on the accept path**
(`just verdict`, the one flow that rebuilds a pristine baseline). The
standalone `just gates` flow has no baseline, so the gate reports
**Skipped, with that reason recorded** — it is never faked by comparing
the candidate against itself. `just gates` is a fast equivalence check,
**not** a fuzz run. The gate is graded strictly: `diff-test` runs the
target's declared `[gates].fuzz` command and parses a
`FUZZ-RESULT iters=<n> divergences=<m>` line; a run that never prints one
**fails**, rather than passing by silence.
*The hard rule.* A machine `accepted` verdict is now **impossible**
without this gate having actually passed. A skipped or failed fuzz gate
caps the verdict at `needs-human-review` — because an accept minted
without differential fuzz is an unverified claim.
*Honest history.* Through Phase 5 this ran as an **out-of-band
per-attempt script** (`scripts/diff-fuzz-<target>.py`), not inside the
pipeline — so **8 of the 10 accepted ledger rows record `fuzz_iters=0`**
even where a 10,000-input audit run is documented in the case study. The
two exceptions are the Phase 0 tokei rows, which record 4,332. Those
out-of-band runs produced real findings — the tokei wins passed only
after canonicalizing benign parallel-walker output ordering (differential
gates must compare *semantics*, not bytes, where the target is
legitimately nondeterministic in presentation), and the cJSON win cleared
quote/number/escape edge cases with 0 divergences.
4. **Sanitizers.** ASan+UBSan (ASan/LSan on the Rust targets) is built and
run against the pinned workload on **every would-be accept** — the
sanitizer gate is machine-enforced, not manual (see below). A flagged
run, *or* a target that declares no sanitizer build at all, caps the
accept at `needs-human-review`.
A **TSan lane** exists on the same accept path, but it only runs when a
target declares `[build].tsan`. **No target declares it today, so there
is currently no TSan coverage** — the lane is wired and dormant, and a
patch touching threads needs that opt-in added before it can be checked.
(MIRI, named in the spec, was never implemented.) The concurrency safety
net that *is* live is the review tier below, which routes any
concurrency-token patch to a human.
## The tier rule (SPEC §8)
@@ -36,7 +86,11 @@ Some changes can pass every gate and still be unsafe to auto-ship. A change
that touches floating-point ordering, concurrency primitives, `unsafe`, or
carries a sanitizer flag is **never** auto-accepted — its verdict is capped
at `needs-human-review`. The tier is decided by the *nature* of the change,
never by the size of the number. The mimalloc allocator swap
never by the size of the number. The routing is machine-enforced: `verdict`
runs a conservative **lexical risk classifier** over the patch's changed
lines (concurrency / unsafe / floating-point token lists — deliberately
over-triggering, not a semantic analysis) and any signal, or the use of
fp-tolerance equivalence mode at all, forces a would-be accept to review. The mimalloc allocator swap
([case study](./case-studies/build-flags.md)) is the canonical example: a
real +4.6% win, correctly held for a human ruling because a global
allocator change's safety is not something one benchmark can clear.
docs/src/introduction.md
+1 -1
View File
@@ -29,7 +29,7 @@ and every one survived the gates above. These are not projections.
| cJSON HTTP service — p50 latency under replayed load | +6.2%, CI [+5.8%, +7.2%] |
| comrak (Rust) — mimalloc allocator swap (class 2) | +4.6%, CI [+3.3%, +5.8%], human-ruled |
Across **five phases and 33 ledger rows**, with **zero shipped false
Across **five phases and 34 ledger rows**, with **zero shipped false
accepts** — including two occasions where the pipeline itself over-accepted
and the audit caught it before anything shipped, each becoming a permanent
new gate.
docs/src/isolation.md
+29
View File
@@ -34,6 +34,35 @@ The SDK's tool allow/deny list is defense-in-depth, not the boundary —
path into the repo is the harnessd Unix socket; the trust layer is owned
by another uid. This is the production shape (`docs/infra.md`).
## Network isolation on the verdict path
Target code should not be able to phone home during a measurement. What
is actually enforced, stated honestly:
- **The agent path is wrapped.** When `harnessd` launches the verdict
pipeline, it execs through `scripts/no-net.sh`, which runs it inside an
unprivileged **network namespace** (`unshare --net --map-current-user`).
`HOTPATH_VERDICT_WRAPPER` overrides the wrapper.
- **It fails closed.** Where namespaces are unavailable the wrapper **exits
97 without running the pipeline** — it does not degrade silently. The one
bypass, `HOTPATH_ALLOW_UNISOLATED=1`, runs the pipeline **with full
network access**; it warns loudly on stderr *and* the resulting ledger row
records the isolation note `no-net.sh (HOTPATH_ALLOW_UNISOLATED=1 —
network NOT isolated)`. A run cannot claim an isolation it did not have.
- **In CI**, the bench workload runs under `docker run --network=none`.
Two limits, stated plainly rather than buried:
- **Only the harnessd (agent) path is wrapped.** A human who invokes
`just verdict` directly runs it **unwrapped on the host**; those rows are
stamped `isolation: "unwrapped-host"` in the ledger. The wrapper is a
property of the agent pipeline, not of the `verdict` binary.
- **Full-container isolation is still an open gap.** The namespace isolates
the *network* — not the filesystem, not the syscall surface. The
fully containerized, seccomp-restricted bench container of SPEC §10 is
**not built**. The agent-side OS boundary above is a separate mechanism
and *is* shipped.
## The two times the pipeline was wrong
Anti-reward-hacking is not "it never fails" — it is "when it fails, the
docs/src/measurement.md
+16
View File
@@ -27,6 +27,22 @@ reproduces). A change is accepted only if the **CI lower bound** clears the
threshold in `config/accept.toml` (default 2%) — not the median. The number
you commit to is the one that survives the interval.
## Beyond wall time: max RSS and PMU counters
Wall time is the **only** metric a verdict is ever decided on. Two
supporting measurements exist alongside it:
- **Max RSS** is captured for every timed run on Unix (`wait4`'s
`ru_maxrss`) and recorded in the run's environment fingerprint as
baseline/candidate medians — so a speedup that quietly pays for itself
in memory is visible next to the number it bought.
- **PMU counters** (cycles, instructions, cache misses, branch misses)
are opt-in diagnostics: `bench-runner compare --perf-stat` wraps each
timed run in `perf stat` and prints per-side medians with the workload
caveat attached. PMU access is unavailable in most VMs, and the
counters never participate in the accept decision — they explain a
wall-time delta, they do not replace it.
## The hardware has to earn trust first
Before any measurement session, the harness runs two self-tests, recorded
docs/src/playbook.md
+4 -2
View File
@@ -8,14 +8,16 @@ the ledger stops you from re-grinding a class that already has a verdict.
|---|---|---|---|
| 1 | Build config | LTO, codegen-units, target-cpu | tokei ✅ +10.4% |
| 2 | Allocator | Global allocator swap (mimalloc) | comrak ✅ +4.6% (human-ruled) |
| 3 | Allocation churn | Remove per-item malloc/free | tokei ✅, cJSON ✅ (+ one overturned) |
| 3 | Allocation churn | Remove per-item malloc/free | tokei ✅; cJSON rejected-bench (+ one overturned) |
| 4 | Data layout | Make the cache hierarchy fit the access pattern | comrak (attempted) |
| 5 | Algorithmic | Eliminate redundant work | tokei ×3 ✅, cJSON ✅ |
| 6 | SIMD | Autovectorization enablement | comrak, tokei (attempted) |
| 7 | Concurrency | Parallelism — always `needs-human-review` | — |
Six classes were exercised across the case studies, with verified wins on
classes 1, 2, 3, and 5 across **both a Rust and a C target**.
classes 1, 2, 3, and 5 — spanning **both a Rust and a C target** (the C
win is the class-5 cJSON rewrite; cJSON's class-3 attempt was an honest
`rejected-bench`, its CI lower bound short of the 2% bar).
## Why the ordering is load-bearing
docs/src/reproduce.md
+13 -1
View File
@@ -5,6 +5,10 @@ through the same commands.
## Prerequisites
- **A Linux/POSIX host.** The pipeline is Unix-only at runtime: `harnessd`
(Unix sockets, `setsid`), service mode, the isolation wrapper, and every
sh-based gate assume it. Windows compiles the workspace and runs the
portable unit tests, but **cannot run the pipeline**.
- Rust (stable) + `just`; `clang`/`cmake` for the C target; `valgrind` for
callgrind profiling. Nightly Rust for the sanitizer gate.
- Targets are vendored fresh (their `workspace/` is gitignored); each
@@ -21,6 +25,14 @@ just verdict <t> <bin> ... # gates + bench vs pristine baseline + ledger row
just report <run-id> # ROI report from a ledger row
```
> **`just gates` does not fuzz.** Differential fuzz differs the candidate
> against a *pristine baseline*, and only `just verdict` builds one — so on
> the `gates` flow the fuzz layer reports **Skipped**, with the reason
> recorded, rather than comparing the candidate against itself. Use `gates`
> as a fast equivalence check; the fuzz gate (and the sanitizer lanes) run
> on the accept path, where an accept is impossible without a passed fuzz
> gate.
Service mode (Phase 4):
```
@@ -42,7 +54,7 @@ Every accepted win can be re-audited independently:
```
python3 scripts/audit-attempt.py <run-id> # mechanical re-check
python3 scripts/diff-fuzz-<target>.py <dir> 10000 <candidate-bin> # 10k fuzz
python3 scripts/diff-fuzz-<target>.py <dir> 10000 <baseline-bin> <candidate-bin>
```
The ledger itself is the record: `results/ledger.sqlite`, append-only,
docs/src/research-forks.md
+13
View File
@@ -52,6 +52,19 @@ tolerance is defensible for the workload, exactly as with the mimalloc
allocator swap. The machine measures and gates; the human ratifies the
tolerance.
> **Scope, stated honestly.** The FP-tolerance *mechanism* is in the gate
> pipeline — `diff-test` honors an `equivalence.toml` policy (it needs a
> committed `[corpus].golden_reference`, and fails closed without one), and
> `verdict` auto-routes any fp-tolerance run to `needs-human-review`. But
> **`matmul` is not wired into that pipeline**: it has an `equivalence.toml`
> and **no `target.toml`**, which is the file `diff-test`/`verdict` load a
> target from. So the kernel lane is demonstrated **script-driven**
> (`scripts/kernel-lane-demo.sh`, using the `fp-compare` binary and
> `bench-runner compare` directly), and the `phase5-matmul-opt` ledger row
> was recorded outside the automated verdict path. Promoting `matmul` to a
> real pipeline target is open work — the policy is closed; that target's
> wiring is not.
**This is the GPU lane in miniature.** The environment has no GPU, so the
GPU fork is shown on the CPU as what it actually is — the same trust
machinery with a different timer. A Triton or CUDA kernel versus a reference
docs/src/roadmap.md
+20 -2
View File
@@ -44,5 +44,23 @@ Full write-up: [Research forks](./research-forks.md).
The end goal is not vendored OSS case studies — it is a **real production
system**. The same gates that matter for a JSON parser matter far more for
a live CMS: an equivalence failure there is a customer-facing bug, and the
no-network sandbox and append-only audit trail are exactly what make it
safe to point an optimization agent at code that a business depends on.
network-isolated verdict path and append-only audit trail are what would
make it safe to point an optimization agent at code that a business depends
on.
## Known gaps (what would have to close first)
Honesty about the remaining edges, since the product is trust:
- **Full-container isolation.** The verdict pipeline is network-isolated on
the agent path (`scripts/no-net.sh`); the seccomp-restricted bench
container of SPEC §10 is **not built**, and a human running `just verdict`
directly is unwrapped.
- **Test-suite pins.** The verify-or-refuse machinery exists, but **no
target ships a `TESTSUITE.sha256` yet** — so "the agent cannot edit the
upstream tests" is currently a mechanism awaiting its pins.
- **TSan.** The lane runs on the accept path only when a target declares
`[build].tsan`; **none does today**, so there is no TSan coverage. Class-7
(concurrency) work depends on that opt-in landing.
- **Kernel lane wiring.** `matmul` demonstrates FP-tolerance equivalence
script-driven; it is not yet a `target.toml` pipeline target.
justfile
+12 -5
View File
@@ -23,10 +23,15 @@ compare baseline candidate:
cargo run -p bench-runner -- --config config/accept.toml compare \
--baseline "{{baseline}}" --candidate "{{candidate}}"
# Verify a target corpus against its pinned manifest
# (wired through diff-test CLI once it exists; manual check for now)
# Verify a target corpus against its pinned manifest (never rewrites it)
pin-check target:
cd corpora/{{target}} && sh gen-corpus.sh
cd corpora/{{target}} && sh gen-corpus.sh --check
# Deliberately re-pin a target corpus manifest — a trust-layer change;
# review and commit the new MANIFEST.sha256 with justification
pin-corpus target:
@echo "WARNING: rewriting corpora/{{target}}/MANIFEST.sha256 — deliberate human action (corpora/README.md)"
cd corpora/{{target}} && sh gen-corpus.sh --pin
# Automated calibration: N A/A sessions (<5% false-positive required) +
# N injected-slowdown sessions (>=95% detection required), JSON evidence
@@ -34,8 +39,10 @@ calibrate cmd out sessions="20":
cargo run -p bench-runner -- --config config/accept.toml calibrate \
--cmd "{{cmd}}" --sessions {{sessions}} --out "{{out}}"
# Equivalence gates for a target: corpus pin (refuses on mismatch) ->
# upstream tests -> golden replay. Fuzz/sanitizers stay per-attempt.
# Equivalence gates for a target: corpus + test-suite pins (refuse on
# mismatch) -> upstream tests -> golden replay. Differential fuzz needs a
# pristine baseline to differ against, so it reports Skipped here and runs
# for real on the accept path; sanitizers likewise run in verdict.
gates target:
cargo run -p diff-test -- {{target}}
results/adversarial-review.md
+16
View File
@@ -57,3 +57,19 @@ consistent, with the compound tokei figure stated as an approximation
the case study and is now enforced by the report generator too.
- `phase0-comrak-002` (mimalloc) is `needs-human-review`, awaiting a ruling
(see `results/rulings/`).
## Erratum (2026-07-13)
Finding 1's separation claim above — "only the historical overturned row
is `accepted + false`, so the rule cleanly separates them" — was too
strong. **Five** accepted rows carry `sanitizers_clean: false`:
`phase1-verdict-smoke-002`, `phase1-ipc-smoke-001`, `phase2-tokei-001`,
`phase2-tokei-003`, and `phase2-comrak-010`. All five pre-date the
machine-enforced sanitizer lane — the Phase 1 rows are harness smoke
tests, and the two Phase 2 tokei wins were sanitizer-audited by hand
during the 100% Phase 2 audit (see the Phase 2 case study) — but the
ledger's machine record cannot distinguish "manually audited clean" from
"never run", so the report generator's not-shippable rule flags all five,
not just the overturned row. That is the correct conservative behavior;
only the description of *why* the rule separates cleanly was wrong. The
original text stands unedited above; this note is the correction.
results/phase2/case-study-agent-loop.md
+11
View File
@@ -230,3 +230,14 @@ remains the deployment shape.
pipeline into machine-enforced sanitizer gating.
- **OS-level tool-boundary isolation:** met — was the tracked hardening
gap; shipped and verified this phase.
---
> **Note (2026-07-13).** The differential-fuzz and ASan/LSan runs
> described above ran **out-of-band** — per-attempt scripts during the
> 100% human audit, exactly as this case study records ("fuzz and
> sanitizers are per-attempt manual in this phase") — so the Phase 2
> ledger rows carry `fuzz_iters=0` in their machine record. The fuzz
> gate is now pipeline-integrated (`diff-test` runs the target's
> declared fuzz command); rows written after this date record the
> iteration count actually executed.
results/phase3/01-tokei-algorithmic-rust.md
+9
View File
@@ -62,3 +62,12 @@ increments, all on the same 40%-of-runtime function — and one honest
rejection in between (`phase2-tokei-009`, a real +5.7% median whose
noisy CI lower bound of +0.2% could not clear the bar, so it does not
count).
---
> **Note (2026-07-13).** The 10,000-input differential fuzz cited above
> ran **out-of-band** (`scripts/diff-fuzz-tokei.py`, per-attempt), not
> inside the verdict pipeline — so these ledger rows record
> `fuzz_iters=0` in their machine record. The fuzz gate is now
> pipeline-integrated; rows written after this date record the
> iteration count actually executed.
results/phase3/04-cjson-c-target.md
+9
View File
@@ -84,3 +84,12 @@ about the trust story is Rust-specific. The cheapest safe lever (skip a
per-number allocation) wasn't enough; the algorithmic lever (stop doing a
redundant heavyweight parse) was — and the pipeline proved it the same
way it proves everything else.
---
> **Note (2026-07-13).** The 10,000-input differential fuzz cited above
> ran **out-of-band** (`scripts/diff-fuzz-cjson.py`, per-attempt), not
> inside the verdict pipeline — so the `phase3-cjson-002` ledger row
> records `fuzz_iters=0` in its machine record. The fuzz gate is now
> pipeline-integrated; rows written after this date record the
> iteration count actually executed.
results/phase3/README.md
+10
View File
@@ -66,3 +66,13 @@ fuzz, and sanitizers. Every rejection is a real mechanism that simply
didn't clear the 2% CI-lower-bound bar — and it is in the ledger too,
because a rejection is a complete, valid outcome. Nothing shipped that
the pipeline couldn't prove.
> **Note (2026-07-13).** "Every win survived … a 10k-input differential
> fuzz, and sanitizers" is true of the **evidence**, not of the machine
> record at the time: through Phase 5 the fuzz ran **out-of-band**
> (`scripts/diff-fuzz-<target>.py`, per-attempt) and the Phase 3 rows
> therefore carry `fuzz_iters=0`. The fuzz gate is now integrated into the
> accept path, where a passed fuzz gate is a **precondition** for a machine
> `accepted` verdict; rows written after this date record the iteration
> count actually executed. See the per-case-study notes in
> `01-tokei-algorithmic-rust.md` and `04-cjson-c-target.md`.
results/phase4/README.md
+25 -3
View File
@@ -36,9 +36,11 @@ numbers.
## The result (`phase4-cjson-service`, ledger)
The accepted batch win `phase3-cjson-002` (+8.85% throughput) was
measured as a **service under a 150 rps coordinated-omission-correct
measured as a **service under a 150 rps-target coordinated-omission-correct
replay**, baseline (pristine cJSON v1.7.19) vs candidate (the banked
win), 20 interleaved rounds, server pinned, 40,000 requests, **0 drops**:
win), 20 interleaved rounds, server pinned, **0 drops** — the ledger row
records `sessions=20`, `rate_rps=150`, `drop_rate=0.0`; the total request
count was **not machine-recorded** (see errata below):
| metric | baseline | candidate | speedup | 95% CI |
|---|---|---|---|---|
@@ -63,7 +65,10 @@ project in one line.
## The ROI, generated mechanically
`just report phase4-cjson-service --service-json results/phase4/cjson-service.json`
→ [`cjson-service-roi.md`](cjson-service-roi.md). On a 500-core fleet at
→ [`cjson-service-roi.md`](cjson-service-roi.md). *(Note, 2026-07-13: the
raw `cjson-service.json` was not preserved — see errata below — so this
command is not reproducible as written; the generated report and the
ledger row are the surviving evidence.)* On a 500-core fleet at
$0.04/core-hour, the +6.2% p50 win returns **27.5 cores / $9,621 per
year at the CI lower bound** (median 29.4 cores / $10,290) — and the
report quotes the lower bound, because the point estimate is not a
@@ -79,3 +84,20 @@ that produced it, so the number survives hostile review.
ledger row and the calibrated service JSON with no hand-editing;
throughput→cores→dollars and latency percentiles, CIs and methodology
inline.
## Errata (2026-07-13)
- **"40,000 requests" retracted.** This report originally stated the
replay comprised 40,000 requests. The `phase4-cjson-service` ledger row
records only `sessions=20`, `rate_rps=150`, and `drop_rate=0.0` — the
per-run request count was never machine-recorded, so the 40,000 figure
(an arithmetic reconstruction, not evidence) is withdrawn. The prose
above now states the ledger-backed facts only. The claimed result is
unaffected: p50 latency +6.2%, 95% CI [+5.8%, +7.2%], under the 150
rps-target CO-correct replay, 20 interleaved rounds, 0 drops.
- **`results/phase4/cjson-service.json` was not preserved.** The
`.gitignore` rule `results/**/*.json` swept the raw service-latency
JSON that the report command consumes, so it no longer exists in the
repository. A `!results/phase4/*.json` carve-out has been added so
future service-mode evidence JSONs are committed alongside the
calibration JSONs.
results/phase5/README.md
+28
View File
@@ -137,3 +137,31 @@ it generalize?" objection with the two hardest cases the spec named:
Zero shipped false accepts, still. The product is trust, and trust
transferred.
---
## Erratum (2026-07-13) — how far the kernel lane is actually wired
The write-up above describes the FP-tolerance gate as though `matmul` runs
through the standard pipeline. It does not, and the distinction matters:
- **The policy is in the pipeline.** `EquivalencePolicy` is honored by
`diff-test`'s gate sequence (an fp-tolerance target needs a committed
`[corpus].golden_reference`; the gate fails closed without one), and
`crates/verdict` treats fp-tolerance equivalence as an automatic
`needs-human-review` signal. That machinery is real and unit-tested.
- **`matmul` is not a pipeline target.** It ships `targets/matmul/kernel.c`
and `targets/matmul/equivalence.toml` but **no `target.toml`** — the file
`diff-test`/`verdict` load a target from. So the lane cannot currently be
run via `just gates` or `just verdict` at all.
- **Therefore the demonstration is script-driven.**
`scripts/kernel-lane-demo.sh` invokes the `fp-compare` binary and
`bench-runner compare` directly, and the `phase5-matmul-opt` ledger row
was recorded **outside** the automated verdict path.
Nothing in the measured result changes — the 3.23× [3.16, 3.26] speedup, the
244,901/262,144 differing values, the rejected +0.5 perturbation, and the
`needs-human-review` verdict all stand as printed. What was overstated is the
*integration*: the equivalence policy is closed; wiring this target into the
pipeline is open work. The original text stands unedited above; this note is
the correction.
scripts/diff-fuzz-cjson.py
+103 -19
View File
@@ -4,7 +4,21 @@ produce byte-identical stdout (serialization + checksum) AND identical exit
code on mutated JSON. cJSON is a deterministic single-threaded parser, so
raw byte comparison needs no canonicalization.
python3 scripts/diff-fuzz-cjson.py <workdir> <iters> <candidate-bin>
python3 scripts/diff-fuzz-cjson.py <workdir> <iters> <baseline-bin> <candidate-bin>
All four arguments are required and are supplied by the harness: diff-test
substitutes {iters}, {baseline} and {candidate} into the target's
[gates].fuzz template and runs it from the repo root. Both binary paths may
be repo-root-relative OR absolute (the accept path rebuilds the pristine
baseline to an absolute path) — relative ones resolve against the repo root.
Nothing here may hardcode a baseline location: the harness owns where the
pristine build lands, and a hardcoded path is a gate that always hard-fails.
Final stdout line is the machine contract diff-test parses:
FUZZ-RESULT iters=<n> divergences=<m>. Exit 0 iff the run completed with
zero divergences, 1 on divergence (the line is still printed, with the real
count), 2 on a usage/setup error — which prints no FUZZ-RESULT and is
therefore graded a hard gate FAILURE, never a pass.
Seeds: slices of the pinned corpus plus hand JSON fragments. Mutations
target parser edges: structural tokens, string escapes, number forms
@@ -15,13 +29,12 @@ import random
import subprocess
import sys
from pathlib import Path
from typing import NoReturn
ROOT = Path("/home/user/peltier.io")
BASE = ROOT / "targets/cjson/baseline/cjson-bench"
CAND = ROOT / sys.argv[3] if len(sys.argv) > 3 else ROOT / "targets/cjson/baseline/cjson-bench"
ROOT = Path(__file__).resolve().parents[1]
CORPUS = ROOT / "corpora/cjson/input/big.json"
WORK = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("/tmp/difffuzz-cjson")
ITERS = int(sys.argv[2]) if len(sys.argv) > 2 else 10000
USAGE = "usage: diff-fuzz-cjson.py <workdir> <iters> <baseline-bin> <candidate-bin>"
TOKENS = [
b"{", b"}", b"[", b"]", b'"', b":", b",", b"\\", b"\\n", b"\\u00e9", b"\\uD83D",
@@ -38,6 +51,59 @@ SEEDS_STATIC = [
]
def die(msg: str) -> NoReturn:
"""Fatal setup error: exit 2 having printed NO FUZZ-RESULT line. diff-test
grades a fuzz command that never reports the line as a hard gate failure,
which is the right posture for a gate that could not run at all — it must
never be gradeable as a pass."""
print(f"diff-fuzz-cjson: {msg}", file=sys.stderr)
raise SystemExit(2)
def parse_args(argv: list[str]) -> tuple[Path, int, Path, Path]:
"""<workdir> <iters> <baseline-bin> <candidate-bin>, all positional."""
if len(argv) != 5:
die(f"expected 4 arguments, got {len(argv) - 1}\n{USAGE}")
work, raw_iters, baseline, candidate = argv[1:5]
if not raw_iters.isdigit():
die(f"iters must be a non-negative integer, got {raw_iters!r}\n{USAGE}")
iters = int(raw_iters)
if iters < 1:
# A zero-iteration run would print FUZZ-RESULT iters=0 divergences=0
# and exit 0 — a fuzz gate that passes without fuzzing anything.
die(f"iters must be >= 1, got {iters}\n{USAGE}")
# `ROOT / p` keeps a repo-root-relative path anchored at the repo root and
# leaves an absolute path untouched (pathlib drops the left operand when
# the right side is absolute). The harness supplies either form.
return ROOT / work, iters, ROOT / baseline, ROOT / candidate
def need_binary(path: Path, label: str) -> Path:
if not path.is_file():
die(
f"{label} binary not found: {path}\nthe harness substitutes "
"{baseline}/{candidate} in [gates].fuzz — check targets/cjson/target.toml"
)
return path
def keep_dir(work: Path, tag: str) -> Path:
"""First unused divergence-<tag> directory under `work`.
The RNG is seeded, so re-running in the same work dir re-diverges at the
same iteration: the kept-divergence directory already exists. The old
plain `mkdir()` raised FileExistsError there and killed the run *before*
the FUZZ-RESULT line, so diff-test reported a genuine divergence — the
whole point of the gate — as "fuzz command did not report FUZZ-RESULT".
This never raises; it picks the next free name instead.
"""
path, n = work / f"divergence-{tag}", 2
while path.exists():
path = work / f"divergence-{tag}-{n}"
n += 1
return path
def mutate(data: bytes, rng: random.Random) -> bytes:
b = bytearray(data)
for _ in range(rng.randrange(1, 8)):
@@ -65,36 +131,54 @@ def run(binary: Path, path: Path) -> tuple[bytes, int]:
return r.stdout, r.returncode
def main() -> int:
def main(argv: list[str]) -> int:
work, iters, baseline, candidate = parse_args(argv)
need_binary(baseline, "baseline")
need_binary(candidate, "candidate")
if not CORPUS.is_file():
die(f"corpus not found: {CORPUS} (generate it with corpora/cjson/gen-corpus.sh)")
rng = random.Random(0x1CE_B00C)
corpus = CORPUS.read_bytes()
seeds = list(SEEDS_STATIC)
# windows of the real corpus (often invalid mid-slice — exercises the
# error path, which must diverge identically or not at all)
for o in range(0, len(corpus) - 4096, len(corpus) // 200):
# error path, which must diverge identically or not at all). The max(1, …)
# guards are no-ops on the pinned 5.6 MB corpus (identical seed set, so
# the fuzz stays reproducible) and stop a degenerate tiny corpus from
# raising a zero-step ValueError before FUZZ-RESULT is ever printed.
for o in range(0, max(1, len(corpus) - 4096), max(1, len(corpus) // 200)):
for n in (64, 512, 4096):
seeds.append(corpus[o : o + n])
WORK.mkdir(parents=True, exist_ok=True)
case = WORK / "case.json"
work.mkdir(parents=True, exist_ok=True)
case = work / "case.json"
divergences = 0
for i in range(ITERS):
for i in range(iters):
payload = mutate(rng.choice(seeds), rng)
case.write_bytes(payload)
a_out, a_rc = run(BASE, case)
b_out, b_rc = run(CAND, case)
a_out, a_rc = run(baseline, case)
b_out, b_rc = run(candidate, case)
if a_out != b_out or a_rc != b_rc:
divergences += 1
keep = WORK / f"divergence-{i}"
keep.mkdir(exist_ok=True)
# Keeping the reproducer is best-effort: an I/O failure here must
# never cost the FUZZ-RESULT line and turn a real divergence into
# an unparseable gate error (see keep_dir).
try:
keep = keep_dir(work, str(i))
keep.mkdir(parents=True, exist_ok=True)
(keep / "input.json").write_bytes(payload)
(keep / "baseline.out").write_bytes(a_out + f"\nrc={a_rc}".encode())
(keep / "candidate.out").write_bytes(b_out + f"\nrc={b_rc}".encode())
print(f"DIVERGENCE at iter {i} (rc {a_rc} vs {b_rc}); kept in {keep}")
except OSError as e:
print(f"DIVERGENCE at iter {i} (rc {a_rc} vs {b_rc}); "
f"could not keep artifacts: {e}")
if (i + 1) % 1000 == 0:
print(f"{i + 1}/{ITERS}, {divergences} divergences", flush=True)
print(f"done: {ITERS} inputs, {divergences} divergences")
print(f"{i + 1}/{iters}, {divergences} divergences", flush=True)
print(f"done: {iters} inputs, {divergences} divergences")
# Machine contract, always the final stdout line on any completing run.
print(f"FUZZ-RESULT iters={iters} divergences={divergences}", flush=True)
return 1 if divergences else 0
if __name__ == "__main__":
sys.exit(main())
sys.exit(main(sys.argv))
scripts/diff-fuzz-comrak.py
+104 -18
View File
@@ -4,7 +4,21 @@ produce byte-identical HTML (stdout), stderr, and exit code on mutated
markdown. comrak is a single-threaded stdin->stdout filter, so raw byte
comparison needs no canonicalization.
python3 scripts/diff-fuzz-comrak.py <workdir> <iters> <candidate-bin>
python3 scripts/diff-fuzz-comrak.py <workdir> <iters> <baseline-bin> <candidate-bin>
All four arguments are required and are supplied by the harness: diff-test
substitutes {iters}, {baseline} and {candidate} into the target's
[gates].fuzz template and runs it from the repo root. Both binary paths may
be repo-root-relative OR absolute (the accept path rebuilds the pristine
baseline to an absolute path) — relative ones resolve against the repo root.
Nothing here may hardcode a baseline location: the harness owns where the
pristine build lands, and a hardcoded path is a gate that always hard-fails.
Final stdout line is the machine contract diff-test parses:
FUZZ-RESULT iters=<n> divergences=<m>. Exit 0 iff the run completed with
zero divergences, 1 on divergence (the line is still printed, with the real
count), 2 on a usage/setup error — which prints no FUZZ-RESULT and is
therefore graded a hard gate FAILURE, never a pass.
Seeds: windows of the pinned Pro Git corpus. Mutations focus on inline/
block structure the parser and teardown paths exercise: emphasis, code
@@ -15,13 +29,12 @@ import random
import subprocess
import sys
from pathlib import Path
from typing import NoReturn
ROOT = Path("/home/user/peltier.io")
BASE = ROOT / "targets/comrak/baseline/release/comrak"
CAND = ROOT / sys.argv[3]
ROOT = Path(__file__).resolve().parents[1]
CORPUS = ROOT / "corpora/comrak/progit.md"
WORK = Path(sys.argv[1])
ITERS = int(sys.argv[2])
USAGE = "usage: diff-fuzz-comrak.py <workdir> <iters> <baseline-bin> <candidate-bin>"
TOKENS = [
b"**", b"*", b"__", b"_", b"`", b"```", b"~~~", b"~~", b"[", b"](", b")",
@@ -30,6 +43,59 @@ TOKENS = [
]
def die(msg: str) -> NoReturn:
"""Fatal setup error: exit 2 having printed NO FUZZ-RESULT line. diff-test
grades a fuzz command that never reports the line as a hard gate failure,
which is the right posture for a gate that could not run at all — it must
never be gradeable as a pass."""
print(f"diff-fuzz-comrak: {msg}", file=sys.stderr)
raise SystemExit(2)
def parse_args(argv: list[str]) -> tuple[Path, int, Path, Path]:
"""<workdir> <iters> <baseline-bin> <candidate-bin>, all positional."""
if len(argv) != 5:
die(f"expected 4 arguments, got {len(argv) - 1}\n{USAGE}")
work, raw_iters, baseline, candidate = argv[1:5]
if not raw_iters.isdigit():
die(f"iters must be a non-negative integer, got {raw_iters!r}\n{USAGE}")
iters = int(raw_iters)
if iters < 1:
# A zero-iteration run would print FUZZ-RESULT iters=0 divergences=0
# and exit 0 — a fuzz gate that passes without fuzzing anything.
die(f"iters must be >= 1, got {iters}\n{USAGE}")
# `ROOT / p` keeps a repo-root-relative path anchored at the repo root and
# leaves an absolute path untouched (pathlib drops the left operand when
# the right side is absolute). The harness supplies either form.
return ROOT / work, iters, ROOT / baseline, ROOT / candidate
def need_binary(path: Path, label: str) -> Path:
if not path.is_file():
die(
f"{label} binary not found: {path}\nthe harness substitutes "
"{baseline}/{candidate} in [gates].fuzz — check targets/comrak/target.toml"
)
return path
def keep_dir(work: Path, tag: str) -> Path:
"""First unused divergence-<tag> directory under `work`.
The RNG is seeded, so re-running in the same work dir re-diverges at the
same iteration: the kept-divergence directory already exists. The old
plain `mkdir()` raised FileExistsError there and killed the run *before*
the FUZZ-RESULT line, so diff-test reported a genuine divergence — the
whole point of the gate — as "fuzz command did not report FUZZ-RESULT".
This never raises; it picks the next free name instead.
"""
path, n = work / f"divergence-{tag}", 2
while path.exists():
path = work / f"divergence-{tag}-{n}"
n += 1
return path
def mutate(data: bytes, rng: random.Random) -> bytes:
b = bytearray(data)
for _ in range(rng.randrange(1, 8)):
@@ -57,34 +123,54 @@ def run(binary: Path, payload: bytes) -> tuple[bytes, bytes, int]:
return r.stdout, r.stderr, r.returncode
def main() -> int:
def main(argv: list[str]) -> int:
work, iters, baseline, candidate = parse_args(argv)
need_binary(baseline, "baseline")
need_binary(candidate, "candidate")
if not CORPUS.is_file():
die(f"corpus not found: {CORPUS} (generate it with corpora/comrak/gen-corpus.sh)")
rng = random.Random(0xC0FFEE)
corpus = CORPUS.read_bytes()
# Seed windows across the whole book, varied sizes.
# Seed windows across the whole book, varied sizes. The max(1, …) guards
# are no-ops on the pinned 11 MB corpus (identical seed set, so the fuzz
# stays reproducible); they stop a degenerate tiny corpus from raising a
# zero-step ValueError, or yielding no seeds at all, before FUZZ-RESULT
# is ever printed.
seeds = [
corpus[o : o + n]
for o in range(0, len(corpus) - 65536, len(corpus) // 300)
for o in range(0, max(1, len(corpus) - 65536), max(1, len(corpus) // 300))
for n in (512, 4096, 32768)
]
WORK.mkdir(parents=True, exist_ok=True)
if not seeds:
die(f"corpus is empty, no seeds to fuzz from: {CORPUS}")
work.mkdir(parents=True, exist_ok=True)
divergences = 0
for i in range(ITERS):
for i in range(iters):
payload = mutate(rng.choice(seeds), rng)
a = run(BASE, payload)
b = run(CAND, payload)
a = run(baseline, payload)
b = run(candidate, payload)
if a != b:
divergences += 1
keep = WORK / f"divergence-{i}"
keep.mkdir()
# Keeping the reproducer is best-effort: an I/O failure here must
# never cost the FUZZ-RESULT line and turn a real divergence into
# an unparseable gate error (see keep_dir).
try:
keep = keep_dir(work, str(i))
keep.mkdir(parents=True, exist_ok=True)
(keep / "input.md").write_bytes(payload)
(keep / "baseline.out").write_bytes(a[0] + b"\n--stderr--\n" + a[1] + f"\nrc={a[2]}".encode())
(keep / "candidate.out").write_bytes(b[0] + b"\n--stderr--\n" + b[1] + f"\nrc={b[2]}".encode())
print(f"DIVERGENCE at iter {i}; kept in {keep}")
except OSError as e:
print(f"DIVERGENCE at iter {i}; could not keep artifacts: {e}")
if (i + 1) % 1000 == 0:
print(f"{i + 1}/{ITERS}, {divergences} divergences", flush=True)
print(f"done: {ITERS} inputs, {divergences} divergences")
print(f"{i + 1}/{iters}, {divergences} divergences", flush=True)
print(f"done: {iters} inputs, {divergences} divergences")
# Machine contract, always the final stdout line on any completing run.
print(f"FUZZ-RESULT iters={iters} divergences={divergences}", flush=True)
return 1 if divergences else 0
if __name__ == "__main__":
sys.exit(main())
sys.exit(main(sys.argv))
scripts/diff-fuzz-tokei.py
+110 -20
View File
@@ -1,8 +1,25 @@
#!/usr/bin/env python3
"""Differential fuzz for phase2-tokei-001: pristine vs candidate tokei
"""Differential fuzz for tokei attempts: pristine vs candidate tokei
must produce byte-identical output on mutated source inputs targeting
the patched code path (quote/comment token scanning).
python3 scripts/diff-fuzz-tokei.py <workdir> <iters> <baseline-bin> <candidate-bin>
All four arguments are required and are supplied by the harness: diff-test
substitutes {iters}, {baseline} and {candidate} into the target's
[gates].fuzz template and runs it from the repo root. Both binary paths may
be repo-root-relative OR absolute (the accept path rebuilds the pristine
baseline to an absolute path) — relative ones resolve against the repo root.
Nothing here may hardcode a baseline location: the harness owns where the
pristine build lands, and a hardcoded path is a gate that always hard-fails.
Final stdout line is the machine contract diff-test parses:
FUZZ-RESULT iters=<n> divergences=<m>. Exit 0 iff the run completed with
zero divergences, 1 on divergence (the line is still printed, with the real
count), 2 on a usage/setup error — which prints no FUZZ-RESULT and is
therefore graded a hard gate FAILURE, never a pass. `iters` counts mutated
files (tokei scans a directory, so files are batched per invocation).
Seeds: files from the pinned corpus (many languages). Mutations focus on
quote/comment structure: inserting/deleting/duplicating quote chars,
comment openers/closers, backslashes, and random bytes; plus boundary
@@ -13,13 +30,12 @@ import random
import subprocess
import sys
from pathlib import Path
from typing import NoReturn
ROOT = Path("/home/user/peltier.io")
BASE = ROOT / "targets/tokei/baseline/release/tokei"
CAND = ROOT / "targets/tokei/candidate-4e3da7e6bd1c/release/tokei"
ROOT = Path(__file__).resolve().parents[1]
CORPUS = ROOT / "corpora/tokei/input/copy1"
WORK = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("/tmp/difffuzz")
ITERS = int(sys.argv[2]) if len(sys.argv) > 2 else 10000
USAGE = "usage: diff-fuzz-tokei.py <workdir> <iters> <baseline-bin> <candidate-bin>"
TOKENS = [
b'"', b"'", b'`', b'"""', b"'''", b"/*", b"*/", b"//", b"#", b"--",
@@ -27,6 +43,60 @@ TOKENS = [
b"r#\"", b"\"#", b"=begin", b"=end", b"'''", b'f"""', b"%{", b"}",
]
def die(msg: str) -> NoReturn:
"""Fatal setup error: exit 2 having printed NO FUZZ-RESULT line. diff-test
grades a fuzz command that never reports the line as a hard gate failure,
which is the right posture for a gate that could not run at all — it must
never be gradeable as a pass."""
print(f"diff-fuzz-tokei: {msg}", file=sys.stderr)
raise SystemExit(2)
def parse_args(argv: list[str]) -> tuple[Path, int, Path, Path]:
"""<workdir> <iters> <baseline-bin> <candidate-bin>, all positional."""
if len(argv) != 5:
die(f"expected 4 arguments, got {len(argv) - 1}\n{USAGE}")
work, raw_iters, baseline, candidate = argv[1:5]
if not raw_iters.isdigit():
die(f"iters must be a non-negative integer, got {raw_iters!r}\n{USAGE}")
iters = int(raw_iters)
if iters < 1:
# A zero-iteration run would print FUZZ-RESULT iters=0 divergences=0
# and exit 0 — a fuzz gate that passes without fuzzing anything.
die(f"iters must be >= 1, got {iters}\n{USAGE}")
# `ROOT / p` keeps a repo-root-relative path anchored at the repo root and
# leaves an absolute path untouched (pathlib drops the left operand when
# the right side is absolute). The harness supplies either form.
return ROOT / work, iters, ROOT / baseline, ROOT / candidate
def need_binary(path: Path, label: str) -> Path:
if not path.is_file():
die(
f"{label} binary not found: {path}\nthe harness substitutes "
"{baseline}/{candidate} in [gates].fuzz — check targets/tokei/target.toml"
)
return path
def keep_dir(work: Path, tag: str) -> Path:
"""First unused divergence-<tag> directory under `work`.
The RNG is seeded, so re-running in the same work dir re-diverges at the
same batch: the kept-divergence directory already exists. The old plain
`rename()` raised (the destination is a non-empty directory) and killed
the run *before* the FUZZ-RESULT line, so diff-test reported a genuine
divergence — the whole point of the gate — as "fuzz command did not
report FUZZ-RESULT". This never raises; it picks the next free name.
"""
path, n = work / f"divergence-{tag}", 2
while path.exists():
path = work / f"divergence-{tag}-{n}"
n += 1
return path
def mutate(data: bytes, rng: random.Random) -> bytes:
ops = rng.randrange(1, 6)
b = bytearray(data)
@@ -51,6 +121,7 @@ def mutate(data: bytes, rng: random.Random) -> bytes:
b[pos:pos] = b"\n"
return bytes(b)
def canonicalize(payload: bytes) -> str:
"""Parse tokei JSON and sort every list so benign output-order
nondeterminism (parallel walker arrival order) doesn't count as a
@@ -78,43 +149,62 @@ def run(binary: Path, d: Path) -> tuple[str, bytes, int]:
canon = f"UNPARSEABLE:{e}:{r.stdout[:200]!r}"
return canon + f"||rc={r.returncode}", r.stdout, r.returncode
def main() -> int:
def main(argv: list[str]) -> int:
work, iters, baseline, candidate = parse_args(argv)
need_binary(baseline, "baseline")
need_binary(candidate, "candidate")
if not CORPUS.is_dir():
die(f"corpus not found: {CORPUS} (generate it with corpora/tokei/gen-corpus.sh)")
rng = random.Random(0xB16B00B5)
seeds = sorted(CORPUS.rglob("*"))
seeds = [p for p in seeds if p.is_file() and p.stat().st_size < 200_000][:400]
if not seeds:
print("no seeds found", file=sys.stderr)
return 2
WORK.mkdir(parents=True, exist_ok=True)
fdir = WORK / "case"
die(f"no seeds found under {CORPUS}")
work.mkdir(parents=True, exist_ok=True)
fdir = work / "case"
divergences = 0
done = 0
runs = 0
# Batch: mutate BATCH files per tokei invocation (tokei scans a dir),
# keeping each seed's extension so language detection stays realistic.
BATCH = 50
while done < ITERS:
while done < iters:
if fdir.exists():
for f in fdir.iterdir():
f.unlink()
fdir.mkdir(exist_ok=True)
for j in range(min(BATCH, ITERS - done)):
batch = min(BATCH, iters - done)
for j in range(batch):
seed = rng.choice(seeds)
name = f"m{done + j}{seed.suffix or '.txt'}"
(fdir / name).write_bytes(mutate(seed.read_bytes(), rng))
a_canon, a_raw, a_rc = run(BASE, fdir)
b_canon, b_raw, b_rc = run(CAND, fdir)
a_canon, a_raw, a_rc = run(baseline, fdir)
b_canon, b_raw, b_rc = run(candidate, fdir)
if a_canon != b_canon:
divergences += 1
keep = WORK / f"divergence-{done}"
# Keeping the reproducer is best-effort: an I/O failure here must
# never cost the FUZZ-RESULT line and turn a real divergence into
# an unparseable gate error (see keep_dir). If the rename fails,
# fdir survives and the next batch cleans it out.
try:
keep = keep_dir(work, str(done))
fdir.rename(keep)
(keep / "_baseline.out").write_bytes(a_raw + f"\nrc={a_rc}".encode())
(keep / "_candidate.out").write_bytes(b_raw + f"\nrc={b_rc}".encode())
print(f"DIVERGENCE at batch starting {done}; inputs+outputs kept in {keep}")
done += BATCH
except OSError as e:
print(f"DIVERGENCE at batch starting {done}; could not keep artifacts: {e}")
done += batch
runs += 1
if done % 1000 == 0:
print(f"{done}/{ITERS} mutated files, {divergences} divergences", flush=True)
print(f"done: {done} mutated files across {done // BATCH} runs, {divergences} divergences")
print(f"{done}/{iters} mutated files, {divergences} divergences", flush=True)
print(f"done: {done} mutated files across {runs} runs, {divergences} divergences")
# Machine contract, always the final stdout line on any completing run.
print(f"FUZZ-RESULT iters={done} divergences={divergences}", flush=True)
return 1 if divergences else 0
if __name__ == "__main__":
sys.exit(main())
sys.exit(main(sys.argv))
scripts/no-net.sh
Executable
+25
View File
@@ -0,0 +1,25 @@
#!/bin/sh
# Network-isolation wrapper for the local verdict path (SPEC §10): exec the
# given command inside a fresh network namespace so target code built and
# run by the pipeline cannot reach the network. harnessd invokes it as
# `no-net.sh sh -c <pipeline-script>` (HOTPATH_VERDICT_WRAPPER overrides).
# CI gets the same guarantee from `docker --network=none` instead.
set -eu
if [ $# -eq 0 ]; then
echo "usage: scripts/no-net.sh <command> [args...]" >&2
exit 2
fi
if unshare --net --map-current-user true 2>/dev/null; then
exec unshare --net --map-current-user -- "$@"
fi
if [ "${HOTPATH_ALLOW_UNISOLATED:-}" = "1" ]; then
echo "no-net.sh: WARNING: unshare --net unavailable and HOTPATH_ALLOW_UNISOLATED=1 —" >&2
echo "no-net.sh: WARNING: running WITHOUT network isolation. Target code can reach the network." >&2
exec "$@"
fi
echo "no-net.sh: cannot create a network namespace (unshare --net --map-current-user failed)." >&2
echo "no-net.sh: refusing to run target code unisolated. Options:" >&2
echo "no-net.sh: - run on a host with unprivileged user namespaces enabled" >&2
echo "no-net.sh: - in CI, run the whole pipeline under docker --network=none" >&2
echo "no-net.sh: - set HOTPATH_ALLOW_UNISOLATED=1 to override (loud warning, not for accepts)" >&2
exit 97
scripts/pin-testsuite.sh
Executable
+63
View File
@@ -0,0 +1,63 @@
#!/bin/sh
# Pin an upstream test suite: hash every file under the given repo-root-
# relative paths (typically targets/<t>/workspace/tests/...) into
# corpora/<target>/TESTSUITE.sha256, sha256sum two-space format. diff-test
# verifies this pin before running upstream tests; targets/fetch.sh
# verifies it after every fetch. Run from repo root:
# scripts/pin-testsuite.sh <target> <path>...
set -eu
if [ $# -lt 2 ]; then
echo "usage: scripts/pin-testsuite.sh <target> <path>..." >&2
exit 2
fi
target="$1"
shift
out="corpora/$target/TESTSUITE.sha256"
if [ ! -d "corpora/$target" ]; then
echo "pin-testsuite: corpora/$target does not exist (run from repo root?)" >&2
exit 1
fi
for p in "$@"; do
if [ ! -e "$p" ]; then
echo "pin-testsuite: $p not found (paths are repo-root-relative)" >&2
exit 1
fi
done
# Build the pin out-of-place and install it only on success: a run that fails
# anywhere below must not truncate, replace, or half-write an existing pin —
# that pin is what diff-test refuses to run against on mismatch.
list="$out.files.$$"
tmp="$out.new.$$"
trap 'rm -f "$list" "$tmp"' EXIT INT TERM
# LC_ALL=C: byte-order sort, so the same tree pins identically on every
# machine. -print0/-z: filenames with spaces or newlines survive intact.
find "$@" -type f -print0 | LC_ALL=C sort -z > "$list"
count=$(tr -dc '\0' < "$list" | wc -c | tr -d ' ')
if [ "$count" -eq 0 ]; then
# Was: `find ... | xargs -0 sha256sum > "$out"` — with no matches, xargs
# still ran sha256sum with zero file arguments, so it hashed STDIN (the
# empty pipe) and wrote a bogus "<sha-of-empty> -" entry, then reported
# SUCCESS. That is a pin of nothing that later "verifies" by reading
# stdin. Refuse instead, and leave any existing pin untouched.
echo "pin-testsuite: no regular files under: $* — refusing to write $out" >&2
exit 1
fi
# --text is GNU's default on Linux (a no-op there); it is spelled out because
# sha256sum defaults to *binary* mode on some platforms, emitting "<hash> *path"
# — a one-space-plus-asterisk form that diff-test's pin verifier (which splits
# on the two-space separator) does not parse. Paths stay repo-root-relative:
# find was handed repo-root-relative paths and we run from the repo root.
xargs -0 sha256sum --text -- < "$list" > "$tmp"
lines=$(wc -l < "$tmp" | tr -d ' ')
if [ "$lines" -ne "$count" ]; then
echo "pin-testsuite: hashed $lines of $count files — refusing to write a partial pin" >&2
exit 1
fi
mv "$tmp" "$out"
echo "wrote $out ($count files)"
echo "pinning a test suite is a deliberate human action (corpora/README.md):"
echo "commit the new pin with justification."
targets/cjson/target.toml
+6
View File
@@ -37,6 +37,12 @@ workload = "5.6 MB synthetic JSON (20k records: nested objects/arrays, escaped/u
tests = "cmake -S targets/cjson/workspace -B targets/cjson/testbuild -DENABLE_CJSON_TEST=On -DCMAKE_BUILD_TYPE=Debug >/dev/null 2>&1 && cmake --build targets/cjson/testbuild -j >/dev/null 2>&1 && ctest --test-dir targets/cjson/testbuild --output-on-failure"
# Golden: full re-serialization + checksum of the corpus, byte-hashed.
golden = "{binary} corpora/cjson/input/big.json 1"
# Differential fuzz, pipeline-integrated: diff-test substitutes {iters},
# {baseline} (the pristine binary the accept path rebuilds) and {candidate},
# then parses the script's final FUZZ-RESULT line. The harness owns where the
# pristine build lands, so the script takes the baseline from argv and never
# hardcodes a path; paths may be repo-root-relative or absolute.
fuzz = "python3 scripts/diff-fuzz-cjson.py targets/cjson/fuzz-work {iters} {baseline} {candidate}"
[corpus]
manifest = "corpora/cjson/MANIFEST.sha256"
targets/comrak/target.toml
+6
View File
@@ -33,6 +33,12 @@ workload = "Pro Git book, all translations, 11.0 MB concatenated markdown, singl
tests = "cargo test --release --manifest-path targets/comrak/workspace/Cargo.toml"
# Golden replay: corpus through the binary, stdout must be byte-identical.
golden = "cat corpora/comrak/progit.md | {binary}"
# Differential fuzz, pipeline-integrated: diff-test substitutes {iters},
# {baseline} (the pristine binary the accept path rebuilds) and {candidate},
# then parses the script's final FUZZ-RESULT line. The harness owns where the
# pristine build lands, so the script takes the baseline from argv and never
# hardcodes a path; paths may be repo-root-relative or absolute.
fuzz = "python3 scripts/diff-fuzz-comrak.py targets/comrak/fuzz-work {iters} {baseline} {candidate}"
[corpus]
manifest = "corpora/comrak/MANIFEST.sha256"
targets/fetch.sh
+12
View File
@@ -21,4 +21,16 @@ grep -A0 '^"vendor/' "$toml" | while IFS= read -r line; do
git -C "$ws/$sub" fetch -q origin "$scommit" 2>/dev/null || true
git -C "$ws/$sub" checkout -q "$scommit"
done
# Upstream test-suite pin: verify against repo root when present; a
# mismatch means the fetched suite is not the one that was audited.
suite="corpora/$name/TESTSUITE.sha256"
if [ -f "$suite" ]; then
if ! sha256sum -c --quiet "$suite"; then
echo "fetch: upstream test-suite pin MISMATCH for $name ($suite) — refusing" >&2
exit 1
fi
echo "$name test suite verified against $suite"
else
echo "note: no $suite — pin the suite with: scripts/pin-testsuite.sh $name <paths...>"
fi
echo "$name pinned at $commit"
targets/tokei/target.toml
+6
View File
@@ -23,6 +23,12 @@ workload = "8 replicas of pinned progit + comrak/src trees, 221 MB, 4360 files,
[gates]
tests = "cargo test --release --manifest-path targets/tokei/workspace/Cargo.toml"
golden = "RAYON_NUM_THREADS=1 {binary} --sort code corpora/tokei/input"
# Differential fuzz, pipeline-integrated: diff-test substitutes {iters},
# {baseline} (the pristine binary the accept path rebuilds) and {candidate},
# then parses the script's final FUZZ-RESULT line. The harness owns where the
# pristine build lands, so the script takes the baseline from argv and never
# hardcodes a path; paths may be repo-root-relative or absolute.
fuzz = "python3 scripts/diff-fuzz-tokei.py targets/tokei/fuzz-work {iters} {baseline} {candidate}"
[corpus]
manifest = "corpora/tokei/MANIFEST.sha256"