Verdicts say what; nothing said why. A rejection that reads only
rejected-bench teaches nothing, and the loop's compounding value is that
the next hypothesis is better than the last. This ships the teaching layer,
built so it can never contaminate the thing it explains.
New crate crates/explain (`just explain <run-id>`):
- Advisory, strictly off the accept path: runs after the verdict is
recorded; nothing it emits feeds gates, bench, verdict, or ledger. Not
agent-facing: harnessd stays at seven ops, by design (SPEC §3.7 says why
- better hypotheses must never become a feedback channel into
measurement).
- Ledger-only inputs, deterministic: consumes exactly one row via the new
ledger::attempt_row; no config, no clock, no re-benching. Identical row,
byte-identical text - proven live (cmp on two runs of a real row).
- Record vs inference is typographic: restatements print plain; anything
beyond is prefixed `inference:` so an explanation can never be quoted as
a measurement.
- The narrative must agree with the machine verdict. The first live run
caught this class of bug: an accepted row without a recorded threshold
was narrated as "a policy rejection" two lines under "verdict: accepted".
Accepted rows are now always described by their CI lower bound (the
defensible claim); rejected-bench rows classify against 1.0 (measured
regression / null result / real-but-below-bar); NHR rows name the
recorded cap and state the numbers were not the deciding factor;
rejected-gate rows name the failing gate. Regression-tested.
- Historical rows are flagged, not laundered: accepted with
sanitizers_clean=false prints the doctrine caution (comrak-010 shows it);
fuzz_iters=0 accepts are named pre-integration rows.
verdict now records accept_threshold (the bar in force for that run) into
the bench env fingerprint, so rows are self-contained going forward -
explain never substitutes today's config for a historical bar, and says so
on older rows instead.
SPEC §3.7 (Added 2026-07-20) carries the invariants; book chapter
docs/src/explain.md shows real output from real rows (the accepted cJSON
win, the caught false-accept, the kernel-lane NHR); CLAUDE.md, README,
justfile, and the architecture layout updated (the layout also gains the
previously-omitted crates/policy line).
Verified: 21 workspace test suites green (6 new explain tests incl.
determinism and verdict-narrative agreement), clippy -D warnings clean,
fmt clean, mdbook builds, live runs against phase3-cjson-002,
phase5-matmul-opt, phase2-comrak-010, and phase4-cjson-service.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An adversarial audit found the enforcement story ahead of the code: gates
that were documented as hard gates were advisory, manual, or absent, and
several human-facing claims outran the machine record. This makes the
claims true.
High findings
- Differential fuzz was documentation, not enforcement: verdict hardcoded
fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case
studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real
pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and
it is inherently baseline-vs-candidate, so it runs on the accept path and
reports Skipped in the standalone `just gates` flow. New hard rule: a
machine `accepted` verdict is impossible unless the fuzz gate actually
passed — otherwise it caps at needs-human-review.
- The agent's system prompt claimed FP/concurrency/unsafe patches were
"routed to human review automatically". No such mechanism existed. Added
a conservative lexical risk classifier (crates/verdict/src/risk.rs) that
forces needs-human-review on any would-be accept carrying those signals;
fp-tolerance equivalence always signals.
- The local verdict path ran agent-influenced code unsandboxed. It now runs
under scripts/no-net.sh (network namespace) and fails closed. Upstream
test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when
present (scripts/pin-testsuite.sh generates them).
Also fixed, found while fixing the above
- The pipeline script's `set -e` was inert (brace group as the left operand
of `&&`), so a failing candidate build did not abort the run and verdict
could bench a stale binary. Now a scoped subshell whose status is branched
on at top level.
- A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path)
stranded RUNNING.lock and left read_verdict polling forever. Failure
accounting now lives outside the wrapper.
- harnessd's patch allowlist ignored git rename/copy headers and --no-prefix
diffs; the rename-escape case is now a committed regression test.
- The workspace did not compile on Windows (ungated Unix APIs). cfg-gated;
71 portable tests now run on both platforms.
- pin-check regenerated the corpus manifest instead of verifying it; it is
now verify-only, with a deliberate `just pin-corpus` to re-pin.
- Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics
(compare --perf-stat), never part of the accept decision.
Docs now match the code, including what is NOT yet enforced: no target ships
a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not
wired to the fp-tolerance gate — each is stated as mechanized-but-inert
rather than done. Corrected the stale "33 ledger rows" (34), the playbook's
class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation
claim (five accepted rows carry sanitizers_clean=false), the unsupported
"40,000 requests" figure, and the blanket "historical rows record
fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
crates/policy: reads the append-only ledger and ranks playbook classes by
the Wilson score lower bound of their machine-verified win rate — the same
'quote the lower bound' discipline the rest of the project uses. A class
that won 1/1 does not outrank one that won 8/10 (wider interval); untried
classes fall back to the cheapest-first prior; tried-and-failed classes
rank last. Advisory only — the gates still decide every verdict.
On the current 33-row ledger it ranks class 5 (algorithmic) first, then
class 3, then class 1 — the proven winners — with class 7 (untried) above
the evidence-against classes. Wins count only machine-sanitizer-verified
accepts (conservatively excluding pre-gate accepts and the overturned
comrak-010): the policy trusts the ledger's machine record, not narrative.
4 unit tests (Wilson monotonicity, small-sample penalty, ranking order).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
- bench-runner service mode (SPEC §3.1 mode c): a coordinated-omission-
correct, open-loop, fixed-rate load generator; latency measured from
the intended send time so queueing is counted, not hidden. Interleaved
A/B of two server binaries, exact percentiles (bounded sample, no HDR
bucketing error), bootstrap p50/p99 CIs via the existing ratio-CI
machine. service-calibrate: A/A false-positive + injected latency-
regression detection (calibration in progress).
- targets/cjson/service.c: minimal single-threaded HTTP server wrapping
the patched cJSON.c (trust-layer, outside the workspace allowlist);
env-toggled per-request busy-wait for the injection calibration only.
- report generator (crates/report bin): reads a ledger row +
optional service-latency JSON, renders throughput→cores→dollars and/or
latency percentiles, every figure with its CI, pricing source, and the
methodology printed inline (SPEC §9); flags any non-accepted row so a
rejected change never reads as a committed saving.
- ledger::report_row; justfile service/service-calibrate/report.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
harnessd gains --socket mode (Unix socket, trusted uid, mutating ops
serialized); the harness client speaks socket when
HOTPATH_HARNESS_SOCKET is set. scripts/agent-isolated.sh launches an
attempt with the agent tree confined — mountns mode (repo bind-mounted
read-only, CAP_SYS_ADMIN/CAP_SYS_PTRACE dropped; no credential copying)
or user mode (unprivileged hpagent, bench-metal shape).
scripts/isolation-check.sh + just isolation-check verify the boundary
from the agent's side: 19 checks, both modes, all passing — repo
writes, forged ledger INSERTs, trust-binary replacement, rw remounts,
and userns re-entry all refused; the harnessd socket door works.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
Adds crates/verdict ('just verdict'): runs the core gates on the
candidate binary, then an interleaved A/B bench vs the baseline with
the CI-lower-bound rule, and writes the append-only ledger row with
per-side median CIs, workload statement, and env fingerprint. Gate
failure short-circuits to rejected-gate; --needs-human-review caps an
accepted verdict (never upgrades). CPU pinning moves into accept.toml
(pin_prefix, machine-specific, applied to both sides identically).
The smoke test caught a real trust gap: pointing --baseline-bin at the
workspace build (phase1-verdict-smoke-001) measured patched-vs-patched
and returned null, because gate runs rebuild the workspace binary from
the patched tree. Fix: --rebuild-baseline stashes the patch, builds
the baseline into isolated targets/<name>/baseline/, and restores the
patch — SPEC 3.1's pristine-checkout rule, now enforced by the
harness. Re-run (phase1-verdict-smoke-002) reproduces the Phase 0
result mechanically: +10.4% median, 95% CI [1.0932, 1.1137], accepted.
Both smoke rows are in the ledger, including the null one.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
Adds the diff-test CLI ('just gates <target>'): loads
targets/<name>/target.toml (new [corpus] section: manifest, root,
golden_sha256; commands now repo-root-relative), verifies the corpus
manifest FIRST and refuses to run any gate on mismatch, then runs the
upstream test suite and golden replay (stdout sha256 vs pinned hash).
Differential fuzz and sanitizers remain per-attempt manual steps in
Phase 1 and are reported as skipped-with-reason; all-skipped is not a
pass.
Verified end-to-end on both vendored targets (comrak: 1 pinned file,
848 tests, golden PASS; tokei: 545 pinned files, 250 tests, golden
PASS) and the refusal path (tampered corpus -> 'HASH MISMATCH ...
refusing to run' before any gate executes).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
Adds 'bench-runner calibrate': N A/A sessions counted against a <5%
false-positive bar, and N sessions with a synthetic slowdown injected
as a real busy-wait inside the candidate's timed window, counted
against a >=95% detection bar (detected = ratio CI entirely below 1.0).
Bootstrap seed varies per session so resampling noise is not
correlated across the very statistic being validated. JSON evidence
includes per-session CIs, injected wait, acceptance verdict, and env
fingerprint.
First run against the real comrak workload (20+20 sessions, pinned
core): false-positive rate 0.000, detection rate 0.950 -> PASS.
Evidence: results/calibration/2026-07-06-comrak-full-calibration.json.
Closes the SPEC 3.1 acceptance-criteria gap; remaining Phase 1 gaps:
no-net containers, perf profiling, diff-test orchestration, agent IPC.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
Commit the spec and scaffold the system it describes:
- SPEC.md: the full v0.1 engineering spec
- crates/: trust-layer Rust workspace, unit-tested
- bench-runner: interleaved ABAB scheduling with discarded warm-ups,
deterministic bootstrap CIs over the ratio of medians,
CI-lower-bound accept rule, env fingerprinting, and a CLI with
compare + A/A self-test modes (A/A verified passing end-to-end)
- diff-test: gate outcome model, sha256 corpus pin verifier that
refuses on mismatch, per-target equivalence policy parser
(byte-identical default, explicit fp-tolerance opt-in)
- ledger: append-only SQLite attempt schema with UPDATE/DELETE
refused by triggers, anti-double-attempt query
- report: cores-saved and dollars/yr ROI math over speedup CIs,
pricing.toml inputs
- playbook/: classes 1-7 with preconditions, procedure, verification
notes, and known failure modes per class
- agent/: Python package skeleton — six-tool contract and prompting
spine, harness IPC deferred to Phase 1
- config/: accept.toml (2% threshold, 30 runs/side), pricing.toml
- targets/, corpora/, results/calibration/: layout and policy docs
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
benbrownandClaude Fable 5