Adds crates/verdict ('just verdict'): runs the core gates on the
candidate binary, then an interleaved A/B bench vs the baseline with
the CI-lower-bound rule, and writes the append-only ledger row with
per-side median CIs, workload statement, and env fingerprint. Gate
failure short-circuits to rejected-gate; --needs-human-review caps an
accepted verdict (never upgrades). CPU pinning moves into accept.toml
(pin_prefix, machine-specific, applied to both sides identically).
The smoke test caught a real trust gap: pointing --baseline-bin at the
workspace build (phase1-verdict-smoke-001) measured patched-vs-patched
and returned null, because gate runs rebuild the workspace binary from
the patched tree. Fix: --rebuild-baseline stashes the patch, builds
the baseline into isolated targets/<name>/baseline/, and restores the
patch — SPEC 3.1's pristine-checkout rule, now enforced by the
harness. Re-run (phase1-verdict-smoke-002) reproduces the Phase 0
result mechanically: +10.4% median, 95% CI [1.0932, 1.1137], accepted.
Both smoke rows are in the ledger, including the null one.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
Adds 'bench-runner calibrate': N A/A sessions counted against a <5%
false-positive bar, and N sessions with a synthetic slowdown injected
as a real busy-wait inside the candidate's timed window, counted
against a >=95% detection bar (detected = ratio CI entirely below 1.0).
Bootstrap seed varies per session so resampling noise is not
correlated across the very statistic being validated. JSON evidence
includes per-session CIs, injected wait, acceptance verdict, and env
fingerprint.
First run against the real comrak workload (20+20 sessions, pinned
core): false-positive rate 0.000, detection rate 0.950 -> PASS.
Evidence: results/calibration/2026-07-06-comrak-full-calibration.json.
Closes the SPEC 3.1 acceptance-criteria gap; remaining Phase 1 gaps:
no-net containers, perf profiling, diff-test orchestration, agent IPC.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
Attempt 4 (phase0-tokei-002, class 3, ACCEPTED — first auto-accept):
every file was read through encoding_rs_io's DecodeReaderBytes via
generic read_to_end, which zero-fills every grown buffer chunk (~221MB
memset/run, 8.5% of Ir) and grows without a size hint. Patch: fs::read
with a BOM-gated fallback to the identical decode path (a UTF-8-BOM
file transcodes invalid sequences, so the fallback is required for
equivalence). Gates: golden byte-identical; differential fuzz old-vs-
new binaries 4,332 iterations (BOM grid, invalid UTF-8, binary junk,
randomized) zero divergence; 250 upstream tests green; ASan+LSan clean
with ASan-run output hashing to golden. Bench: +9.9% [8.0%, 12.7%].
Attempt 5 (phase0-tokei-003, ship tree, ACCEPTED): stacks fat LTO+cgu=1
(real +1.6% standalone but under threshold, phase0-tokei-001) on the
accepted patch; benched as a whole vs pristine: +10.4% [8.5%, 12.0%].
Median clears the Phase 0 exit bar; CI lower bound 8.5%.
Case study written (results/phase0/case-study-tokei.md), ledger at 5
attempts, CLAUDE.md phase line moved to Phase 1, next hypotheses
logged (tokei class-5 byte-skip table; comrak class-3 arena).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
Commit the spec and scaffold the system it describes:
- SPEC.md: the full v0.1 engineering spec
- crates/: trust-layer Rust workspace, unit-tested
- bench-runner: interleaved ABAB scheduling with discarded warm-ups,
deterministic bootstrap CIs over the ratio of medians,
CI-lower-bound accept rule, env fingerprinting, and a CLI with
compare + A/A self-test modes (A/A verified passing end-to-end)
- diff-test: gate outcome model, sha256 corpus pin verifier that
refuses on mismatch, per-target equivalence policy parser
(byte-identical default, explicit fp-tolerance opt-in)
- ledger: append-only SQLite attempt schema with UPDATE/DELETE
refused by triggers, anti-double-attempt query
- report: cores-saved and dollars/yr ROI math over speedup CIs,
pricing.toml inputs
- playbook/: classes 1-7 with preconditions, procedure, verification
notes, and known failure modes per class
- agent/: Python package skeleton — six-tool contract and prompting
spine, harness IPC deferred to Phase 1
- config/: accept.toml (2% threshold, 30 runs/side), pricing.toml
- targets/, corpora/, results/calibration/: layout and policy docs
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
Claude
benbrownandGitHub