explain: post-verdict diagnostics — SPEC §3.7, crate, docs

Verdicts say what; nothing said why. A rejection that reads only
rejected-bench teaches nothing, and the loop's compounding value is that
the next hypothesis is better than the last. This ships the teaching layer,
built so it can never contaminate the thing it explains.

New crate crates/explain (`just explain <run-id>`):

- Advisory, strictly off the accept path: runs after the verdict is
  recorded; nothing it emits feeds gates, bench, verdict, or ledger. Not
  agent-facing: harnessd stays at seven ops, by design (SPEC §3.7 says why
  - better hypotheses must never become a feedback channel into
  measurement).
- Ledger-only inputs, deterministic: consumes exactly one row via the new
  ledger::attempt_row; no config, no clock, no re-benching. Identical row,
  byte-identical text - proven live (cmp on two runs of a real row).
- Record vs inference is typographic: restatements print plain; anything
  beyond is prefixed `inference:` so an explanation can never be quoted as
  a measurement.
- The narrative must agree with the machine verdict. The first live run
  caught this class of bug: an accepted row without a recorded threshold
  was narrated as "a policy rejection" two lines under "verdict: accepted".
  Accepted rows are now always described by their CI lower bound (the
  defensible claim); rejected-bench rows classify against 1.0 (measured
  regression / null result / real-but-below-bar); NHR rows name the
  recorded cap and state the numbers were not the deciding factor;
  rejected-gate rows name the failing gate. Regression-tested.
- Historical rows are flagged, not laundered: accepted with
  sanitizers_clean=false prints the doctrine caution (comrak-010 shows it);
  fuzz_iters=0 accepts are named pre-integration rows.

verdict now records accept_threshold (the bar in force for that run) into
the bench env fingerprint, so rows are self-contained going forward -
explain never substitutes today's config for a historical bar, and says so
on older rows instead.

SPEC §3.7 (Added 2026-07-20) carries the invariants; book chapter
docs/src/explain.md shows real output from real rows (the accepted cJSON
win, the caught false-accept, the kernel-lane NHR); CLAUDE.md, README,
justfile, and the architecture layout updated (the layout also gains the
previously-omitted crates/policy line).

Verified: 21 workspace test suites green (6 new explain tests incl.
determinism and verdict-narrative agreement), clippy -D warnings clean,
fmt clean, mdbook builds, live runs against phase3-cjson-002,
phase5-matmul-opt, phase2-comrak-010, and phase4-cjson-service.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-20 09:53:14 -04:00
co-authored by Claude Fable 5
parent 48ef299cec
commit 444e5294fa
14 changed files with 751 additions and 0 deletions
+3
View File
@@ -25,6 +25,7 @@ Full engineering spec: `SPEC.md`. Read it before starting any phase.
- `crates/diff-test` — behavioral equivalence: corpus/test-suite pin checks + upstream test suite + golden I/O replay + differential fuzz (a baseline-vs-candidate gate; sanitizers run on the accept path in `crates/verdict`, and `diff-test` reports them as skipped-with-reason) - `crates/diff-test` — behavioral equivalence: corpus/test-suite pin checks + upstream test suite + golden I/O replay + differential fuzz (a baseline-vs-candidate gate; sanitizers run on the accept path in `crates/verdict`, and `diff-test` reports them as skipped-with-reason)
- `crates/ledger` — append-only SQLite record of every attempt: hypothesis, patch, gate results, bench deltas, verdict, cost - `crates/ledger` — append-only SQLite record of every attempt: hypothesis, patch, gate results, bench deltas, verdict, cost
- `crates/report` — ROI generator: cores saved × $/core-hr, latency percentile deltas, CIs, workload caveats printed on every number - `crates/report` — ROI generator: cores saved × $/core-hr, latency percentile deltas, CIs, workload caveats printed on every number
- `crates/explain` — advisory post-verdict diagnosis (SPEC §3.7): one ledger row → why it won or lost. Ledger-only inputs, deterministic (identical row = identical bytes), `inference:`-labeled lines, verdict-narrative agreement enforced, historical accepts flagged rather than laundered. Strictly off the accept path; not exposed to the agent (harnessd stays seven ops)
- `agent/` — Claude Agent SDK (Python) loop, prompts, tool definitions - `agent/` — Claude Agent SDK (Python) loop, prompts, tool definitions
- `targets/` — vendored OSS targets for case studies (permissive licenses only) - `targets/` — vendored OSS targets for case studies (permissive licenses only)
- `playbook/` — optimization classes, ordered, with preconditions and known risks - `playbook/` — optimization classes, ordered, with preconditions and known risks
@@ -58,6 +59,8 @@ just pin-corpus <target> # deliberate corpus re-pin (writes MANIFEST.sha256)
just calibrate <cmd> <out> # automated A/A + regression-injection calibration just calibrate <cmd> <out> # automated A/A + regression-injection calibration
just verdict <t> <bin> ... # gates + bench vs pristine-rebuilt baseline + ledger row just verdict <t> <bin> ... # gates + bench vs pristine-rebuilt baseline + ledger row
just report <run-id> # ROI report from a ledger row (CIs + methodology inline) just report <run-id> # ROI report from a ledger row (CIs + methodology inline)
just explain <run-id> # advisory post-verdict diagnosis of a row (SPEC §3.7):
# why it won/lost, ledger-only inputs, never changes a verdict
just agent-attempt <t> <id> # one unattended agent attempt behind the OS boundary just agent-attempt <t> <id> # one unattended agent attempt behind the OS boundary
just isolation-check # verify the SPEC §10 OS boundary (19 checks, both modes) just isolation-check # verify the SPEC §10 OS boundary (19 checks, both modes)
just service <b> <c> <doc> # service-mode latency A/B, CO-correct, p50/p99 CIs just service <b> <c> <doc> # service-mode latency A/B, CO-correct, p50/p99 CIs
Generated
+10
View File
@@ -207,6 +207,16 @@ version = "1.0.2"
source = "registry+https://github.com/rust-lang/crates.io-index" source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f"
[[package]]
name = "explain"
version = "0.1.0"
dependencies = [
"anyhow",
"clap",
"ledger",
"serde_json",
]
[[package]] [[package]]
name = "fallible-iterator" name = "fallible-iterator"
version = "0.3.0" version = "0.3.0"
+1
View File
@@ -10,6 +10,7 @@ members = [
"crates/verdict", "crates/verdict",
"crates/harnessd", "crates/harnessd",
"crates/policy", "crates/policy",
"crates/explain",
] ]
[workspace.package] [workspace.package]
+1
View File
@@ -61,6 +61,7 @@ just gates <target> # corpus pin + upstream tests + golden replay
# (fuzz needs a baseline — it runs in `just verdict`) # (fuzz needs a baseline — it runs in `just verdict`)
just verdict <t> <bin> ... # gates + fuzz + sanitizers + bench vs pristine baseline just verdict <t> <bin> ... # gates + fuzz + sanitizers + bench vs pristine baseline
just report <run-id> # ROI report from a ledger row just report <run-id> # ROI report from a ledger row
just explain <run-id> # why a row won/lost — advisory, from the record only
just isolation-check # 19 OS-boundary checks (both modes) just isolation-check # 19 OS-boundary checks (both modes)
``` ```
+14
View File
@@ -90,6 +90,20 @@ Consumes ledger rows, emits per-engagement ROI report:
- Optional: joules saved (RAPL) for the sustainability line. *(Amended 2026-07-13: never implemented — RAPL energy is not captured anywhere in the harness, so no report emits a joules figure. See the §3.1 amendment.)* - Optional: joules saved (RAPL) for the sustainability line. *(Amended 2026-07-13: never implemented — RAPL energy is not captured anywhere in the harness, so no report emits a joules figure. See the §3.1 amendment.)*
- Every figure carries CI + workload description + environment fingerprint. Caveats are printed on the report, not in an appendix. - Every figure carries CI + workload description + environment fingerprint. Caveats are printed on the report, not in an appendix.
### 3.7 `explain/` (post-verdict diagnostics)
> **Added 2026-07-20.** Verdicts say *what*; this component says *why*. It exists to make the next hypothesis better, and it is built so it can never contaminate the thing it explains.
Requirements:
- **Advisory, strictly off the accept path.** Runs only after a verdict is recorded; nothing it emits feeds gates, the bench, the verdict, or the ledger.
- **Ledger-only inputs, deterministic output.** Consumes exactly one row via `ledger::attempt_row` — no config files, no clock, no re-benching. Identical row ⇒ byte-identical text; the header states this so a regenerated explanation is checkable.
- **Record vs. inference is typographically explicit.** Restatements of the machine record (verdict, medians, CI, workload, gate fields) are printed plain; anything beyond restatement is prefixed `inference:` so no explanation can be quoted as if the machine measured it.
- **The narrative must agree with the machine verdict.** An `accepted` row is described by its CI *lower bound* — the defensible claim — never re-litigated against today's config. A `rejected-bench` row is classified against 1.0: measured regression (CI < 1.0), null result (CI straddles 1.0), or real-but-below-bar. A `needs-human-review` row names the recorded cap (risk signals, missing differential-fuzz pass, fp-tolerance tier) and states that the numbers were not the deciding factor. A `rejected-gate` row names the failing gate.
- **Historical rows are flagged, not laundered.** An accepted row with `sanitizers_clean=false` or `fuzz_iters=0` gets an explicit caution naming the doctrine it predates (see the §3.2 amendments and `results/rulings/`).
- **Rows are self-contained going forward:** `verdict` now records `accept_threshold` (the bar in force for that run) in the bench env fingerprint. For older rows, explain prints that the bar was not machine-recorded — it never substitutes today's `config/accept.toml` for a historical one.
- CLI: `just explain <run-id>` → `cargo run -p explain -- --run-id <id> [--db results/ledger.sqlite]`.
- **Deliberately not agent-facing yet.** The harnessd surface stays at seven ops; explanations are a human/CLI tool. Any future exposure to the agent loop must remain read-only, advisory, and outside the accept path — the loop's compounding benefit (better next hypotheses) must never become a feedback channel into measurement.
## 4. Optimization playbook v0 (strict order) ## 4. Optimization playbook v0 (strict order)
1. **Build configuration**: opt level, `-march`/`target-cpu`, LTO (thin → fat), PGO, BOLT where applicable. 1. **Build configuration**: opt level, `-march`/`target-cpu`, LTO (thin → fat), PGO, BOLT where applicable.
+13
View File
@@ -0,0 +1,13 @@
[package]
name = "explain"
description = "Advisory post-verdict diagnosis of a ledger row: why it won or lost, derived solely from the machine record, never on the accept path"
version.workspace = true
edition.workspace = true
license.workspace = true
repository.workspace = true
[dependencies]
anyhow.workspace = true
clap.workspace = true
ledger = { path = "../ledger" }
serde_json.workspace = true
+525
View File
@@ -0,0 +1,525 @@
//! Post-verdict diagnosis (SPEC §3.7). Turns one ledger row into a plain-text
//! explanation of WHY the attempt won or lost.
//!
//! Invariants, in order of importance:
//! - **Advisory.** Runs after the verdict is written; cannot change one.
//! - **Ledger-only inputs.** Every statement derives from the row passed in —
//! no config files, no re-benching, no clock. Same row, same bytes out.
//! - **Verdicts are records; explanations are inferences.** Lines that go
//! beyond restating the row are prefixed `inference:` so nobody can quote
//! an explanation as if the machine measured it.
use serde_json::Value;
/// Everything explain reads, pulled out of the row once.
pub struct Facts {
pub run_id: String,
pub target: String,
pub verdict: String,
pub hypothesis: String,
pub hotspot: String,
pub playbook_class: i64,
pub timestamp: String,
pub patch: String,
// gates
pub upstream_tests: bool,
pub golden_replay: bool,
pub fuzz_iters: u64,
pub fuzz_divergence: bool,
pub sanitizers_clean: bool,
pub tsan_clean: Option<bool>,
pub risk_signals: Vec<String>,
pub equivalence_mode: Option<String>,
// bench (absent when gates failed before benching)
pub bench: Option<BenchFacts>,
}
pub struct BenchFacts {
pub baseline_median: f64,
pub candidate_median: f64,
pub speedup_median: f64,
pub speedup_lo: f64,
pub speedup_hi: f64,
pub workload: Option<String>,
pub isolation: Option<String>,
/// Accept bar in force for this run — recorded per-row since 2026-07-20;
/// None on older rows (and explain must not substitute today's config).
pub accept_threshold: Option<f64>,
pub max_rss_baseline_kib: Option<u64>,
pub max_rss_candidate_kib: Option<u64>,
}
impl Facts {
pub fn from_row(row: &Value) -> Option<Facts> {
let s = |k: &str| row.get(k)?.as_str().map(str::to_string);
let gates = row.get("gates")?;
let bench = row.get("bench").filter(|b| !b.is_null()).map(|b| {
let env = b.get("env_fingerprint");
let ci = b.get("speedup_ci").and_then(|c| c.as_array().cloned());
let ci_at = |i: usize| -> f64 {
ci.as_ref()
.and_then(|c| c.get(i))
.and_then(|v| v.as_f64())
.unwrap_or(f64::NAN)
};
let rss = env.and_then(|e| e.get("max_rss_kib"));
BenchFacts {
baseline_median: b
.get("baseline_median")
.and_then(|v| v.as_f64())
.unwrap_or(f64::NAN),
candidate_median: b
.get("candidate_median")
.and_then(|v| v.as_f64())
.unwrap_or(f64::NAN),
speedup_median: b
.get("speedup_median")
.and_then(|v| v.as_f64())
.unwrap_or(f64::NAN),
speedup_lo: ci_at(0),
speedup_hi: ci_at(1),
workload: env
.and_then(|e| e.get("workload"))
.and_then(|v| v.as_str())
.map(str::to_string),
isolation: env
.and_then(|e| e.get("isolation"))
.and_then(|v| v.as_str())
.map(str::to_string),
accept_threshold: env
.and_then(|e| e.get("accept_threshold"))
.and_then(|v| v.as_f64()),
max_rss_baseline_kib: rss
.and_then(|r| r.get("baseline_median"))
.and_then(|v| v.as_u64()),
max_rss_candidate_kib: rss
.and_then(|r| r.get("candidate_median"))
.and_then(|v| v.as_u64()),
}
});
Some(Facts {
run_id: s("run_id")?,
target: s("target")?,
verdict: s("verdict")?,
hypothesis: s("hypothesis")?,
hotspot: s("hotspot")?,
playbook_class: row.get("playbook_class")?.as_i64()?,
timestamp: s("timestamp")?,
patch: s("patch")?,
upstream_tests: gates
.get("upstream_tests")
.and_then(|v| v.as_bool())
.unwrap_or(false),
golden_replay: gates
.get("golden_replay")
.and_then(|v| v.as_bool())
.unwrap_or(false),
fuzz_iters: gates
.get("fuzz_iters")
.and_then(|v| v.as_u64())
.unwrap_or(0),
fuzz_divergence: gates
.get("fuzz_divergence")
.and_then(|v| v.as_bool())
.unwrap_or(false),
sanitizers_clean: gates
.get("sanitizers_clean")
.and_then(|v| v.as_bool())
.unwrap_or(false),
tsan_clean: gates.get("tsan_clean").and_then(|v| v.as_bool()),
risk_signals: gates
.get("risk_signals")
.and_then(|v| v.as_array())
.map(|a| {
a.iter()
.filter_map(|x| x.as_str().map(str::to_string))
.collect()
})
.unwrap_or_default(),
equivalence_mode: gates
.get("equivalence_mode")
.and_then(|v| v.as_str())
.map(str::to_string),
bench,
})
}
}
/// Where the bootstrap CI of the speedup ratio landed, relative to 1.0 and
/// (when the row records it) the accept bar.
#[derive(Debug, PartialEq)]
pub enum CiClass {
/// Lower bound clears the recorded bar.
MeetsBar,
/// CI excludes 1.0 upward, but does not clear the bar (or no bar recorded).
RealBelowBar,
/// CI straddles 1.0 — indistinguishable from no change.
Indistinguishable,
/// CI entirely below 1.0 — a measured regression.
Regression,
}
pub fn classify_ci(lo: f64, hi: f64, threshold: Option<f64>) -> CiClass {
if hi < 1.0 {
CiClass::Regression
} else if lo <= 1.0 {
CiClass::Indistinguishable
} else if let Some(t) = threshold {
if lo >= 1.0 + t {
CiClass::MeetsBar
} else {
CiClass::RealBelowBar
}
} else {
CiClass::RealBelowBar
}
}
pub struct DiffStats {
pub files: usize,
pub added: usize,
pub removed: usize,
}
/// Mechanical shape of the stored unified diff. None when the row carries a
/// placeholder instead of a real patch (build-config attempts, smoke rows).
pub fn diff_stats(patch: &str) -> Option<DiffStats> {
if !patch.contains("--- ") || !patch.contains("+++ ") {
return None;
}
let mut files = 0;
let mut added = 0;
let mut removed = 0;
for line in patch.lines() {
if line.starts_with("+++ ") {
files += 1;
} else if line.starts_with('+') && !line.starts_with("+++") {
added += 1;
} else if line.starts_with('-') && !line.starts_with("---") {
removed += 1;
}
}
Some(DiffStats {
files,
added,
removed,
})
}
fn pct(x: f64) -> String {
format!("{:+.1}%", (x - 1.0) * 100.0)
}
/// The whole report. Pure function of the facts: byte-identical on re-run.
pub fn explain(f: &Facts) -> String {
let mut out = String::new();
let mut line = |s: String| {
out.push_str(&s);
out.push('\n');
};
line(format!("explain {} — advisory diagnosis", f.run_id));
line(
"derived solely from the ledger row; regeneration is byte-identical; \
this can never change a verdict"
.to_string(),
);
line(String::new());
line(format!(
"attempt: {} | class {} | hotspot: {} | recorded {}",
f.target, f.playbook_class, f.hotspot, f.timestamp
));
line(format!("hypothesis: {}", f.hypothesis));
if let Some(d) = diff_stats(&f.patch) {
line(format!(
"patch shape: {} file(s), +{} / -{} lines",
d.files, d.added, d.removed
));
} else {
line(
"patch shape: no source diff recorded (build-config or out-of-band attempt)"
.to_string(),
);
}
line(String::new());
line(format!("verdict (machine record): {}", f.verdict));
// --- what decided it -------------------------------------------------
match f.verdict.as_str() {
"rejected-gate" => {
let failed: Vec<&str> = [
(!f.upstream_tests, "upstream test suite"),
(!f.golden_replay, "golden replay"),
(f.fuzz_divergence, "differential fuzz (divergence found)"),
]
.iter()
.filter(|(cond, _)| *cond)
.map(|(_, name)| *name)
.collect();
if failed.is_empty() {
line("decided by: a gate failure (the failing gate is not individually recorded in this row)".to_string());
} else {
line(format!("decided by: gate failure — {}", failed.join(", ")));
}
line(
"inference: the patch changed observable behavior or broke the build contract; \
the bench (if any ran) is irrelevant until equivalence holds"
.to_string(),
);
}
"accepted" | "rejected-bench" | "needs-human-review" => {
if let Some(b) = &f.bench {
line(format!(
"bench: baseline median {:.4}s, candidate median {:.4}s — speedup {:.4} ({}), 95% CI [{:.4}, {:.4}]",
b.baseline_median, b.candidate_median, b.speedup_median,
pct(b.speedup_median), b.speedup_lo, b.speedup_hi
));
if let Some(w) = &b.workload {
line(format!("workload: {}", w));
}
match &b.accept_threshold {
Some(t) => line(format!(
"accept bar recorded for this run: CI lower bound >= {:.2} (1 + threshold {:.2})",
1.0 + t, t
)),
None => line(
"accept bar: not machine-recorded for this row (rows before 2026-07-20); \
the rule is CI lower bound >= 1 + threshold from config/accept.toml at run time"
.to_string(),
),
}
// The narrative must agree with the machine verdict: an
// `accepted` row cleared the bar in force at run time by
// definition, even when the row predates per-row threshold
// recording — never describe an accept as a rejection.
match f.verdict.as_str() {
"accepted" => {
line(format!(
"inference: the defensible claim is the lower bound — \"at least {}\" on this workload, not the median {}",
pct(b.speedup_lo), pct(b.speedup_median)
));
if let Some(t) = b.accept_threshold {
if b.speedup_lo < 1.0 + t + 0.01 {
line("inference: thin margin over the bar — re-run on calibrated hardware before any external claim".to_string());
}
}
}
"needs-human-review" => {
line("inference: the numbers were not the cap — the review routing below decided this verdict".to_string());
}
_ => match classify_ci(b.speedup_lo, b.speedup_hi, b.accept_threshold) {
CiClass::MeetsBar => {
line("inference: the CI lower bound clears the recorded bar yet the verdict is a rejection — inconsistent row, inspect by hand".to_string());
}
CiClass::RealBelowBar => {
line(format!(
"inference: the improvement is real (CI excludes 1.0) but below the ship bar — \
a policy rejection, not noise; the measured effect is {} to {}",
pct(b.speedup_lo), pct(b.speedup_hi)
));
}
CiClass::Indistinguishable => {
line("inference: indistinguishable from no change — the CI straddles 1.0. \
This is a null result, not a small win"
.to_string());
}
CiClass::Regression => {
line(format!(
"inference: a measured regression — the candidate is {} to {} SLOWER; report it as one",
pct(2.0 - b.speedup_hi), pct(2.0 - b.speedup_lo)
));
}
},
}
if let (Some(rb), Some(rc)) = (b.max_rss_baseline_kib, b.max_rss_candidate_kib) {
if rb > 0 {
let ratio = rc as f64 / rb as f64;
line(format!(
"max RSS medians: baseline {} KiB, candidate {} KiB ({}) — context, never an accept metric",
rb, rc, pct(ratio)
));
if f.verdict == "accepted" && ratio > 1.05 {
line("inference: the speedup is partly bought with memory — note the trade in any claim".to_string());
}
}
}
if let Some(iso) = &b.isolation {
if iso.contains("NOT isolated") || iso == "unwrapped-host" {
line(format!(
"isolation note on record: \"{}\" — this run did not have the network-namespace boundary",
iso
));
}
}
} else {
line("bench: no evidence recorded on this row".to_string());
}
}
other => line(format!(
"verdict string '{}' is outside the known set — inspect the row by hand",
other
)),
}
// --- caps and caveats -------------------------------------------------
if f.verdict == "needs-human-review" {
if !f.risk_signals.is_empty() {
line(format!(
"routed to human review by the lexical risk classifier (SPEC §8): {}",
f.risk_signals.join(", ")
));
line("inference: the numbers do not clear a risk class — concurrency/unsafe/FP changes can be \
correct on this workload and wrong in production"
.to_string());
} else if f.fuzz_iters == 0 {
line("no passed differential-fuzz gate on record — the no-fuzz-no-accept rule caps any would-be accept at review".to_string());
}
if f.equivalence_mode.as_deref() == Some("fp-tolerance") {
line(
"equivalence used the fp-tolerance tier — always a review signal (SPEC §8)"
.to_string(),
);
}
}
if f.verdict == "accepted" {
if !f.sanitizers_clean {
line("CAUTION: accepted with sanitizers_clean=false — this row predates the machine-enforced \
sanitizer lane; under current doctrine it could not be machine-accepted. See results/rulings/"
.to_string());
}
match f.fuzz_iters {
0 => line(
"fuzz: 0 iterations in the machine record — a pre-integration row; differential fuzz \
ran out-of-band via scripts through Phase 5 (an accept today requires a passed in-pipeline run)"
.to_string(),
),
n => line(format!("fuzz: {} iterations executed, no divergence", n)),
}
if let Some(false) = f.tsan_clean {
line("CAUTION: TSan lane flagged this attempt".to_string());
}
}
out
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn ci_classification_covers_all_regions() {
assert_eq!(classify_ci(0.91, 0.96, Some(0.02)), CiClass::Regression);
assert_eq!(
classify_ci(0.98, 1.01, Some(0.02)),
CiClass::Indistinguishable
);
assert_eq!(classify_ci(1.005, 1.015, Some(0.02)), CiClass::RealBelowBar);
assert_eq!(classify_ci(1.03, 1.06, Some(0.02)), CiClass::MeetsBar);
// No recorded bar: a real improvement can never be called MeetsBar.
assert_eq!(classify_ci(1.08, 1.10, None), CiClass::RealBelowBar);
assert_eq!(
classify_ci(1.0, 1.02, Some(0.02)),
CiClass::Indistinguishable
);
}
#[test]
fn diff_stats_counts_and_rejects_placeholders() {
let d = diff_stats("--- a/f.c\n+++ b/f.c\n@@\n-old\n+new\n+more\n").unwrap();
assert_eq!((d.files, d.added, d.removed), (1, 2, 1));
assert!(diff_stats("(no source patch)").is_none());
}
fn row(verdict: &str, gates: serde_json::Value, bench: serde_json::Value) -> serde_json::Value {
serde_json::json!({
"run_id": "t-001", "timestamp": "2026-07-20T00:00:00Z", "target": "demo",
"target_commit": "abc", "phase": 6, "hotspot": "hot() (40%)",
"playbook_class": 5, "hypothesis": "loop fusion cuts passes",
"patch": "--- a/x\n+++ b/x\n-a\n+b\n", "gates": gates, "bench": bench,
"verdict": verdict, "tokens_spent": 0, "wall_time_s": 1.0
})
}
fn bench(lo: f64, hi: f64, thr: Option<f64>) -> serde_json::Value {
let mut env = serde_json::json!({"workload": "10k records", "isolation": "no-net.sh"});
if let Some(t) = thr {
env["accept_threshold"] = serde_json::json!(t);
}
serde_json::json!({
"baseline_median": 2.0, "baseline_ci": [1.9, 2.1],
"candidate_median": 1.8, "candidate_ci": [1.7, 1.9],
"speedup_median": (lo + hi) / 2.0, "speedup_ci": [lo, hi],
"env_fingerprint": env
})
}
#[test]
fn explanation_is_deterministic_and_labels_inference() {
let r = row(
"accepted",
serde_json::json!({"upstream_tests": true, "golden_replay": true,
"fuzz_iters": 10000, "fuzz_divergence": false, "sanitizers_clean": true}),
bench(1.05, 1.09, Some(0.02)),
);
let f = Facts::from_row(&r).unwrap();
let a = explain(&f);
let b = explain(&f);
assert_eq!(a, b, "explanations must be byte-identical on re-run");
assert!(a.contains("advisory"));
assert!(a.contains("inference:"));
assert!(
a.contains("at least +5.0%"),
"claim must be the lower bound: {a}"
);
assert!(a.contains("10000 iterations"));
}
#[test]
fn regression_and_null_results_are_named_plainly() {
let g = serde_json::json!({"upstream_tests": true, "golden_replay": true,
"fuzz_iters": 0, "fuzz_divergence": false, "sanitizers_clean": false});
let reg = Facts::from_row(&row(
"rejected-bench",
g.clone(),
bench(0.92, 0.97, Some(0.02)),
))
.unwrap();
assert!(explain(&reg).contains("SLOWER"));
let null =
Facts::from_row(&row("rejected-bench", g, bench(0.99, 1.01, Some(0.02)))).unwrap();
assert!(explain(&null).contains("indistinguishable from no change"));
}
#[test]
fn historical_accept_without_sanitizers_is_flagged() {
let r = row(
"accepted",
serde_json::json!({"upstream_tests": true, "golden_replay": true,
"fuzz_iters": 0, "fuzz_divergence": false, "sanitizers_clean": false}),
bench(1.08, 1.12, None),
);
let text = explain(&Facts::from_row(&r).unwrap());
assert!(text.contains("sanitizers_clean=false"));
assert!(text.contains("not machine-recorded for this row"));
assert!(text.contains("out-of-band"));
// An accepted row cleared the bar in force at run time by definition:
// the missing per-row threshold must never read as a rejection.
assert!(!text.contains("policy rejection"), "{text}");
assert!(text.contains("at least +8.0%"));
}
#[test]
fn nhr_names_its_cap() {
let r = row(
"needs-human-review",
serde_json::json!({"upstream_tests": true, "golden_replay": true,
"fuzz_iters": 10000, "fuzz_divergence": false, "sanitizers_clean": true,
"risk_signals": ["concurrency"], "equivalence_mode": "byte-identical"}),
bench(1.10, 1.15, Some(0.02)),
);
let text = explain(&Facts::from_row(&r).unwrap());
assert!(text.contains("lexical risk classifier"));
assert!(text.contains("concurrency"));
}
}
+34
View File
@@ -0,0 +1,34 @@
//! `just explain <run-id>` — advisory post-verdict diagnosis (SPEC §3.7).
//! Reads ONE ledger row and prints why the attempt won or lost. Never on the
//! accept path: the verdict this reads was final before this binary ran.
use anyhow::{anyhow, Context, Result};
use clap::Parser;
use explain::{explain, Facts};
use std::path::PathBuf;
#[derive(Parser)]
#[command(
name = "explain",
about = "Advisory diagnosis of a ledger row (SPEC §3.7) — derived solely from the machine record"
)]
struct Cli {
/// Ledger run_id to explain.
#[arg(long)]
run_id: String,
#[arg(long, default_value = "results/ledger.sqlite")]
db: PathBuf,
}
fn main() -> Result<()> {
let cli = Cli::parse();
let ledger = ledger::Ledger::open(&cli.db)
.with_context(|| format!("opening ledger {}", cli.db.display()))?;
let row = ledger
.attempt_row(&cli.run_id)?
.ok_or_else(|| anyhow!("no ledger row with run_id '{}'", cli.run_id))?;
let facts = Facts::from_row(&row)
.ok_or_else(|| anyhow!("row '{}' is missing required fields", cli.run_id))?;
print!("{}", explain(&facts));
Ok(())
}
+69
View File
@@ -308,6 +308,75 @@ impl Ledger {
} }
} }
/// The full machine record of one attempt — every column, with the
/// gates/bench JSON columns parsed. Read-only; this is the entire input
/// surface of `crates/explain` (SPEC §3.7), which must derive from the
/// ledger row and nothing else.
pub fn attempt_row(&self, run_id: &str) -> Result<Option<serde_json::Value>, LedgerError> {
let row = self.conn.query_row(
"SELECT timestamp, target, target_commit, phase, hotspot, playbook_class, \
hypothesis, patch, gates, bench, verdict, tokens_spent, wall_time_s \
FROM attempts WHERE run_id = ?1",
[run_id],
|r| {
Ok((
r.get::<_, String>(0)?,
r.get::<_, String>(1)?,
r.get::<_, String>(2)?,
r.get::<_, i64>(3)?,
r.get::<_, String>(4)?,
r.get::<_, i64>(5)?,
r.get::<_, String>(6)?,
r.get::<_, String>(7)?,
r.get::<_, String>(8)?,
r.get::<_, Option<String>>(9)?,
r.get::<_, String>(10)?,
r.get::<_, i64>(11)?,
r.get::<_, f64>(12)?,
))
},
);
match row {
Ok((
ts,
target,
commit,
phase,
hotspot,
class,
hyp,
patch,
gates,
bench,
verdict,
tokens,
wall,
)) => {
let gates: serde_json::Value = serde_json::from_str(&gates)?;
let bench: Option<serde_json::Value> =
bench.map(|b| serde_json::from_str(&b)).transpose()?;
Ok(Some(serde_json::json!({
"run_id": run_id,
"timestamp": ts,
"target": target,
"target_commit": commit,
"phase": phase,
"hotspot": hotspot,
"playbook_class": class,
"hypothesis": hyp,
"patch": patch,
"gates": gates,
"bench": bench,
"verdict": verdict,
"tokens_spent": tokens,
"wall_time_s": wall,
})))
}
Err(rusqlite::Error::QueryReturnedNoRows) => Ok(None),
Err(e) => Err(e.into()),
}
}
pub fn count(&self) -> Result<u64, LedgerError> { pub fn count(&self) -> Result<u64, LedgerError> {
Ok(self Ok(self
.conn .conn
+4
View File
@@ -472,6 +472,10 @@ fn main() -> Result<()> {
"workload": spec.bench.workload, "workload": spec.bench.workload,
"target_commit": spec.source.commit, "target_commit": spec.source.commit,
"isolation": cli.isolation_note.as_deref().unwrap_or("unwrapped-host"), "isolation": cli.isolation_note.as_deref().unwrap_or("unwrapped-host"),
// The accept bar in force for THIS run, recorded so the row is
// self-contained: `explain` must never guess a historical
// threshold from today's config (SPEC §3.7).
"accept_threshold": cfg.threshold,
"gates_detail": "diff-fuzz gated in-pipeline against the pristine baseline (an accept requires it to have passed); ASan+UBSan enforced on the accept path, TSan when configured", "gates_detail": "diff-fuzz gated in-pipeline against the pristine baseline (an accept requires it to have passed); ASan+UBSan enforced on the accept path, TSan when configured",
}); });
if !samples.baseline_max_rss_kib.is_empty() && !samples.candidate_max_rss_kib.is_empty() { if !samples.baseline_max_rss_kib.is_empty() && !samples.candidate_max_rss_kib.is_empty() {
+1
View File
@@ -25,5 +25,6 @@
- [Consume it: the peltier skill](./skill.md) - [Consume it: the peltier skill](./skill.md)
- [Reproduce it yourself](./reproduce.md) - [Reproduce it yourself](./reproduce.md)
- [Explain: why a row won or lost](./explain.md)
- [The trust boundary (infra)](./infra.md) - [The trust boundary (infra)](./infra.md)
- [Roadmap & scope](./roadmap.md) - [Roadmap & scope](./roadmap.md)
+3
View File
@@ -11,6 +11,9 @@ crates/ Trust layer — agent has NO write access (SPEC §10)
diff-test/ Equivalence gates, corpus hash-pinning, per-target spec diff-test/ Equivalence gates, corpus hash-pinning, per-target spec
ledger/ Append-only SQLite attempt ledger (enforced by triggers) ledger/ Append-only SQLite attempt ledger (enforced by triggers)
report/ ROI: speedup CI → cores → dollars, methodology inline report/ ROI: speedup CI → cores → dollars, methodology inline
explain/ Advisory post-verdict diagnosis of a ledger row (why it
won or lost) — never on the accept path
policy/ Learned class-selection prior from the ledger (advisory)
verdict/ The pipeline in one command: gates → bench → ledger row verdict/ The pipeline in one command: gates → bench → ledger row
harnessd/ The one door the agent talks through (7 JSON ops) harnessd/ The one door the agent talks through (7 JSON ops)
agent/ Untrusted proposer (Claude Agent SDK, Python) agent/ Untrusted proposer (Claude Agent SDK, Python)
+68
View File
@@ -0,0 +1,68 @@
# Explain: why a row won or lost
Verdicts say *what*. `just explain <run-id>` says *why* — an advisory
diagnosis of one ledger row, built so it can never contaminate the thing it
explains:
- **Ledger-only inputs, deterministic.** It reads exactly one row and nothing
else — no config, no clock, no re-benching. The same row produces
byte-identical text, so a regenerated explanation is checkable.
- **Strictly off the accept path.** It runs after the verdict is written;
nothing it says feeds gates, the bench, or the ledger.
- **Record vs. inference is explicit.** Restatements of the machine record are
printed plain; anything beyond that is prefixed `inference:` — so an
explanation can never be quoted as if the machine measured it.
- **The narrative must agree with the verdict.** An accepted row is described
by its CI *lower bound* (the defensible claim). A rejected bench is
classified honestly: measured regression, null result, or
real-but-below-bar. A `needs-human-review` row names the recorded cap and
says plainly that the numbers were not the deciding factor.
## Real output, real rows
The shipped cJSON win ([`phase3-cjson-002`](./case-studies/cjson.md)):
```
verdict (machine record): accepted
bench: baseline median 0.4263s, candidate median 0.3916s — speedup 1.0885 (+8.8%), 95% CI [1.0745, 1.1001]
workload: 5.6 MB synthetic JSON (20k records: nested objects/arrays, escaped/unicode strings, ints, dense floats), parsed+serialized 2x, single thread
inference: the defensible claim is the lower bound — "at least +7.4%" on this workload, not the median +8.8%
```
The [caught false-accept](./case-studies/comrak-false-accept.md)
(`phase2-comrak-010`) — explain flags it instead of laundering it:
```
inference: the defensible claim is the lower bound — "at least +7.9%" on this workload, not the median +10.7%
CAUTION: accepted with sanitizers_clean=false — this row predates the machine-enforced
sanitizer lane; under current doctrine it could not be machine-accepted. See results/rulings/
```
The kernel-lane row (`phase5-matmul-opt`, `needs-human-review`):
```
verdict (machine record): needs-human-review
bench: … speedup 3.2301 (+223.0%), 95% CI [3.1600, 3.2594]
inference: the numbers were not the cap — the review routing below decided this verdict
```
That last line is the point: a 3.2× speedup with a tight interval still does
not argue with the review routing. Explanations restate the record and reason
*from* it — they do not re-litigate it.
## Why this exists
The loop's compounding value is not any single verdict — it is that the next
hypothesis is better than the last. A rejection that says only
`rejected-bench` teaches nothing; one that says "real improvement, below the
ship bar" or "indistinguishable from no change — a null result, not a small
win" or "a measured regression, report it as one" points at three different
next moves. Explain is that teaching layer, kept deliberately outside the
measurement it learns from — see SPEC §3.7 for the invariants, including the
one that keeps it out of the agent-facing surface: better hypotheses must
never become a feedback channel into the accept path.
Rows written from 2026-07-20 on also record the accept bar in force for that
run (`accept_threshold` in the env fingerprint), so an explanation never has
to guess a historical threshold from today's config — for older rows it says
so explicitly instead.
+5
View File
@@ -92,6 +92,11 @@ service-calibrate server doc out *flags:
report run_id *flags: report run_id *flags:
cargo run -p report -- --run-id "{{run_id}}" {{flags}} cargo run -p report -- --run-id "{{run_id}}" {{flags}}
# Advisory post-verdict diagnosis of a ledger row (SPEC §3.7): why it won
# or lost, derived solely from the machine record. Never changes a verdict.
explain run_id:
cargo run -p explain -- --run-id "{{run_id}}"
# Learned playbook-class ranking from the ledger (SPEC §13 research fork) # Learned playbook-class ranking from the ledger (SPEC §13 research fork)
policy *flags: policy *flags:
cargo run -p policy -- {{flags}} cargo run -p policy -- {{flags}}