diff --git a/CLAUDE.md b/CLAUDE.md index 12dce14..29241ea 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -25,6 +25,7 @@ Full engineering spec: `SPEC.md`. Read it before starting any phase. - `crates/diff-test` — behavioral equivalence: corpus/test-suite pin checks + upstream test suite + golden I/O replay + differential fuzz (a baseline-vs-candidate gate; sanitizers run on the accept path in `crates/verdict`, and `diff-test` reports them as skipped-with-reason) - `crates/ledger` — append-only SQLite record of every attempt: hypothesis, patch, gate results, bench deltas, verdict, cost - `crates/report` — ROI generator: cores saved × $/core-hr, latency percentile deltas, CIs, workload caveats printed on every number +- `crates/explain` — advisory post-verdict diagnosis (SPEC §3.7): one ledger row → why it won or lost. Ledger-only inputs, deterministic (identical row = identical bytes), `inference:`-labeled lines, verdict-narrative agreement enforced, historical accepts flagged rather than laundered. Strictly off the accept path; not exposed to the agent (harnessd stays seven ops) - `agent/` — Claude Agent SDK (Python) loop, prompts, tool definitions - `targets/` — vendored OSS targets for case studies (permissive licenses only) - `playbook/` — optimization classes, ordered, with preconditions and known risks @@ -58,6 +59,8 @@ just pin-corpus # deliberate corpus re-pin (writes MANIFEST.sha256) just calibrate # automated A/A + regression-injection calibration just verdict ... # gates + bench vs pristine-rebuilt baseline + ledger row just report # ROI report from a ledger row (CIs + methodology inline) +just explain # advisory post-verdict diagnosis of a row (SPEC §3.7): + # why it won/lost, ledger-only inputs, never changes a verdict just agent-attempt # one unattended agent attempt behind the OS boundary just isolation-check # verify the SPEC §10 OS boundary (19 checks, both modes) just service # service-mode latency A/B, CO-correct, p50/p99 CIs diff --git a/Cargo.lock b/Cargo.lock index 069b70d..eaf43b6 100644 --- a/Cargo.lock +++ b/Cargo.lock @@ -207,6 +207,16 @@ version = "1.0.2" source = "registry+https://github.com/rust-lang/crates.io-index" checksum = "877a4ace8713b0bcf2a4e7eec82529c029f1d0619886d18145fea96c3ffe5c0f" +[[package]] +name = "explain" +version = "0.1.0" +dependencies = [ + "anyhow", + "clap", + "ledger", + "serde_json", +] + [[package]] name = "fallible-iterator" version = "0.3.0" diff --git a/Cargo.toml b/Cargo.toml index bccb3b8..da7023b 100644 --- a/Cargo.toml +++ b/Cargo.toml @@ -10,6 +10,7 @@ members = [ "crates/verdict", "crates/harnessd", "crates/policy", + "crates/explain", ] [workspace.package] diff --git a/README.md b/README.md index 37e371b..162e3b5 100644 --- a/README.md +++ b/README.md @@ -61,6 +61,7 @@ just gates # corpus pin + upstream tests + golden replay # (fuzz needs a baseline — it runs in `just verdict`) just verdict ... # gates + fuzz + sanitizers + bench vs pristine baseline just report # ROI report from a ledger row +just explain # why a row won/lost — advisory, from the record only just isolation-check # 19 OS-boundary checks (both modes) ``` diff --git a/SPEC.md b/SPEC.md index 2267e8e..4a03d7b 100644 --- a/SPEC.md +++ b/SPEC.md @@ -90,6 +90,20 @@ Consumes ledger rows, emits per-engagement ROI report: - Optional: joules saved (RAPL) for the sustainability line. *(Amended 2026-07-13: never implemented — RAPL energy is not captured anywhere in the harness, so no report emits a joules figure. See the §3.1 amendment.)* - Every figure carries CI + workload description + environment fingerprint. Caveats are printed on the report, not in an appendix. +### 3.7 `explain/` (post-verdict diagnostics) + +> **Added 2026-07-20.** Verdicts say *what*; this component says *why*. It exists to make the next hypothesis better, and it is built so it can never contaminate the thing it explains. + +Requirements: +- **Advisory, strictly off the accept path.** Runs only after a verdict is recorded; nothing it emits feeds gates, the bench, the verdict, or the ledger. +- **Ledger-only inputs, deterministic output.** Consumes exactly one row via `ledger::attempt_row` — no config files, no clock, no re-benching. Identical row ⇒ byte-identical text; the header states this so a regenerated explanation is checkable. +- **Record vs. inference is typographically explicit.** Restatements of the machine record (verdict, medians, CI, workload, gate fields) are printed plain; anything beyond restatement is prefixed `inference:` so no explanation can be quoted as if the machine measured it. +- **The narrative must agree with the machine verdict.** An `accepted` row is described by its CI *lower bound* — the defensible claim — never re-litigated against today's config. A `rejected-bench` row is classified against 1.0: measured regression (CI < 1.0), null result (CI straddles 1.0), or real-but-below-bar. A `needs-human-review` row names the recorded cap (risk signals, missing differential-fuzz pass, fp-tolerance tier) and states that the numbers were not the deciding factor. A `rejected-gate` row names the failing gate. +- **Historical rows are flagged, not laundered.** An accepted row with `sanitizers_clean=false` or `fuzz_iters=0` gets an explicit caution naming the doctrine it predates (see the §3.2 amendments and `results/rulings/`). +- **Rows are self-contained going forward:** `verdict` now records `accept_threshold` (the bar in force for that run) in the bench env fingerprint. For older rows, explain prints that the bar was not machine-recorded — it never substitutes today's `config/accept.toml` for a historical one. +- CLI: `just explain ` → `cargo run -p explain -- --run-id [--db results/ledger.sqlite]`. +- **Deliberately not agent-facing yet.** The harnessd surface stays at seven ops; explanations are a human/CLI tool. Any future exposure to the agent loop must remain read-only, advisory, and outside the accept path — the loop's compounding benefit (better next hypotheses) must never become a feedback channel into measurement. + ## 4. Optimization playbook v0 (strict order) 1. **Build configuration**: opt level, `-march`/`target-cpu`, LTO (thin → fat), PGO, BOLT where applicable. diff --git a/crates/explain/Cargo.toml b/crates/explain/Cargo.toml new file mode 100644 index 0000000..64a553e --- /dev/null +++ b/crates/explain/Cargo.toml @@ -0,0 +1,13 @@ +[package] +name = "explain" +description = "Advisory post-verdict diagnosis of a ledger row: why it won or lost, derived solely from the machine record, never on the accept path" +version.workspace = true +edition.workspace = true +license.workspace = true +repository.workspace = true + +[dependencies] +anyhow.workspace = true +clap.workspace = true +ledger = { path = "../ledger" } +serde_json.workspace = true diff --git a/crates/explain/src/lib.rs b/crates/explain/src/lib.rs new file mode 100644 index 0000000..80b7bd7 --- /dev/null +++ b/crates/explain/src/lib.rs @@ -0,0 +1,525 @@ +//! Post-verdict diagnosis (SPEC §3.7). Turns one ledger row into a plain-text +//! explanation of WHY the attempt won or lost. +//! +//! Invariants, in order of importance: +//! - **Advisory.** Runs after the verdict is written; cannot change one. +//! - **Ledger-only inputs.** Every statement derives from the row passed in — +//! no config files, no re-benching, no clock. Same row, same bytes out. +//! - **Verdicts are records; explanations are inferences.** Lines that go +//! beyond restating the row are prefixed `inference:` so nobody can quote +//! an explanation as if the machine measured it. + +use serde_json::Value; + +/// Everything explain reads, pulled out of the row once. +pub struct Facts { + pub run_id: String, + pub target: String, + pub verdict: String, + pub hypothesis: String, + pub hotspot: String, + pub playbook_class: i64, + pub timestamp: String, + pub patch: String, + // gates + pub upstream_tests: bool, + pub golden_replay: bool, + pub fuzz_iters: u64, + pub fuzz_divergence: bool, + pub sanitizers_clean: bool, + pub tsan_clean: Option, + pub risk_signals: Vec, + pub equivalence_mode: Option, + // bench (absent when gates failed before benching) + pub bench: Option, +} + +pub struct BenchFacts { + pub baseline_median: f64, + pub candidate_median: f64, + pub speedup_median: f64, + pub speedup_lo: f64, + pub speedup_hi: f64, + pub workload: Option, + pub isolation: Option, + /// Accept bar in force for this run — recorded per-row since 2026-07-20; + /// None on older rows (and explain must not substitute today's config). + pub accept_threshold: Option, + pub max_rss_baseline_kib: Option, + pub max_rss_candidate_kib: Option, +} + +impl Facts { + pub fn from_row(row: &Value) -> Option { + let s = |k: &str| row.get(k)?.as_str().map(str::to_string); + let gates = row.get("gates")?; + let bench = row.get("bench").filter(|b| !b.is_null()).map(|b| { + let env = b.get("env_fingerprint"); + let ci = b.get("speedup_ci").and_then(|c| c.as_array().cloned()); + let ci_at = |i: usize| -> f64 { + ci.as_ref() + .and_then(|c| c.get(i)) + .and_then(|v| v.as_f64()) + .unwrap_or(f64::NAN) + }; + let rss = env.and_then(|e| e.get("max_rss_kib")); + BenchFacts { + baseline_median: b + .get("baseline_median") + .and_then(|v| v.as_f64()) + .unwrap_or(f64::NAN), + candidate_median: b + .get("candidate_median") + .and_then(|v| v.as_f64()) + .unwrap_or(f64::NAN), + speedup_median: b + .get("speedup_median") + .and_then(|v| v.as_f64()) + .unwrap_or(f64::NAN), + speedup_lo: ci_at(0), + speedup_hi: ci_at(1), + workload: env + .and_then(|e| e.get("workload")) + .and_then(|v| v.as_str()) + .map(str::to_string), + isolation: env + .and_then(|e| e.get("isolation")) + .and_then(|v| v.as_str()) + .map(str::to_string), + accept_threshold: env + .and_then(|e| e.get("accept_threshold")) + .and_then(|v| v.as_f64()), + max_rss_baseline_kib: rss + .and_then(|r| r.get("baseline_median")) + .and_then(|v| v.as_u64()), + max_rss_candidate_kib: rss + .and_then(|r| r.get("candidate_median")) + .and_then(|v| v.as_u64()), + } + }); + Some(Facts { + run_id: s("run_id")?, + target: s("target")?, + verdict: s("verdict")?, + hypothesis: s("hypothesis")?, + hotspot: s("hotspot")?, + playbook_class: row.get("playbook_class")?.as_i64()?, + timestamp: s("timestamp")?, + patch: s("patch")?, + upstream_tests: gates + .get("upstream_tests") + .and_then(|v| v.as_bool()) + .unwrap_or(false), + golden_replay: gates + .get("golden_replay") + .and_then(|v| v.as_bool()) + .unwrap_or(false), + fuzz_iters: gates + .get("fuzz_iters") + .and_then(|v| v.as_u64()) + .unwrap_or(0), + fuzz_divergence: gates + .get("fuzz_divergence") + .and_then(|v| v.as_bool()) + .unwrap_or(false), + sanitizers_clean: gates + .get("sanitizers_clean") + .and_then(|v| v.as_bool()) + .unwrap_or(false), + tsan_clean: gates.get("tsan_clean").and_then(|v| v.as_bool()), + risk_signals: gates + .get("risk_signals") + .and_then(|v| v.as_array()) + .map(|a| { + a.iter() + .filter_map(|x| x.as_str().map(str::to_string)) + .collect() + }) + .unwrap_or_default(), + equivalence_mode: gates + .get("equivalence_mode") + .and_then(|v| v.as_str()) + .map(str::to_string), + bench, + }) + } +} + +/// Where the bootstrap CI of the speedup ratio landed, relative to 1.0 and +/// (when the row records it) the accept bar. +#[derive(Debug, PartialEq)] +pub enum CiClass { + /// Lower bound clears the recorded bar. + MeetsBar, + /// CI excludes 1.0 upward, but does not clear the bar (or no bar recorded). + RealBelowBar, + /// CI straddles 1.0 — indistinguishable from no change. + Indistinguishable, + /// CI entirely below 1.0 — a measured regression. + Regression, +} + +pub fn classify_ci(lo: f64, hi: f64, threshold: Option) -> CiClass { + if hi < 1.0 { + CiClass::Regression + } else if lo <= 1.0 { + CiClass::Indistinguishable + } else if let Some(t) = threshold { + if lo >= 1.0 + t { + CiClass::MeetsBar + } else { + CiClass::RealBelowBar + } + } else { + CiClass::RealBelowBar + } +} + +pub struct DiffStats { + pub files: usize, + pub added: usize, + pub removed: usize, +} + +/// Mechanical shape of the stored unified diff. None when the row carries a +/// placeholder instead of a real patch (build-config attempts, smoke rows). +pub fn diff_stats(patch: &str) -> Option { + if !patch.contains("--- ") || !patch.contains("+++ ") { + return None; + } + let mut files = 0; + let mut added = 0; + let mut removed = 0; + for line in patch.lines() { + if line.starts_with("+++ ") { + files += 1; + } else if line.starts_with('+') && !line.starts_with("+++") { + added += 1; + } else if line.starts_with('-') && !line.starts_with("---") { + removed += 1; + } + } + Some(DiffStats { + files, + added, + removed, + }) +} + +fn pct(x: f64) -> String { + format!("{:+.1}%", (x - 1.0) * 100.0) +} + +/// The whole report. Pure function of the facts: byte-identical on re-run. +pub fn explain(f: &Facts) -> String { + let mut out = String::new(); + let mut line = |s: String| { + out.push_str(&s); + out.push('\n'); + }; + + line(format!("explain {} — advisory diagnosis", f.run_id)); + line( + "derived solely from the ledger row; regeneration is byte-identical; \ + this can never change a verdict" + .to_string(), + ); + line(String::new()); + line(format!( + "attempt: {} | class {} | hotspot: {} | recorded {}", + f.target, f.playbook_class, f.hotspot, f.timestamp + )); + line(format!("hypothesis: {}", f.hypothesis)); + if let Some(d) = diff_stats(&f.patch) { + line(format!( + "patch shape: {} file(s), +{} / -{} lines", + d.files, d.added, d.removed + )); + } else { + line( + "patch shape: no source diff recorded (build-config or out-of-band attempt)" + .to_string(), + ); + } + line(String::new()); + line(format!("verdict (machine record): {}", f.verdict)); + + // --- what decided it ------------------------------------------------- + match f.verdict.as_str() { + "rejected-gate" => { + let failed: Vec<&str> = [ + (!f.upstream_tests, "upstream test suite"), + (!f.golden_replay, "golden replay"), + (f.fuzz_divergence, "differential fuzz (divergence found)"), + ] + .iter() + .filter(|(cond, _)| *cond) + .map(|(_, name)| *name) + .collect(); + if failed.is_empty() { + line("decided by: a gate failure (the failing gate is not individually recorded in this row)".to_string()); + } else { + line(format!("decided by: gate failure — {}", failed.join(", "))); + } + line( + "inference: the patch changed observable behavior or broke the build contract; \ + the bench (if any ran) is irrelevant until equivalence holds" + .to_string(), + ); + } + "accepted" | "rejected-bench" | "needs-human-review" => { + if let Some(b) = &f.bench { + line(format!( + "bench: baseline median {:.4}s, candidate median {:.4}s — speedup {:.4} ({}), 95% CI [{:.4}, {:.4}]", + b.baseline_median, b.candidate_median, b.speedup_median, + pct(b.speedup_median), b.speedup_lo, b.speedup_hi + )); + if let Some(w) = &b.workload { + line(format!("workload: {}", w)); + } + match &b.accept_threshold { + Some(t) => line(format!( + "accept bar recorded for this run: CI lower bound >= {:.2} (1 + threshold {:.2})", + 1.0 + t, t + )), + None => line( + "accept bar: not machine-recorded for this row (rows before 2026-07-20); \ + the rule is CI lower bound >= 1 + threshold from config/accept.toml at run time" + .to_string(), + ), + } + // The narrative must agree with the machine verdict: an + // `accepted` row cleared the bar in force at run time by + // definition, even when the row predates per-row threshold + // recording — never describe an accept as a rejection. + match f.verdict.as_str() { + "accepted" => { + line(format!( + "inference: the defensible claim is the lower bound — \"at least {}\" on this workload, not the median {}", + pct(b.speedup_lo), pct(b.speedup_median) + )); + if let Some(t) = b.accept_threshold { + if b.speedup_lo < 1.0 + t + 0.01 { + line("inference: thin margin over the bar — re-run on calibrated hardware before any external claim".to_string()); + } + } + } + "needs-human-review" => { + line("inference: the numbers were not the cap — the review routing below decided this verdict".to_string()); + } + _ => match classify_ci(b.speedup_lo, b.speedup_hi, b.accept_threshold) { + CiClass::MeetsBar => { + line("inference: the CI lower bound clears the recorded bar yet the verdict is a rejection — inconsistent row, inspect by hand".to_string()); + } + CiClass::RealBelowBar => { + line(format!( + "inference: the improvement is real (CI excludes 1.0) but below the ship bar — \ + a policy rejection, not noise; the measured effect is {} to {}", + pct(b.speedup_lo), pct(b.speedup_hi) + )); + } + CiClass::Indistinguishable => { + line("inference: indistinguishable from no change — the CI straddles 1.0. \ + This is a null result, not a small win" + .to_string()); + } + CiClass::Regression => { + line(format!( + "inference: a measured regression — the candidate is {} to {} SLOWER; report it as one", + pct(2.0 - b.speedup_hi), pct(2.0 - b.speedup_lo) + )); + } + }, + } + if let (Some(rb), Some(rc)) = (b.max_rss_baseline_kib, b.max_rss_candidate_kib) { + if rb > 0 { + let ratio = rc as f64 / rb as f64; + line(format!( + "max RSS medians: baseline {} KiB, candidate {} KiB ({}) — context, never an accept metric", + rb, rc, pct(ratio) + )); + if f.verdict == "accepted" && ratio > 1.05 { + line("inference: the speedup is partly bought with memory — note the trade in any claim".to_string()); + } + } + } + if let Some(iso) = &b.isolation { + if iso.contains("NOT isolated") || iso == "unwrapped-host" { + line(format!( + "isolation note on record: \"{}\" — this run did not have the network-namespace boundary", + iso + )); + } + } + } else { + line("bench: no evidence recorded on this row".to_string()); + } + } + other => line(format!( + "verdict string '{}' is outside the known set — inspect the row by hand", + other + )), + } + + // --- caps and caveats ------------------------------------------------- + if f.verdict == "needs-human-review" { + if !f.risk_signals.is_empty() { + line(format!( + "routed to human review by the lexical risk classifier (SPEC §8): {}", + f.risk_signals.join(", ") + )); + line("inference: the numbers do not clear a risk class — concurrency/unsafe/FP changes can be \ + correct on this workload and wrong in production" + .to_string()); + } else if f.fuzz_iters == 0 { + line("no passed differential-fuzz gate on record — the no-fuzz-no-accept rule caps any would-be accept at review".to_string()); + } + if f.equivalence_mode.as_deref() == Some("fp-tolerance") { + line( + "equivalence used the fp-tolerance tier — always a review signal (SPEC §8)" + .to_string(), + ); + } + } + if f.verdict == "accepted" { + if !f.sanitizers_clean { + line("CAUTION: accepted with sanitizers_clean=false — this row predates the machine-enforced \ + sanitizer lane; under current doctrine it could not be machine-accepted. See results/rulings/" + .to_string()); + } + match f.fuzz_iters { + 0 => line( + "fuzz: 0 iterations in the machine record — a pre-integration row; differential fuzz \ + ran out-of-band via scripts through Phase 5 (an accept today requires a passed in-pipeline run)" + .to_string(), + ), + n => line(format!("fuzz: {} iterations executed, no divergence", n)), + } + if let Some(false) = f.tsan_clean { + line("CAUTION: TSan lane flagged this attempt".to_string()); + } + } + + out +} + +#[cfg(test)] +mod tests { + use super::*; + + #[test] + fn ci_classification_covers_all_regions() { + assert_eq!(classify_ci(0.91, 0.96, Some(0.02)), CiClass::Regression); + assert_eq!( + classify_ci(0.98, 1.01, Some(0.02)), + CiClass::Indistinguishable + ); + assert_eq!(classify_ci(1.005, 1.015, Some(0.02)), CiClass::RealBelowBar); + assert_eq!(classify_ci(1.03, 1.06, Some(0.02)), CiClass::MeetsBar); + // No recorded bar: a real improvement can never be called MeetsBar. + assert_eq!(classify_ci(1.08, 1.10, None), CiClass::RealBelowBar); + assert_eq!( + classify_ci(1.0, 1.02, Some(0.02)), + CiClass::Indistinguishable + ); + } + + #[test] + fn diff_stats_counts_and_rejects_placeholders() { + let d = diff_stats("--- a/f.c\n+++ b/f.c\n@@\n-old\n+new\n+more\n").unwrap(); + assert_eq!((d.files, d.added, d.removed), (1, 2, 1)); + assert!(diff_stats("(no source patch)").is_none()); + } + + fn row(verdict: &str, gates: serde_json::Value, bench: serde_json::Value) -> serde_json::Value { + serde_json::json!({ + "run_id": "t-001", "timestamp": "2026-07-20T00:00:00Z", "target": "demo", + "target_commit": "abc", "phase": 6, "hotspot": "hot() (40%)", + "playbook_class": 5, "hypothesis": "loop fusion cuts passes", + "patch": "--- a/x\n+++ b/x\n-a\n+b\n", "gates": gates, "bench": bench, + "verdict": verdict, "tokens_spent": 0, "wall_time_s": 1.0 + }) + } + + fn bench(lo: f64, hi: f64, thr: Option) -> serde_json::Value { + let mut env = serde_json::json!({"workload": "10k records", "isolation": "no-net.sh"}); + if let Some(t) = thr { + env["accept_threshold"] = serde_json::json!(t); + } + serde_json::json!({ + "baseline_median": 2.0, "baseline_ci": [1.9, 2.1], + "candidate_median": 1.8, "candidate_ci": [1.7, 1.9], + "speedup_median": (lo + hi) / 2.0, "speedup_ci": [lo, hi], + "env_fingerprint": env + }) + } + + #[test] + fn explanation_is_deterministic_and_labels_inference() { + let r = row( + "accepted", + serde_json::json!({"upstream_tests": true, "golden_replay": true, + "fuzz_iters": 10000, "fuzz_divergence": false, "sanitizers_clean": true}), + bench(1.05, 1.09, Some(0.02)), + ); + let f = Facts::from_row(&r).unwrap(); + let a = explain(&f); + let b = explain(&f); + assert_eq!(a, b, "explanations must be byte-identical on re-run"); + assert!(a.contains("advisory")); + assert!(a.contains("inference:")); + assert!( + a.contains("at least +5.0%"), + "claim must be the lower bound: {a}" + ); + assert!(a.contains("10000 iterations")); + } + + #[test] + fn regression_and_null_results_are_named_plainly() { + let g = serde_json::json!({"upstream_tests": true, "golden_replay": true, + "fuzz_iters": 0, "fuzz_divergence": false, "sanitizers_clean": false}); + let reg = Facts::from_row(&row( + "rejected-bench", + g.clone(), + bench(0.92, 0.97, Some(0.02)), + )) + .unwrap(); + assert!(explain(®).contains("SLOWER")); + let null = + Facts::from_row(&row("rejected-bench", g, bench(0.99, 1.01, Some(0.02)))).unwrap(); + assert!(explain(&null).contains("indistinguishable from no change")); + } + + #[test] + fn historical_accept_without_sanitizers_is_flagged() { + let r = row( + "accepted", + serde_json::json!({"upstream_tests": true, "golden_replay": true, + "fuzz_iters": 0, "fuzz_divergence": false, "sanitizers_clean": false}), + bench(1.08, 1.12, None), + ); + let text = explain(&Facts::from_row(&r).unwrap()); + assert!(text.contains("sanitizers_clean=false")); + assert!(text.contains("not machine-recorded for this row")); + assert!(text.contains("out-of-band")); + // An accepted row cleared the bar in force at run time by definition: + // the missing per-row threshold must never read as a rejection. + assert!(!text.contains("policy rejection"), "{text}"); + assert!(text.contains("at least +8.0%")); + } + + #[test] + fn nhr_names_its_cap() { + let r = row( + "needs-human-review", + serde_json::json!({"upstream_tests": true, "golden_replay": true, + "fuzz_iters": 10000, "fuzz_divergence": false, "sanitizers_clean": true, + "risk_signals": ["concurrency"], "equivalence_mode": "byte-identical"}), + bench(1.10, 1.15, Some(0.02)), + ); + let text = explain(&Facts::from_row(&r).unwrap()); + assert!(text.contains("lexical risk classifier")); + assert!(text.contains("concurrency")); + } +} diff --git a/crates/explain/src/main.rs b/crates/explain/src/main.rs new file mode 100644 index 0000000..8c06d54 --- /dev/null +++ b/crates/explain/src/main.rs @@ -0,0 +1,34 @@ +//! `just explain ` — advisory post-verdict diagnosis (SPEC §3.7). +//! Reads ONE ledger row and prints why the attempt won or lost. Never on the +//! accept path: the verdict this reads was final before this binary ran. + +use anyhow::{anyhow, Context, Result}; +use clap::Parser; +use explain::{explain, Facts}; +use std::path::PathBuf; + +#[derive(Parser)] +#[command( + name = "explain", + about = "Advisory diagnosis of a ledger row (SPEC §3.7) — derived solely from the machine record" +)] +struct Cli { + /// Ledger run_id to explain. + #[arg(long)] + run_id: String, + #[arg(long, default_value = "results/ledger.sqlite")] + db: PathBuf, +} + +fn main() -> Result<()> { + let cli = Cli::parse(); + let ledger = ledger::Ledger::open(&cli.db) + .with_context(|| format!("opening ledger {}", cli.db.display()))?; + let row = ledger + .attempt_row(&cli.run_id)? + .ok_or_else(|| anyhow!("no ledger row with run_id '{}'", cli.run_id))?; + let facts = Facts::from_row(&row) + .ok_or_else(|| anyhow!("row '{}' is missing required fields", cli.run_id))?; + print!("{}", explain(&facts)); + Ok(()) +} diff --git a/crates/ledger/src/lib.rs b/crates/ledger/src/lib.rs index 98ab99d..c1c88b1 100644 --- a/crates/ledger/src/lib.rs +++ b/crates/ledger/src/lib.rs @@ -308,6 +308,75 @@ impl Ledger { } } + /// The full machine record of one attempt — every column, with the + /// gates/bench JSON columns parsed. Read-only; this is the entire input + /// surface of `crates/explain` (SPEC §3.7), which must derive from the + /// ledger row and nothing else. + pub fn attempt_row(&self, run_id: &str) -> Result, LedgerError> { + let row = self.conn.query_row( + "SELECT timestamp, target, target_commit, phase, hotspot, playbook_class, \ + hypothesis, patch, gates, bench, verdict, tokens_spent, wall_time_s \ + FROM attempts WHERE run_id = ?1", + [run_id], + |r| { + Ok(( + r.get::<_, String>(0)?, + r.get::<_, String>(1)?, + r.get::<_, String>(2)?, + r.get::<_, i64>(3)?, + r.get::<_, String>(4)?, + r.get::<_, i64>(5)?, + r.get::<_, String>(6)?, + r.get::<_, String>(7)?, + r.get::<_, String>(8)?, + r.get::<_, Option>(9)?, + r.get::<_, String>(10)?, + r.get::<_, i64>(11)?, + r.get::<_, f64>(12)?, + )) + }, + ); + match row { + Ok(( + ts, + target, + commit, + phase, + hotspot, + class, + hyp, + patch, + gates, + bench, + verdict, + tokens, + wall, + )) => { + let gates: serde_json::Value = serde_json::from_str(&gates)?; + let bench: Option = + bench.map(|b| serde_json::from_str(&b)).transpose()?; + Ok(Some(serde_json::json!({ + "run_id": run_id, + "timestamp": ts, + "target": target, + "target_commit": commit, + "phase": phase, + "hotspot": hotspot, + "playbook_class": class, + "hypothesis": hyp, + "patch": patch, + "gates": gates, + "bench": bench, + "verdict": verdict, + "tokens_spent": tokens, + "wall_time_s": wall, + }))) + } + Err(rusqlite::Error::QueryReturnedNoRows) => Ok(None), + Err(e) => Err(e.into()), + } + } + pub fn count(&self) -> Result { Ok(self .conn diff --git a/crates/verdict/src/main.rs b/crates/verdict/src/main.rs index ff43226..195c380 100644 --- a/crates/verdict/src/main.rs +++ b/crates/verdict/src/main.rs @@ -472,6 +472,10 @@ fn main() -> Result<()> { "workload": spec.bench.workload, "target_commit": spec.source.commit, "isolation": cli.isolation_note.as_deref().unwrap_or("unwrapped-host"), + // The accept bar in force for THIS run, recorded so the row is + // self-contained: `explain` must never guess a historical + // threshold from today's config (SPEC §3.7). + "accept_threshold": cfg.threshold, "gates_detail": "diff-fuzz gated in-pipeline against the pristine baseline (an accept requires it to have passed); ASan+UBSan enforced on the accept path, TSan when configured", }); if !samples.baseline_max_rss_kib.is_empty() && !samples.candidate_max_rss_kib.is_empty() { diff --git a/docs/src/SUMMARY.md b/docs/src/SUMMARY.md index 5061cca..466c5be 100644 --- a/docs/src/SUMMARY.md +++ b/docs/src/SUMMARY.md @@ -25,5 +25,6 @@ - [Consume it: the peltier skill](./skill.md) - [Reproduce it yourself](./reproduce.md) +- [Explain: why a row won or lost](./explain.md) - [The trust boundary (infra)](./infra.md) - [Roadmap & scope](./roadmap.md) diff --git a/docs/src/architecture.md b/docs/src/architecture.md index 4076cd0..ab644ac 100644 --- a/docs/src/architecture.md +++ b/docs/src/architecture.md @@ -11,6 +11,9 @@ crates/ Trust layer — agent has NO write access (SPEC §10) diff-test/ Equivalence gates, corpus hash-pinning, per-target spec ledger/ Append-only SQLite attempt ledger (enforced by triggers) report/ ROI: speedup CI → cores → dollars, methodology inline + explain/ Advisory post-verdict diagnosis of a ledger row (why it + won or lost) — never on the accept path + policy/ Learned class-selection prior from the ledger (advisory) verdict/ The pipeline in one command: gates → bench → ledger row harnessd/ The one door the agent talks through (7 JSON ops) agent/ Untrusted proposer (Claude Agent SDK, Python) diff --git a/docs/src/explain.md b/docs/src/explain.md new file mode 100644 index 0000000..03b2d14 --- /dev/null +++ b/docs/src/explain.md @@ -0,0 +1,68 @@ +# Explain: why a row won or lost + +Verdicts say *what*. `just explain ` says *why* — an advisory +diagnosis of one ledger row, built so it can never contaminate the thing it +explains: + +- **Ledger-only inputs, deterministic.** It reads exactly one row and nothing + else — no config, no clock, no re-benching. The same row produces + byte-identical text, so a regenerated explanation is checkable. +- **Strictly off the accept path.** It runs after the verdict is written; + nothing it says feeds gates, the bench, or the ledger. +- **Record vs. inference is explicit.** Restatements of the machine record are + printed plain; anything beyond that is prefixed `inference:` — so an + explanation can never be quoted as if the machine measured it. +- **The narrative must agree with the verdict.** An accepted row is described + by its CI *lower bound* (the defensible claim). A rejected bench is + classified honestly: measured regression, null result, or + real-but-below-bar. A `needs-human-review` row names the recorded cap and + says plainly that the numbers were not the deciding factor. + +## Real output, real rows + +The shipped cJSON win ([`phase3-cjson-002`](./case-studies/cjson.md)): + +``` +verdict (machine record): accepted +bench: baseline median 0.4263s, candidate median 0.3916s — speedup 1.0885 (+8.8%), 95% CI [1.0745, 1.1001] +workload: 5.6 MB synthetic JSON (20k records: nested objects/arrays, escaped/unicode strings, ints, dense floats), parsed+serialized 2x, single thread +inference: the defensible claim is the lower bound — "at least +7.4%" on this workload, not the median +8.8% +``` + +The [caught false-accept](./case-studies/comrak-false-accept.md) +(`phase2-comrak-010`) — explain flags it instead of laundering it: + +``` +inference: the defensible claim is the lower bound — "at least +7.9%" on this workload, not the median +10.7% +CAUTION: accepted with sanitizers_clean=false — this row predates the machine-enforced +sanitizer lane; under current doctrine it could not be machine-accepted. See results/rulings/ +``` + +The kernel-lane row (`phase5-matmul-opt`, `needs-human-review`): + +``` +verdict (machine record): needs-human-review +bench: … speedup 3.2301 (+223.0%), 95% CI [3.1600, 3.2594] +inference: the numbers were not the cap — the review routing below decided this verdict +``` + +That last line is the point: a 3.2× speedup with a tight interval still does +not argue with the review routing. Explanations restate the record and reason +*from* it — they do not re-litigate it. + +## Why this exists + +The loop's compounding value is not any single verdict — it is that the next +hypothesis is better than the last. A rejection that says only +`rejected-bench` teaches nothing; one that says "real improvement, below the +ship bar" or "indistinguishable from no change — a null result, not a small +win" or "a measured regression, report it as one" points at three different +next moves. Explain is that teaching layer, kept deliberately outside the +measurement it learns from — see SPEC §3.7 for the invariants, including the +one that keeps it out of the agent-facing surface: better hypotheses must +never become a feedback channel into the accept path. + +Rows written from 2026-07-20 on also record the accept bar in force for that +run (`accept_threshold` in the env fingerprint), so an explanation never has +to guess a historical threshold from today's config — for older rows it says +so explicitly instead. diff --git a/justfile b/justfile index d65952a..1158474 100644 --- a/justfile +++ b/justfile @@ -92,6 +92,11 @@ service-calibrate server doc out *flags: report run_id *flags: cargo run -p report -- --run-id "{{run_id}}" {{flags}} +# Advisory post-verdict diagnosis of a ledger row (SPEC §3.7): why it won +# or lost, derived solely from the machine record. Never changes a verdict. +explain run_id: + cargo run -p explain -- --run-id "{{run_id}}" + # Learned playbook-class ranking from the ledger (SPEC §13 research fork) policy *flags: cargo run -p policy -- {{flags}}