explain: post-verdict diagnostics — SPEC §3.7, crate, docs

Verdicts say what; nothing said why. A rejection that reads only
rejected-bench teaches nothing, and the loop's compounding value is that
the next hypothesis is better than the last. This ships the teaching layer,
built so it can never contaminate the thing it explains.

New crate crates/explain (`just explain <run-id>`):

- Advisory, strictly off the accept path: runs after the verdict is
  recorded; nothing it emits feeds gates, bench, verdict, or ledger. Not
  agent-facing: harnessd stays at seven ops, by design (SPEC §3.7 says why
  - better hypotheses must never become a feedback channel into
  measurement).
- Ledger-only inputs, deterministic: consumes exactly one row via the new
  ledger::attempt_row; no config, no clock, no re-benching. Identical row,
  byte-identical text - proven live (cmp on two runs of a real row).
- Record vs inference is typographic: restatements print plain; anything
  beyond is prefixed `inference:` so an explanation can never be quoted as
  a measurement.
- The narrative must agree with the machine verdict. The first live run
  caught this class of bug: an accepted row without a recorded threshold
  was narrated as "a policy rejection" two lines under "verdict: accepted".
  Accepted rows are now always described by their CI lower bound (the
  defensible claim); rejected-bench rows classify against 1.0 (measured
  regression / null result / real-but-below-bar); NHR rows name the
  recorded cap and state the numbers were not the deciding factor;
  rejected-gate rows name the failing gate. Regression-tested.
- Historical rows are flagged, not laundered: accepted with
  sanitizers_clean=false prints the doctrine caution (comrak-010 shows it);
  fuzz_iters=0 accepts are named pre-integration rows.

verdict now records accept_threshold (the bar in force for that run) into
the bench env fingerprint, so rows are self-contained going forward -
explain never substitutes today's config for a historical bar, and says so
on older rows instead.

SPEC §3.7 (Added 2026-07-20) carries the invariants; book chapter
docs/src/explain.md shows real output from real rows (the accepted cJSON
win, the caught false-accept, the kernel-lane NHR); CLAUDE.md, README,
justfile, and the architecture layout updated (the layout also gains the
previously-omitted crates/policy line).

Verified: 21 workspace test suites green (6 new explain tests incl.
determinism and verdict-narrative agreement), clippy -D warnings clean,
fmt clean, mdbook builds, live runs against phase3-cjson-002,
phase5-matmul-opt, phase2-comrak-010, and phase4-cjson-service.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
2026-07-20 09:53:14 -04:00
co-authored by Claude Fable 5
parent 48ef299cec
commit 444e5294fa
14 changed files with 751 additions and 0 deletions
+1
View File
@@ -25,5 +25,6 @@
- [Consume it: the peltier skill](./skill.md)
- [Reproduce it yourself](./reproduce.md)
- [Explain: why a row won or lost](./explain.md)
- [The trust boundary (infra)](./infra.md)
- [Roadmap & scope](./roadmap.md)
+3
View File
@@ -11,6 +11,9 @@ crates/ Trust layer — agent has NO write access (SPEC §10)
diff-test/ Equivalence gates, corpus hash-pinning, per-target spec
ledger/ Append-only SQLite attempt ledger (enforced by triggers)
report/ ROI: speedup CI → cores → dollars, methodology inline
explain/ Advisory post-verdict diagnosis of a ledger row (why it
won or lost) — never on the accept path
policy/ Learned class-selection prior from the ledger (advisory)
verdict/ The pipeline in one command: gates → bench → ledger row
harnessd/ The one door the agent talks through (7 JSON ops)
agent/ Untrusted proposer (Claude Agent SDK, Python)
+68
View File
@@ -0,0 +1,68 @@
# Explain: why a row won or lost
Verdicts say *what*. `just explain <run-id>` says *why* — an advisory
diagnosis of one ledger row, built so it can never contaminate the thing it
explains:
- **Ledger-only inputs, deterministic.** It reads exactly one row and nothing
else — no config, no clock, no re-benching. The same row produces
byte-identical text, so a regenerated explanation is checkable.
- **Strictly off the accept path.** It runs after the verdict is written;
nothing it says feeds gates, the bench, or the ledger.
- **Record vs. inference is explicit.** Restatements of the machine record are
printed plain; anything beyond that is prefixed `inference:` — so an
explanation can never be quoted as if the machine measured it.
- **The narrative must agree with the verdict.** An accepted row is described
by its CI *lower bound* (the defensible claim). A rejected bench is
classified honestly: measured regression, null result, or
real-but-below-bar. A `needs-human-review` row names the recorded cap and
says plainly that the numbers were not the deciding factor.
## Real output, real rows
The shipped cJSON win ([`phase3-cjson-002`](./case-studies/cjson.md)):
```
verdict (machine record): accepted
bench: baseline median 0.4263s, candidate median 0.3916s — speedup 1.0885 (+8.8%), 95% CI [1.0745, 1.1001]
workload: 5.6 MB synthetic JSON (20k records: nested objects/arrays, escaped/unicode strings, ints, dense floats), parsed+serialized 2x, single thread
inference: the defensible claim is the lower bound — "at least +7.4%" on this workload, not the median +8.8%
```
The [caught false-accept](./case-studies/comrak-false-accept.md)
(`phase2-comrak-010`) — explain flags it instead of laundering it:
```
inference: the defensible claim is the lower bound — "at least +7.9%" on this workload, not the median +10.7%
CAUTION: accepted with sanitizers_clean=false — this row predates the machine-enforced
sanitizer lane; under current doctrine it could not be machine-accepted. See results/rulings/
```
The kernel-lane row (`phase5-matmul-opt`, `needs-human-review`):
```
verdict (machine record): needs-human-review
bench: … speedup 3.2301 (+223.0%), 95% CI [3.1600, 3.2594]
inference: the numbers were not the cap — the review routing below decided this verdict
```
That last line is the point: a 3.2× speedup with a tight interval still does
not argue with the review routing. Explanations restate the record and reason
*from* it — they do not re-litigate it.
## Why this exists
The loop's compounding value is not any single verdict — it is that the next
hypothesis is better than the last. A rejection that says only
`rejected-bench` teaches nothing; one that says "real improvement, below the
ship bar" or "indistinguishable from no change — a null result, not a small
win" or "a measured regression, report it as one" points at three different
next moves. Explain is that teaching layer, kept deliberately outside the
measurement it learns from — see SPEC §3.7 for the invariants, including the
one that keeps it out of the agent-facing surface: better hypotheses must
never become a feedback channel into the accept path.
Rows written from 2026-07-20 on also record the accept bar in force for that
run (`accept_threshold` in the env fingerprint), so an explanation never has
to guess a historical threshold from today's config — for older rows it says
so explicitly instead.