mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
explain: post-verdict diagnostics — SPEC §3.7, crate, docs
Verdicts say what; nothing said why. A rejection that reads only rejected-bench teaches nothing, and the loop's compounding value is that the next hypothesis is better than the last. This ships the teaching layer, built so it can never contaminate the thing it explains. New crate crates/explain (`just explain <run-id>`): - Advisory, strictly off the accept path: runs after the verdict is recorded; nothing it emits feeds gates, bench, verdict, or ledger. Not agent-facing: harnessd stays at seven ops, by design (SPEC §3.7 says why - better hypotheses must never become a feedback channel into measurement). - Ledger-only inputs, deterministic: consumes exactly one row via the new ledger::attempt_row; no config, no clock, no re-benching. Identical row, byte-identical text - proven live (cmp on two runs of a real row). - Record vs inference is typographic: restatements print plain; anything beyond is prefixed `inference:` so an explanation can never be quoted as a measurement. - The narrative must agree with the machine verdict. The first live run caught this class of bug: an accepted row without a recorded threshold was narrated as "a policy rejection" two lines under "verdict: accepted". Accepted rows are now always described by their CI lower bound (the defensible claim); rejected-bench rows classify against 1.0 (measured regression / null result / real-but-below-bar); NHR rows name the recorded cap and state the numbers were not the deciding factor; rejected-gate rows name the failing gate. Regression-tested. - Historical rows are flagged, not laundered: accepted with sanitizers_clean=false prints the doctrine caution (comrak-010 shows it); fuzz_iters=0 accepts are named pre-integration rows. verdict now records accept_threshold (the bar in force for that run) into the bench env fingerprint, so rows are self-contained going forward - explain never substitutes today's config for a historical bar, and says so on older rows instead. SPEC §3.7 (Added 2026-07-20) carries the invariants; book chapter docs/src/explain.md shows real output from real rows (the accepted cJSON win, the caught false-accept, the kernel-lane NHR); CLAUDE.md, README, justfile, and the architecture layout updated (the layout also gains the previously-omitted crates/policy line). Verified: 21 workspace test suites green (6 new explain tests incl. determinism and verdict-narrative agreement), clippy -D warnings clean, fmt clean, mdbook builds, live runs against phase3-cjson-002, phase5-matmul-opt, phase2-comrak-010, and phase4-cjson-service. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -25,6 +25,7 @@ Full engineering spec: `SPEC.md`. Read it before starting any phase.
|
||||
- `crates/diff-test` — behavioral equivalence: corpus/test-suite pin checks + upstream test suite + golden I/O replay + differential fuzz (a baseline-vs-candidate gate; sanitizers run on the accept path in `crates/verdict`, and `diff-test` reports them as skipped-with-reason)
|
||||
- `crates/ledger` — append-only SQLite record of every attempt: hypothesis, patch, gate results, bench deltas, verdict, cost
|
||||
- `crates/report` — ROI generator: cores saved × $/core-hr, latency percentile deltas, CIs, workload caveats printed on every number
|
||||
- `crates/explain` — advisory post-verdict diagnosis (SPEC §3.7): one ledger row → why it won or lost. Ledger-only inputs, deterministic (identical row = identical bytes), `inference:`-labeled lines, verdict-narrative agreement enforced, historical accepts flagged rather than laundered. Strictly off the accept path; not exposed to the agent (harnessd stays seven ops)
|
||||
- `agent/` — Claude Agent SDK (Python) loop, prompts, tool definitions
|
||||
- `targets/` — vendored OSS targets for case studies (permissive licenses only)
|
||||
- `playbook/` — optimization classes, ordered, with preconditions and known risks
|
||||
@@ -58,6 +59,8 @@ just pin-corpus <target> # deliberate corpus re-pin (writes MANIFEST.sha256)
|
||||
just calibrate <cmd> <out> # automated A/A + regression-injection calibration
|
||||
just verdict <t> <bin> ... # gates + bench vs pristine-rebuilt baseline + ledger row
|
||||
just report <run-id> # ROI report from a ledger row (CIs + methodology inline)
|
||||
just explain <run-id> # advisory post-verdict diagnosis of a row (SPEC §3.7):
|
||||
# why it won/lost, ledger-only inputs, never changes a verdict
|
||||
just agent-attempt <t> <id> # one unattended agent attempt behind the OS boundary
|
||||
just isolation-check # verify the SPEC §10 OS boundary (19 checks, both modes)
|
||||
just service <b> <c> <doc> # service-mode latency A/B, CO-correct, p50/p99 CIs
|
||||
|
||||
Reference in New Issue
Block a user