From 24127bbcb4702540601a4048ce6766ce056ce3b3 Mon Sep 17 00:00:00 2001 From: Ben Brown Date: Mon, 20 Jul 2026 12:52:44 -0400 Subject: [PATCH] =?UTF-8?q?docs:=20name=20the=20statistical=20identity=20?= =?UTF-8?q?=E2=80=94=20Bayesian=20detective,=20frequentist=20court?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The two halves of the system run on opposite statistical philosophies, on purpose, and until now the docs never said so. The optimizer is a Bayesian detective: playbook prior, ledger evidence, policy ranking, explain feedback — belief-updating that decides what to try next. The trust layer is a frequentist court: equivalence gates, per-machine calibrated error rates (A/A false-positive bound, injected-regression power bound), and a CI-lower-bound accept rule with no prior anywhere on the accept path — because its audience is a hostile reviewer who shares none of your priors. The load-bearing edge is the return path: evidence flows from court to detective (ledger rows + explain); authority never flows the other way. Invariant, now stated in four places: priors steer where we look; they never touch what we conclude. A wrong prior costs compute — each wasted attempt honestly ledgered. A prior in the verdict would cost a false claim, the one unrecoverable failure for a project whose product is trust. - thesis.md: full section with the two-box diagram incl. the return edge, in docket order (equivalence before measurement); notes that estimated payoff is a belief and dollar figures are minted only by the mechanical report from accepted rows, and that even the learning layer bets on Wilson LOWER bounds — the detective is a pessimist too. - README: the one-breath version, linking the full argument. - SPEC §1: dated note mapping each half to its sections. - CLAUDE.md: the mental model, one paragraph under Mission. Framing credit: emerged from a design conversation on whether peltier is "Bayesian engineering" — it is exactly half of one. Co-Authored-By: Claude Fable 5 --- CLAUDE.md | 2 ++ README.md | 8 +++++++ SPEC.md | 2 ++ docs/src/thesis.md | 59 ++++++++++++++++++++++++++++++++++++++++++++++ 4 files changed, 71 insertions(+) diff --git a/CLAUDE.md b/CLAUDE.md index 29241ea..e88354c 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -6,6 +6,8 @@ An agent that profiles real binaries and services, isolates hot paths, proposes optimizations, and accepts a change **only** when (a) behavioral-equivalence gates pass and (b) the benchmark improvement is statistically significant on trusted infrastructure. Output: verified performance deltas translated into ROI reports nobody can dispute. +Mental model: **the optimizer is a Bayesian detective; the trust layer is a frequentist court.** The detective (agent + playbook prior + policy ranking + ledger evidence + explain feedback) chooses what to try; the court (gates + calibrated bench + CI-lower-bound rule) decides what is true, with measured error rates and no priors on the accept path. Evidence flows back; authority never does. Priors steer where we look — they never touch what we conclude. + Full engineering spec: `SPEC.md`. Read it before starting any phase. ## Non-negotiables (re-read every session) diff --git a/README.md b/README.md index 162e3b5..09b0ef6 100644 --- a/README.md +++ b/README.md @@ -15,6 +15,14 @@ deltas with the methodology attached** — a stopwatch on pinned hardware, a proof of equivalence, and an ROI figure whose confidence interval survives hostile review. +The design in one breath: **the optimizer is a Bayesian detective; the +trust layer is a frequentist court.** The detective holds priors, updates +them from the ledger, and picks the next bet; the court runs calibrated +procedures with measured error rates and no priors at all. Evidence flows +back from court to detective — authority never does. Priors steer where we +look; they never touch what we conclude. +[The full argument](https://bdbrown4.github.io/peltier.io/thesis.html#the-detective-and-the-court). + 📖 **Full documentation: the [peltier.io book](https://bdbrown4.github.io/peltier.io/)** · Design spec: [SPEC.md](SPEC.md) · Charter: [CLAUDE.md](CLAUDE.md) diff --git a/SPEC.md b/SPEC.md index 4a03d7b..b7275bb 100644 --- a/SPEC.md +++ b/SPEC.md @@ -9,6 +9,8 @@ Why this wins: - **The metric defends itself.** Same inputs, same outputs, fewer cycles, with confidence intervals. No correlation-causation problem is possible by construction. - **The ledger compounds.** Every accepted/rejected attempt with full evidence is future RL training data and, eventually, the substrate for learned optimization policies (see §13, research fork). +> **Added 2026-07-20 — the statistical identity, named.** The optimizer is a **Bayesian detective**; the trust layer is a **frequentist court**. The detective (agent loop, playbook prior, §13 policy ranking, ledger evidence, §3.7 explain feedback) decides *what to try next* by belief-updating over accumulated evidence. The court (§3.1–§3.3: gates, calibrated bench, CI-lower-bound accept rule) decides *what is true* via procedures with empirically measured error rates — no prior appears anywhere on the accept path. Evidence flows from court to detective (ledger rows, explanations); authority never flows the other way. Invariant: **priors steer where we look; they never touch what we conclude.** A wrong prior costs compute; a prior in the verdict would cost trust. + ## 2. System overview ``` diff --git a/docs/src/thesis.md b/docs/src/thesis.md index 923d9b9..a323d32 100644 --- a/docs/src/thesis.md +++ b/docs/src/thesis.md @@ -46,3 +46,62 @@ cannot touch the thresholds. It can only propose a patch and ask the harness to judge it. The [architecture](./architecture.md) is designed so that even a compromised or adversarial proposer cannot manufacture a false win. + +## The detective and the court + +> The optimizer is a **Bayesian detective**. The trust layer is a +> **frequentist court**. + +That sentence is the whole design. The two halves run on opposite +statistical philosophies, deliberately: + +``` + BAYESIAN DETECTIVE FREQUENTIST COURT + the optimizer: agent loop, the trust layer: gates, + playbook prior, policy ranking calibrated bench, verdict + + ┌───────────────────────────┐ ┌────────────────────────────┐ + │ generate hypotheses │ │ prove equivalence first │ + │ prioritize the docket │ proposes │ calibrate the instrument │ + │ estimate likely payoff │ ─────────▶ │ measure, interleaved A/B │ + └───────────────────────────┘ │ reject noise (CI lower │ + ▲ │ bound vs. threshold) │ + │ └─────────────┬──────────────┘ + │ ▼ + │ evidence flows back verdict + └───── ledger row + explain ◀──────────────┘ +``` + +The **detective** reasons like a Bayesian: it holds a prior over where wins +live (the [playbook's](./playbook.md) cheapest-first ordering), updates it +from accumulated evidence (the append-only ledger — failures included, +because failures are the most informative updates), ranks its next bets +(`crates/policy`, by the Wilson *lower* bound of each class's proven win +rate — even the learning layer bets pessimistically), and reads the +[post-verdict diagnosis](./explain.md) so the next hypothesis is sharper +than the last. Its "estimate likely payoff" is a belief, and stays one: +dollar figures are minted only by the court's +[mechanical report](./roi.md), only from accepted rows. + +The **court** reasons like a frequentist, because its audience is a hostile +reviewer who shares none of your priors. Its verdicts are decision +procedures with *measured* long-run error rates: the A/A calibration bounds +the false-positive rate and the injected-regression test bounds the power — +per machine, empirically, before any real measurement is believed. The +accept rule is a conservative bound, not a posterior; there is no prior +anywhere on the accept path, so there is nothing to argue about except the +evidence. + +The return edge is what makes the detective Bayesian at all — and note its +direction. **Evidence flows back; authority never does.** The invariant +underneath the whole system: + +> **Priors may steer where you look. They may never touch what you +> conclude.** + +A wrong prior in the detective costs compute — a few wasted attempts, each +honestly ledgered. A prior in the court could cost a false claim, the one +unrecoverable failure for a project whose product is trust. The gates are +what make speculative, prior-driven exploration *safe*: the detective can +believe anything it likes, because nothing it believes survives into a +number.