Merge pull request #24 from bdbrown4/docs/detective-court

docs: name the statistical identity — Bayesian detective, frequentist court
This commit is contained in:
2026-07-20 13:04:56 -04:00
committed by GitHub
4 changed files with 71 additions and 0 deletions
+2
View File
@@ -6,6 +6,8 @@
An agent that profiles real binaries and services, isolates hot paths, proposes optimizations, and accepts a change **only** when (a) behavioral-equivalence gates pass and (b) the benchmark improvement is statistically significant on trusted infrastructure. Output: verified performance deltas translated into ROI reports nobody can dispute. An agent that profiles real binaries and services, isolates hot paths, proposes optimizations, and accepts a change **only** when (a) behavioral-equivalence gates pass and (b) the benchmark improvement is statistically significant on trusted infrastructure. Output: verified performance deltas translated into ROI reports nobody can dispute.
Mental model: **the optimizer is a Bayesian detective; the trust layer is a frequentist court.** The detective (agent + playbook prior + policy ranking + ledger evidence + explain feedback) chooses what to try; the court (gates + calibrated bench + CI-lower-bound rule) decides what is true, with measured error rates and no priors on the accept path. Evidence flows back; authority never does. Priors steer where we look — they never touch what we conclude.
Full engineering spec: `SPEC.md`. Read it before starting any phase. Full engineering spec: `SPEC.md`. Read it before starting any phase.
## Non-negotiables (re-read every session) ## Non-negotiables (re-read every session)
+8
View File
@@ -15,6 +15,14 @@ deltas with the methodology attached** — a stopwatch on pinned hardware, a
proof of equivalence, and an ROI figure whose confidence interval survives proof of equivalence, and an ROI figure whose confidence interval survives
hostile review. hostile review.
The design in one breath: **the optimizer is a Bayesian detective; the
trust layer is a frequentist court.** The detective holds priors, updates
them from the ledger, and picks the next bet; the court runs calibrated
procedures with measured error rates and no priors at all. Evidence flows
back from court to detective — authority never does. Priors steer where we
look; they never touch what we conclude.
[The full argument](https://bdbrown4.github.io/peltier.io/thesis.html#the-detective-and-the-court).
📖 **Full documentation: the [peltier.io book](https://bdbrown4.github.io/peltier.io/)** 📖 **Full documentation: the [peltier.io book](https://bdbrown4.github.io/peltier.io/)**
· Design spec: [SPEC.md](SPEC.md) · Charter: [CLAUDE.md](CLAUDE.md) · Design spec: [SPEC.md](SPEC.md) · Charter: [CLAUDE.md](CLAUDE.md)
+2
View File
@@ -9,6 +9,8 @@ Why this wins:
- **The metric defends itself.** Same inputs, same outputs, fewer cycles, with confidence intervals. No correlation-causation problem is possible by construction. - **The metric defends itself.** Same inputs, same outputs, fewer cycles, with confidence intervals. No correlation-causation problem is possible by construction.
- **The ledger compounds.** Every accepted/rejected attempt with full evidence is future RL training data and, eventually, the substrate for learned optimization policies (see §13, research fork). - **The ledger compounds.** Every accepted/rejected attempt with full evidence is future RL training data and, eventually, the substrate for learned optimization policies (see §13, research fork).
> **Added 2026-07-20 — the statistical identity, named.** The optimizer is a **Bayesian detective**; the trust layer is a **frequentist court**. The detective (agent loop, playbook prior, §13 policy ranking, ledger evidence, §3.7 explain feedback) decides *what to try next* by belief-updating over accumulated evidence. The court (§3.1–§3.3: gates, calibrated bench, CI-lower-bound accept rule) decides *what is true* via procedures with empirically measured error rates — no prior appears anywhere on the accept path. Evidence flows from court to detective (ledger rows, explanations); authority never flows the other way. Invariant: **priors steer where we look; they never touch what we conclude.** A wrong prior costs compute; a prior in the verdict would cost trust.
## 2. System overview ## 2. System overview
``` ```
+59
View File
@@ -46,3 +46,62 @@ cannot touch the thresholds. It can only propose a patch and ask the
harness to judge it. The [architecture](./architecture.md) is designed so harness to judge it. The [architecture](./architecture.md) is designed so
that even a compromised or adversarial proposer cannot manufacture a false that even a compromised or adversarial proposer cannot manufacture a false
win. win.
## The detective and the court
> The optimizer is a **Bayesian detective**. The trust layer is a
> **frequentist court**.
That sentence is the whole design. The two halves run on opposite
statistical philosophies, deliberately:
```
BAYESIAN DETECTIVE FREQUENTIST COURT
the optimizer: agent loop, the trust layer: gates,
playbook prior, policy ranking calibrated bench, verdict
┌───────────────────────────┐ ┌────────────────────────────┐
│ generate hypotheses │ │ prove equivalence first │
│ prioritize the docket │ proposes │ calibrate the instrument │
│ estimate likely payoff │ ─────────▶ │ measure, interleaved A/B │
└───────────────────────────┘ │ reject noise (CI lower │
▲ │ bound vs. threshold) │
│ └─────────────┬──────────────┘
│ ▼
│ evidence flows back verdict
└───── ledger row + explain ◀──────────────┘
```
The **detective** reasons like a Bayesian: it holds a prior over where wins
live (the [playbook's](./playbook.md) cheapest-first ordering), updates it
from accumulated evidence (the append-only ledger — failures included,
because failures are the most informative updates), ranks its next bets
(`crates/policy`, by the Wilson *lower* bound of each class's proven win
rate — even the learning layer bets pessimistically), and reads the
[post-verdict diagnosis](./explain.md) so the next hypothesis is sharper
than the last. Its "estimate likely payoff" is a belief, and stays one:
dollar figures are minted only by the court's
[mechanical report](./roi.md), only from accepted rows.
The **court** reasons like a frequentist, because its audience is a hostile
reviewer who shares none of your priors. Its verdicts are decision
procedures with *measured* long-run error rates: the A/A calibration bounds
the false-positive rate and the injected-regression test bounds the power —
per machine, empirically, before any real measurement is believed. The
accept rule is a conservative bound, not a posterior; there is no prior
anywhere on the accept path, so there is nothing to argue about except the
evidence.
The return edge is what makes the detective Bayesian at all — and note its
direction. **Evidence flows back; authority never does.** The invariant
underneath the whole system:
> **Priors may steer where you look. They may never touch what you
> conclude.**
A wrong prior in the detective costs compute — a few wasted attempts, each
honestly ledgered. A prior in the court could cost a false claim, the one
unrecoverable failure for a project whose product is trust. The gates are
what make speculative, prior-driven exploration *safe*: the detective can
believe anything it likes, because nothing it believes survives into a
number.