mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
docs: name the statistical identity — Bayesian detective, frequentist court
The two halves of the system run on opposite statistical philosophies, on purpose, and until now the docs never said so. The optimizer is a Bayesian detective: playbook prior, ledger evidence, policy ranking, explain feedback — belief-updating that decides what to try next. The trust layer is a frequentist court: equivalence gates, per-machine calibrated error rates (A/A false-positive bound, injected-regression power bound), and a CI-lower-bound accept rule with no prior anywhere on the accept path — because its audience is a hostile reviewer who shares none of your priors. The load-bearing edge is the return path: evidence flows from court to detective (ledger rows + explain); authority never flows the other way. Invariant, now stated in four places: priors steer where we look; they never touch what we conclude. A wrong prior costs compute — each wasted attempt honestly ledgered. A prior in the verdict would cost a false claim, the one unrecoverable failure for a project whose product is trust. - thesis.md: full section with the two-box diagram incl. the return edge, in docket order (equivalence before measurement); notes that estimated payoff is a belief and dollar figures are minted only by the mechanical report from accepted rows, and that even the learning layer bets on Wilson LOWER bounds — the detective is a pessimist too. - README: the one-breath version, linking the full argument. - SPEC §1: dated note mapping each half to its sections. - CLAUDE.md: the mental model, one paragraph under Mission. Framing credit: emerged from a design conversation on whether peltier is "Bayesian engineering" — it is exactly half of one. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
This commit is contained in:
@@ -6,6 +6,8 @@
|
||||
|
||||
An agent that profiles real binaries and services, isolates hot paths, proposes optimizations, and accepts a change **only** when (a) behavioral-equivalence gates pass and (b) the benchmark improvement is statistically significant on trusted infrastructure. Output: verified performance deltas translated into ROI reports nobody can dispute.
|
||||
|
||||
Mental model: **the optimizer is a Bayesian detective; the trust layer is a frequentist court.** The detective (agent + playbook prior + policy ranking + ledger evidence + explain feedback) chooses what to try; the court (gates + calibrated bench + CI-lower-bound rule) decides what is true, with measured error rates and no priors on the accept path. Evidence flows back; authority never does. Priors steer where we look — they never touch what we conclude.
|
||||
|
||||
Full engineering spec: `SPEC.md`. Read it before starting any phase.
|
||||
|
||||
## Non-negotiables (re-read every session)
|
||||
|
||||
@@ -15,6 +15,14 @@ deltas with the methodology attached** — a stopwatch on pinned hardware, a
|
||||
proof of equivalence, and an ROI figure whose confidence interval survives
|
||||
hostile review.
|
||||
|
||||
The design in one breath: **the optimizer is a Bayesian detective; the
|
||||
trust layer is a frequentist court.** The detective holds priors, updates
|
||||
them from the ledger, and picks the next bet; the court runs calibrated
|
||||
procedures with measured error rates and no priors at all. Evidence flows
|
||||
back from court to detective — authority never does. Priors steer where we
|
||||
look; they never touch what we conclude.
|
||||
[The full argument](https://bdbrown4.github.io/peltier.io/thesis.html#the-detective-and-the-court).
|
||||
|
||||
📖 **Full documentation: the [peltier.io book](https://bdbrown4.github.io/peltier.io/)**
|
||||
· Design spec: [SPEC.md](SPEC.md) · Charter: [CLAUDE.md](CLAUDE.md)
|
||||
|
||||
|
||||
@@ -9,6 +9,8 @@ Why this wins:
|
||||
- **The metric defends itself.** Same inputs, same outputs, fewer cycles, with confidence intervals. No correlation-causation problem is possible by construction.
|
||||
- **The ledger compounds.** Every accepted/rejected attempt with full evidence is future RL training data and, eventually, the substrate for learned optimization policies (see §13, research fork).
|
||||
|
||||
> **Added 2026-07-20 — the statistical identity, named.** The optimizer is a **Bayesian detective**; the trust layer is a **frequentist court**. The detective (agent loop, playbook prior, §13 policy ranking, ledger evidence, §3.7 explain feedback) decides *what to try next* by belief-updating over accumulated evidence. The court (§3.1–§3.3: gates, calibrated bench, CI-lower-bound accept rule) decides *what is true* via procedures with empirically measured error rates — no prior appears anywhere on the accept path. Evidence flows from court to detective (ledger rows, explanations); authority never flows the other way. Invariant: **priors steer where we look; they never touch what we conclude.** A wrong prior costs compute; a prior in the verdict would cost trust.
|
||||
|
||||
## 2. System overview
|
||||
|
||||
```
|
||||
|
||||
@@ -46,3 +46,62 @@ cannot touch the thresholds. It can only propose a patch and ask the
|
||||
harness to judge it. The [architecture](./architecture.md) is designed so
|
||||
that even a compromised or adversarial proposer cannot manufacture a false
|
||||
win.
|
||||
|
||||
## The detective and the court
|
||||
|
||||
> The optimizer is a **Bayesian detective**. The trust layer is a
|
||||
> **frequentist court**.
|
||||
|
||||
That sentence is the whole design. The two halves run on opposite
|
||||
statistical philosophies, deliberately:
|
||||
|
||||
```
|
||||
BAYESIAN DETECTIVE FREQUENTIST COURT
|
||||
the optimizer: agent loop, the trust layer: gates,
|
||||
playbook prior, policy ranking calibrated bench, verdict
|
||||
|
||||
┌───────────────────────────┐ ┌────────────────────────────┐
|
||||
│ generate hypotheses │ │ prove equivalence first │
|
||||
│ prioritize the docket │ proposes │ calibrate the instrument │
|
||||
│ estimate likely payoff │ ─────────▶ │ measure, interleaved A/B │
|
||||
└───────────────────────────┘ │ reject noise (CI lower │
|
||||
▲ │ bound vs. threshold) │
|
||||
│ └─────────────┬──────────────┘
|
||||
│ ▼
|
||||
│ evidence flows back verdict
|
||||
└───── ledger row + explain ◀──────────────┘
|
||||
```
|
||||
|
||||
The **detective** reasons like a Bayesian: it holds a prior over where wins
|
||||
live (the [playbook's](./playbook.md) cheapest-first ordering), updates it
|
||||
from accumulated evidence (the append-only ledger — failures included,
|
||||
because failures are the most informative updates), ranks its next bets
|
||||
(`crates/policy`, by the Wilson *lower* bound of each class's proven win
|
||||
rate — even the learning layer bets pessimistically), and reads the
|
||||
[post-verdict diagnosis](./explain.md) so the next hypothesis is sharper
|
||||
than the last. Its "estimate likely payoff" is a belief, and stays one:
|
||||
dollar figures are minted only by the court's
|
||||
[mechanical report](./roi.md), only from accepted rows.
|
||||
|
||||
The **court** reasons like a frequentist, because its audience is a hostile
|
||||
reviewer who shares none of your priors. Its verdicts are decision
|
||||
procedures with *measured* long-run error rates: the A/A calibration bounds
|
||||
the false-positive rate and the injected-regression test bounds the power —
|
||||
per machine, empirically, before any real measurement is believed. The
|
||||
accept rule is a conservative bound, not a posterior; there is no prior
|
||||
anywhere on the accept path, so there is nothing to argue about except the
|
||||
evidence.
|
||||
|
||||
The return edge is what makes the detective Bayesian at all — and note its
|
||||
direction. **Evidence flows back; authority never does.** The invariant
|
||||
underneath the whole system:
|
||||
|
||||
> **Priors may steer where you look. They may never touch what you
|
||||
> conclude.**
|
||||
|
||||
A wrong prior in the detective costs compute — a few wasted attempts, each
|
||||
honestly ledgered. A prior in the court could cost a false claim, the one
|
||||
unrecoverable failure for a project whose product is trust. The gates are
|
||||
what make speculative, prior-driven exploration *safe*: the detective can
|
||||
believe anything it likes, because nothing it believes survives into a
|
||||
number.
|
||||
|
||||
Reference in New Issue
Block a user