mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
docs: full mdBook site + GitHub Pages deploy; rewrite README as project front page
- docs/: an mdBook documentation site — thesis, architecture, measurement
discipline, equivalence gates, agent loop, isolation/anti-reward-hacking,
playbook, ROI, reproduce, roadmap, and the five case studies (pulled from
their single source in results/ via {{#include}}). Builds clean (21
pages).
- .github/workflows/docs.yml: builds mdBook and deploys to GitHub Pages on
push to main (one-time setup: Settings → Pages → Source = GitHub Actions).
- README.md: rewritten as the project front page — the thesis, the verified
results table, layout, quick start, how it stays honest, links to the
book. Replaces the stale Phase-0-era text.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
This commit is contained in:
Claude
@@ -0,0 +1,53 @@
|
|||||||
|
name: docs
|
||||||
|
|
||||||
|
# Builds the mdBook site in docs/ and deploys it to GitHub Pages.
|
||||||
|
# One-time setup: repo Settings → Pages → Source = "GitHub Actions".
|
||||||
|
|
||||||
|
on:
|
||||||
|
push:
|
||||||
|
branches: [main]
|
||||||
|
paths:
|
||||||
|
- "docs/**"
|
||||||
|
- ".github/workflows/docs.yml"
|
||||||
|
workflow_dispatch:
|
||||||
|
|
||||||
|
permissions:
|
||||||
|
contents: read
|
||||||
|
pages: write
|
||||||
|
id-token: write
|
||||||
|
|
||||||
|
# Allow one concurrent deployment; don't cancel an in-progress prod deploy.
|
||||||
|
concurrency:
|
||||||
|
group: pages
|
||||||
|
cancel-in-progress: false
|
||||||
|
|
||||||
|
jobs:
|
||||||
|
build:
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
env:
|
||||||
|
MDBOOK_VERSION: "0.4.40"
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- name: Install mdBook
|
||||||
|
run: |
|
||||||
|
mkdir -p "$HOME/.local/bin"
|
||||||
|
url="https://github.com/rust-lang/mdBook/releases/download/v${MDBOOK_VERSION}/mdbook-v${MDBOOK_VERSION}-x86_64-unknown-linux-gnu.tar.gz"
|
||||||
|
curl -sSL "$url" | tar -xz -C "$HOME/.local/bin"
|
||||||
|
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
|
||||||
|
- name: Build
|
||||||
|
run: mdbook build docs
|
||||||
|
- name: Upload artifact
|
||||||
|
uses: actions/upload-pages-artifact@v3
|
||||||
|
with:
|
||||||
|
path: docs/book
|
||||||
|
|
||||||
|
deploy:
|
||||||
|
needs: build
|
||||||
|
runs-on: ubuntu-latest
|
||||||
|
environment:
|
||||||
|
name: github-pages
|
||||||
|
url: ${{ steps.deployment.outputs.page_url }}
|
||||||
|
steps:
|
||||||
|
- name: Deploy to GitHub Pages
|
||||||
|
id: deployment
|
||||||
|
uses: actions/deploy-pages@v4
|
||||||
@@ -249,3 +249,4 @@ targets/cjson/cand-*/
|
|||||||
targets/cjson/asan*/
|
targets/cjson/asan*/
|
||||||
targets/cjson/svc-baseline/
|
targets/cjson/svc-baseline/
|
||||||
targets/cjson/svc-candidate/
|
targets/cjson/svc-candidate/
|
||||||
|
docs/book/
|
||||||
|
|||||||
@@ -1,72 +1,88 @@
|
|||||||
# peltier.io
|
# hotpath
|
||||||
|
|
||||||
Spot cooling for hot paths. A profile-guided optimization agent that only ships wins it can prove.
|
> Spot cooling for hot paths. A profile-guided optimization agent that
|
||||||
|
> only ships wins it can prove.
|
||||||
|
|
||||||
hotpath finds the 10–90% left on the floor between "the compiler" and
|
hotpath profiles real binaries and services, isolates hot paths, proposes
|
||||||
"hire a perf consultant," with two properties no consultant offers:
|
optimizations, and accepts a change **only** when two independent bars are
|
||||||
**verified equivalence** (a change provably preserves behavior or is
|
cleared: **behavioral equivalence** (upstream tests + byte-identical golden
|
||||||
discarded) and **self-verifying ROI** (the value claim is a stopwatch on
|
replay + differential fuzzing + sanitizers) and **statistical significance**
|
||||||
trusted infrastructure, not a survey).
|
(the speedup's bootstrap 95% CI lower bound clears a threshold on
|
||||||
|
calibrated hardware). A change without both is discarded and logged.
|
||||||
|
|
||||||
Full design: **[SPEC.md](SPEC.md)**.
|
Consultants sell surveys; compilers sell flags. hotpath sells **verified
|
||||||
|
deltas with the methodology attached** — a stopwatch on pinned hardware, a
|
||||||
|
proof of equivalence, and an ROI figure whose confidence interval survives
|
||||||
|
hostile review.
|
||||||
|
|
||||||
|
📖 **Full documentation: the [hotpath book](https://bdbrown4.github.io/peltier.io/)**
|
||||||
|
· Design spec: [SPEC.md](SPEC.md) · Charter: [CLAUDE.md](CLAUDE.md)
|
||||||
|
|
||||||
|
## Verified results
|
||||||
|
|
||||||
|
Every number carries its 95% CI and its workload, and every one survived the
|
||||||
|
gates above. Across **five phases and 33 ledger rows, zero shipped false
|
||||||
|
accepts** — including two pipeline over-accepts the audit caught before
|
||||||
|
anything shipped, each becoming a permanent new gate.
|
||||||
|
|
||||||
|
| Target | Win | Verified |
|
||||||
|
|---|---|---|
|
||||||
|
| tokei (Rust) — LTO (class 1) | +10.4% | CI [+8.5%, +12.0%] |
|
||||||
|
| tokei — three class-5 algorithmic wins | +10.7% / +3.7% / +9.9% | each CI-significant |
|
||||||
|
| cJSON (C) — number-handling rewrite (class 5) | +8.85% | CI [+7.5%, +10.0%] |
|
||||||
|
| cJSON HTTP service — p50 latency under replay | +6.2% | CI [+5.8%, +7.2%] |
|
||||||
|
| comrak (Rust) — mimalloc (class 2) | +4.6% | CI [+3.3%, +5.8%], [human-ruled](results/rulings/phase0-comrak-002.md) |
|
||||||
|
|
||||||
## Layout
|
## Layout
|
||||||
|
|
||||||
```
|
```
|
||||||
crates/ Trust layer (agent has NO write access — SPEC §10)
|
crates/ Trust layer — agent has NO write access (SPEC §10)
|
||||||
bench-runner/ Interleaved A/B timing, bootstrap CIs, A/A calibration
|
bench-runner/ Interleaved A/B, bootstrap CIs, A/A calibration, service mode
|
||||||
diff-test/ Equivalence gates, corpus hash-pinning, per-target policy
|
diff-test/ Equivalence gates, corpus hash-pinning, per-target spec
|
||||||
ledger/ Append-only SQLite attempt ledger (enforced by triggers)
|
ledger/ Append-only SQLite attempt ledger (mutation-refusing triggers)
|
||||||
report/ ROI math: speedup CI → cores → dollars, caveats included
|
report/ ROI: speedup CI → cores → dollars, methodology inline
|
||||||
|
verdict/ The pipeline in one command: gates → bench → ledger row
|
||||||
|
harnessd/ The one door the agent talks through (7 JSON ops)
|
||||||
agent/ Untrusted proposer (Claude Agent SDK, Python)
|
agent/ Untrusted proposer (Claude Agent SDK, Python)
|
||||||
playbook/ Optimization classes 1–7, tried strictly cheapest-first
|
playbook/ Optimization classes 1–7, tried strictly cheapest-first
|
||||||
config/ accept.toml (thresholds), pricing.toml (ROI inputs)
|
config/ accept.toml (thresholds), pricing.toml (ROI inputs)
|
||||||
targets/ Per-target checkouts — the only agent-writable path
|
targets/ Vendored OSS targets — the only agent-writable path
|
||||||
corpora/ Hash-pinned golden-replay inputs (read-only to agent)
|
corpora/ Hash-pinned golden-replay inputs (read-only to agent)
|
||||||
results/ Calibration evidence, per-engagement outputs
|
results/ Calibration evidence, case studies, generated ROI reports
|
||||||
|
docs/ The mdBook site (deployed to GitHub Pages)
|
||||||
```
|
```
|
||||||
|
|
||||||
## Quick start
|
## Quick start
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
cargo test --workspace # trust-layer unit tests
|
just build / test / lint # trust-layer workspace
|
||||||
just aa # A/A self-test: must yield a null verdict
|
just aa # A/A self-test — must yield a null verdict
|
||||||
just compare "cmd-a" "cmd-b" # interleaved A/B with bootstrap CI
|
just gates <target> # corpus pin + upstream tests + golden replay
|
||||||
|
just verdict <t> <bin> ... # gates + bench vs pristine baseline + ledger row
|
||||||
|
just report <run-id> # ROI report from a ledger row
|
||||||
|
just isolation-check # 19 OS-boundary checks (both modes)
|
||||||
```
|
```
|
||||||
|
|
||||||
## Status
|
Full command reference and reproduction steps:
|
||||||
|
[Reproduce it yourself](https://bdbrown4.github.io/peltier.io/reproduce.html).
|
||||||
|
|
||||||
**Phase 0 complete** (see SPEC §5). Five attempts in the ledger across
|
## How it stays honest
|
||||||
two pinned targets (comrak 0.53.0, tokei 14.0.0), A/A-calibrated,
|
|
||||||
every number with CI and workload:
|
|
||||||
|
|
||||||
- **tokei +10.4% median [95% CI +8.5%, +12.0%]** — read-path buffer
|
- The agent speaks to the trust layer through **seven read-only-plus-one-
|
||||||
zeroing eliminated + fat LTO, byte-identical, 4,332-iteration
|
write JSON operations**; it has no shell, cannot write outside a target
|
||||||
diff-fuzz clean, sanitizers clean, auto-accepted
|
workspace, and cannot touch the ledger or thresholds.
|
||||||
(`results/phase0/case-study-tokei.md`). Meets the Phase 0 exit bar.
|
- The **baseline is rebuilt from a pristine checkout** every session — a
|
||||||
- comrak +4.6% [+3.3%, +5.8%] — mimalloc swap, `needs-human-review`
|
patch never becomes its own comparison.
|
||||||
on an LSan reachability flag (`results/phase0/case-study-comrak.md`).
|
- The **ledger is append-only** (SQLite triggers refuse UPDATE/DELETE);
|
||||||
- Two null/sub-threshold results correctly rejected and logged.
|
accepted rows are written only by the `verdict` binary after the gates.
|
||||||
|
- The agent process runs under **OS-level isolation** (read-only mount
|
||||||
|
namespace or an unprivileged uid) — the boundary is filesystem
|
||||||
|
permissions, not a prompt.
|
||||||
|
|
||||||
Next: Phase 1 trust-layer hardening (containers, perf, automated
|
See [the case studies](https://bdbrown4.github.io/peltier.io/case-studies/overview.html)
|
||||||
calibration, gate orchestration, agent IPC).
|
— especially [the caught false-accept](https://bdbrown4.github.io/peltier.io/case-studies/comrak-false-accept.html),
|
||||||
|
where the pipeline over-accepted a leaking patch and the audit overturned
|
||||||
Done:
|
it before it shipped.
|
||||||
- Workspace with the four trust-layer crates; core statistics
|
|
||||||
(interleaved scheduling, bootstrap ratio CIs, CI-lower-bound accept
|
|
||||||
rule), append-only ledger schema with mutation-refusing triggers,
|
|
||||||
corpus hash-pin verifier, equivalence-policy parser, ROI formulas —
|
|
||||||
all unit-tested.
|
|
||||||
- bench-runner CLI: whole-program `compare` and `aa` modes with env
|
|
||||||
fingerprinting.
|
|
||||||
- Playbook classes 1–7; agent tool contract and prompting spine.
|
|
||||||
|
|
||||||
Next (Phase 1 exit criteria):
|
|
||||||
- `perf stat` counters + max RSS + RAPL capture per run.
|
|
||||||
- Automated A/A calibration sessions and regression-injection self-test
|
|
||||||
recorded to `results/calibration/` (<5% FP, ≥95% detection).
|
|
||||||
- diff-test orchestration end-to-end on two real targets; harness IPC
|
|
||||||
for the agent tool surface.
|
|
||||||
|
|
||||||
## License
|
## License
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,20 @@
|
|||||||
|
[book]
|
||||||
|
title = "hotpath"
|
||||||
|
description = "A profile-guided optimization agent that only ships wins it can prove."
|
||||||
|
authors = ["hotpath"]
|
||||||
|
language = "en"
|
||||||
|
src = "src"
|
||||||
|
|
||||||
|
[output.html]
|
||||||
|
default-theme = "ayu"
|
||||||
|
preferred-dark-theme = "ayu"
|
||||||
|
git-repository-url = "https://github.com/bdbrown4/peltier.io"
|
||||||
|
edit-url-template = "https://github.com/bdbrown4/peltier.io/edit/main/docs/{path}"
|
||||||
|
additional-css = ["theme/custom.css"]
|
||||||
|
|
||||||
|
[output.html.fold]
|
||||||
|
enable = true
|
||||||
|
level = 1
|
||||||
|
|
||||||
|
[output.html.playground]
|
||||||
|
runnable = false
|
||||||
@@ -0,0 +1,27 @@
|
|||||||
|
# Summary
|
||||||
|
|
||||||
|
[hotpath](./introduction.md)
|
||||||
|
|
||||||
|
- [The thesis: trust is the product](./thesis.md)
|
||||||
|
- [Architecture](./architecture.md)
|
||||||
|
- [Measurement discipline](./measurement.md)
|
||||||
|
- [Equivalence gates](./equivalence.md)
|
||||||
|
- [The agent loop](./agent-loop.md)
|
||||||
|
- [Isolation & anti-reward-hacking](./isolation.md)
|
||||||
|
- [The playbook](./playbook.md)
|
||||||
|
- [ROI: from stopwatch to dollars](./roi.md)
|
||||||
|
|
||||||
|
# Case studies
|
||||||
|
|
||||||
|
- [Overview](./case-studies/overview.md)
|
||||||
|
- [tokei — compounding algorithmic wins (Rust)](./case-studies/tokei.md)
|
||||||
|
- [Cheap wins first: build flags & allocators](./case-studies/build-flags.md)
|
||||||
|
- [The caught false-accept (comrak)](./case-studies/comrak-false-accept.md)
|
||||||
|
- [cJSON — the cross-language proof (C)](./case-studies/cjson.md)
|
||||||
|
- [Services & scale: latency under load](./case-studies/service.md)
|
||||||
|
|
||||||
|
# Reference
|
||||||
|
|
||||||
|
- [Reproduce it yourself](./reproduce.md)
|
||||||
|
- [The trust boundary (infra)](./infra.md)
|
||||||
|
- [Roadmap & scope](./roadmap.md)
|
||||||
@@ -0,0 +1,52 @@
|
|||||||
|
# The agent loop
|
||||||
|
|
||||||
|
The proposer is a Claude Agent SDK loop (`agent/hotpath_agent/`) driving
|
||||||
|
the seven harness tools over a standalone MCP server. It has no built-in
|
||||||
|
file, shell, or web tools in its allowlist — only the harness door.
|
||||||
|
|
||||||
|
## One attempt, start to finish
|
||||||
|
|
||||||
|
1. **Read the ledger.** What classes and hypotheses were already tried on
|
||||||
|
this target? The ledger is the anti-double-attempt memory: a class may
|
||||||
|
be re-entered with a *materially new* hypothesis, never a duplicate.
|
||||||
|
2. **Read the profile.** callgrind cache-sim ranks hotspots with per-
|
||||||
|
function cache-miss and branch-mispredict columns, so the agent can tell
|
||||||
|
an allocation-churn hotspot from a branch-bound one and pick the right
|
||||||
|
playbook class.
|
||||||
|
3. **Pick the cheapest untried class** whose profile preconditions are met,
|
||||||
|
and read its playbook entry.
|
||||||
|
4. **Read the source** (windowed, at the pinned commit) and **propose a
|
||||||
|
patch** with a stated hypothesis.
|
||||||
|
5. **Run the verdict** and **poll** for the result.
|
||||||
|
6. **Report it verbatim.** A rejection with a clean ledger row is a
|
||||||
|
*successful* outcome. Do not iterate more than twice on a rejected
|
||||||
|
hypothesis without a new idea.
|
||||||
|
|
||||||
|
## What the loop proved about itself
|
||||||
|
|
||||||
|
Across an unattended grind of 20+ audited attempts, the loop demonstrated
|
||||||
|
the behavior the architecture is designed to produce:
|
||||||
|
|
||||||
|
- It chose the cheapest untried class each time, reading the ledger to
|
||||||
|
avoid re-grinding a dead end.
|
||||||
|
- Every patch went through `propose_patch` (path-allowlist + `git apply`)
|
||||||
|
and the full gate+bench pipeline. Malformed diffs were refused by the
|
||||||
|
harness, not the agent's goodwill.
|
||||||
|
- **It twice refused to fabricate a verdict it could not read back.** When
|
||||||
|
a multi-minute pipeline outran the MCP transport's timeout, the agent
|
||||||
|
checked the ledger, confirmed the row wasn't written, and reported the
|
||||||
|
verdict as *unreadable, not invented* — asking a human to read the
|
||||||
|
ledger rather than guessing a number. That refusal, under a real failure
|
||||||
|
mode, is the anti-reward-hacking design working.
|
||||||
|
|
||||||
|
## The honest-failure record
|
||||||
|
|
||||||
|
The ledger records **every** attempt, including the rejections, because a
|
||||||
|
rejection is training data and a demonstration of the bar. Of the audited
|
||||||
|
unattended attempts, most were honest rejections — real mechanisms that
|
||||||
|
simply didn't clear the 2% CI-lower-bound bar — a handful were verified
|
||||||
|
wins, and one was a `rejected-gate` where golden replay caught the agent's
|
||||||
|
patch producing wrong output that a green test suite would have shipped.
|
||||||
|
|
||||||
|
The load-bearing number is **zero shipped false accepts** across the whole
|
||||||
|
project.
|
||||||
@@ -0,0 +1,61 @@
|
|||||||
|
# Architecture
|
||||||
|
|
||||||
|
hotpath is split into a **trust layer** (Rust, the agent cannot write it)
|
||||||
|
and an **untrusted proposer** (the agent). The split is structural, not
|
||||||
|
advisory: the guarantees hold even if the proposer is adversarial.
|
||||||
|
|
||||||
|
```
|
||||||
|
crates/ Trust layer — agent has NO write access (SPEC §10)
|
||||||
|
bench-runner/ Interleaved A/B timing, bootstrap CIs, A/A calibration,
|
||||||
|
service-mode latency (coordinated-omission correct)
|
||||||
|
diff-test/ Equivalence gates, corpus hash-pinning, per-target spec
|
||||||
|
ledger/ Append-only SQLite attempt ledger (enforced by triggers)
|
||||||
|
report/ ROI: speedup CI → cores → dollars, methodology inline
|
||||||
|
verdict/ The pipeline in one command: gates → bench → ledger row
|
||||||
|
harnessd/ The one door the agent talks through (7 JSON ops)
|
||||||
|
agent/ Untrusted proposer (Claude Agent SDK, Python)
|
||||||
|
playbook/ Optimization classes 1–7, tried strictly cheapest-first
|
||||||
|
config/ accept.toml (thresholds), pricing.toml (ROI inputs)
|
||||||
|
targets/ Vendored OSS targets — the only agent-writable path
|
||||||
|
corpora/ Hash-pinned golden-replay inputs (read-only to agent)
|
||||||
|
results/ Calibration evidence, case studies, generated reports
|
||||||
|
```
|
||||||
|
|
||||||
|
## The seven-tool boundary
|
||||||
|
|
||||||
|
The agent never runs a shell, never writes a file, never touches the
|
||||||
|
ledger directly. It speaks to `harnessd` — the only door — through exactly
|
||||||
|
seven JSON operations:
|
||||||
|
|
||||||
|
| Tool | What it can do |
|
||||||
|
|---|---|
|
||||||
|
| `read_profile` | Read a target's ranked hotspots (read-only) |
|
||||||
|
| `read_ledger` | Read prior attempts — the anti-double-attempt memory |
|
||||||
|
| `read_playbook` | Read an optimization class's preconditions/procedure |
|
||||||
|
| `read_target_source` | Read a workspace file at the pinned commit (windowed) |
|
||||||
|
| `propose_patch` | Submit a unified diff — path-allowlisted, then `git apply` |
|
||||||
|
| `run_verdict` | Launch the gate+bench pipeline (detached) |
|
||||||
|
| `read_verdict` | Poll the append-only ledger for the result |
|
||||||
|
|
||||||
|
Note what is *absent*: no write outside `targets/<name>/workspace`, no
|
||||||
|
shell, no way to run the bench or write the ledger except through the
|
||||||
|
`verdict` binary, which only records a pass after the full gated pipeline.
|
||||||
|
|
||||||
|
## The data flow of one attempt
|
||||||
|
|
||||||
|
```
|
||||||
|
profile ─▶ agent hypothesizes ─▶ propose_patch ─▶ [path allowlist + git apply]
|
||||||
|
│
|
||||||
|
ledger row ◀─ verdict ◀─ bench (interleaved A/B) ◀─ gates (tests, golden,
|
||||||
|
(append-only) vs pristine-rebuilt fuzz, sanitizers)
|
||||||
|
baseline
|
||||||
|
```
|
||||||
|
|
||||||
|
The baseline is **rebuilt from a pristine checkout every session** — never
|
||||||
|
from the agent's workspace — so a patch cannot poison its own comparison.
|
||||||
|
The verdict is decided by the CI-lower-bound rule and written to an
|
||||||
|
append-only ledger the agent cannot mutate.
|
||||||
|
|
||||||
|
The next chapters walk each stage: [measurement](./measurement.md),
|
||||||
|
[equivalence gates](./equivalence.md), [the agent loop](./agent-loop.md),
|
||||||
|
and the [isolation](./isolation.md) that makes the boundary load-bearing.
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
{{#include ../../../results/phase3/02-build-flags-and-allocators.md}}
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
{{#include ../../../results/phase3/04-cjson-c-target.md}}
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
{{#include ../../../results/phase3/03-comrak-caught-false-accept.md}}
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
# Case studies
|
||||||
|
|
||||||
|
Five verified case studies, each reproducible from a pinned corpus through
|
||||||
|
the same gated pipeline, each number carrying its 95% CI and workload.
|
||||||
|
|
||||||
|
- [tokei — compounding algorithmic wins (Rust)](./tokei.md) — three class-5
|
||||||
|
wins on the same hot function, each unattended and audited.
|
||||||
|
- [Cheap wins first: build flags & allocators](./build-flags.md) — LTO
|
||||||
|
accepted (+10%), mimalloc held for a human ruling then accepted-scoped.
|
||||||
|
- [The caught false-accept (comrak)](./comrak-false-accept.md) — the
|
||||||
|
leaking teardown the bench loved, overturned by audit. The most important
|
||||||
|
study.
|
||||||
|
- [cJSON — the cross-language proof (C)](./cjson.md) — a sub-threshold win
|
||||||
|
rejected, then a byte-identical win accepted, on a C target.
|
||||||
|
- [Services & scale: latency under load](./service.md) — the cJSON win
|
||||||
|
measured as a latency delta under coordinated-omission-correct replay,
|
||||||
|
turned into a dollar figure mechanically.
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
{{#include ../../../results/phase4/README.md}}
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
{{#include ../../../results/phase3/01-tokei-algorithmic-rust.md}}
|
||||||
@@ -0,0 +1,52 @@
|
|||||||
|
# Equivalence gates
|
||||||
|
|
||||||
|
A faster binary that behaves differently is not an optimization — it is a
|
||||||
|
regression with good marketing. Before any speedup is even measured, the
|
||||||
|
candidate must prove it does the same thing the baseline does. These are
|
||||||
|
**hard gates**: a failure is `rejected-gate` and the bench never runs.
|
||||||
|
|
||||||
|
## The layers
|
||||||
|
|
||||||
|
1. **Upstream test suite.** The target's own tests, run against the patched
|
||||||
|
code. For cJSON that is the vendored Unity/CTest suite (19 tests:
|
||||||
|
`parse_number`, `parse_string`, `print_*`, `misc_tests`, …); for the Rust
|
||||||
|
targets, `cargo test`.
|
||||||
|
|
||||||
|
2. **Golden replay — byte-identical.** The candidate processes the
|
||||||
|
hash-pinned corpus and its output is hashed and compared to a pinned
|
||||||
|
`GOLDEN.sha256`. A code counter that miscounts one line, a JSON printer
|
||||||
|
that serializes one float differently — caught here. The corpus itself
|
||||||
|
is pinned via `MANIFEST.sha256` and the harness refuses to run on a
|
||||||
|
mismatch.
|
||||||
|
|
||||||
|
3. **Differential fuzzing.** Pristine vs candidate on thousands of mutated
|
||||||
|
inputs, comparing output. Every accepted win cleared **10,000 mutated
|
||||||
|
inputs with 0 divergences** — the tokei wins after canonicalizing benign
|
||||||
|
parallel-walker output ordering (a real finding: differential gates must
|
||||||
|
compare *semantics*, not bytes, where the target is legitimately
|
||||||
|
nondeterministic in presentation), the cJSON win on quote/number/escape
|
||||||
|
edge cases.
|
||||||
|
|
||||||
|
4. **Sanitizers.** ASan/LSan for Rust, ASan+UBSan for C. Run on **every
|
||||||
|
would-be accept** — this is machine-enforced, not manual (see below).
|
||||||
|
|
||||||
|
## The tier rule (SPEC §8)
|
||||||
|
|
||||||
|
Some changes can pass every gate and still be unsafe to auto-ship. A change
|
||||||
|
that touches floating-point ordering, concurrency primitives, `unsafe`, or
|
||||||
|
carries a sanitizer flag is **never** auto-accepted — its verdict is capped
|
||||||
|
at `needs-human-review`. The tier is decided by the *nature* of the change,
|
||||||
|
never by the size of the number. The mimalloc allocator swap
|
||||||
|
([case study](./case-studies/build-flags.md)) is the canonical example: a
|
||||||
|
real +4.6% win, correctly held for a human ruling because a global
|
||||||
|
allocator change's safety is not something one benchmark can clear.
|
||||||
|
|
||||||
|
## When a gate caught the pipeline itself
|
||||||
|
|
||||||
|
The sanitizer gate exists in its current, machine-enforced form because it
|
||||||
|
once *wasn't*. The pipeline auto-accepted a comrak patch that skipped AST
|
||||||
|
teardown for a +10.7% win — and the audit's manual LeakSanitizer run found
|
||||||
|
it leaks the arena at exit. It was overturned before shipping and the fix
|
||||||
|
made `verdict` build with ASan/UBSan and run the pinned workload on every
|
||||||
|
accept, capping any flagged win at `needs-human-review`. The full story is
|
||||||
|
[the caught false-accept](./case-studies/comrak-false-accept.md).
|
||||||
@@ -0,0 +1 @@
|
|||||||
|
{{#include ../../docs/infra.md}}
|
||||||
@@ -0,0 +1,46 @@
|
|||||||
|
# hotpath
|
||||||
|
|
||||||
|
> Spot cooling for hot paths. A profile-guided optimization agent that
|
||||||
|
> only ships wins it can prove.
|
||||||
|
|
||||||
|
hotpath profiles real binaries and services, isolates hot paths, proposes
|
||||||
|
optimizations, and accepts a change **only** when two independent bars are
|
||||||
|
cleared:
|
||||||
|
|
||||||
|
1. **Behavioral equivalence** — the change provably preserves behavior
|
||||||
|
(upstream tests, byte-identical golden replay, differential fuzzing,
|
||||||
|
sanitizers) or it is discarded.
|
||||||
|
2. **Statistical significance** — the speedup's bootstrap 95% CI lower
|
||||||
|
bound clears a threshold on trusted, calibrated hardware.
|
||||||
|
|
||||||
|
Everything else is commentary. A change without both is not a win; it is a
|
||||||
|
ledger row explaining why not.
|
||||||
|
|
||||||
|
## What it has actually done
|
||||||
|
|
||||||
|
Every number here carries its 95% confidence interval and its workload,
|
||||||
|
and every one survived the gates above. These are not projections.
|
||||||
|
|
||||||
|
| Result | Verified |
|
||||||
|
|---|---|
|
||||||
|
| tokei (Rust) — three compounding class-5 algorithmic wins | +10.7%, +3.7%, +9.9%, each CI-significant |
|
||||||
|
| tokei — link-time optimization (class 1) | +10.4%, CI [+8.5%, +12.0%] |
|
||||||
|
| cJSON (C) — number-handling rewrite (class 5) | +8.85%, CI [+7.5%, +10.0%] |
|
||||||
|
| cJSON HTTP service — p50 latency under replayed load | +6.2%, CI [+5.8%, +7.2%] |
|
||||||
|
| comrak (Rust) — mimalloc allocator swap (class 2) | +4.6%, CI [+3.3%, +5.8%], human-ruled |
|
||||||
|
|
||||||
|
Across **five phases and 33 ledger rows**, with **zero shipped false
|
||||||
|
accepts** — including two occasions where the pipeline itself over-accepted
|
||||||
|
and the audit caught it before anything shipped, each becoming a permanent
|
||||||
|
new gate.
|
||||||
|
|
||||||
|
## The differentiator
|
||||||
|
|
||||||
|
Consultants sell surveys; compilers sell flags. hotpath sells **verified
|
||||||
|
deltas with the methodology attached** — a stopwatch on pinned hardware,
|
||||||
|
a proof of equivalence, and an ROI figure whose confidence interval you
|
||||||
|
can quote to a hostile reviewer. That is the entire product, and this site
|
||||||
|
is how it works.
|
||||||
|
|
||||||
|
Start with [the thesis](./thesis.md), or jump to a
|
||||||
|
[case study](./case-studies/overview.md).
|
||||||
@@ -0,0 +1,53 @@
|
|||||||
|
# Isolation & anti-reward-hacking
|
||||||
|
|
||||||
|
The guarantees only matter if the proposer cannot cheat. hotpath's defense
|
||||||
|
is **structural**: the agent is denied the capabilities it would need to
|
||||||
|
manufacture a false win, at the OS level, not by prompt.
|
||||||
|
|
||||||
|
## What holds even if the proposer is adversarial
|
||||||
|
|
||||||
|
- **Writes go through one door.** `propose_patch` is the only harness-
|
||||||
|
mediated write path: a diff is path-allowlisted (relative paths only, no
|
||||||
|
`..`, no `.git`, nothing outside `targets/<name>/workspace`) and then
|
||||||
|
`git apply`-ed rooted in that workspace. A patch aimed at `config/` or
|
||||||
|
`crates/` is refused before any apply. Verified against a malicious
|
||||||
|
accept-threshold diff.
|
||||||
|
- **The ledger is append-only.** `UPDATE` and `DELETE` are refused by
|
||||||
|
SQLite triggers; `accepted` rows are written only by the `verdict`
|
||||||
|
binary after the gated pipeline. The agent cannot forge a passing
|
||||||
|
verdict through the harness.
|
||||||
|
- **The baseline is unbypassable.** It is rebuilt from a pristine checkout;
|
||||||
|
the agent's workspace never becomes the comparison.
|
||||||
|
|
||||||
|
## OS-level isolation (SPEC §10)
|
||||||
|
|
||||||
|
The SDK's tool allow/deny list is defense-in-depth, not the boundary —
|
||||||
|
`scripts/agent-isolated.sh` provides the real one, verified by
|
||||||
|
`just isolation-check` (19 checks, both modes):
|
||||||
|
|
||||||
|
- **`mountns` mode** — the whole agent process tree runs in a private mount
|
||||||
|
namespace with the repo bind-mounted **read-only** and `CAP_SYS_ADMIN`/
|
||||||
|
`CAP_SYS_PTRACE` dropped from the bounding set. Repo writes fail with
|
||||||
|
`EROFS`; a forged `INSERT`, a direct `verdict` invocation, a `mount -o
|
||||||
|
remount,rw`, and even userns re-entry to undo the mount are all refused.
|
||||||
|
- **`user` mode** — the agent runs as an unprivileged user whose only write
|
||||||
|
path into the repo is the harnessd Unix socket; the trust layer is owned
|
||||||
|
by another uid. This is the production shape (`docs/infra.md`).
|
||||||
|
|
||||||
|
## The two times the pipeline was wrong
|
||||||
|
|
||||||
|
Anti-reward-hacking is not "it never fails" — it is "when it fails, the
|
||||||
|
audit catches it and the system gets harder." That happened twice, and
|
||||||
|
both are documented, not hidden:
|
||||||
|
|
||||||
|
1. A leaking teardown patch the bench loved was **auto-accepted** and then
|
||||||
|
**overturned** by the 100% human audit's LeakSanitizer run. The fix made
|
||||||
|
the sanitizer gate machine-enforced on every accept.
|
||||||
|
2. During the [adversarial review](https://github.com/bdbrown4/peltier.io/blob/main/results/adversarial-review.md),
|
||||||
|
the ROI report generator was found to mint a clean dollar figure for that
|
||||||
|
same overturned row (it trusted the immutable ledger's historical
|
||||||
|
`accepted` verdict). The fix made the report treat an accepted-but-not-
|
||||||
|
sanitizer-clean row as **not shippable**.
|
||||||
|
|
||||||
|
Both defects were found by attacking the system's own output, fixed, and
|
||||||
|
turned into permanent gates. **Zero false accepts shipped.**
|
||||||
@@ -0,0 +1,64 @@
|
|||||||
|
# Measurement discipline
|
||||||
|
|
||||||
|
The product is a trustworthy number, so the measurement is where most of
|
||||||
|
the engineering went. The reference is Mytkowicz et al., *Producing Wrong
|
||||||
|
Data Without Doing Anything Obviously Wrong* — the default state of a
|
||||||
|
benchmark is "subtly lying."
|
||||||
|
|
||||||
|
## Interleaved A/B, never sequential
|
||||||
|
|
||||||
|
Baseline and candidate run **interleaved** (ABABAB…), never in sequential
|
||||||
|
blocks. Thermal drift, a background cron job, a noisy neighbor — anything
|
||||||
|
that changes over the run hits both sides equally instead of biasing one.
|
||||||
|
Warm-up runs are discarded.
|
||||||
|
|
||||||
|
## The baseline is always pristine
|
||||||
|
|
||||||
|
The baseline binary is **rebuilt from a clean checkout** each session, with
|
||||||
|
the candidate's patch stashed. A patch can never silently become part of
|
||||||
|
its own baseline (a real bug that bit an early smoke test, now closed by
|
||||||
|
`verdict --rebuild-baseline`).
|
||||||
|
|
||||||
|
## Bootstrap CIs, and the lower-bound rule
|
||||||
|
|
||||||
|
Each side yields ≥30 measured samples. The statistic is the **bootstrap
|
||||||
|
95% CI of the ratio-of-medians** (deterministic xorshift seed, so a result
|
||||||
|
reproduces). A change is accepted only if the **CI lower bound** clears the
|
||||||
|
threshold in `config/accept.toml` (default 2%) — not the median. The number
|
||||||
|
you commit to is the one that survives the interval.
|
||||||
|
|
||||||
|
## The hardware has to earn trust first
|
||||||
|
|
||||||
|
Before any measurement session, the harness runs two self-tests, recorded
|
||||||
|
in `results/calibration/`:
|
||||||
|
|
||||||
|
- **A/A**: run the *same* binary as both sides. It must produce a null
|
||||||
|
verdict. A "speedup" from identical binaries is a calibration failure.
|
||||||
|
- **Regression injection**: a synthetic 5% slowdown (a real busy-wait in
|
||||||
|
the timed window) must be detected ≥95% of the time.
|
||||||
|
|
||||||
|
CPU pinning (`taskset -c 2`) alone tightened the A/A confidence interval
|
||||||
|
~5× on the dev container. Every calibration anchor — local and on GitHub's
|
||||||
|
shared runners — passed both bars before its numbers were trusted.
|
||||||
|
|
||||||
|
## Service mode: coordinated omission
|
||||||
|
|
||||||
|
Phase 4 measures latency under load, where the classic trap is
|
||||||
|
**coordinated omission**: a closed-loop client that waits for a slow
|
||||||
|
response simply sends fewer requests during the stall, so the stall never
|
||||||
|
appears in the latency histogram. hotpath's load generator is **open-loop**
|
||||||
|
— requests are scheduled at a fixed rate and each latency is measured from
|
||||||
|
its *intended* send time. When the server falls behind, the requests queued
|
||||||
|
behind it each carry the full delay a real client would have seen. Nothing
|
||||||
|
is omitted.
|
||||||
|
|
||||||
|
The service measurement is calibrated the same way: 0/10 A/A false
|
||||||
|
positives, 10/10 injected-5%-latency-regression detections
|
||||||
|
(`results/calibration/cjson-service-aa.json`).
|
||||||
|
|
||||||
|
> **Percentiles are claimed only when the measurement can pin them down.**
|
||||||
|
> The cJSON service's p50 latency win was CI-tight and accepted (+6.2%,
|
||||||
|
> [+5.8%, +7.2%]); its p99 CI was [0.07, 4.97] — uselessly wide on a
|
||||||
|
> single-worker loopback tail — so it was **rejected**, even though the
|
||||||
|
> median pointed the right way. A number the measurement can't pin down
|
||||||
|
> does not ship.
|
||||||
@@ -0,0 +1,33 @@
|
|||||||
|
# The playbook
|
||||||
|
|
||||||
|
Optimizations are tried in a fixed order — **cheapest and safest first** —
|
||||||
|
because cost-per-attempt and risk both rise as you go down the list, and
|
||||||
|
the ledger stops you from re-grinding a class that already has a verdict.
|
||||||
|
|
||||||
|
| # | Class | Mechanism | Verified on |
|
||||||
|
|---|---|---|---|
|
||||||
|
| 1 | Build config | LTO, codegen-units, target-cpu | tokei ✅ +10.4% |
|
||||||
|
| 2 | Allocator | Global allocator swap (mimalloc) | comrak ✅ +4.6% (human-ruled) |
|
||||||
|
| 3 | Allocation churn | Remove per-item malloc/free | tokei ✅, cJSON ✅ (+ one overturned) |
|
||||||
|
| 4 | Data layout | Make the cache hierarchy fit the access pattern | comrak (attempted) |
|
||||||
|
| 5 | Algorithmic | Eliminate redundant work | tokei ×3 ✅, cJSON ✅ |
|
||||||
|
| 6 | SIMD | Autovectorization enablement | comrak, tokei (attempted) |
|
||||||
|
| 7 | Concurrency | Parallelism — always `needs-human-review` | — |
|
||||||
|
|
||||||
|
Six classes were exercised across the case studies, with verified wins on
|
||||||
|
classes 1, 2, 3, and 5 across **both a Rust and a C target**.
|
||||||
|
|
||||||
|
## Why the ordering is load-bearing
|
||||||
|
|
||||||
|
The cheapest class produced the project's single largest individual win:
|
||||||
|
LTO on tokei, **+10.4%**, a pure build-config change with no source edit.
|
||||||
|
The lesson the ledger enforces is the one every performance engineer learns
|
||||||
|
the hard way — *you do not hand-optimize a hot loop before you have turned
|
||||||
|
on the optimizer's own switches.* An agent that jumped straight to SIMD
|
||||||
|
would have missed a bigger, safer, free win sitting one config flag away.
|
||||||
|
|
||||||
|
Each class entry (`playbook/0N-*.md`) carries its **preconditions** (the
|
||||||
|
profile signature that justifies trying it), its **procedure**, its
|
||||||
|
**verification** requirements, and its **known failure modes**. The agent
|
||||||
|
reads the profile, matches the signature, and picks the cheapest class
|
||||||
|
whose preconditions are met and that isn't already spent on this hotspot.
|
||||||
@@ -0,0 +1,50 @@
|
|||||||
|
# Reproduce it yourself
|
||||||
|
|
||||||
|
Nothing here is a screenshot. Every number regenerates from a pinned corpus
|
||||||
|
through the same commands.
|
||||||
|
|
||||||
|
## Prerequisites
|
||||||
|
|
||||||
|
- Rust (stable) + `just`; `clang`/`cmake` for the C target; `valgrind` for
|
||||||
|
callgrind profiling. Nightly Rust for the sanitizer gate.
|
||||||
|
- Targets are vendored fresh (their `workspace/` is gitignored); each
|
||||||
|
`targets/<name>/target.toml` has the `[fetch]` recipe.
|
||||||
|
|
||||||
|
## The commands
|
||||||
|
|
||||||
|
```
|
||||||
|
just build / test / lint # trust-layer workspace
|
||||||
|
just aa [cmd] # A/A self-test — must be a null verdict
|
||||||
|
just calibrate <cmd> <out> # A/A false-positive + injection detection
|
||||||
|
just gates <target> # corpus pin + upstream tests + golden replay
|
||||||
|
just verdict <t> <bin> ... # gates + bench vs pristine baseline + ledger row
|
||||||
|
just report <run-id> # ROI report from a ledger row
|
||||||
|
```
|
||||||
|
|
||||||
|
Service mode (Phase 4):
|
||||||
|
|
||||||
|
```
|
||||||
|
just service <baseline-bin> <candidate-bin> <doc> # interleaved p50/p99 CIs
|
||||||
|
just service-calibrate <server> <doc> <out> # latency A/A + injection
|
||||||
|
just report <run-id> --service-json <json> # latency + ROI report
|
||||||
|
```
|
||||||
|
|
||||||
|
The agent loop, behind OS isolation:
|
||||||
|
|
||||||
|
```
|
||||||
|
just isolation-check # 19 boundary checks, both modes
|
||||||
|
scripts/agent-isolated.sh <target> <run-id> # one unattended attempt, confined
|
||||||
|
```
|
||||||
|
|
||||||
|
## Auditing a result
|
||||||
|
|
||||||
|
Every accepted win can be re-audited independently:
|
||||||
|
|
||||||
|
```
|
||||||
|
python3 scripts/audit-attempt.py <run-id> # mechanical re-check
|
||||||
|
python3 scripts/diff-fuzz-<target>.py <dir> 10000 <candidate-bin> # 10k fuzz
|
||||||
|
```
|
||||||
|
|
||||||
|
The ledger itself is the record: `results/ledger.sqlite`, append-only,
|
||||||
|
every attempt (win or loss) with its hypothesis, gates, bench evidence, and
|
||||||
|
verdict.
|
||||||
@@ -0,0 +1,42 @@
|
|||||||
|
# Roadmap & scope
|
||||||
|
|
||||||
|
## Done — the core loop, end to end
|
||||||
|
|
||||||
|
| Phase | Scope | Result |
|
||||||
|
|---|---|---|
|
||||||
|
| **0** | Manual dry run, two Rust targets | tokei +10.4% verified; methodology written |
|
||||||
|
| **1** | Trust layer (bench-runner, diff-test, ledger) | A/A <5% FP, injection ≥95%, gates on 2 targets; calibrated free on GitHub runners |
|
||||||
|
| **2** | Unattended agent loop | 20 audited attempts, 3 verified wins, **0 shipped false accepts**, OS-isolated |
|
||||||
|
| **3** | Playbook + proof, C/C++ | cross-language cJSON win (+8.85%), 4 case studies, 6 playbook classes |
|
||||||
|
| **4** | Services & scale | cJSON service +6.2% p50 under CO-correct replay, mechanical ROI report |
|
||||||
|
|
||||||
|
The `profile → hypothesize → patch → gated-verdict → ROI` loop runs
|
||||||
|
unattended, OS-isolated, across **Rust and C**, **batch and service**, from
|
||||||
|
a pinned corpus to a dollar figure — with zero shipped false accepts across
|
||||||
|
all phases, and the two pipeline over-accepts both caught by audit and
|
||||||
|
turned into permanent gates.
|
||||||
|
|
||||||
|
## Frontier — the research forks (SPEC §13)
|
||||||
|
|
||||||
|
Deliberately out of the core roadmap, these are where the largest gaps
|
||||||
|
live:
|
||||||
|
|
||||||
|
- **GPU kernel lane.** Triton/CUDA targets where 2–10× gaps are common;
|
||||||
|
correctness via reference-kernel differential testing; reward = measured
|
||||||
|
kernel time. The trust machinery (differential equivalence, interleaved
|
||||||
|
timing, append-only ledger) transfers directly; the profiler and bench
|
||||||
|
become kernel-time-aware, and it needs GPU hardware to run.
|
||||||
|
- **Learned optimization policies.** The ledger is a growing dataset of
|
||||||
|
`(profile signature, class, hypothesis) → verdict`. That is training data
|
||||||
|
for a policy that *ranks* which class to try first on a new hotspot,
|
||||||
|
turning the fixed cheapest-first ordering into a learned prior. Prior art:
|
||||||
|
MLGO, AlphaDev, Meta's LLM Compiler; ProGraML-style IR graphs as the
|
||||||
|
program representation.
|
||||||
|
|
||||||
|
## The real target
|
||||||
|
|
||||||
|
The end goal is not vendored OSS case studies — it is a **real production
|
||||||
|
system**. The same gates that matter for a JSON parser matter far more for
|
||||||
|
a live CMS: an equivalence failure there is a customer-facing bug, and the
|
||||||
|
no-network sandbox and append-only audit trail are exactly what make it
|
||||||
|
safe to point an optimization agent at code that a business depends on.
|
||||||
@@ -0,0 +1,39 @@
|
|||||||
|
# ROI: from stopwatch to dollars
|
||||||
|
|
||||||
|
The report is the pitch artifact, and it answers exactly one question:
|
||||||
|
*what did the stopwatch say, and what does that buy?* It is generated
|
||||||
|
**mechanically** — `just report <run-id>` reads a ledger row and the
|
||||||
|
calibrated bench and renders the numbers; nothing is hand-typed.
|
||||||
|
|
||||||
|
## The three conversions
|
||||||
|
|
||||||
|
- **Throughput → cores → dollars.** A speedup `s` on a CPU-bound fleet of
|
||||||
|
`N` cores returns `N × (1 − 1/s)` cores; at a `$/core-hour` rate that is
|
||||||
|
an annual saving. Both flow through the speedup's confidence interval, so
|
||||||
|
the report quotes a *range*.
|
||||||
|
- **Latency → percentile deltas → value.** Under replayed load, the p50/p99
|
||||||
|
latency speedups with their CIs; a p50 service-time reduction maps to
|
||||||
|
throughput capacity for a CPU-bound service.
|
||||||
|
- **Batch windows → wall-clock hours** returned to a schedule.
|
||||||
|
|
||||||
|
## The rule that makes it survive review
|
||||||
|
|
||||||
|
Every figure carries its CI and its workload, and the **methodology ships
|
||||||
|
inside the report** — the interleaved A/B, the pristine baseline, the
|
||||||
|
bootstrap CI, the gates, the calibration. And the number the report tells
|
||||||
|
you to quote is the **CI lower bound**, never the median:
|
||||||
|
|
||||||
|
> On a 500-core fleet at $0.04/core-hour, the cJSON service win (+6.2% p50
|
||||||
|
> latency) returns **27.5 cores / $9,621 per year at the CI lower bound**
|
||||||
|
> (median 29.4 / $10,290). The point estimate is not a promise.
|
||||||
|
|
||||||
|
## What the report refuses to do
|
||||||
|
|
||||||
|
It will not mint a clean dollar figure for a change that wasn't shipped. A
|
||||||
|
row whose verdict isn't `accepted`, **or** an `accepted` row that isn't
|
||||||
|
sanitizer-clean (a pre-gate accept the audit overturned), gets a bold **Not
|
||||||
|
shipped** banner before any number. That guard was itself added by the
|
||||||
|
[adversarial review](https://github.com/bdbrown4/peltier.io/blob/main/results/adversarial-review.md),
|
||||||
|
which caught the generator pitching the overturned comrak patch. The pitch
|
||||||
|
artifact is held to the same standard as everything else: it cannot claim
|
||||||
|
what wasn't proven.
|
||||||
@@ -0,0 +1,48 @@
|
|||||||
|
# The thesis: trust is the product
|
||||||
|
|
||||||
|
Performance work has a credibility problem. "We made it 15% faster" is
|
||||||
|
almost always some mix of a noisy benchmark, a workload that flatters the
|
||||||
|
change, and a behavioral difference nobody checked. The number is real to
|
||||||
|
the person who measured it and worthless to everyone downstream.
|
||||||
|
|
||||||
|
hotpath is built on the opposite premise: **a performance claim is a
|
||||||
|
liability until it is proven, and the proof ships with the number.**
|
||||||
|
|
||||||
|
## Three non-negotiables
|
||||||
|
|
||||||
|
1. **No unverified performance claims.** A change without passing
|
||||||
|
equivalence gates *and* a statistically significant benchmark delta
|
||||||
|
does not exist. It is discarded and logged.
|
||||||
|
2. **Measure before optimizing.** No patch is proposed until profile data
|
||||||
|
identifies the hot path. A single benchmark run is never trusted.
|
||||||
|
3. **Cheap wins first.** Build flags → LTO → PGO → allocator swap → *then*
|
||||||
|
code changes. The ledger prevents re-attempting a dead end.
|
||||||
|
|
||||||
|
## What "proven" means here
|
||||||
|
|
||||||
|
A win must clear **both** of two independent bars, because either one alone
|
||||||
|
is gameable:
|
||||||
|
|
||||||
|
- **Equivalence** stops "optimizations" that are actually behavior changes.
|
||||||
|
A JSON parser that's faster because it stopped validating numbers is not
|
||||||
|
faster; it's broken. Byte-identical golden replay and differential
|
||||||
|
fuzzing catch this.
|
||||||
|
- **Significance** stops noise from masquerading as a win. The bar is not
|
||||||
|
"the median improved" — it is "the bootstrap 95% CI *lower bound* clears
|
||||||
|
the threshold," measured by interleaving baseline and candidate on pinned
|
||||||
|
hardware that has passed its own A/A calibration.
|
||||||
|
|
||||||
|
A real +5.7% median that couldn't clear the significance bar (its CI lower
|
||||||
|
bound was +0.2%) was **rejected** — and that rejection is in the ledger, because
|
||||||
|
a rejection is a complete, valid outcome.
|
||||||
|
|
||||||
|
## Why an agent
|
||||||
|
|
||||||
|
The loop — profile, hypothesize, patch, gate, measure, record — is
|
||||||
|
mechanical and tedious, and the temptation to fudge a number is exactly
|
||||||
|
what a tired human succumbs to. An agent proposes; the *trust layer*
|
||||||
|
disposes. The agent cannot write the ledger, cannot move the baseline,
|
||||||
|
cannot touch the thresholds. It can only propose a patch and ask the
|
||||||
|
harness to judge it. The [architecture](./architecture.md) is designed so
|
||||||
|
that even a compromised or adversarial proposer cannot manufacture a false
|
||||||
|
win.
|
||||||
Vendored
+20
@@ -0,0 +1,20 @@
|
|||||||
|
/* hotpath docs — small refinements over the default mdBook theme. */
|
||||||
|
|
||||||
|
.content h1 {
|
||||||
|
border-bottom: 2px solid var(--links);
|
||||||
|
padding-bottom: 0.2em;
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Verified-number tables read as evidence: tighten and emphasize. */
|
||||||
|
.content table td:first-child {
|
||||||
|
font-weight: 500;
|
||||||
|
}
|
||||||
|
|
||||||
|
blockquote {
|
||||||
|
border-inline-start: 4px solid var(--links);
|
||||||
|
}
|
||||||
|
|
||||||
|
/* Inline verdict-ish code (accepted / rejected-bench / needs-human-review) */
|
||||||
|
.content code {
|
||||||
|
font-variant-ligatures: none;
|
||||||
|
}
|
||||||
Reference in New Issue
Block a user