mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
docs: full mdBook site + GitHub Pages deploy; rewrite README as project front page
- docs/: an mdBook documentation site — thesis, architecture, measurement
discipline, equivalence gates, agent loop, isolation/anti-reward-hacking,
playbook, ROI, reproduce, roadmap, and the five case studies (pulled from
their single source in results/ via {{#include}}). Builds clean (21
pages).
- .github/workflows/docs.yml: builds mdBook and deploys to GitHub Pages on
push to main (one-time setup: Settings → Pages → Source = GitHub Actions).
- README.md: rewritten as the project front page — the thesis, the verified
results table, layout, quick start, how it stays honest, links to the
book. Replaces the stale Phase-0-era text.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
This commit is contained in:
@@ -0,0 +1,53 @@
|
||||
name: docs
|
||||
|
||||
# Builds the mdBook site in docs/ and deploys it to GitHub Pages.
|
||||
# One-time setup: repo Settings → Pages → Source = "GitHub Actions".
|
||||
|
||||
on:
|
||||
push:
|
||||
branches: [main]
|
||||
paths:
|
||||
- "docs/**"
|
||||
- ".github/workflows/docs.yml"
|
||||
workflow_dispatch:
|
||||
|
||||
permissions:
|
||||
contents: read
|
||||
pages: write
|
||||
id-token: write
|
||||
|
||||
# Allow one concurrent deployment; don't cancel an in-progress prod deploy.
|
||||
concurrency:
|
||||
group: pages
|
||||
cancel-in-progress: false
|
||||
|
||||
jobs:
|
||||
build:
|
||||
runs-on: ubuntu-latest
|
||||
env:
|
||||
MDBOOK_VERSION: "0.4.40"
|
||||
steps:
|
||||
- uses: actions/checkout@v4
|
||||
- name: Install mdBook
|
||||
run: |
|
||||
mkdir -p "$HOME/.local/bin"
|
||||
url="https://github.com/rust-lang/mdBook/releases/download/v${MDBOOK_VERSION}/mdbook-v${MDBOOK_VERSION}-x86_64-unknown-linux-gnu.tar.gz"
|
||||
curl -sSL "$url" | tar -xz -C "$HOME/.local/bin"
|
||||
echo "$HOME/.local/bin" >> "$GITHUB_PATH"
|
||||
- name: Build
|
||||
run: mdbook build docs
|
||||
- name: Upload artifact
|
||||
uses: actions/upload-pages-artifact@v3
|
||||
with:
|
||||
path: docs/book
|
||||
|
||||
deploy:
|
||||
needs: build
|
||||
runs-on: ubuntu-latest
|
||||
environment:
|
||||
name: github-pages
|
||||
url: ${{ steps.deployment.outputs.page_url }}
|
||||
steps:
|
||||
- name: Deploy to GitHub Pages
|
||||
id: deployment
|
||||
uses: actions/deploy-pages@v4
|
||||
@@ -249,3 +249,4 @@ targets/cjson/cand-*/
|
||||
targets/cjson/asan*/
|
||||
targets/cjson/svc-baseline/
|
||||
targets/cjson/svc-candidate/
|
||||
docs/book/
|
||||
|
||||
@@ -1,72 +1,88 @@
|
||||
# peltier.io
|
||||
# hotpath
|
||||
|
||||
Spot cooling for hot paths. A profile-guided optimization agent that only ships wins it can prove.
|
||||
> Spot cooling for hot paths. A profile-guided optimization agent that
|
||||
> only ships wins it can prove.
|
||||
|
||||
hotpath finds the 10–90% left on the floor between "the compiler" and
|
||||
"hire a perf consultant," with two properties no consultant offers:
|
||||
**verified equivalence** (a change provably preserves behavior or is
|
||||
discarded) and **self-verifying ROI** (the value claim is a stopwatch on
|
||||
trusted infrastructure, not a survey).
|
||||
hotpath profiles real binaries and services, isolates hot paths, proposes
|
||||
optimizations, and accepts a change **only** when two independent bars are
|
||||
cleared: **behavioral equivalence** (upstream tests + byte-identical golden
|
||||
replay + differential fuzzing + sanitizers) and **statistical significance**
|
||||
(the speedup's bootstrap 95% CI lower bound clears a threshold on
|
||||
calibrated hardware). A change without both is discarded and logged.
|
||||
|
||||
Full design: **[SPEC.md](SPEC.md)**.
|
||||
Consultants sell surveys; compilers sell flags. hotpath sells **verified
|
||||
deltas with the methodology attached** — a stopwatch on pinned hardware, a
|
||||
proof of equivalence, and an ROI figure whose confidence interval survives
|
||||
hostile review.
|
||||
|
||||
📖 **Full documentation: the [hotpath book](https://bdbrown4.github.io/peltier.io/)**
|
||||
· Design spec: [SPEC.md](SPEC.md) · Charter: [CLAUDE.md](CLAUDE.md)
|
||||
|
||||
## Verified results
|
||||
|
||||
Every number carries its 95% CI and its workload, and every one survived the
|
||||
gates above. Across **five phases and 33 ledger rows, zero shipped false
|
||||
accepts** — including two pipeline over-accepts the audit caught before
|
||||
anything shipped, each becoming a permanent new gate.
|
||||
|
||||
| Target | Win | Verified |
|
||||
|---|---|---|
|
||||
| tokei (Rust) — LTO (class 1) | +10.4% | CI [+8.5%, +12.0%] |
|
||||
| tokei — three class-5 algorithmic wins | +10.7% / +3.7% / +9.9% | each CI-significant |
|
||||
| cJSON (C) — number-handling rewrite (class 5) | +8.85% | CI [+7.5%, +10.0%] |
|
||||
| cJSON HTTP service — p50 latency under replay | +6.2% | CI [+5.8%, +7.2%] |
|
||||
| comrak (Rust) — mimalloc (class 2) | +4.6% | CI [+3.3%, +5.8%], [human-ruled](results/rulings/phase0-comrak-002.md) |
|
||||
|
||||
## Layout
|
||||
|
||||
```
|
||||
crates/ Trust layer (agent has NO write access — SPEC §10)
|
||||
bench-runner/ Interleaved A/B timing, bootstrap CIs, A/A calibration
|
||||
diff-test/ Equivalence gates, corpus hash-pinning, per-target policy
|
||||
ledger/ Append-only SQLite attempt ledger (enforced by triggers)
|
||||
report/ ROI math: speedup CI → cores → dollars, caveats included
|
||||
crates/ Trust layer — agent has NO write access (SPEC §10)
|
||||
bench-runner/ Interleaved A/B, bootstrap CIs, A/A calibration, service mode
|
||||
diff-test/ Equivalence gates, corpus hash-pinning, per-target spec
|
||||
ledger/ Append-only SQLite attempt ledger (mutation-refusing triggers)
|
||||
report/ ROI: speedup CI → cores → dollars, methodology inline
|
||||
verdict/ The pipeline in one command: gates → bench → ledger row
|
||||
harnessd/ The one door the agent talks through (7 JSON ops)
|
||||
agent/ Untrusted proposer (Claude Agent SDK, Python)
|
||||
playbook/ Optimization classes 1–7, tried strictly cheapest-first
|
||||
config/ accept.toml (thresholds), pricing.toml (ROI inputs)
|
||||
targets/ Per-target checkouts — the only agent-writable path
|
||||
targets/ Vendored OSS targets — the only agent-writable path
|
||||
corpora/ Hash-pinned golden-replay inputs (read-only to agent)
|
||||
results/ Calibration evidence, per-engagement outputs
|
||||
results/ Calibration evidence, case studies, generated ROI reports
|
||||
docs/ The mdBook site (deployed to GitHub Pages)
|
||||
```
|
||||
|
||||
## Quick start
|
||||
|
||||
```sh
|
||||
cargo test --workspace # trust-layer unit tests
|
||||
just aa # A/A self-test: must yield a null verdict
|
||||
just compare "cmd-a" "cmd-b" # interleaved A/B with bootstrap CI
|
||||
just build / test / lint # trust-layer workspace
|
||||
just aa # A/A self-test — must yield a null verdict
|
||||
just gates <target> # corpus pin + upstream tests + golden replay
|
||||
just verdict <t> <bin> ... # gates + bench vs pristine baseline + ledger row
|
||||
just report <run-id> # ROI report from a ledger row
|
||||
just isolation-check # 19 OS-boundary checks (both modes)
|
||||
```
|
||||
|
||||
## Status
|
||||
Full command reference and reproduction steps:
|
||||
[Reproduce it yourself](https://bdbrown4.github.io/peltier.io/reproduce.html).
|
||||
|
||||
**Phase 0 complete** (see SPEC §5). Five attempts in the ledger across
|
||||
two pinned targets (comrak 0.53.0, tokei 14.0.0), A/A-calibrated,
|
||||
every number with CI and workload:
|
||||
## How it stays honest
|
||||
|
||||
- **tokei +10.4% median [95% CI +8.5%, +12.0%]** — read-path buffer
|
||||
zeroing eliminated + fat LTO, byte-identical, 4,332-iteration
|
||||
diff-fuzz clean, sanitizers clean, auto-accepted
|
||||
(`results/phase0/case-study-tokei.md`). Meets the Phase 0 exit bar.
|
||||
- comrak +4.6% [+3.3%, +5.8%] — mimalloc swap, `needs-human-review`
|
||||
on an LSan reachability flag (`results/phase0/case-study-comrak.md`).
|
||||
- Two null/sub-threshold results correctly rejected and logged.
|
||||
- The agent speaks to the trust layer through **seven read-only-plus-one-
|
||||
write JSON operations**; it has no shell, cannot write outside a target
|
||||
workspace, and cannot touch the ledger or thresholds.
|
||||
- The **baseline is rebuilt from a pristine checkout** every session — a
|
||||
patch never becomes its own comparison.
|
||||
- The **ledger is append-only** (SQLite triggers refuse UPDATE/DELETE);
|
||||
accepted rows are written only by the `verdict` binary after the gates.
|
||||
- The agent process runs under **OS-level isolation** (read-only mount
|
||||
namespace or an unprivileged uid) — the boundary is filesystem
|
||||
permissions, not a prompt.
|
||||
|
||||
Next: Phase 1 trust-layer hardening (containers, perf, automated
|
||||
calibration, gate orchestration, agent IPC).
|
||||
|
||||
Done:
|
||||
- Workspace with the four trust-layer crates; core statistics
|
||||
(interleaved scheduling, bootstrap ratio CIs, CI-lower-bound accept
|
||||
rule), append-only ledger schema with mutation-refusing triggers,
|
||||
corpus hash-pin verifier, equivalence-policy parser, ROI formulas —
|
||||
all unit-tested.
|
||||
- bench-runner CLI: whole-program `compare` and `aa` modes with env
|
||||
fingerprinting.
|
||||
- Playbook classes 1–7; agent tool contract and prompting spine.
|
||||
|
||||
Next (Phase 1 exit criteria):
|
||||
- `perf stat` counters + max RSS + RAPL capture per run.
|
||||
- Automated A/A calibration sessions and regression-injection self-test
|
||||
recorded to `results/calibration/` (<5% FP, ≥95% detection).
|
||||
- diff-test orchestration end-to-end on two real targets; harness IPC
|
||||
for the agent tool surface.
|
||||
See [the case studies](https://bdbrown4.github.io/peltier.io/case-studies/overview.html)
|
||||
— especially [the caught false-accept](https://bdbrown4.github.io/peltier.io/case-studies/comrak-false-accept.html),
|
||||
where the pipeline over-accepted a leaking patch and the audit overturned
|
||||
it before it shipped.
|
||||
|
||||
## License
|
||||
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
[book]
|
||||
title = "hotpath"
|
||||
description = "A profile-guided optimization agent that only ships wins it can prove."
|
||||
authors = ["hotpath"]
|
||||
language = "en"
|
||||
src = "src"
|
||||
|
||||
[output.html]
|
||||
default-theme = "ayu"
|
||||
preferred-dark-theme = "ayu"
|
||||
git-repository-url = "https://github.com/bdbrown4/peltier.io"
|
||||
edit-url-template = "https://github.com/bdbrown4/peltier.io/edit/main/docs/{path}"
|
||||
additional-css = ["theme/custom.css"]
|
||||
|
||||
[output.html.fold]
|
||||
enable = true
|
||||
level = 1
|
||||
|
||||
[output.html.playground]
|
||||
runnable = false
|
||||
@@ -0,0 +1,27 @@
|
||||
# Summary
|
||||
|
||||
[hotpath](./introduction.md)
|
||||
|
||||
- [The thesis: trust is the product](./thesis.md)
|
||||
- [Architecture](./architecture.md)
|
||||
- [Measurement discipline](./measurement.md)
|
||||
- [Equivalence gates](./equivalence.md)
|
||||
- [The agent loop](./agent-loop.md)
|
||||
- [Isolation & anti-reward-hacking](./isolation.md)
|
||||
- [The playbook](./playbook.md)
|
||||
- [ROI: from stopwatch to dollars](./roi.md)
|
||||
|
||||
# Case studies
|
||||
|
||||
- [Overview](./case-studies/overview.md)
|
||||
- [tokei — compounding algorithmic wins (Rust)](./case-studies/tokei.md)
|
||||
- [Cheap wins first: build flags & allocators](./case-studies/build-flags.md)
|
||||
- [The caught false-accept (comrak)](./case-studies/comrak-false-accept.md)
|
||||
- [cJSON — the cross-language proof (C)](./case-studies/cjson.md)
|
||||
- [Services & scale: latency under load](./case-studies/service.md)
|
||||
|
||||
# Reference
|
||||
|
||||
- [Reproduce it yourself](./reproduce.md)
|
||||
- [The trust boundary (infra)](./infra.md)
|
||||
- [Roadmap & scope](./roadmap.md)
|
||||
@@ -0,0 +1,52 @@
|
||||
# The agent loop
|
||||
|
||||
The proposer is a Claude Agent SDK loop (`agent/hotpath_agent/`) driving
|
||||
the seven harness tools over a standalone MCP server. It has no built-in
|
||||
file, shell, or web tools in its allowlist — only the harness door.
|
||||
|
||||
## One attempt, start to finish
|
||||
|
||||
1. **Read the ledger.** What classes and hypotheses were already tried on
|
||||
this target? The ledger is the anti-double-attempt memory: a class may
|
||||
be re-entered with a *materially new* hypothesis, never a duplicate.
|
||||
2. **Read the profile.** callgrind cache-sim ranks hotspots with per-
|
||||
function cache-miss and branch-mispredict columns, so the agent can tell
|
||||
an allocation-churn hotspot from a branch-bound one and pick the right
|
||||
playbook class.
|
||||
3. **Pick the cheapest untried class** whose profile preconditions are met,
|
||||
and read its playbook entry.
|
||||
4. **Read the source** (windowed, at the pinned commit) and **propose a
|
||||
patch** with a stated hypothesis.
|
||||
5. **Run the verdict** and **poll** for the result.
|
||||
6. **Report it verbatim.** A rejection with a clean ledger row is a
|
||||
*successful* outcome. Do not iterate more than twice on a rejected
|
||||
hypothesis without a new idea.
|
||||
|
||||
## What the loop proved about itself
|
||||
|
||||
Across an unattended grind of 20+ audited attempts, the loop demonstrated
|
||||
the behavior the architecture is designed to produce:
|
||||
|
||||
- It chose the cheapest untried class each time, reading the ledger to
|
||||
avoid re-grinding a dead end.
|
||||
- Every patch went through `propose_patch` (path-allowlist + `git apply`)
|
||||
and the full gate+bench pipeline. Malformed diffs were refused by the
|
||||
harness, not the agent's goodwill.
|
||||
- **It twice refused to fabricate a verdict it could not read back.** When
|
||||
a multi-minute pipeline outran the MCP transport's timeout, the agent
|
||||
checked the ledger, confirmed the row wasn't written, and reported the
|
||||
verdict as *unreadable, not invented* — asking a human to read the
|
||||
ledger rather than guessing a number. That refusal, under a real failure
|
||||
mode, is the anti-reward-hacking design working.
|
||||
|
||||
## The honest-failure record
|
||||
|
||||
The ledger records **every** attempt, including the rejections, because a
|
||||
rejection is training data and a demonstration of the bar. Of the audited
|
||||
unattended attempts, most were honest rejections — real mechanisms that
|
||||
simply didn't clear the 2% CI-lower-bound bar — a handful were verified
|
||||
wins, and one was a `rejected-gate` where golden replay caught the agent's
|
||||
patch producing wrong output that a green test suite would have shipped.
|
||||
|
||||
The load-bearing number is **zero shipped false accepts** across the whole
|
||||
project.
|
||||
@@ -0,0 +1,61 @@
|
||||
# Architecture
|
||||
|
||||
hotpath is split into a **trust layer** (Rust, the agent cannot write it)
|
||||
and an **untrusted proposer** (the agent). The split is structural, not
|
||||
advisory: the guarantees hold even if the proposer is adversarial.
|
||||
|
||||
```
|
||||
crates/ Trust layer — agent has NO write access (SPEC §10)
|
||||
bench-runner/ Interleaved A/B timing, bootstrap CIs, A/A calibration,
|
||||
service-mode latency (coordinated-omission correct)
|
||||
diff-test/ Equivalence gates, corpus hash-pinning, per-target spec
|
||||
ledger/ Append-only SQLite attempt ledger (enforced by triggers)
|
||||
report/ ROI: speedup CI → cores → dollars, methodology inline
|
||||
verdict/ The pipeline in one command: gates → bench → ledger row
|
||||
harnessd/ The one door the agent talks through (7 JSON ops)
|
||||
agent/ Untrusted proposer (Claude Agent SDK, Python)
|
||||
playbook/ Optimization classes 1–7, tried strictly cheapest-first
|
||||
config/ accept.toml (thresholds), pricing.toml (ROI inputs)
|
||||
targets/ Vendored OSS targets — the only agent-writable path
|
||||
corpora/ Hash-pinned golden-replay inputs (read-only to agent)
|
||||
results/ Calibration evidence, case studies, generated reports
|
||||
```
|
||||
|
||||
## The seven-tool boundary
|
||||
|
||||
The agent never runs a shell, never writes a file, never touches the
|
||||
ledger directly. It speaks to `harnessd` — the only door — through exactly
|
||||
seven JSON operations:
|
||||
|
||||
| Tool | What it can do |
|
||||
|---|---|
|
||||
| `read_profile` | Read a target's ranked hotspots (read-only) |
|
||||
| `read_ledger` | Read prior attempts — the anti-double-attempt memory |
|
||||
| `read_playbook` | Read an optimization class's preconditions/procedure |
|
||||
| `read_target_source` | Read a workspace file at the pinned commit (windowed) |
|
||||
| `propose_patch` | Submit a unified diff — path-allowlisted, then `git apply` |
|
||||
| `run_verdict` | Launch the gate+bench pipeline (detached) |
|
||||
| `read_verdict` | Poll the append-only ledger for the result |
|
||||
|
||||
Note what is *absent*: no write outside `targets/<name>/workspace`, no
|
||||
shell, no way to run the bench or write the ledger except through the
|
||||
`verdict` binary, which only records a pass after the full gated pipeline.
|
||||
|
||||
## The data flow of one attempt
|
||||
|
||||
```
|
||||
profile ─▶ agent hypothesizes ─▶ propose_patch ─▶ [path allowlist + git apply]
|
||||
│
|
||||
ledger row ◀─ verdict ◀─ bench (interleaved A/B) ◀─ gates (tests, golden,
|
||||
(append-only) vs pristine-rebuilt fuzz, sanitizers)
|
||||
baseline
|
||||
```
|
||||
|
||||
The baseline is **rebuilt from a pristine checkout every session** — never
|
||||
from the agent's workspace — so a patch cannot poison its own comparison.
|
||||
The verdict is decided by the CI-lower-bound rule and written to an
|
||||
append-only ledger the agent cannot mutate.
|
||||
|
||||
The next chapters walk each stage: [measurement](./measurement.md),
|
||||
[equivalence gates](./equivalence.md), [the agent loop](./agent-loop.md),
|
||||
and the [isolation](./isolation.md) that makes the boundary load-bearing.
|
||||
@@ -0,0 +1 @@
|
||||
{{#include ../../../results/phase3/02-build-flags-and-allocators.md}}
|
||||
@@ -0,0 +1 @@
|
||||
{{#include ../../../results/phase3/04-cjson-c-target.md}}
|
||||
@@ -0,0 +1 @@
|
||||
{{#include ../../../results/phase3/03-comrak-caught-false-accept.md}}
|
||||
@@ -0,0 +1,17 @@
|
||||
# Case studies
|
||||
|
||||
Five verified case studies, each reproducible from a pinned corpus through
|
||||
the same gated pipeline, each number carrying its 95% CI and workload.
|
||||
|
||||
- [tokei — compounding algorithmic wins (Rust)](./tokei.md) — three class-5
|
||||
wins on the same hot function, each unattended and audited.
|
||||
- [Cheap wins first: build flags & allocators](./build-flags.md) — LTO
|
||||
accepted (+10%), mimalloc held for a human ruling then accepted-scoped.
|
||||
- [The caught false-accept (comrak)](./comrak-false-accept.md) — the
|
||||
leaking teardown the bench loved, overturned by audit. The most important
|
||||
study.
|
||||
- [cJSON — the cross-language proof (C)](./cjson.md) — a sub-threshold win
|
||||
rejected, then a byte-identical win accepted, on a C target.
|
||||
- [Services & scale: latency under load](./service.md) — the cJSON win
|
||||
measured as a latency delta under coordinated-omission-correct replay,
|
||||
turned into a dollar figure mechanically.
|
||||
@@ -0,0 +1 @@
|
||||
{{#include ../../../results/phase4/README.md}}
|
||||
@@ -0,0 +1 @@
|
||||
{{#include ../../../results/phase3/01-tokei-algorithmic-rust.md}}
|
||||
@@ -0,0 +1,52 @@
|
||||
# Equivalence gates
|
||||
|
||||
A faster binary that behaves differently is not an optimization — it is a
|
||||
regression with good marketing. Before any speedup is even measured, the
|
||||
candidate must prove it does the same thing the baseline does. These are
|
||||
**hard gates**: a failure is `rejected-gate` and the bench never runs.
|
||||
|
||||
## The layers
|
||||
|
||||
1. **Upstream test suite.** The target's own tests, run against the patched
|
||||
code. For cJSON that is the vendored Unity/CTest suite (19 tests:
|
||||
`parse_number`, `parse_string`, `print_*`, `misc_tests`, …); for the Rust
|
||||
targets, `cargo test`.
|
||||
|
||||
2. **Golden replay — byte-identical.** The candidate processes the
|
||||
hash-pinned corpus and its output is hashed and compared to a pinned
|
||||
`GOLDEN.sha256`. A code counter that miscounts one line, a JSON printer
|
||||
that serializes one float differently — caught here. The corpus itself
|
||||
is pinned via `MANIFEST.sha256` and the harness refuses to run on a
|
||||
mismatch.
|
||||
|
||||
3. **Differential fuzzing.** Pristine vs candidate on thousands of mutated
|
||||
inputs, comparing output. Every accepted win cleared **10,000 mutated
|
||||
inputs with 0 divergences** — the tokei wins after canonicalizing benign
|
||||
parallel-walker output ordering (a real finding: differential gates must
|
||||
compare *semantics*, not bytes, where the target is legitimately
|
||||
nondeterministic in presentation), the cJSON win on quote/number/escape
|
||||
edge cases.
|
||||
|
||||
4. **Sanitizers.** ASan/LSan for Rust, ASan+UBSan for C. Run on **every
|
||||
would-be accept** — this is machine-enforced, not manual (see below).
|
||||
|
||||
## The tier rule (SPEC §8)
|
||||
|
||||
Some changes can pass every gate and still be unsafe to auto-ship. A change
|
||||
that touches floating-point ordering, concurrency primitives, `unsafe`, or
|
||||
carries a sanitizer flag is **never** auto-accepted — its verdict is capped
|
||||
at `needs-human-review`. The tier is decided by the *nature* of the change,
|
||||
never by the size of the number. The mimalloc allocator swap
|
||||
([case study](./case-studies/build-flags.md)) is the canonical example: a
|
||||
real +4.6% win, correctly held for a human ruling because a global
|
||||
allocator change's safety is not something one benchmark can clear.
|
||||
|
||||
## When a gate caught the pipeline itself
|
||||
|
||||
The sanitizer gate exists in its current, machine-enforced form because it
|
||||
once *wasn't*. The pipeline auto-accepted a comrak patch that skipped AST
|
||||
teardown for a +10.7% win — and the audit's manual LeakSanitizer run found
|
||||
it leaks the arena at exit. It was overturned before shipping and the fix
|
||||
made `verdict` build with ASan/UBSan and run the pinned workload on every
|
||||
accept, capping any flagged win at `needs-human-review`. The full story is
|
||||
[the caught false-accept](./case-studies/comrak-false-accept.md).
|
||||
@@ -0,0 +1 @@
|
||||
{{#include ../../docs/infra.md}}
|
||||
@@ -0,0 +1,46 @@
|
||||
# hotpath
|
||||
|
||||
> Spot cooling for hot paths. A profile-guided optimization agent that
|
||||
> only ships wins it can prove.
|
||||
|
||||
hotpath profiles real binaries and services, isolates hot paths, proposes
|
||||
optimizations, and accepts a change **only** when two independent bars are
|
||||
cleared:
|
||||
|
||||
1. **Behavioral equivalence** — the change provably preserves behavior
|
||||
(upstream tests, byte-identical golden replay, differential fuzzing,
|
||||
sanitizers) or it is discarded.
|
||||
2. **Statistical significance** — the speedup's bootstrap 95% CI lower
|
||||
bound clears a threshold on trusted, calibrated hardware.
|
||||
|
||||
Everything else is commentary. A change without both is not a win; it is a
|
||||
ledger row explaining why not.
|
||||
|
||||
## What it has actually done
|
||||
|
||||
Every number here carries its 95% confidence interval and its workload,
|
||||
and every one survived the gates above. These are not projections.
|
||||
|
||||
| Result | Verified |
|
||||
|---|---|
|
||||
| tokei (Rust) — three compounding class-5 algorithmic wins | +10.7%, +3.7%, +9.9%, each CI-significant |
|
||||
| tokei — link-time optimization (class 1) | +10.4%, CI [+8.5%, +12.0%] |
|
||||
| cJSON (C) — number-handling rewrite (class 5) | +8.85%, CI [+7.5%, +10.0%] |
|
||||
| cJSON HTTP service — p50 latency under replayed load | +6.2%, CI [+5.8%, +7.2%] |
|
||||
| comrak (Rust) — mimalloc allocator swap (class 2) | +4.6%, CI [+3.3%, +5.8%], human-ruled |
|
||||
|
||||
Across **five phases and 33 ledger rows**, with **zero shipped false
|
||||
accepts** — including two occasions where the pipeline itself over-accepted
|
||||
and the audit caught it before anything shipped, each becoming a permanent
|
||||
new gate.
|
||||
|
||||
## The differentiator
|
||||
|
||||
Consultants sell surveys; compilers sell flags. hotpath sells **verified
|
||||
deltas with the methodology attached** — a stopwatch on pinned hardware,
|
||||
a proof of equivalence, and an ROI figure whose confidence interval you
|
||||
can quote to a hostile reviewer. That is the entire product, and this site
|
||||
is how it works.
|
||||
|
||||
Start with [the thesis](./thesis.md), or jump to a
|
||||
[case study](./case-studies/overview.md).
|
||||
@@ -0,0 +1,53 @@
|
||||
# Isolation & anti-reward-hacking
|
||||
|
||||
The guarantees only matter if the proposer cannot cheat. hotpath's defense
|
||||
is **structural**: the agent is denied the capabilities it would need to
|
||||
manufacture a false win, at the OS level, not by prompt.
|
||||
|
||||
## What holds even if the proposer is adversarial
|
||||
|
||||
- **Writes go through one door.** `propose_patch` is the only harness-
|
||||
mediated write path: a diff is path-allowlisted (relative paths only, no
|
||||
`..`, no `.git`, nothing outside `targets/<name>/workspace`) and then
|
||||
`git apply`-ed rooted in that workspace. A patch aimed at `config/` or
|
||||
`crates/` is refused before any apply. Verified against a malicious
|
||||
accept-threshold diff.
|
||||
- **The ledger is append-only.** `UPDATE` and `DELETE` are refused by
|
||||
SQLite triggers; `accepted` rows are written only by the `verdict`
|
||||
binary after the gated pipeline. The agent cannot forge a passing
|
||||
verdict through the harness.
|
||||
- **The baseline is unbypassable.** It is rebuilt from a pristine checkout;
|
||||
the agent's workspace never becomes the comparison.
|
||||
|
||||
## OS-level isolation (SPEC §10)
|
||||
|
||||
The SDK's tool allow/deny list is defense-in-depth, not the boundary —
|
||||
`scripts/agent-isolated.sh` provides the real one, verified by
|
||||
`just isolation-check` (19 checks, both modes):
|
||||
|
||||
- **`mountns` mode** — the whole agent process tree runs in a private mount
|
||||
namespace with the repo bind-mounted **read-only** and `CAP_SYS_ADMIN`/
|
||||
`CAP_SYS_PTRACE` dropped from the bounding set. Repo writes fail with
|
||||
`EROFS`; a forged `INSERT`, a direct `verdict` invocation, a `mount -o
|
||||
remount,rw`, and even userns re-entry to undo the mount are all refused.
|
||||
- **`user` mode** — the agent runs as an unprivileged user whose only write
|
||||
path into the repo is the harnessd Unix socket; the trust layer is owned
|
||||
by another uid. This is the production shape (`docs/infra.md`).
|
||||
|
||||
## The two times the pipeline was wrong
|
||||
|
||||
Anti-reward-hacking is not "it never fails" — it is "when it fails, the
|
||||
audit catches it and the system gets harder." That happened twice, and
|
||||
both are documented, not hidden:
|
||||
|
||||
1. A leaking teardown patch the bench loved was **auto-accepted** and then
|
||||
**overturned** by the 100% human audit's LeakSanitizer run. The fix made
|
||||
the sanitizer gate machine-enforced on every accept.
|
||||
2. During the [adversarial review](https://github.com/bdbrown4/peltier.io/blob/main/results/adversarial-review.md),
|
||||
the ROI report generator was found to mint a clean dollar figure for that
|
||||
same overturned row (it trusted the immutable ledger's historical
|
||||
`accepted` verdict). The fix made the report treat an accepted-but-not-
|
||||
sanitizer-clean row as **not shippable**.
|
||||
|
||||
Both defects were found by attacking the system's own output, fixed, and
|
||||
turned into permanent gates. **Zero false accepts shipped.**
|
||||
@@ -0,0 +1,64 @@
|
||||
# Measurement discipline
|
||||
|
||||
The product is a trustworthy number, so the measurement is where most of
|
||||
the engineering went. The reference is Mytkowicz et al., *Producing Wrong
|
||||
Data Without Doing Anything Obviously Wrong* — the default state of a
|
||||
benchmark is "subtly lying."
|
||||
|
||||
## Interleaved A/B, never sequential
|
||||
|
||||
Baseline and candidate run **interleaved** (ABABAB…), never in sequential
|
||||
blocks. Thermal drift, a background cron job, a noisy neighbor — anything
|
||||
that changes over the run hits both sides equally instead of biasing one.
|
||||
Warm-up runs are discarded.
|
||||
|
||||
## The baseline is always pristine
|
||||
|
||||
The baseline binary is **rebuilt from a clean checkout** each session, with
|
||||
the candidate's patch stashed. A patch can never silently become part of
|
||||
its own baseline (a real bug that bit an early smoke test, now closed by
|
||||
`verdict --rebuild-baseline`).
|
||||
|
||||
## Bootstrap CIs, and the lower-bound rule
|
||||
|
||||
Each side yields ≥30 measured samples. The statistic is the **bootstrap
|
||||
95% CI of the ratio-of-medians** (deterministic xorshift seed, so a result
|
||||
reproduces). A change is accepted only if the **CI lower bound** clears the
|
||||
threshold in `config/accept.toml` (default 2%) — not the median. The number
|
||||
you commit to is the one that survives the interval.
|
||||
|
||||
## The hardware has to earn trust first
|
||||
|
||||
Before any measurement session, the harness runs two self-tests, recorded
|
||||
in `results/calibration/`:
|
||||
|
||||
- **A/A**: run the *same* binary as both sides. It must produce a null
|
||||
verdict. A "speedup" from identical binaries is a calibration failure.
|
||||
- **Regression injection**: a synthetic 5% slowdown (a real busy-wait in
|
||||
the timed window) must be detected ≥95% of the time.
|
||||
|
||||
CPU pinning (`taskset -c 2`) alone tightened the A/A confidence interval
|
||||
~5× on the dev container. Every calibration anchor — local and on GitHub's
|
||||
shared runners — passed both bars before its numbers were trusted.
|
||||
|
||||
## Service mode: coordinated omission
|
||||
|
||||
Phase 4 measures latency under load, where the classic trap is
|
||||
**coordinated omission**: a closed-loop client that waits for a slow
|
||||
response simply sends fewer requests during the stall, so the stall never
|
||||
appears in the latency histogram. hotpath's load generator is **open-loop**
|
||||
— requests are scheduled at a fixed rate and each latency is measured from
|
||||
its *intended* send time. When the server falls behind, the requests queued
|
||||
behind it each carry the full delay a real client would have seen. Nothing
|
||||
is omitted.
|
||||
|
||||
The service measurement is calibrated the same way: 0/10 A/A false
|
||||
positives, 10/10 injected-5%-latency-regression detections
|
||||
(`results/calibration/cjson-service-aa.json`).
|
||||
|
||||
> **Percentiles are claimed only when the measurement can pin them down.**
|
||||
> The cJSON service's p50 latency win was CI-tight and accepted (+6.2%,
|
||||
> [+5.8%, +7.2%]); its p99 CI was [0.07, 4.97] — uselessly wide on a
|
||||
> single-worker loopback tail — so it was **rejected**, even though the
|
||||
> median pointed the right way. A number the measurement can't pin down
|
||||
> does not ship.
|
||||
@@ -0,0 +1,33 @@
|
||||
# The playbook
|
||||
|
||||
Optimizations are tried in a fixed order — **cheapest and safest first** —
|
||||
because cost-per-attempt and risk both rise as you go down the list, and
|
||||
the ledger stops you from re-grinding a class that already has a verdict.
|
||||
|
||||
| # | Class | Mechanism | Verified on |
|
||||
|---|---|---|---|
|
||||
| 1 | Build config | LTO, codegen-units, target-cpu | tokei ✅ +10.4% |
|
||||
| 2 | Allocator | Global allocator swap (mimalloc) | comrak ✅ +4.6% (human-ruled) |
|
||||
| 3 | Allocation churn | Remove per-item malloc/free | tokei ✅, cJSON ✅ (+ one overturned) |
|
||||
| 4 | Data layout | Make the cache hierarchy fit the access pattern | comrak (attempted) |
|
||||
| 5 | Algorithmic | Eliminate redundant work | tokei ×3 ✅, cJSON ✅ |
|
||||
| 6 | SIMD | Autovectorization enablement | comrak, tokei (attempted) |
|
||||
| 7 | Concurrency | Parallelism — always `needs-human-review` | — |
|
||||
|
||||
Six classes were exercised across the case studies, with verified wins on
|
||||
classes 1, 2, 3, and 5 across **both a Rust and a C target**.
|
||||
|
||||
## Why the ordering is load-bearing
|
||||
|
||||
The cheapest class produced the project's single largest individual win:
|
||||
LTO on tokei, **+10.4%**, a pure build-config change with no source edit.
|
||||
The lesson the ledger enforces is the one every performance engineer learns
|
||||
the hard way — *you do not hand-optimize a hot loop before you have turned
|
||||
on the optimizer's own switches.* An agent that jumped straight to SIMD
|
||||
would have missed a bigger, safer, free win sitting one config flag away.
|
||||
|
||||
Each class entry (`playbook/0N-*.md`) carries its **preconditions** (the
|
||||
profile signature that justifies trying it), its **procedure**, its
|
||||
**verification** requirements, and its **known failure modes**. The agent
|
||||
reads the profile, matches the signature, and picks the cheapest class
|
||||
whose preconditions are met and that isn't already spent on this hotspot.
|
||||
@@ -0,0 +1,50 @@
|
||||
# Reproduce it yourself
|
||||
|
||||
Nothing here is a screenshot. Every number regenerates from a pinned corpus
|
||||
through the same commands.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Rust (stable) + `just`; `clang`/`cmake` for the C target; `valgrind` for
|
||||
callgrind profiling. Nightly Rust for the sanitizer gate.
|
||||
- Targets are vendored fresh (their `workspace/` is gitignored); each
|
||||
`targets/<name>/target.toml` has the `[fetch]` recipe.
|
||||
|
||||
## The commands
|
||||
|
||||
```
|
||||
just build / test / lint # trust-layer workspace
|
||||
just aa [cmd] # A/A self-test — must be a null verdict
|
||||
just calibrate <cmd> <out> # A/A false-positive + injection detection
|
||||
just gates <target> # corpus pin + upstream tests + golden replay
|
||||
just verdict <t> <bin> ... # gates + bench vs pristine baseline + ledger row
|
||||
just report <run-id> # ROI report from a ledger row
|
||||
```
|
||||
|
||||
Service mode (Phase 4):
|
||||
|
||||
```
|
||||
just service <baseline-bin> <candidate-bin> <doc> # interleaved p50/p99 CIs
|
||||
just service-calibrate <server> <doc> <out> # latency A/A + injection
|
||||
just report <run-id> --service-json <json> # latency + ROI report
|
||||
```
|
||||
|
||||
The agent loop, behind OS isolation:
|
||||
|
||||
```
|
||||
just isolation-check # 19 boundary checks, both modes
|
||||
scripts/agent-isolated.sh <target> <run-id> # one unattended attempt, confined
|
||||
```
|
||||
|
||||
## Auditing a result
|
||||
|
||||
Every accepted win can be re-audited independently:
|
||||
|
||||
```
|
||||
python3 scripts/audit-attempt.py <run-id> # mechanical re-check
|
||||
python3 scripts/diff-fuzz-<target>.py <dir> 10000 <candidate-bin> # 10k fuzz
|
||||
```
|
||||
|
||||
The ledger itself is the record: `results/ledger.sqlite`, append-only,
|
||||
every attempt (win or loss) with its hypothesis, gates, bench evidence, and
|
||||
verdict.
|
||||
@@ -0,0 +1,42 @@
|
||||
# Roadmap & scope
|
||||
|
||||
## Done — the core loop, end to end
|
||||
|
||||
| Phase | Scope | Result |
|
||||
|---|---|---|
|
||||
| **0** | Manual dry run, two Rust targets | tokei +10.4% verified; methodology written |
|
||||
| **1** | Trust layer (bench-runner, diff-test, ledger) | A/A <5% FP, injection ≥95%, gates on 2 targets; calibrated free on GitHub runners |
|
||||
| **2** | Unattended agent loop | 20 audited attempts, 3 verified wins, **0 shipped false accepts**, OS-isolated |
|
||||
| **3** | Playbook + proof, C/C++ | cross-language cJSON win (+8.85%), 4 case studies, 6 playbook classes |
|
||||
| **4** | Services & scale | cJSON service +6.2% p50 under CO-correct replay, mechanical ROI report |
|
||||
|
||||
The `profile → hypothesize → patch → gated-verdict → ROI` loop runs
|
||||
unattended, OS-isolated, across **Rust and C**, **batch and service**, from
|
||||
a pinned corpus to a dollar figure — with zero shipped false accepts across
|
||||
all phases, and the two pipeline over-accepts both caught by audit and
|
||||
turned into permanent gates.
|
||||
|
||||
## Frontier — the research forks (SPEC §13)
|
||||
|
||||
Deliberately out of the core roadmap, these are where the largest gaps
|
||||
live:
|
||||
|
||||
- **GPU kernel lane.** Triton/CUDA targets where 2–10× gaps are common;
|
||||
correctness via reference-kernel differential testing; reward = measured
|
||||
kernel time. The trust machinery (differential equivalence, interleaved
|
||||
timing, append-only ledger) transfers directly; the profiler and bench
|
||||
become kernel-time-aware, and it needs GPU hardware to run.
|
||||
- **Learned optimization policies.** The ledger is a growing dataset of
|
||||
`(profile signature, class, hypothesis) → verdict`. That is training data
|
||||
for a policy that *ranks* which class to try first on a new hotspot,
|
||||
turning the fixed cheapest-first ordering into a learned prior. Prior art:
|
||||
MLGO, AlphaDev, Meta's LLM Compiler; ProGraML-style IR graphs as the
|
||||
program representation.
|
||||
|
||||
## The real target
|
||||
|
||||
The end goal is not vendored OSS case studies — it is a **real production
|
||||
system**. The same gates that matter for a JSON parser matter far more for
|
||||
a live CMS: an equivalence failure there is a customer-facing bug, and the
|
||||
no-network sandbox and append-only audit trail are exactly what make it
|
||||
safe to point an optimization agent at code that a business depends on.
|
||||
@@ -0,0 +1,39 @@
|
||||
# ROI: from stopwatch to dollars
|
||||
|
||||
The report is the pitch artifact, and it answers exactly one question:
|
||||
*what did the stopwatch say, and what does that buy?* It is generated
|
||||
**mechanically** — `just report <run-id>` reads a ledger row and the
|
||||
calibrated bench and renders the numbers; nothing is hand-typed.
|
||||
|
||||
## The three conversions
|
||||
|
||||
- **Throughput → cores → dollars.** A speedup `s` on a CPU-bound fleet of
|
||||
`N` cores returns `N × (1 − 1/s)` cores; at a `$/core-hour` rate that is
|
||||
an annual saving. Both flow through the speedup's confidence interval, so
|
||||
the report quotes a *range*.
|
||||
- **Latency → percentile deltas → value.** Under replayed load, the p50/p99
|
||||
latency speedups with their CIs; a p50 service-time reduction maps to
|
||||
throughput capacity for a CPU-bound service.
|
||||
- **Batch windows → wall-clock hours** returned to a schedule.
|
||||
|
||||
## The rule that makes it survive review
|
||||
|
||||
Every figure carries its CI and its workload, and the **methodology ships
|
||||
inside the report** — the interleaved A/B, the pristine baseline, the
|
||||
bootstrap CI, the gates, the calibration. And the number the report tells
|
||||
you to quote is the **CI lower bound**, never the median:
|
||||
|
||||
> On a 500-core fleet at $0.04/core-hour, the cJSON service win (+6.2% p50
|
||||
> latency) returns **27.5 cores / $9,621 per year at the CI lower bound**
|
||||
> (median 29.4 / $10,290). The point estimate is not a promise.
|
||||
|
||||
## What the report refuses to do
|
||||
|
||||
It will not mint a clean dollar figure for a change that wasn't shipped. A
|
||||
row whose verdict isn't `accepted`, **or** an `accepted` row that isn't
|
||||
sanitizer-clean (a pre-gate accept the audit overturned), gets a bold **Not
|
||||
shipped** banner before any number. That guard was itself added by the
|
||||
[adversarial review](https://github.com/bdbrown4/peltier.io/blob/main/results/adversarial-review.md),
|
||||
which caught the generator pitching the overturned comrak patch. The pitch
|
||||
artifact is held to the same standard as everything else: it cannot claim
|
||||
what wasn't proven.
|
||||
@@ -0,0 +1,48 @@
|
||||
# The thesis: trust is the product
|
||||
|
||||
Performance work has a credibility problem. "We made it 15% faster" is
|
||||
almost always some mix of a noisy benchmark, a workload that flatters the
|
||||
change, and a behavioral difference nobody checked. The number is real to
|
||||
the person who measured it and worthless to everyone downstream.
|
||||
|
||||
hotpath is built on the opposite premise: **a performance claim is a
|
||||
liability until it is proven, and the proof ships with the number.**
|
||||
|
||||
## Three non-negotiables
|
||||
|
||||
1. **No unverified performance claims.** A change without passing
|
||||
equivalence gates *and* a statistically significant benchmark delta
|
||||
does not exist. It is discarded and logged.
|
||||
2. **Measure before optimizing.** No patch is proposed until profile data
|
||||
identifies the hot path. A single benchmark run is never trusted.
|
||||
3. **Cheap wins first.** Build flags → LTO → PGO → allocator swap → *then*
|
||||
code changes. The ledger prevents re-attempting a dead end.
|
||||
|
||||
## What "proven" means here
|
||||
|
||||
A win must clear **both** of two independent bars, because either one alone
|
||||
is gameable:
|
||||
|
||||
- **Equivalence** stops "optimizations" that are actually behavior changes.
|
||||
A JSON parser that's faster because it stopped validating numbers is not
|
||||
faster; it's broken. Byte-identical golden replay and differential
|
||||
fuzzing catch this.
|
||||
- **Significance** stops noise from masquerading as a win. The bar is not
|
||||
"the median improved" — it is "the bootstrap 95% CI *lower bound* clears
|
||||
the threshold," measured by interleaving baseline and candidate on pinned
|
||||
hardware that has passed its own A/A calibration.
|
||||
|
||||
A real +5.7% median that couldn't clear the significance bar (its CI lower
|
||||
bound was +0.2%) was **rejected** — and that rejection is in the ledger, because
|
||||
a rejection is a complete, valid outcome.
|
||||
|
||||
## Why an agent
|
||||
|
||||
The loop — profile, hypothesize, patch, gate, measure, record — is
|
||||
mechanical and tedious, and the temptation to fudge a number is exactly
|
||||
what a tired human succumbs to. An agent proposes; the *trust layer*
|
||||
disposes. The agent cannot write the ledger, cannot move the baseline,
|
||||
cannot touch the thresholds. It can only propose a patch and ask the
|
||||
harness to judge it. The [architecture](./architecture.md) is designed so
|
||||
that even a compromised or adversarial proposer cannot manufacture a false
|
||||
win.
|
||||
Vendored
+20
@@ -0,0 +1,20 @@
|
||||
/* hotpath docs — small refinements over the default mdBook theme. */
|
||||
|
||||
.content h1 {
|
||||
border-bottom: 2px solid var(--links);
|
||||
padding-bottom: 0.2em;
|
||||
}
|
||||
|
||||
/* Verified-number tables read as evidence: tighten and emphasize. */
|
||||
.content table td:first-child {
|
||||
font-weight: 500;
|
||||
}
|
||||
|
||||
blockquote {
|
||||
border-inline-start: 4px solid var(--links);
|
||||
}
|
||||
|
||||
/* Inline verdict-ish code (accepted / rejected-bench / needs-human-review) */
|
||||
.content code {
|
||||
font-variant-ligatures: none;
|
||||
}
|
||||
Reference in New Issue
Block a user