From 5d6c259b30b45bf86a8b4d970343422d01c9ec39 Mon Sep 17 00:00:00 2001 From: Claude Date: Sun, 12 Jul 2026 00:20:20 +0000 Subject: [PATCH] docs: full mdBook site + GitHub Pages deploy; rewrite README as project front page MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit - docs/: an mdBook documentation site — thesis, architecture, measurement discipline, equivalence gates, agent loop, isolation/anti-reward-hacking, playbook, ROI, reproduce, roadmap, and the five case studies (pulled from their single source in results/ via {{#include}}). Builds clean (21 pages). - .github/workflows/docs.yml: builds mdBook and deploys to GitHub Pages on push to main (one-time setup: Settings → Pages → Source = GitHub Actions). - README.md: rewritten as the project front page — the thesis, the verified results table, layout, quick start, how it stays honest, links to the book. Replaces the stale Phase-0-era text. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2 --- .github/workflows/docs.yml | 53 +++++++++ .gitignore | 1 + README.md | 112 +++++++++++-------- docs/book.toml | 20 ++++ docs/src/SUMMARY.md | 27 +++++ docs/src/agent-loop.md | 52 +++++++++ docs/src/architecture.md | 61 ++++++++++ docs/src/case-studies/build-flags.md | 1 + docs/src/case-studies/cjson.md | 1 + docs/src/case-studies/comrak-false-accept.md | 1 + docs/src/case-studies/overview.md | 17 +++ docs/src/case-studies/service.md | 1 + docs/src/case-studies/tokei.md | 1 + docs/src/equivalence.md | 52 +++++++++ docs/src/infra.md | 1 + docs/src/introduction.md | 46 ++++++++ docs/src/isolation.md | 53 +++++++++ docs/src/measurement.md | 64 +++++++++++ docs/src/playbook.md | 33 ++++++ docs/src/reproduce.md | 50 +++++++++ docs/src/roadmap.md | 42 +++++++ docs/src/roi.md | 39 +++++++ docs/src/thesis.md | 48 ++++++++ docs/theme/custom.css | 20 ++++ 24 files changed, 748 insertions(+), 48 deletions(-) create mode 100644 .github/workflows/docs.yml create mode 100644 docs/book.toml create mode 100644 docs/src/SUMMARY.md create mode 100644 docs/src/agent-loop.md create mode 100644 docs/src/architecture.md create mode 100644 docs/src/case-studies/build-flags.md create mode 100644 docs/src/case-studies/cjson.md create mode 100644 docs/src/case-studies/comrak-false-accept.md create mode 100644 docs/src/case-studies/overview.md create mode 100644 docs/src/case-studies/service.md create mode 100644 docs/src/case-studies/tokei.md create mode 100644 docs/src/equivalence.md create mode 100644 docs/src/infra.md create mode 100644 docs/src/introduction.md create mode 100644 docs/src/isolation.md create mode 100644 docs/src/measurement.md create mode 100644 docs/src/playbook.md create mode 100644 docs/src/reproduce.md create mode 100644 docs/src/roadmap.md create mode 100644 docs/src/roi.md create mode 100644 docs/src/thesis.md create mode 100644 docs/theme/custom.css diff --git a/.github/workflows/docs.yml b/.github/workflows/docs.yml new file mode 100644 index 0000000..6715ad4 --- /dev/null +++ b/.github/workflows/docs.yml @@ -0,0 +1,53 @@ +name: docs + +# Builds the mdBook site in docs/ and deploys it to GitHub Pages. +# One-time setup: repo Settings → Pages → Source = "GitHub Actions". + +on: + push: + branches: [main] + paths: + - "docs/**" + - ".github/workflows/docs.yml" + workflow_dispatch: + +permissions: + contents: read + pages: write + id-token: write + +# Allow one concurrent deployment; don't cancel an in-progress prod deploy. +concurrency: + group: pages + cancel-in-progress: false + +jobs: + build: + runs-on: ubuntu-latest + env: + MDBOOK_VERSION: "0.4.40" + steps: + - uses: actions/checkout@v4 + - name: Install mdBook + run: | + mkdir -p "$HOME/.local/bin" + url="https://github.com/rust-lang/mdBook/releases/download/v${MDBOOK_VERSION}/mdbook-v${MDBOOK_VERSION}-x86_64-unknown-linux-gnu.tar.gz" + curl -sSL "$url" | tar -xz -C "$HOME/.local/bin" + echo "$HOME/.local/bin" >> "$GITHUB_PATH" + - name: Build + run: mdbook build docs + - name: Upload artifact + uses: actions/upload-pages-artifact@v3 + with: + path: docs/book + + deploy: + needs: build + runs-on: ubuntu-latest + environment: + name: github-pages + url: ${{ steps.deployment.outputs.page_url }} + steps: + - name: Deploy to GitHub Pages + id: deployment + uses: actions/deploy-pages@v4 diff --git a/.gitignore b/.gitignore index 6ded5d1..6cf1a9e 100644 --- a/.gitignore +++ b/.gitignore @@ -249,3 +249,4 @@ targets/cjson/cand-*/ targets/cjson/asan*/ targets/cjson/svc-baseline/ targets/cjson/svc-candidate/ +docs/book/ diff --git a/README.md b/README.md index 44968a0..8179f3e 100644 --- a/README.md +++ b/README.md @@ -1,72 +1,88 @@ -# peltier.io +# hotpath -Spot cooling for hot paths. A profile-guided optimization agent that only ships wins it can prove. +> Spot cooling for hot paths. A profile-guided optimization agent that +> only ships wins it can prove. -hotpath finds the 10–90% left on the floor between "the compiler" and -"hire a perf consultant," with two properties no consultant offers: -**verified equivalence** (a change provably preserves behavior or is -discarded) and **self-verifying ROI** (the value claim is a stopwatch on -trusted infrastructure, not a survey). +hotpath profiles real binaries and services, isolates hot paths, proposes +optimizations, and accepts a change **only** when two independent bars are +cleared: **behavioral equivalence** (upstream tests + byte-identical golden +replay + differential fuzzing + sanitizers) and **statistical significance** +(the speedup's bootstrap 95% CI lower bound clears a threshold on +calibrated hardware). A change without both is discarded and logged. -Full design: **[SPEC.md](SPEC.md)**. +Consultants sell surveys; compilers sell flags. hotpath sells **verified +deltas with the methodology attached** — a stopwatch on pinned hardware, a +proof of equivalence, and an ROI figure whose confidence interval survives +hostile review. + +📖 **Full documentation: the [hotpath book](https://bdbrown4.github.io/peltier.io/)** +· Design spec: [SPEC.md](SPEC.md) · Charter: [CLAUDE.md](CLAUDE.md) + +## Verified results + +Every number carries its 95% CI and its workload, and every one survived the +gates above. Across **five phases and 33 ledger rows, zero shipped false +accepts** — including two pipeline over-accepts the audit caught before +anything shipped, each becoming a permanent new gate. + +| Target | Win | Verified | +|---|---|---| +| tokei (Rust) — LTO (class 1) | +10.4% | CI [+8.5%, +12.0%] | +| tokei — three class-5 algorithmic wins | +10.7% / +3.7% / +9.9% | each CI-significant | +| cJSON (C) — number-handling rewrite (class 5) | +8.85% | CI [+7.5%, +10.0%] | +| cJSON HTTP service — p50 latency under replay | +6.2% | CI [+5.8%, +7.2%] | +| comrak (Rust) — mimalloc (class 2) | +4.6% | CI [+3.3%, +5.8%], [human-ruled](results/rulings/phase0-comrak-002.md) | ## Layout ``` -crates/ Trust layer (agent has NO write access — SPEC §10) - bench-runner/ Interleaved A/B timing, bootstrap CIs, A/A calibration - diff-test/ Equivalence gates, corpus hash-pinning, per-target policy - ledger/ Append-only SQLite attempt ledger (enforced by triggers) - report/ ROI math: speedup CI → cores → dollars, caveats included +crates/ Trust layer — agent has NO write access (SPEC §10) + bench-runner/ Interleaved A/B, bootstrap CIs, A/A calibration, service mode + diff-test/ Equivalence gates, corpus hash-pinning, per-target spec + ledger/ Append-only SQLite attempt ledger (mutation-refusing triggers) + report/ ROI: speedup CI → cores → dollars, methodology inline + verdict/ The pipeline in one command: gates → bench → ledger row + harnessd/ The one door the agent talks through (7 JSON ops) agent/ Untrusted proposer (Claude Agent SDK, Python) playbook/ Optimization classes 1–7, tried strictly cheapest-first config/ accept.toml (thresholds), pricing.toml (ROI inputs) -targets/ Per-target checkouts — the only agent-writable path +targets/ Vendored OSS targets — the only agent-writable path corpora/ Hash-pinned golden-replay inputs (read-only to agent) -results/ Calibration evidence, per-engagement outputs +results/ Calibration evidence, case studies, generated ROI reports +docs/ The mdBook site (deployed to GitHub Pages) ``` ## Quick start ```sh -cargo test --workspace # trust-layer unit tests -just aa # A/A self-test: must yield a null verdict -just compare "cmd-a" "cmd-b" # interleaved A/B with bootstrap CI +just build / test / lint # trust-layer workspace +just aa # A/A self-test — must yield a null verdict +just gates # corpus pin + upstream tests + golden replay +just verdict ... # gates + bench vs pristine baseline + ledger row +just report # ROI report from a ledger row +just isolation-check # 19 OS-boundary checks (both modes) ``` -## Status +Full command reference and reproduction steps: +[Reproduce it yourself](https://bdbrown4.github.io/peltier.io/reproduce.html). -**Phase 0 complete** (see SPEC §5). Five attempts in the ledger across -two pinned targets (comrak 0.53.0, tokei 14.0.0), A/A-calibrated, -every number with CI and workload: +## How it stays honest -- **tokei +10.4% median [95% CI +8.5%, +12.0%]** — read-path buffer - zeroing eliminated + fat LTO, byte-identical, 4,332-iteration - diff-fuzz clean, sanitizers clean, auto-accepted - (`results/phase0/case-study-tokei.md`). Meets the Phase 0 exit bar. -- comrak +4.6% [+3.3%, +5.8%] — mimalloc swap, `needs-human-review` - on an LSan reachability flag (`results/phase0/case-study-comrak.md`). -- Two null/sub-threshold results correctly rejected and logged. +- The agent speaks to the trust layer through **seven read-only-plus-one- + write JSON operations**; it has no shell, cannot write outside a target + workspace, and cannot touch the ledger or thresholds. +- The **baseline is rebuilt from a pristine checkout** every session — a + patch never becomes its own comparison. +- The **ledger is append-only** (SQLite triggers refuse UPDATE/DELETE); + accepted rows are written only by the `verdict` binary after the gates. +- The agent process runs under **OS-level isolation** (read-only mount + namespace or an unprivileged uid) — the boundary is filesystem + permissions, not a prompt. -Next: Phase 1 trust-layer hardening (containers, perf, automated -calibration, gate orchestration, agent IPC). - -Done: -- Workspace with the four trust-layer crates; core statistics - (interleaved scheduling, bootstrap ratio CIs, CI-lower-bound accept - rule), append-only ledger schema with mutation-refusing triggers, - corpus hash-pin verifier, equivalence-policy parser, ROI formulas — - all unit-tested. -- bench-runner CLI: whole-program `compare` and `aa` modes with env - fingerprinting. -- Playbook classes 1–7; agent tool contract and prompting spine. - -Next (Phase 1 exit criteria): -- `perf stat` counters + max RSS + RAPL capture per run. -- Automated A/A calibration sessions and regression-injection self-test - recorded to `results/calibration/` (<5% FP, ≥95% detection). -- diff-test orchestration end-to-end on two real targets; harness IPC - for the agent tool surface. +See [the case studies](https://bdbrown4.github.io/peltier.io/case-studies/overview.html) +— especially [the caught false-accept](https://bdbrown4.github.io/peltier.io/case-studies/comrak-false-accept.html), +where the pipeline over-accepted a leaking patch and the audit overturned +it before it shipped. ## License diff --git a/docs/book.toml b/docs/book.toml new file mode 100644 index 0000000..14d0dd8 --- /dev/null +++ b/docs/book.toml @@ -0,0 +1,20 @@ +[book] +title = "hotpath" +description = "A profile-guided optimization agent that only ships wins it can prove." +authors = ["hotpath"] +language = "en" +src = "src" + +[output.html] +default-theme = "ayu" +preferred-dark-theme = "ayu" +git-repository-url = "https://github.com/bdbrown4/peltier.io" +edit-url-template = "https://github.com/bdbrown4/peltier.io/edit/main/docs/{path}" +additional-css = ["theme/custom.css"] + +[output.html.fold] +enable = true +level = 1 + +[output.html.playground] +runnable = false diff --git a/docs/src/SUMMARY.md b/docs/src/SUMMARY.md new file mode 100644 index 0000000..e8e0b0a --- /dev/null +++ b/docs/src/SUMMARY.md @@ -0,0 +1,27 @@ +# Summary + +[hotpath](./introduction.md) + +- [The thesis: trust is the product](./thesis.md) +- [Architecture](./architecture.md) +- [Measurement discipline](./measurement.md) +- [Equivalence gates](./equivalence.md) +- [The agent loop](./agent-loop.md) +- [Isolation & anti-reward-hacking](./isolation.md) +- [The playbook](./playbook.md) +- [ROI: from stopwatch to dollars](./roi.md) + +# Case studies + +- [Overview](./case-studies/overview.md) +- [tokei — compounding algorithmic wins (Rust)](./case-studies/tokei.md) +- [Cheap wins first: build flags & allocators](./case-studies/build-flags.md) +- [The caught false-accept (comrak)](./case-studies/comrak-false-accept.md) +- [cJSON — the cross-language proof (C)](./case-studies/cjson.md) +- [Services & scale: latency under load](./case-studies/service.md) + +# Reference + +- [Reproduce it yourself](./reproduce.md) +- [The trust boundary (infra)](./infra.md) +- [Roadmap & scope](./roadmap.md) diff --git a/docs/src/agent-loop.md b/docs/src/agent-loop.md new file mode 100644 index 0000000..0110774 --- /dev/null +++ b/docs/src/agent-loop.md @@ -0,0 +1,52 @@ +# The agent loop + +The proposer is a Claude Agent SDK loop (`agent/hotpath_agent/`) driving +the seven harness tools over a standalone MCP server. It has no built-in +file, shell, or web tools in its allowlist — only the harness door. + +## One attempt, start to finish + +1. **Read the ledger.** What classes and hypotheses were already tried on + this target? The ledger is the anti-double-attempt memory: a class may + be re-entered with a *materially new* hypothesis, never a duplicate. +2. **Read the profile.** callgrind cache-sim ranks hotspots with per- + function cache-miss and branch-mispredict columns, so the agent can tell + an allocation-churn hotspot from a branch-bound one and pick the right + playbook class. +3. **Pick the cheapest untried class** whose profile preconditions are met, + and read its playbook entry. +4. **Read the source** (windowed, at the pinned commit) and **propose a + patch** with a stated hypothesis. +5. **Run the verdict** and **poll** for the result. +6. **Report it verbatim.** A rejection with a clean ledger row is a + *successful* outcome. Do not iterate more than twice on a rejected + hypothesis without a new idea. + +## What the loop proved about itself + +Across an unattended grind of 20+ audited attempts, the loop demonstrated +the behavior the architecture is designed to produce: + +- It chose the cheapest untried class each time, reading the ledger to + avoid re-grinding a dead end. +- Every patch went through `propose_patch` (path-allowlist + `git apply`) + and the full gate+bench pipeline. Malformed diffs were refused by the + harness, not the agent's goodwill. +- **It twice refused to fabricate a verdict it could not read back.** When + a multi-minute pipeline outran the MCP transport's timeout, the agent + checked the ledger, confirmed the row wasn't written, and reported the + verdict as *unreadable, not invented* — asking a human to read the + ledger rather than guessing a number. That refusal, under a real failure + mode, is the anti-reward-hacking design working. + +## The honest-failure record + +The ledger records **every** attempt, including the rejections, because a +rejection is training data and a demonstration of the bar. Of the audited +unattended attempts, most were honest rejections — real mechanisms that +simply didn't clear the 2% CI-lower-bound bar — a handful were verified +wins, and one was a `rejected-gate` where golden replay caught the agent's +patch producing wrong output that a green test suite would have shipped. + +The load-bearing number is **zero shipped false accepts** across the whole +project. diff --git a/docs/src/architecture.md b/docs/src/architecture.md new file mode 100644 index 0000000..8291207 --- /dev/null +++ b/docs/src/architecture.md @@ -0,0 +1,61 @@ +# Architecture + +hotpath is split into a **trust layer** (Rust, the agent cannot write it) +and an **untrusted proposer** (the agent). The split is structural, not +advisory: the guarantees hold even if the proposer is adversarial. + +``` +crates/ Trust layer — agent has NO write access (SPEC §10) + bench-runner/ Interleaved A/B timing, bootstrap CIs, A/A calibration, + service-mode latency (coordinated-omission correct) + diff-test/ Equivalence gates, corpus hash-pinning, per-target spec + ledger/ Append-only SQLite attempt ledger (enforced by triggers) + report/ ROI: speedup CI → cores → dollars, methodology inline + verdict/ The pipeline in one command: gates → bench → ledger row + harnessd/ The one door the agent talks through (7 JSON ops) +agent/ Untrusted proposer (Claude Agent SDK, Python) +playbook/ Optimization classes 1–7, tried strictly cheapest-first +config/ accept.toml (thresholds), pricing.toml (ROI inputs) +targets/ Vendored OSS targets — the only agent-writable path +corpora/ Hash-pinned golden-replay inputs (read-only to agent) +results/ Calibration evidence, case studies, generated reports +``` + +## The seven-tool boundary + +The agent never runs a shell, never writes a file, never touches the +ledger directly. It speaks to `harnessd` — the only door — through exactly +seven JSON operations: + +| Tool | What it can do | +|---|---| +| `read_profile` | Read a target's ranked hotspots (read-only) | +| `read_ledger` | Read prior attempts — the anti-double-attempt memory | +| `read_playbook` | Read an optimization class's preconditions/procedure | +| `read_target_source` | Read a workspace file at the pinned commit (windowed) | +| `propose_patch` | Submit a unified diff — path-allowlisted, then `git apply` | +| `run_verdict` | Launch the gate+bench pipeline (detached) | +| `read_verdict` | Poll the append-only ledger for the result | + +Note what is *absent*: no write outside `targets//workspace`, no +shell, no way to run the bench or write the ledger except through the +`verdict` binary, which only records a pass after the full gated pipeline. + +## The data flow of one attempt + +``` +profile ─▶ agent hypothesizes ─▶ propose_patch ─▶ [path allowlist + git apply] + │ + ledger row ◀─ verdict ◀─ bench (interleaved A/B) ◀─ gates (tests, golden, + (append-only) vs pristine-rebuilt fuzz, sanitizers) + baseline +``` + +The baseline is **rebuilt from a pristine checkout every session** — never +from the agent's workspace — so a patch cannot poison its own comparison. +The verdict is decided by the CI-lower-bound rule and written to an +append-only ledger the agent cannot mutate. + +The next chapters walk each stage: [measurement](./measurement.md), +[equivalence gates](./equivalence.md), [the agent loop](./agent-loop.md), +and the [isolation](./isolation.md) that makes the boundary load-bearing. diff --git a/docs/src/case-studies/build-flags.md b/docs/src/case-studies/build-flags.md new file mode 100644 index 0000000..03b3a2d --- /dev/null +++ b/docs/src/case-studies/build-flags.md @@ -0,0 +1 @@ +{{#include ../../../results/phase3/02-build-flags-and-allocators.md}} diff --git a/docs/src/case-studies/cjson.md b/docs/src/case-studies/cjson.md new file mode 100644 index 0000000..feade0a --- /dev/null +++ b/docs/src/case-studies/cjson.md @@ -0,0 +1 @@ +{{#include ../../../results/phase3/04-cjson-c-target.md}} diff --git a/docs/src/case-studies/comrak-false-accept.md b/docs/src/case-studies/comrak-false-accept.md new file mode 100644 index 0000000..f84e908 --- /dev/null +++ b/docs/src/case-studies/comrak-false-accept.md @@ -0,0 +1 @@ +{{#include ../../../results/phase3/03-comrak-caught-false-accept.md}} diff --git a/docs/src/case-studies/overview.md b/docs/src/case-studies/overview.md new file mode 100644 index 0000000..348d46c --- /dev/null +++ b/docs/src/case-studies/overview.md @@ -0,0 +1,17 @@ +# Case studies + +Five verified case studies, each reproducible from a pinned corpus through +the same gated pipeline, each number carrying its 95% CI and workload. + +- [tokei — compounding algorithmic wins (Rust)](./tokei.md) — three class-5 + wins on the same hot function, each unattended and audited. +- [Cheap wins first: build flags & allocators](./build-flags.md) — LTO + accepted (+10%), mimalloc held for a human ruling then accepted-scoped. +- [The caught false-accept (comrak)](./comrak-false-accept.md) — the + leaking teardown the bench loved, overturned by audit. The most important + study. +- [cJSON — the cross-language proof (C)](./cjson.md) — a sub-threshold win + rejected, then a byte-identical win accepted, on a C target. +- [Services & scale: latency under load](./service.md) — the cJSON win + measured as a latency delta under coordinated-omission-correct replay, + turned into a dollar figure mechanically. diff --git a/docs/src/case-studies/service.md b/docs/src/case-studies/service.md new file mode 100644 index 0000000..fa1cddc --- /dev/null +++ b/docs/src/case-studies/service.md @@ -0,0 +1 @@ +{{#include ../../../results/phase4/README.md}} diff --git a/docs/src/case-studies/tokei.md b/docs/src/case-studies/tokei.md new file mode 100644 index 0000000..0d86db1 --- /dev/null +++ b/docs/src/case-studies/tokei.md @@ -0,0 +1 @@ +{{#include ../../../results/phase3/01-tokei-algorithmic-rust.md}} diff --git a/docs/src/equivalence.md b/docs/src/equivalence.md new file mode 100644 index 0000000..9ab7ef6 --- /dev/null +++ b/docs/src/equivalence.md @@ -0,0 +1,52 @@ +# Equivalence gates + +A faster binary that behaves differently is not an optimization — it is a +regression with good marketing. Before any speedup is even measured, the +candidate must prove it does the same thing the baseline does. These are +**hard gates**: a failure is `rejected-gate` and the bench never runs. + +## The layers + +1. **Upstream test suite.** The target's own tests, run against the patched + code. For cJSON that is the vendored Unity/CTest suite (19 tests: + `parse_number`, `parse_string`, `print_*`, `misc_tests`, …); for the Rust + targets, `cargo test`. + +2. **Golden replay — byte-identical.** The candidate processes the + hash-pinned corpus and its output is hashed and compared to a pinned + `GOLDEN.sha256`. A code counter that miscounts one line, a JSON printer + that serializes one float differently — caught here. The corpus itself + is pinned via `MANIFEST.sha256` and the harness refuses to run on a + mismatch. + +3. **Differential fuzzing.** Pristine vs candidate on thousands of mutated + inputs, comparing output. Every accepted win cleared **10,000 mutated + inputs with 0 divergences** — the tokei wins after canonicalizing benign + parallel-walker output ordering (a real finding: differential gates must + compare *semantics*, not bytes, where the target is legitimately + nondeterministic in presentation), the cJSON win on quote/number/escape + edge cases. + +4. **Sanitizers.** ASan/LSan for Rust, ASan+UBSan for C. Run on **every + would-be accept** — this is machine-enforced, not manual (see below). + +## The tier rule (SPEC §8) + +Some changes can pass every gate and still be unsafe to auto-ship. A change +that touches floating-point ordering, concurrency primitives, `unsafe`, or +carries a sanitizer flag is **never** auto-accepted — its verdict is capped +at `needs-human-review`. The tier is decided by the *nature* of the change, +never by the size of the number. The mimalloc allocator swap +([case study](./case-studies/build-flags.md)) is the canonical example: a +real +4.6% win, correctly held for a human ruling because a global +allocator change's safety is not something one benchmark can clear. + +## When a gate caught the pipeline itself + +The sanitizer gate exists in its current, machine-enforced form because it +once *wasn't*. The pipeline auto-accepted a comrak patch that skipped AST +teardown for a +10.7% win — and the audit's manual LeakSanitizer run found +it leaks the arena at exit. It was overturned before shipping and the fix +made `verdict` build with ASan/UBSan and run the pinned workload on every +accept, capping any flagged win at `needs-human-review`. The full story is +[the caught false-accept](./case-studies/comrak-false-accept.md). diff --git a/docs/src/infra.md b/docs/src/infra.md new file mode 100644 index 0000000..54bdf73 --- /dev/null +++ b/docs/src/infra.md @@ -0,0 +1 @@ +{{#include ../../docs/infra.md}} diff --git a/docs/src/introduction.md b/docs/src/introduction.md new file mode 100644 index 0000000..5af7e55 --- /dev/null +++ b/docs/src/introduction.md @@ -0,0 +1,46 @@ +# hotpath + +> Spot cooling for hot paths. A profile-guided optimization agent that +> only ships wins it can prove. + +hotpath profiles real binaries and services, isolates hot paths, proposes +optimizations, and accepts a change **only** when two independent bars are +cleared: + +1. **Behavioral equivalence** — the change provably preserves behavior + (upstream tests, byte-identical golden replay, differential fuzzing, + sanitizers) or it is discarded. +2. **Statistical significance** — the speedup's bootstrap 95% CI lower + bound clears a threshold on trusted, calibrated hardware. + +Everything else is commentary. A change without both is not a win; it is a +ledger row explaining why not. + +## What it has actually done + +Every number here carries its 95% confidence interval and its workload, +and every one survived the gates above. These are not projections. + +| Result | Verified | +|---|---| +| tokei (Rust) — three compounding class-5 algorithmic wins | +10.7%, +3.7%, +9.9%, each CI-significant | +| tokei — link-time optimization (class 1) | +10.4%, CI [+8.5%, +12.0%] | +| cJSON (C) — number-handling rewrite (class 5) | +8.85%, CI [+7.5%, +10.0%] | +| cJSON HTTP service — p50 latency under replayed load | +6.2%, CI [+5.8%, +7.2%] | +| comrak (Rust) — mimalloc allocator swap (class 2) | +4.6%, CI [+3.3%, +5.8%], human-ruled | + +Across **five phases and 33 ledger rows**, with **zero shipped false +accepts** — including two occasions where the pipeline itself over-accepted +and the audit caught it before anything shipped, each becoming a permanent +new gate. + +## The differentiator + +Consultants sell surveys; compilers sell flags. hotpath sells **verified +deltas with the methodology attached** — a stopwatch on pinned hardware, +a proof of equivalence, and an ROI figure whose confidence interval you +can quote to a hostile reviewer. That is the entire product, and this site +is how it works. + +Start with [the thesis](./thesis.md), or jump to a +[case study](./case-studies/overview.md). diff --git a/docs/src/isolation.md b/docs/src/isolation.md new file mode 100644 index 0000000..34f65da --- /dev/null +++ b/docs/src/isolation.md @@ -0,0 +1,53 @@ +# Isolation & anti-reward-hacking + +The guarantees only matter if the proposer cannot cheat. hotpath's defense +is **structural**: the agent is denied the capabilities it would need to +manufacture a false win, at the OS level, not by prompt. + +## What holds even if the proposer is adversarial + +- **Writes go through one door.** `propose_patch` is the only harness- + mediated write path: a diff is path-allowlisted (relative paths only, no + `..`, no `.git`, nothing outside `targets//workspace`) and then + `git apply`-ed rooted in that workspace. A patch aimed at `config/` or + `crates/` is refused before any apply. Verified against a malicious + accept-threshold diff. +- **The ledger is append-only.** `UPDATE` and `DELETE` are refused by + SQLite triggers; `accepted` rows are written only by the `verdict` + binary after the gated pipeline. The agent cannot forge a passing + verdict through the harness. +- **The baseline is unbypassable.** It is rebuilt from a pristine checkout; + the agent's workspace never becomes the comparison. + +## OS-level isolation (SPEC §10) + +The SDK's tool allow/deny list is defense-in-depth, not the boundary — +`scripts/agent-isolated.sh` provides the real one, verified by +`just isolation-check` (19 checks, both modes): + +- **`mountns` mode** — the whole agent process tree runs in a private mount + namespace with the repo bind-mounted **read-only** and `CAP_SYS_ADMIN`/ + `CAP_SYS_PTRACE` dropped from the bounding set. Repo writes fail with + `EROFS`; a forged `INSERT`, a direct `verdict` invocation, a `mount -o + remount,rw`, and even userns re-entry to undo the mount are all refused. +- **`user` mode** — the agent runs as an unprivileged user whose only write + path into the repo is the harnessd Unix socket; the trust layer is owned + by another uid. This is the production shape (`docs/infra.md`). + +## The two times the pipeline was wrong + +Anti-reward-hacking is not "it never fails" — it is "when it fails, the +audit catches it and the system gets harder." That happened twice, and +both are documented, not hidden: + +1. A leaking teardown patch the bench loved was **auto-accepted** and then + **overturned** by the 100% human audit's LeakSanitizer run. The fix made + the sanitizer gate machine-enforced on every accept. +2. During the [adversarial review](https://github.com/bdbrown4/peltier.io/blob/main/results/adversarial-review.md), + the ROI report generator was found to mint a clean dollar figure for that + same overturned row (it trusted the immutable ledger's historical + `accepted` verdict). The fix made the report treat an accepted-but-not- + sanitizer-clean row as **not shippable**. + +Both defects were found by attacking the system's own output, fixed, and +turned into permanent gates. **Zero false accepts shipped.** diff --git a/docs/src/measurement.md b/docs/src/measurement.md new file mode 100644 index 0000000..05ff5d1 --- /dev/null +++ b/docs/src/measurement.md @@ -0,0 +1,64 @@ +# Measurement discipline + +The product is a trustworthy number, so the measurement is where most of +the engineering went. The reference is Mytkowicz et al., *Producing Wrong +Data Without Doing Anything Obviously Wrong* — the default state of a +benchmark is "subtly lying." + +## Interleaved A/B, never sequential + +Baseline and candidate run **interleaved** (ABABAB…), never in sequential +blocks. Thermal drift, a background cron job, a noisy neighbor — anything +that changes over the run hits both sides equally instead of biasing one. +Warm-up runs are discarded. + +## The baseline is always pristine + +The baseline binary is **rebuilt from a clean checkout** each session, with +the candidate's patch stashed. A patch can never silently become part of +its own baseline (a real bug that bit an early smoke test, now closed by +`verdict --rebuild-baseline`). + +## Bootstrap CIs, and the lower-bound rule + +Each side yields ≥30 measured samples. The statistic is the **bootstrap +95% CI of the ratio-of-medians** (deterministic xorshift seed, so a result +reproduces). A change is accepted only if the **CI lower bound** clears the +threshold in `config/accept.toml` (default 2%) — not the median. The number +you commit to is the one that survives the interval. + +## The hardware has to earn trust first + +Before any measurement session, the harness runs two self-tests, recorded +in `results/calibration/`: + +- **A/A**: run the *same* binary as both sides. It must produce a null + verdict. A "speedup" from identical binaries is a calibration failure. +- **Regression injection**: a synthetic 5% slowdown (a real busy-wait in + the timed window) must be detected ≥95% of the time. + +CPU pinning (`taskset -c 2`) alone tightened the A/A confidence interval +~5× on the dev container. Every calibration anchor — local and on GitHub's +shared runners — passed both bars before its numbers were trusted. + +## Service mode: coordinated omission + +Phase 4 measures latency under load, where the classic trap is +**coordinated omission**: a closed-loop client that waits for a slow +response simply sends fewer requests during the stall, so the stall never +appears in the latency histogram. hotpath's load generator is **open-loop** +— requests are scheduled at a fixed rate and each latency is measured from +its *intended* send time. When the server falls behind, the requests queued +behind it each carry the full delay a real client would have seen. Nothing +is omitted. + +The service measurement is calibrated the same way: 0/10 A/A false +positives, 10/10 injected-5%-latency-regression detections +(`results/calibration/cjson-service-aa.json`). + +> **Percentiles are claimed only when the measurement can pin them down.** +> The cJSON service's p50 latency win was CI-tight and accepted (+6.2%, +> [+5.8%, +7.2%]); its p99 CI was [0.07, 4.97] — uselessly wide on a +> single-worker loopback tail — so it was **rejected**, even though the +> median pointed the right way. A number the measurement can't pin down +> does not ship. diff --git a/docs/src/playbook.md b/docs/src/playbook.md new file mode 100644 index 0000000..28ef5b9 --- /dev/null +++ b/docs/src/playbook.md @@ -0,0 +1,33 @@ +# The playbook + +Optimizations are tried in a fixed order — **cheapest and safest first** — +because cost-per-attempt and risk both rise as you go down the list, and +the ledger stops you from re-grinding a class that already has a verdict. + +| # | Class | Mechanism | Verified on | +|---|---|---|---| +| 1 | Build config | LTO, codegen-units, target-cpu | tokei ✅ +10.4% | +| 2 | Allocator | Global allocator swap (mimalloc) | comrak ✅ +4.6% (human-ruled) | +| 3 | Allocation churn | Remove per-item malloc/free | tokei ✅, cJSON ✅ (+ one overturned) | +| 4 | Data layout | Make the cache hierarchy fit the access pattern | comrak (attempted) | +| 5 | Algorithmic | Eliminate redundant work | tokei ×3 ✅, cJSON ✅ | +| 6 | SIMD | Autovectorization enablement | comrak, tokei (attempted) | +| 7 | Concurrency | Parallelism — always `needs-human-review` | — | + +Six classes were exercised across the case studies, with verified wins on +classes 1, 2, 3, and 5 across **both a Rust and a C target**. + +## Why the ordering is load-bearing + +The cheapest class produced the project's single largest individual win: +LTO on tokei, **+10.4%**, a pure build-config change with no source edit. +The lesson the ledger enforces is the one every performance engineer learns +the hard way — *you do not hand-optimize a hot loop before you have turned +on the optimizer's own switches.* An agent that jumped straight to SIMD +would have missed a bigger, safer, free win sitting one config flag away. + +Each class entry (`playbook/0N-*.md`) carries its **preconditions** (the +profile signature that justifies trying it), its **procedure**, its +**verification** requirements, and its **known failure modes**. The agent +reads the profile, matches the signature, and picks the cheapest class +whose preconditions are met and that isn't already spent on this hotspot. diff --git a/docs/src/reproduce.md b/docs/src/reproduce.md new file mode 100644 index 0000000..3306f26 --- /dev/null +++ b/docs/src/reproduce.md @@ -0,0 +1,50 @@ +# Reproduce it yourself + +Nothing here is a screenshot. Every number regenerates from a pinned corpus +through the same commands. + +## Prerequisites + +- Rust (stable) + `just`; `clang`/`cmake` for the C target; `valgrind` for + callgrind profiling. Nightly Rust for the sanitizer gate. +- Targets are vendored fresh (their `workspace/` is gitignored); each + `targets//target.toml` has the `[fetch]` recipe. + +## The commands + +``` +just build / test / lint # trust-layer workspace +just aa [cmd] # A/A self-test — must be a null verdict +just calibrate # A/A false-positive + injection detection +just gates # corpus pin + upstream tests + golden replay +just verdict ... # gates + bench vs pristine baseline + ledger row +just report # ROI report from a ledger row +``` + +Service mode (Phase 4): + +``` +just service # interleaved p50/p99 CIs +just service-calibrate # latency A/A + injection +just report --service-json # latency + ROI report +``` + +The agent loop, behind OS isolation: + +``` +just isolation-check # 19 boundary checks, both modes +scripts/agent-isolated.sh # one unattended attempt, confined +``` + +## Auditing a result + +Every accepted win can be re-audited independently: + +``` +python3 scripts/audit-attempt.py # mechanical re-check +python3 scripts/diff-fuzz-.py 10000 # 10k fuzz +``` + +The ledger itself is the record: `results/ledger.sqlite`, append-only, +every attempt (win or loss) with its hypothesis, gates, bench evidence, and +verdict. diff --git a/docs/src/roadmap.md b/docs/src/roadmap.md new file mode 100644 index 0000000..82100ce --- /dev/null +++ b/docs/src/roadmap.md @@ -0,0 +1,42 @@ +# Roadmap & scope + +## Done — the core loop, end to end + +| Phase | Scope | Result | +|---|---|---| +| **0** | Manual dry run, two Rust targets | tokei +10.4% verified; methodology written | +| **1** | Trust layer (bench-runner, diff-test, ledger) | A/A <5% FP, injection ≥95%, gates on 2 targets; calibrated free on GitHub runners | +| **2** | Unattended agent loop | 20 audited attempts, 3 verified wins, **0 shipped false accepts**, OS-isolated | +| **3** | Playbook + proof, C/C++ | cross-language cJSON win (+8.85%), 4 case studies, 6 playbook classes | +| **4** | Services & scale | cJSON service +6.2% p50 under CO-correct replay, mechanical ROI report | + +The `profile → hypothesize → patch → gated-verdict → ROI` loop runs +unattended, OS-isolated, across **Rust and C**, **batch and service**, from +a pinned corpus to a dollar figure — with zero shipped false accepts across +all phases, and the two pipeline over-accepts both caught by audit and +turned into permanent gates. + +## Frontier — the research forks (SPEC §13) + +Deliberately out of the core roadmap, these are where the largest gaps +live: + +- **GPU kernel lane.** Triton/CUDA targets where 2–10× gaps are common; + correctness via reference-kernel differential testing; reward = measured + kernel time. The trust machinery (differential equivalence, interleaved + timing, append-only ledger) transfers directly; the profiler and bench + become kernel-time-aware, and it needs GPU hardware to run. +- **Learned optimization policies.** The ledger is a growing dataset of + `(profile signature, class, hypothesis) → verdict`. That is training data + for a policy that *ranks* which class to try first on a new hotspot, + turning the fixed cheapest-first ordering into a learned prior. Prior art: + MLGO, AlphaDev, Meta's LLM Compiler; ProGraML-style IR graphs as the + program representation. + +## The real target + +The end goal is not vendored OSS case studies — it is a **real production +system**. The same gates that matter for a JSON parser matter far more for +a live CMS: an equivalence failure there is a customer-facing bug, and the +no-network sandbox and append-only audit trail are exactly what make it +safe to point an optimization agent at code that a business depends on. diff --git a/docs/src/roi.md b/docs/src/roi.md new file mode 100644 index 0000000..969b8a6 --- /dev/null +++ b/docs/src/roi.md @@ -0,0 +1,39 @@ +# ROI: from stopwatch to dollars + +The report is the pitch artifact, and it answers exactly one question: +*what did the stopwatch say, and what does that buy?* It is generated +**mechanically** — `just report ` reads a ledger row and the +calibrated bench and renders the numbers; nothing is hand-typed. + +## The three conversions + +- **Throughput → cores → dollars.** A speedup `s` on a CPU-bound fleet of + `N` cores returns `N × (1 − 1/s)` cores; at a `$/core-hour` rate that is + an annual saving. Both flow through the speedup's confidence interval, so + the report quotes a *range*. +- **Latency → percentile deltas → value.** Under replayed load, the p50/p99 + latency speedups with their CIs; a p50 service-time reduction maps to + throughput capacity for a CPU-bound service. +- **Batch windows → wall-clock hours** returned to a schedule. + +## The rule that makes it survive review + +Every figure carries its CI and its workload, and the **methodology ships +inside the report** — the interleaved A/B, the pristine baseline, the +bootstrap CI, the gates, the calibration. And the number the report tells +you to quote is the **CI lower bound**, never the median: + +> On a 500-core fleet at $0.04/core-hour, the cJSON service win (+6.2% p50 +> latency) returns **27.5 cores / $9,621 per year at the CI lower bound** +> (median 29.4 / $10,290). The point estimate is not a promise. + +## What the report refuses to do + +It will not mint a clean dollar figure for a change that wasn't shipped. A +row whose verdict isn't `accepted`, **or** an `accepted` row that isn't +sanitizer-clean (a pre-gate accept the audit overturned), gets a bold **Not +shipped** banner before any number. That guard was itself added by the +[adversarial review](https://github.com/bdbrown4/peltier.io/blob/main/results/adversarial-review.md), +which caught the generator pitching the overturned comrak patch. The pitch +artifact is held to the same standard as everything else: it cannot claim +what wasn't proven. diff --git a/docs/src/thesis.md b/docs/src/thesis.md new file mode 100644 index 0000000..78c78e9 --- /dev/null +++ b/docs/src/thesis.md @@ -0,0 +1,48 @@ +# The thesis: trust is the product + +Performance work has a credibility problem. "We made it 15% faster" is +almost always some mix of a noisy benchmark, a workload that flatters the +change, and a behavioral difference nobody checked. The number is real to +the person who measured it and worthless to everyone downstream. + +hotpath is built on the opposite premise: **a performance claim is a +liability until it is proven, and the proof ships with the number.** + +## Three non-negotiables + +1. **No unverified performance claims.** A change without passing + equivalence gates *and* a statistically significant benchmark delta + does not exist. It is discarded and logged. +2. **Measure before optimizing.** No patch is proposed until profile data + identifies the hot path. A single benchmark run is never trusted. +3. **Cheap wins first.** Build flags → LTO → PGO → allocator swap → *then* + code changes. The ledger prevents re-attempting a dead end. + +## What "proven" means here + +A win must clear **both** of two independent bars, because either one alone +is gameable: + +- **Equivalence** stops "optimizations" that are actually behavior changes. + A JSON parser that's faster because it stopped validating numbers is not + faster; it's broken. Byte-identical golden replay and differential + fuzzing catch this. +- **Significance** stops noise from masquerading as a win. The bar is not + "the median improved" — it is "the bootstrap 95% CI *lower bound* clears + the threshold," measured by interleaving baseline and candidate on pinned + hardware that has passed its own A/A calibration. + +A real +5.7% median that couldn't clear the significance bar (its CI lower +bound was +0.2%) was **rejected** — and that rejection is in the ledger, because +a rejection is a complete, valid outcome. + +## Why an agent + +The loop — profile, hypothesize, patch, gate, measure, record — is +mechanical and tedious, and the temptation to fudge a number is exactly +what a tired human succumbs to. An agent proposes; the *trust layer* +disposes. The agent cannot write the ledger, cannot move the baseline, +cannot touch the thresholds. It can only propose a patch and ask the +harness to judge it. The [architecture](./architecture.md) is designed so +that even a compromised or adversarial proposer cannot manufacture a false +win. diff --git a/docs/theme/custom.css b/docs/theme/custom.css new file mode 100644 index 0000000..4816833 --- /dev/null +++ b/docs/theme/custom.css @@ -0,0 +1,20 @@ +/* hotpath docs — small refinements over the default mdBook theme. */ + +.content h1 { + border-bottom: 2px solid var(--links); + padding-bottom: 0.2em; +} + +/* Verified-number tables read as evidence: tighten and emphasize. */ +.content table td:first-child { + font-weight: 500; +} + +blockquote { + border-inline-start: 4px solid var(--links); +} + +/* Inline verdict-ish code (accepted / rejected-bench / needs-human-review) */ +.content code { + font-variant-ligatures: none; +}