mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
docs: full mdBook site + GitHub Pages deploy; rewrite README as project front page
- docs/: an mdBook documentation site — thesis, architecture, measurement
discipline, equivalence gates, agent loop, isolation/anti-reward-hacking,
playbook, ROI, reproduce, roadmap, and the five case studies (pulled from
their single source in results/ via {{#include}}). Builds clean (21
pages).
- .github/workflows/docs.yml: builds mdBook and deploys to GitHub Pages on
push to main (one-time setup: Settings → Pages → Source = GitHub Actions).
- README.md: rewritten as the project front page — the thesis, the verified
results table, layout, quick start, how it stays honest, links to the
book. Replaces the stale Phase-0-era text.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
This commit is contained in:
@@ -0,0 +1,27 @@
|
||||
# Summary
|
||||
|
||||
[hotpath](./introduction.md)
|
||||
|
||||
- [The thesis: trust is the product](./thesis.md)
|
||||
- [Architecture](./architecture.md)
|
||||
- [Measurement discipline](./measurement.md)
|
||||
- [Equivalence gates](./equivalence.md)
|
||||
- [The agent loop](./agent-loop.md)
|
||||
- [Isolation & anti-reward-hacking](./isolation.md)
|
||||
- [The playbook](./playbook.md)
|
||||
- [ROI: from stopwatch to dollars](./roi.md)
|
||||
|
||||
# Case studies
|
||||
|
||||
- [Overview](./case-studies/overview.md)
|
||||
- [tokei — compounding algorithmic wins (Rust)](./case-studies/tokei.md)
|
||||
- [Cheap wins first: build flags & allocators](./case-studies/build-flags.md)
|
||||
- [The caught false-accept (comrak)](./case-studies/comrak-false-accept.md)
|
||||
- [cJSON — the cross-language proof (C)](./case-studies/cjson.md)
|
||||
- [Services & scale: latency under load](./case-studies/service.md)
|
||||
|
||||
# Reference
|
||||
|
||||
- [Reproduce it yourself](./reproduce.md)
|
||||
- [The trust boundary (infra)](./infra.md)
|
||||
- [Roadmap & scope](./roadmap.md)
|
||||
@@ -0,0 +1,52 @@
|
||||
# The agent loop
|
||||
|
||||
The proposer is a Claude Agent SDK loop (`agent/hotpath_agent/`) driving
|
||||
the seven harness tools over a standalone MCP server. It has no built-in
|
||||
file, shell, or web tools in its allowlist — only the harness door.
|
||||
|
||||
## One attempt, start to finish
|
||||
|
||||
1. **Read the ledger.** What classes and hypotheses were already tried on
|
||||
this target? The ledger is the anti-double-attempt memory: a class may
|
||||
be re-entered with a *materially new* hypothesis, never a duplicate.
|
||||
2. **Read the profile.** callgrind cache-sim ranks hotspots with per-
|
||||
function cache-miss and branch-mispredict columns, so the agent can tell
|
||||
an allocation-churn hotspot from a branch-bound one and pick the right
|
||||
playbook class.
|
||||
3. **Pick the cheapest untried class** whose profile preconditions are met,
|
||||
and read its playbook entry.
|
||||
4. **Read the source** (windowed, at the pinned commit) and **propose a
|
||||
patch** with a stated hypothesis.
|
||||
5. **Run the verdict** and **poll** for the result.
|
||||
6. **Report it verbatim.** A rejection with a clean ledger row is a
|
||||
*successful* outcome. Do not iterate more than twice on a rejected
|
||||
hypothesis without a new idea.
|
||||
|
||||
## What the loop proved about itself
|
||||
|
||||
Across an unattended grind of 20+ audited attempts, the loop demonstrated
|
||||
the behavior the architecture is designed to produce:
|
||||
|
||||
- It chose the cheapest untried class each time, reading the ledger to
|
||||
avoid re-grinding a dead end.
|
||||
- Every patch went through `propose_patch` (path-allowlist + `git apply`)
|
||||
and the full gate+bench pipeline. Malformed diffs were refused by the
|
||||
harness, not the agent's goodwill.
|
||||
- **It twice refused to fabricate a verdict it could not read back.** When
|
||||
a multi-minute pipeline outran the MCP transport's timeout, the agent
|
||||
checked the ledger, confirmed the row wasn't written, and reported the
|
||||
verdict as *unreadable, not invented* — asking a human to read the
|
||||
ledger rather than guessing a number. That refusal, under a real failure
|
||||
mode, is the anti-reward-hacking design working.
|
||||
|
||||
## The honest-failure record
|
||||
|
||||
The ledger records **every** attempt, including the rejections, because a
|
||||
rejection is training data and a demonstration of the bar. Of the audited
|
||||
unattended attempts, most were honest rejections — real mechanisms that
|
||||
simply didn't clear the 2% CI-lower-bound bar — a handful were verified
|
||||
wins, and one was a `rejected-gate` where golden replay caught the agent's
|
||||
patch producing wrong output that a green test suite would have shipped.
|
||||
|
||||
The load-bearing number is **zero shipped false accepts** across the whole
|
||||
project.
|
||||
@@ -0,0 +1,61 @@
|
||||
# Architecture
|
||||
|
||||
hotpath is split into a **trust layer** (Rust, the agent cannot write it)
|
||||
and an **untrusted proposer** (the agent). The split is structural, not
|
||||
advisory: the guarantees hold even if the proposer is adversarial.
|
||||
|
||||
```
|
||||
crates/ Trust layer — agent has NO write access (SPEC §10)
|
||||
bench-runner/ Interleaved A/B timing, bootstrap CIs, A/A calibration,
|
||||
service-mode latency (coordinated-omission correct)
|
||||
diff-test/ Equivalence gates, corpus hash-pinning, per-target spec
|
||||
ledger/ Append-only SQLite attempt ledger (enforced by triggers)
|
||||
report/ ROI: speedup CI → cores → dollars, methodology inline
|
||||
verdict/ The pipeline in one command: gates → bench → ledger row
|
||||
harnessd/ The one door the agent talks through (7 JSON ops)
|
||||
agent/ Untrusted proposer (Claude Agent SDK, Python)
|
||||
playbook/ Optimization classes 1–7, tried strictly cheapest-first
|
||||
config/ accept.toml (thresholds), pricing.toml (ROI inputs)
|
||||
targets/ Vendored OSS targets — the only agent-writable path
|
||||
corpora/ Hash-pinned golden-replay inputs (read-only to agent)
|
||||
results/ Calibration evidence, case studies, generated reports
|
||||
```
|
||||
|
||||
## The seven-tool boundary
|
||||
|
||||
The agent never runs a shell, never writes a file, never touches the
|
||||
ledger directly. It speaks to `harnessd` — the only door — through exactly
|
||||
seven JSON operations:
|
||||
|
||||
| Tool | What it can do |
|
||||
|---|---|
|
||||
| `read_profile` | Read a target's ranked hotspots (read-only) |
|
||||
| `read_ledger` | Read prior attempts — the anti-double-attempt memory |
|
||||
| `read_playbook` | Read an optimization class's preconditions/procedure |
|
||||
| `read_target_source` | Read a workspace file at the pinned commit (windowed) |
|
||||
| `propose_patch` | Submit a unified diff — path-allowlisted, then `git apply` |
|
||||
| `run_verdict` | Launch the gate+bench pipeline (detached) |
|
||||
| `read_verdict` | Poll the append-only ledger for the result |
|
||||
|
||||
Note what is *absent*: no write outside `targets/<name>/workspace`, no
|
||||
shell, no way to run the bench or write the ledger except through the
|
||||
`verdict` binary, which only records a pass after the full gated pipeline.
|
||||
|
||||
## The data flow of one attempt
|
||||
|
||||
```
|
||||
profile ─▶ agent hypothesizes ─▶ propose_patch ─▶ [path allowlist + git apply]
|
||||
│
|
||||
ledger row ◀─ verdict ◀─ bench (interleaved A/B) ◀─ gates (tests, golden,
|
||||
(append-only) vs pristine-rebuilt fuzz, sanitizers)
|
||||
baseline
|
||||
```
|
||||
|
||||
The baseline is **rebuilt from a pristine checkout every session** — never
|
||||
from the agent's workspace — so a patch cannot poison its own comparison.
|
||||
The verdict is decided by the CI-lower-bound rule and written to an
|
||||
append-only ledger the agent cannot mutate.
|
||||
|
||||
The next chapters walk each stage: [measurement](./measurement.md),
|
||||
[equivalence gates](./equivalence.md), [the agent loop](./agent-loop.md),
|
||||
and the [isolation](./isolation.md) that makes the boundary load-bearing.
|
||||
@@ -0,0 +1 @@
|
||||
{{#include ../../../results/phase3/02-build-flags-and-allocators.md}}
|
||||
@@ -0,0 +1 @@
|
||||
{{#include ../../../results/phase3/04-cjson-c-target.md}}
|
||||
@@ -0,0 +1 @@
|
||||
{{#include ../../../results/phase3/03-comrak-caught-false-accept.md}}
|
||||
@@ -0,0 +1,17 @@
|
||||
# Case studies
|
||||
|
||||
Five verified case studies, each reproducible from a pinned corpus through
|
||||
the same gated pipeline, each number carrying its 95% CI and workload.
|
||||
|
||||
- [tokei — compounding algorithmic wins (Rust)](./tokei.md) — three class-5
|
||||
wins on the same hot function, each unattended and audited.
|
||||
- [Cheap wins first: build flags & allocators](./build-flags.md) — LTO
|
||||
accepted (+10%), mimalloc held for a human ruling then accepted-scoped.
|
||||
- [The caught false-accept (comrak)](./comrak-false-accept.md) — the
|
||||
leaking teardown the bench loved, overturned by audit. The most important
|
||||
study.
|
||||
- [cJSON — the cross-language proof (C)](./cjson.md) — a sub-threshold win
|
||||
rejected, then a byte-identical win accepted, on a C target.
|
||||
- [Services & scale: latency under load](./service.md) — the cJSON win
|
||||
measured as a latency delta under coordinated-omission-correct replay,
|
||||
turned into a dollar figure mechanically.
|
||||
@@ -0,0 +1 @@
|
||||
{{#include ../../../results/phase4/README.md}}
|
||||
@@ -0,0 +1 @@
|
||||
{{#include ../../../results/phase3/01-tokei-algorithmic-rust.md}}
|
||||
@@ -0,0 +1,52 @@
|
||||
# Equivalence gates
|
||||
|
||||
A faster binary that behaves differently is not an optimization — it is a
|
||||
regression with good marketing. Before any speedup is even measured, the
|
||||
candidate must prove it does the same thing the baseline does. These are
|
||||
**hard gates**: a failure is `rejected-gate` and the bench never runs.
|
||||
|
||||
## The layers
|
||||
|
||||
1. **Upstream test suite.** The target's own tests, run against the patched
|
||||
code. For cJSON that is the vendored Unity/CTest suite (19 tests:
|
||||
`parse_number`, `parse_string`, `print_*`, `misc_tests`, …); for the Rust
|
||||
targets, `cargo test`.
|
||||
|
||||
2. **Golden replay — byte-identical.** The candidate processes the
|
||||
hash-pinned corpus and its output is hashed and compared to a pinned
|
||||
`GOLDEN.sha256`. A code counter that miscounts one line, a JSON printer
|
||||
that serializes one float differently — caught here. The corpus itself
|
||||
is pinned via `MANIFEST.sha256` and the harness refuses to run on a
|
||||
mismatch.
|
||||
|
||||
3. **Differential fuzzing.** Pristine vs candidate on thousands of mutated
|
||||
inputs, comparing output. Every accepted win cleared **10,000 mutated
|
||||
inputs with 0 divergences** — the tokei wins after canonicalizing benign
|
||||
parallel-walker output ordering (a real finding: differential gates must
|
||||
compare *semantics*, not bytes, where the target is legitimately
|
||||
nondeterministic in presentation), the cJSON win on quote/number/escape
|
||||
edge cases.
|
||||
|
||||
4. **Sanitizers.** ASan/LSan for Rust, ASan+UBSan for C. Run on **every
|
||||
would-be accept** — this is machine-enforced, not manual (see below).
|
||||
|
||||
## The tier rule (SPEC §8)
|
||||
|
||||
Some changes can pass every gate and still be unsafe to auto-ship. A change
|
||||
that touches floating-point ordering, concurrency primitives, `unsafe`, or
|
||||
carries a sanitizer flag is **never** auto-accepted — its verdict is capped
|
||||
at `needs-human-review`. The tier is decided by the *nature* of the change,
|
||||
never by the size of the number. The mimalloc allocator swap
|
||||
([case study](./case-studies/build-flags.md)) is the canonical example: a
|
||||
real +4.6% win, correctly held for a human ruling because a global
|
||||
allocator change's safety is not something one benchmark can clear.
|
||||
|
||||
## When a gate caught the pipeline itself
|
||||
|
||||
The sanitizer gate exists in its current, machine-enforced form because it
|
||||
once *wasn't*. The pipeline auto-accepted a comrak patch that skipped AST
|
||||
teardown for a +10.7% win — and the audit's manual LeakSanitizer run found
|
||||
it leaks the arena at exit. It was overturned before shipping and the fix
|
||||
made `verdict` build with ASan/UBSan and run the pinned workload on every
|
||||
accept, capping any flagged win at `needs-human-review`. The full story is
|
||||
[the caught false-accept](./case-studies/comrak-false-accept.md).
|
||||
@@ -0,0 +1 @@
|
||||
{{#include ../../docs/infra.md}}
|
||||
@@ -0,0 +1,46 @@
|
||||
# hotpath
|
||||
|
||||
> Spot cooling for hot paths. A profile-guided optimization agent that
|
||||
> only ships wins it can prove.
|
||||
|
||||
hotpath profiles real binaries and services, isolates hot paths, proposes
|
||||
optimizations, and accepts a change **only** when two independent bars are
|
||||
cleared:
|
||||
|
||||
1. **Behavioral equivalence** — the change provably preserves behavior
|
||||
(upstream tests, byte-identical golden replay, differential fuzzing,
|
||||
sanitizers) or it is discarded.
|
||||
2. **Statistical significance** — the speedup's bootstrap 95% CI lower
|
||||
bound clears a threshold on trusted, calibrated hardware.
|
||||
|
||||
Everything else is commentary. A change without both is not a win; it is a
|
||||
ledger row explaining why not.
|
||||
|
||||
## What it has actually done
|
||||
|
||||
Every number here carries its 95% confidence interval and its workload,
|
||||
and every one survived the gates above. These are not projections.
|
||||
|
||||
| Result | Verified |
|
||||
|---|---|
|
||||
| tokei (Rust) — three compounding class-5 algorithmic wins | +10.7%, +3.7%, +9.9%, each CI-significant |
|
||||
| tokei — link-time optimization (class 1) | +10.4%, CI [+8.5%, +12.0%] |
|
||||
| cJSON (C) — number-handling rewrite (class 5) | +8.85%, CI [+7.5%, +10.0%] |
|
||||
| cJSON HTTP service — p50 latency under replayed load | +6.2%, CI [+5.8%, +7.2%] |
|
||||
| comrak (Rust) — mimalloc allocator swap (class 2) | +4.6%, CI [+3.3%, +5.8%], human-ruled |
|
||||
|
||||
Across **five phases and 33 ledger rows**, with **zero shipped false
|
||||
accepts** — including two occasions where the pipeline itself over-accepted
|
||||
and the audit caught it before anything shipped, each becoming a permanent
|
||||
new gate.
|
||||
|
||||
## The differentiator
|
||||
|
||||
Consultants sell surveys; compilers sell flags. hotpath sells **verified
|
||||
deltas with the methodology attached** — a stopwatch on pinned hardware,
|
||||
a proof of equivalence, and an ROI figure whose confidence interval you
|
||||
can quote to a hostile reviewer. That is the entire product, and this site
|
||||
is how it works.
|
||||
|
||||
Start with [the thesis](./thesis.md), or jump to a
|
||||
[case study](./case-studies/overview.md).
|
||||
@@ -0,0 +1,53 @@
|
||||
# Isolation & anti-reward-hacking
|
||||
|
||||
The guarantees only matter if the proposer cannot cheat. hotpath's defense
|
||||
is **structural**: the agent is denied the capabilities it would need to
|
||||
manufacture a false win, at the OS level, not by prompt.
|
||||
|
||||
## What holds even if the proposer is adversarial
|
||||
|
||||
- **Writes go through one door.** `propose_patch` is the only harness-
|
||||
mediated write path: a diff is path-allowlisted (relative paths only, no
|
||||
`..`, no `.git`, nothing outside `targets/<name>/workspace`) and then
|
||||
`git apply`-ed rooted in that workspace. A patch aimed at `config/` or
|
||||
`crates/` is refused before any apply. Verified against a malicious
|
||||
accept-threshold diff.
|
||||
- **The ledger is append-only.** `UPDATE` and `DELETE` are refused by
|
||||
SQLite triggers; `accepted` rows are written only by the `verdict`
|
||||
binary after the gated pipeline. The agent cannot forge a passing
|
||||
verdict through the harness.
|
||||
- **The baseline is unbypassable.** It is rebuilt from a pristine checkout;
|
||||
the agent's workspace never becomes the comparison.
|
||||
|
||||
## OS-level isolation (SPEC §10)
|
||||
|
||||
The SDK's tool allow/deny list is defense-in-depth, not the boundary —
|
||||
`scripts/agent-isolated.sh` provides the real one, verified by
|
||||
`just isolation-check` (19 checks, both modes):
|
||||
|
||||
- **`mountns` mode** — the whole agent process tree runs in a private mount
|
||||
namespace with the repo bind-mounted **read-only** and `CAP_SYS_ADMIN`/
|
||||
`CAP_SYS_PTRACE` dropped from the bounding set. Repo writes fail with
|
||||
`EROFS`; a forged `INSERT`, a direct `verdict` invocation, a `mount -o
|
||||
remount,rw`, and even userns re-entry to undo the mount are all refused.
|
||||
- **`user` mode** — the agent runs as an unprivileged user whose only write
|
||||
path into the repo is the harnessd Unix socket; the trust layer is owned
|
||||
by another uid. This is the production shape (`docs/infra.md`).
|
||||
|
||||
## The two times the pipeline was wrong
|
||||
|
||||
Anti-reward-hacking is not "it never fails" — it is "when it fails, the
|
||||
audit catches it and the system gets harder." That happened twice, and
|
||||
both are documented, not hidden:
|
||||
|
||||
1. A leaking teardown patch the bench loved was **auto-accepted** and then
|
||||
**overturned** by the 100% human audit's LeakSanitizer run. The fix made
|
||||
the sanitizer gate machine-enforced on every accept.
|
||||
2. During the [adversarial review](https://github.com/bdbrown4/peltier.io/blob/main/results/adversarial-review.md),
|
||||
the ROI report generator was found to mint a clean dollar figure for that
|
||||
same overturned row (it trusted the immutable ledger's historical
|
||||
`accepted` verdict). The fix made the report treat an accepted-but-not-
|
||||
sanitizer-clean row as **not shippable**.
|
||||
|
||||
Both defects were found by attacking the system's own output, fixed, and
|
||||
turned into permanent gates. **Zero false accepts shipped.**
|
||||
@@ -0,0 +1,64 @@
|
||||
# Measurement discipline
|
||||
|
||||
The product is a trustworthy number, so the measurement is where most of
|
||||
the engineering went. The reference is Mytkowicz et al., *Producing Wrong
|
||||
Data Without Doing Anything Obviously Wrong* — the default state of a
|
||||
benchmark is "subtly lying."
|
||||
|
||||
## Interleaved A/B, never sequential
|
||||
|
||||
Baseline and candidate run **interleaved** (ABABAB…), never in sequential
|
||||
blocks. Thermal drift, a background cron job, a noisy neighbor — anything
|
||||
that changes over the run hits both sides equally instead of biasing one.
|
||||
Warm-up runs are discarded.
|
||||
|
||||
## The baseline is always pristine
|
||||
|
||||
The baseline binary is **rebuilt from a clean checkout** each session, with
|
||||
the candidate's patch stashed. A patch can never silently become part of
|
||||
its own baseline (a real bug that bit an early smoke test, now closed by
|
||||
`verdict --rebuild-baseline`).
|
||||
|
||||
## Bootstrap CIs, and the lower-bound rule
|
||||
|
||||
Each side yields ≥30 measured samples. The statistic is the **bootstrap
|
||||
95% CI of the ratio-of-medians** (deterministic xorshift seed, so a result
|
||||
reproduces). A change is accepted only if the **CI lower bound** clears the
|
||||
threshold in `config/accept.toml` (default 2%) — not the median. The number
|
||||
you commit to is the one that survives the interval.
|
||||
|
||||
## The hardware has to earn trust first
|
||||
|
||||
Before any measurement session, the harness runs two self-tests, recorded
|
||||
in `results/calibration/`:
|
||||
|
||||
- **A/A**: run the *same* binary as both sides. It must produce a null
|
||||
verdict. A "speedup" from identical binaries is a calibration failure.
|
||||
- **Regression injection**: a synthetic 5% slowdown (a real busy-wait in
|
||||
the timed window) must be detected ≥95% of the time.
|
||||
|
||||
CPU pinning (`taskset -c 2`) alone tightened the A/A confidence interval
|
||||
~5× on the dev container. Every calibration anchor — local and on GitHub's
|
||||
shared runners — passed both bars before its numbers were trusted.
|
||||
|
||||
## Service mode: coordinated omission
|
||||
|
||||
Phase 4 measures latency under load, where the classic trap is
|
||||
**coordinated omission**: a closed-loop client that waits for a slow
|
||||
response simply sends fewer requests during the stall, so the stall never
|
||||
appears in the latency histogram. hotpath's load generator is **open-loop**
|
||||
— requests are scheduled at a fixed rate and each latency is measured from
|
||||
its *intended* send time. When the server falls behind, the requests queued
|
||||
behind it each carry the full delay a real client would have seen. Nothing
|
||||
is omitted.
|
||||
|
||||
The service measurement is calibrated the same way: 0/10 A/A false
|
||||
positives, 10/10 injected-5%-latency-regression detections
|
||||
(`results/calibration/cjson-service-aa.json`).
|
||||
|
||||
> **Percentiles are claimed only when the measurement can pin them down.**
|
||||
> The cJSON service's p50 latency win was CI-tight and accepted (+6.2%,
|
||||
> [+5.8%, +7.2%]); its p99 CI was [0.07, 4.97] — uselessly wide on a
|
||||
> single-worker loopback tail — so it was **rejected**, even though the
|
||||
> median pointed the right way. A number the measurement can't pin down
|
||||
> does not ship.
|
||||
@@ -0,0 +1,33 @@
|
||||
# The playbook
|
||||
|
||||
Optimizations are tried in a fixed order — **cheapest and safest first** —
|
||||
because cost-per-attempt and risk both rise as you go down the list, and
|
||||
the ledger stops you from re-grinding a class that already has a verdict.
|
||||
|
||||
| # | Class | Mechanism | Verified on |
|
||||
|---|---|---|---|
|
||||
| 1 | Build config | LTO, codegen-units, target-cpu | tokei ✅ +10.4% |
|
||||
| 2 | Allocator | Global allocator swap (mimalloc) | comrak ✅ +4.6% (human-ruled) |
|
||||
| 3 | Allocation churn | Remove per-item malloc/free | tokei ✅, cJSON ✅ (+ one overturned) |
|
||||
| 4 | Data layout | Make the cache hierarchy fit the access pattern | comrak (attempted) |
|
||||
| 5 | Algorithmic | Eliminate redundant work | tokei ×3 ✅, cJSON ✅ |
|
||||
| 6 | SIMD | Autovectorization enablement | comrak, tokei (attempted) |
|
||||
| 7 | Concurrency | Parallelism — always `needs-human-review` | — |
|
||||
|
||||
Six classes were exercised across the case studies, with verified wins on
|
||||
classes 1, 2, 3, and 5 across **both a Rust and a C target**.
|
||||
|
||||
## Why the ordering is load-bearing
|
||||
|
||||
The cheapest class produced the project's single largest individual win:
|
||||
LTO on tokei, **+10.4%**, a pure build-config change with no source edit.
|
||||
The lesson the ledger enforces is the one every performance engineer learns
|
||||
the hard way — *you do not hand-optimize a hot loop before you have turned
|
||||
on the optimizer's own switches.* An agent that jumped straight to SIMD
|
||||
would have missed a bigger, safer, free win sitting one config flag away.
|
||||
|
||||
Each class entry (`playbook/0N-*.md`) carries its **preconditions** (the
|
||||
profile signature that justifies trying it), its **procedure**, its
|
||||
**verification** requirements, and its **known failure modes**. The agent
|
||||
reads the profile, matches the signature, and picks the cheapest class
|
||||
whose preconditions are met and that isn't already spent on this hotspot.
|
||||
@@ -0,0 +1,50 @@
|
||||
# Reproduce it yourself
|
||||
|
||||
Nothing here is a screenshot. Every number regenerates from a pinned corpus
|
||||
through the same commands.
|
||||
|
||||
## Prerequisites
|
||||
|
||||
- Rust (stable) + `just`; `clang`/`cmake` for the C target; `valgrind` for
|
||||
callgrind profiling. Nightly Rust for the sanitizer gate.
|
||||
- Targets are vendored fresh (their `workspace/` is gitignored); each
|
||||
`targets/<name>/target.toml` has the `[fetch]` recipe.
|
||||
|
||||
## The commands
|
||||
|
||||
```
|
||||
just build / test / lint # trust-layer workspace
|
||||
just aa [cmd] # A/A self-test — must be a null verdict
|
||||
just calibrate <cmd> <out> # A/A false-positive + injection detection
|
||||
just gates <target> # corpus pin + upstream tests + golden replay
|
||||
just verdict <t> <bin> ... # gates + bench vs pristine baseline + ledger row
|
||||
just report <run-id> # ROI report from a ledger row
|
||||
```
|
||||
|
||||
Service mode (Phase 4):
|
||||
|
||||
```
|
||||
just service <baseline-bin> <candidate-bin> <doc> # interleaved p50/p99 CIs
|
||||
just service-calibrate <server> <doc> <out> # latency A/A + injection
|
||||
just report <run-id> --service-json <json> # latency + ROI report
|
||||
```
|
||||
|
||||
The agent loop, behind OS isolation:
|
||||
|
||||
```
|
||||
just isolation-check # 19 boundary checks, both modes
|
||||
scripts/agent-isolated.sh <target> <run-id> # one unattended attempt, confined
|
||||
```
|
||||
|
||||
## Auditing a result
|
||||
|
||||
Every accepted win can be re-audited independently:
|
||||
|
||||
```
|
||||
python3 scripts/audit-attempt.py <run-id> # mechanical re-check
|
||||
python3 scripts/diff-fuzz-<target>.py <dir> 10000 <candidate-bin> # 10k fuzz
|
||||
```
|
||||
|
||||
The ledger itself is the record: `results/ledger.sqlite`, append-only,
|
||||
every attempt (win or loss) with its hypothesis, gates, bench evidence, and
|
||||
verdict.
|
||||
@@ -0,0 +1,42 @@
|
||||
# Roadmap & scope
|
||||
|
||||
## Done — the core loop, end to end
|
||||
|
||||
| Phase | Scope | Result |
|
||||
|---|---|---|
|
||||
| **0** | Manual dry run, two Rust targets | tokei +10.4% verified; methodology written |
|
||||
| **1** | Trust layer (bench-runner, diff-test, ledger) | A/A <5% FP, injection ≥95%, gates on 2 targets; calibrated free on GitHub runners |
|
||||
| **2** | Unattended agent loop | 20 audited attempts, 3 verified wins, **0 shipped false accepts**, OS-isolated |
|
||||
| **3** | Playbook + proof, C/C++ | cross-language cJSON win (+8.85%), 4 case studies, 6 playbook classes |
|
||||
| **4** | Services & scale | cJSON service +6.2% p50 under CO-correct replay, mechanical ROI report |
|
||||
|
||||
The `profile → hypothesize → patch → gated-verdict → ROI` loop runs
|
||||
unattended, OS-isolated, across **Rust and C**, **batch and service**, from
|
||||
a pinned corpus to a dollar figure — with zero shipped false accepts across
|
||||
all phases, and the two pipeline over-accepts both caught by audit and
|
||||
turned into permanent gates.
|
||||
|
||||
## Frontier — the research forks (SPEC §13)
|
||||
|
||||
Deliberately out of the core roadmap, these are where the largest gaps
|
||||
live:
|
||||
|
||||
- **GPU kernel lane.** Triton/CUDA targets where 2–10× gaps are common;
|
||||
correctness via reference-kernel differential testing; reward = measured
|
||||
kernel time. The trust machinery (differential equivalence, interleaved
|
||||
timing, append-only ledger) transfers directly; the profiler and bench
|
||||
become kernel-time-aware, and it needs GPU hardware to run.
|
||||
- **Learned optimization policies.** The ledger is a growing dataset of
|
||||
`(profile signature, class, hypothesis) → verdict`. That is training data
|
||||
for a policy that *ranks* which class to try first on a new hotspot,
|
||||
turning the fixed cheapest-first ordering into a learned prior. Prior art:
|
||||
MLGO, AlphaDev, Meta's LLM Compiler; ProGraML-style IR graphs as the
|
||||
program representation.
|
||||
|
||||
## The real target
|
||||
|
||||
The end goal is not vendored OSS case studies — it is a **real production
|
||||
system**. The same gates that matter for a JSON parser matter far more for
|
||||
a live CMS: an equivalence failure there is a customer-facing bug, and the
|
||||
no-network sandbox and append-only audit trail are exactly what make it
|
||||
safe to point an optimization agent at code that a business depends on.
|
||||
@@ -0,0 +1,39 @@
|
||||
# ROI: from stopwatch to dollars
|
||||
|
||||
The report is the pitch artifact, and it answers exactly one question:
|
||||
*what did the stopwatch say, and what does that buy?* It is generated
|
||||
**mechanically** — `just report <run-id>` reads a ledger row and the
|
||||
calibrated bench and renders the numbers; nothing is hand-typed.
|
||||
|
||||
## The three conversions
|
||||
|
||||
- **Throughput → cores → dollars.** A speedup `s` on a CPU-bound fleet of
|
||||
`N` cores returns `N × (1 − 1/s)` cores; at a `$/core-hour` rate that is
|
||||
an annual saving. Both flow through the speedup's confidence interval, so
|
||||
the report quotes a *range*.
|
||||
- **Latency → percentile deltas → value.** Under replayed load, the p50/p99
|
||||
latency speedups with their CIs; a p50 service-time reduction maps to
|
||||
throughput capacity for a CPU-bound service.
|
||||
- **Batch windows → wall-clock hours** returned to a schedule.
|
||||
|
||||
## The rule that makes it survive review
|
||||
|
||||
Every figure carries its CI and its workload, and the **methodology ships
|
||||
inside the report** — the interleaved A/B, the pristine baseline, the
|
||||
bootstrap CI, the gates, the calibration. And the number the report tells
|
||||
you to quote is the **CI lower bound**, never the median:
|
||||
|
||||
> On a 500-core fleet at $0.04/core-hour, the cJSON service win (+6.2% p50
|
||||
> latency) returns **27.5 cores / $9,621 per year at the CI lower bound**
|
||||
> (median 29.4 / $10,290). The point estimate is not a promise.
|
||||
|
||||
## What the report refuses to do
|
||||
|
||||
It will not mint a clean dollar figure for a change that wasn't shipped. A
|
||||
row whose verdict isn't `accepted`, **or** an `accepted` row that isn't
|
||||
sanitizer-clean (a pre-gate accept the audit overturned), gets a bold **Not
|
||||
shipped** banner before any number. That guard was itself added by the
|
||||
[adversarial review](https://github.com/bdbrown4/peltier.io/blob/main/results/adversarial-review.md),
|
||||
which caught the generator pitching the overturned comrak patch. The pitch
|
||||
artifact is held to the same standard as everything else: it cannot claim
|
||||
what wasn't proven.
|
||||
@@ -0,0 +1,48 @@
|
||||
# The thesis: trust is the product
|
||||
|
||||
Performance work has a credibility problem. "We made it 15% faster" is
|
||||
almost always some mix of a noisy benchmark, a workload that flatters the
|
||||
change, and a behavioral difference nobody checked. The number is real to
|
||||
the person who measured it and worthless to everyone downstream.
|
||||
|
||||
hotpath is built on the opposite premise: **a performance claim is a
|
||||
liability until it is proven, and the proof ships with the number.**
|
||||
|
||||
## Three non-negotiables
|
||||
|
||||
1. **No unverified performance claims.** A change without passing
|
||||
equivalence gates *and* a statistically significant benchmark delta
|
||||
does not exist. It is discarded and logged.
|
||||
2. **Measure before optimizing.** No patch is proposed until profile data
|
||||
identifies the hot path. A single benchmark run is never trusted.
|
||||
3. **Cheap wins first.** Build flags → LTO → PGO → allocator swap → *then*
|
||||
code changes. The ledger prevents re-attempting a dead end.
|
||||
|
||||
## What "proven" means here
|
||||
|
||||
A win must clear **both** of two independent bars, because either one alone
|
||||
is gameable:
|
||||
|
||||
- **Equivalence** stops "optimizations" that are actually behavior changes.
|
||||
A JSON parser that's faster because it stopped validating numbers is not
|
||||
faster; it's broken. Byte-identical golden replay and differential
|
||||
fuzzing catch this.
|
||||
- **Significance** stops noise from masquerading as a win. The bar is not
|
||||
"the median improved" — it is "the bootstrap 95% CI *lower bound* clears
|
||||
the threshold," measured by interleaving baseline and candidate on pinned
|
||||
hardware that has passed its own A/A calibration.
|
||||
|
||||
A real +5.7% median that couldn't clear the significance bar (its CI lower
|
||||
bound was +0.2%) was **rejected** — and that rejection is in the ledger, because
|
||||
a rejection is a complete, valid outcome.
|
||||
|
||||
## Why an agent
|
||||
|
||||
The loop — profile, hypothesize, patch, gate, measure, record — is
|
||||
mechanical and tedious, and the temptation to fudge a number is exactly
|
||||
what a tired human succumbs to. An agent proposes; the *trust layer*
|
||||
disposes. The agent cannot write the ledger, cannot move the baseline,
|
||||
cannot touch the thresholds. It can only propose a patch and ask the
|
||||
harness to judge it. The [architecture](./architecture.md) is designed so
|
||||
that even a compromised or adversarial proposer cannot manufacture a false
|
||||
win.
|
||||
Reference in New Issue
Block a user