peltier.io/SPEC.md

hotpath — Engineering Spec v0.1

1. Mission & thesis

Performance engineering is scarce artisan labor. Compilers closed the gap at general codegen; nothing closed it at the application and infrastructure layer, where profile-guided rewrites, build configuration, data-layout changes, and kernel-level work still leave 10–90% on the floor. hotpath is an agent that harvests that gap with two properties no consultant offers: verified equivalence (a change provably preserves behavior or is discarded) and self-verifying ROI (the value claim is a stopwatch on trusted infrastructure, not a survey).

Why this wins:

• Empty niche. Between "the compiler" and "hire a perf consultant" there is no product. LLM codegen is crowded; verified optimization is not.
• The metric defends itself. Same inputs, same outputs, fewer cycles, with confidence intervals. No correlation-causation problem is possible by construction.
• The ledger compounds. Every accepted/rejected attempt with full evidence is future RL training data and, eventually, the substrate for learned optimization policies (see §13, research fork).

2. System overview

                        ┌────────────────────────────────────────────┐
                        │              TRUST LAYER                   │
                        │  (agent has NO write access to this box)   │
  ┌─────────┐  patch    │  ┌────────────┐  ┌───────────┐  ┌───────┐  │
  │  agent/ │──────────▶│  │ diff-test  │─▶│bench-runner│─▶│ledger │  │
  │ (Agent  │  via git  │  │ gates      │  │ interleaved│  │SQLite │  │
  │  SDK)   │  apply +  │  └────────────┘  │ A/B + CIs  │  └───┬───┘  │
  └────┬────┘  allowlist│                  └───────────┘      │      │
       │                └──────────────────────────────────────┼──────┘
       │ reads profiles, ledger, playbook                      ▼
       ▼                                                  ┌─────────┐
  ┌──────────┐                                            │ report/ │
  │ profiler │  perf / flamegraph / coz                   │  ROI    │
  │ adapter  │                                            └─────────┘
  └──────────┘

Loop: profile → rank hotspots → select playbook class → hypothesize → patch → gates → bench → verdict → ledger → repeat.

3. Component specs

3.1 bench-runner (the product is this crate)

Requirements:

• Runs baseline and candidate interleaved (ABABAB…), never sequential blocks, to control thermal and background drift.
• Baseline is rebuilt from a pristine checkout each session — never from agent workspace.
• Environment control: CPU pinning (taskset/cgroup cpuset), performance governor, turbo/SMT state recorded (disabled where root permits), ASLR handled explicitly (see §7), no-network container, warm-up runs discarded.
• Statistics: ≥30 measured runs per side (configurable); report median and bootstrap 95% CI of the ratio; effect accepted only if CI lower bound ≥ threshold (config/accept.toml, default 2%).
• A/A self-test mode: run the same binary as both sides; must produce a null verdict. Scheduled automatically before any measurement session.
• Regression-injection self-test: a synthetic 5% slowdown must be detected ≥95% of the time.
• Metrics captured per run: wall time, cycles, instructions, cache misses, branch misses (via perf stat), max RSS, and RAPL energy where available.
• Modes: (a) whole-program via hyperfine-style CLI timing; (b) microbench via criterion harness hooks; (c) service mode (Phase 4): wrk2 / vegeta load replay with coordinated-omission-aware latency recording (HdrHistogram).

Acceptance: passes A/A calibration at <5% false-positive rate and injected-regression detection at ≥95%, documented in results/calibration/.

3.2 diff-test (equivalence gates)

Layered, all must pass:

1. Upstream test suite of the target, unmodified, green.
2. Golden replay: recorded input corpus → byte-identical outputs (stdout, files, exit codes). FP-producing targets declare an explicit tolerance policy in targets/<name>/equivalence.toml; absent a policy, byte-identical is required.
3. Differential fuzzing of changed functions: old vs. new implementation compiled side by side, driven by proptest/cargo-fuzz (Rust) or libFuzzer (C/C++), outputs compared. Minimum 60s or 10k iterations per changed function.
4. Sanitizers on the candidate: ASan + UBSan always; TSan when the diff touches threads, atomics, or locks; MIRI for Rust targets where feasible.

Corpus and test files live outside agent-writable paths and are hash-pinned; diff-test refuses to run if hashes mismatch.

3.3 ledger

Append-only SQLite. One row per attempt: run_id, timestamp, target, target_commit, phase, hotspot (symbol + % of profile), playbook_class, hypothesis, patch (diff), gates {tests, golden, fuzz_iters, sanitizers}, bench {baseline_ci, candidate_ci, delta_ci, env_fingerprint}, verdict {accepted | rejected-gate | rejected-bench | needs-human-review}, tokens_spent, wall_time.

Nothing is ever deleted. The ledger is the audit trail, the anti-double-attempt memory, and the future training set.

3.4 profiler adapter

Phase 0–2: perf record + flamegraphs (inferno) + perf stat; hotspot ranking = exclusive samples by symbol with source mapping. Phase 3: add coz causal profiling — "what does speeding this line up buy end-to-end" — which is the literal bridge from cycles to ROI and the correct prioritizer for multi-threaded targets.

3.5 agent/

Claude Agent SDK (Python). Tools exposed to the model: read_profile, read_ledger, read_playbook, propose_patch(diff, hypothesis), run_verdict, read_target_source (read-only). Deliberately not exposed: shell on the host, writes outside targets/<name>/, any access to crates/, config/, corpora/.

Prompting spine: (1) state hypothesis before patching; (2) prefer the cheapest untried playbook class; (3) max two iterations per rejected hypothesis; (4) a rejection with a clean ledger row is a successful outcome.

3.6 report/

Consumes ledger rows, emits per-engagement ROI report:

• Throughput jobs: cores saved = fleet_cores × (1 − 1/speedup); $ = cores × $/core-hr × hours/yr (rates in config/pricing.toml).
• Services: p50/p95/p99 deltas under replayed traffic, with the business-value mapping left as a customer-supplied parameter and one worked example.
• Optional: joules saved (RAPL) for the sustainability line.
• Every figure carries CI + workload description + environment fingerprint. Caveats are printed on the report, not in an appendix.

4. Optimization playbook v0 (strict order)

1. Build configuration: opt level, -march/target-cpu, LTO (thin → fat), PGO, BOLT where applicable.
2. Allocator swap: mimalloc/jemalloc trial. (Classes 1–2 are the "your 30% was a linker flag" tier — always exhausted first.)
3. Allocation churn: hoisting, reuse, arenas, small-vector.
4. Data layout: struct packing/reordering, AoS→SoA, false-sharing fixes, cache blocking.
5. Algorithmic: complexity class, better std/library primitive, memoization, precomputation.
6. SIMD: autovectorization enablement first (aliasing, bounds, FP flags — FP flags trigger human review), intrinsics last.
7. Concurrency: contention reduction, sharding, lock-free only with needs-human-review.

Each class ships as playbook/NN-name.md: preconditions (profile signature that suggests it), procedure, verification notes, known failure modes.

5. Phase plan & exit criteria

Phase	Scope	Exit criteria
0 Manual dry run (1–2 wknd)	Two CPU-bound OSS targets; loop run by hand with Claude Code as copilot	≥1 verified win ≥10% on one target; written case study with methodology
1 Trust layer (3–4 wknd)	bench-runner + diff-test + ledger	A/A false-positive <5%; injected 5% regression caught ≥95%; gates run on 2 targets end-to-end
2 Agent (3–4 wknd)	Agent SDK loop, Rust targets first	Unattended profile→verdict loop on 1 target; ≥1 auto-accepted win; zero false accepts across ≥20 audited attempts
3 Playbook + proof	Classes 3–7, coz, C/C++ targets	3–5 public case studies; zero regressions shipped; playbook ≥6 classes
4 Services & scale	Load replay, latency mode, ROI reports	One real service workload optimized under replayed traffic; report generated mechanically

Language order: Rust first (criterion/proptest/MIRI make Phases 1–2 cheaper), C/C++ second (the richer legacy hunting ground), services third.

6. Target selection criteria (Phase 0–3)

CPU-bound (>70% user time on profile); meaningful test suite; existing benchmark or easily scripted workload; permissive license; active but not hyper-optimized (skip anything with a dedicated perf team); builds cleanly in container. Good hunting: codecs, parsers, serializers, image processing, compression, static-site generators, linters.

7. Statistical methodology notes

• Interleaved A/B with pristine-checkout baselines is the core defense against drift.
• Layout bias (Mytkowicz et al., ASPLOS'09: link order and env size alone swing results past "real" effects): mitigate by randomizing link order / ASLR across runs and aggregating, Stabilizer-style, rather than fixing one lucky layout. Document the chosen mode in the env fingerprint.
• Report ratios with bootstrap CIs; never means without spread; never a single run, ever.
• Service latency: wrk2 constant-throughput mode + HdrHistogram to avoid coordinated omission; never trust closed-loop load generators for p99 claims.

8. Equivalence policy tiers

• Auto-acceptable: bit-identical outputs across all gates, no FP-ordering changes, no concurrency-primitive changes, sanitizers clean.
• needs-human-review (never auto-accepted): FP reassociation/contraction or -ffast-math-class flags; any diff touching atomics, locks, memory ordering, or thread counts; unsafe blocks / raw pointer arithmetic introduced; anything the sanitizers flag even as warnings.
• Auto-rejected: any gate failure, any divergence under differential fuzz, patch outside allowlist.

9. ROI methodology (the pitch artifact)

The report answers one question: what did the stopwatch say, and what does that cost or buy? Throughput → cores → dollars via public cloud pricing; latency → percentile deltas under recorded production-shaped load, priced by the customer's own latency-value number; batch windows → wall-clock hours returned to the schedule. All three come with CIs and a workload statement. The methodology section ships inside every report so the number survives hostile review — that is the differentiation.

10. Security & anti-reward-hacking (load-bearing)

The agent's incentive is "make the number go up"; the design must make cheating structurally impossible, not merely discouraged:

• Trust layer (crates/, config/, corpora/, upstream tests) is read-only to the agent — enforced by filesystem permissions and a separate-process harness, not by prompt.
• Patches are applied by the harness via git apply after a path-allowlist check; the agent never holds a general shell on the host.
• Baselines rebuilt from pristine checkouts; corpora hash-pinned; diff-test refuses on mismatch.
• Target code executes only in no-network containers (bench containers additionally seccomp-restricted).
• Periodic human audit of accepted wins (100% in Phase 2, sampled thereafter); any false accept is a stop-the-line event.

11. Risk register

Risk	Mitigation
Measurement bias / noise	§7 discipline; A/A tests before every session; env fingerprints
Overfitting to the benchmark	Holdout workloads; Phase 4 uses recorded real traffic only
FP equivalence gaps	Explicit per-target tolerance policy; default byte-identical; human review tier
Concurrency bugs introduced	TSan gate; human-review tier for all concurrency diffs
UB introduced	UBSan always; MIRI for Rust; human-review for unsafe
Agent games the harness	§10 structural controls; audits
License/IP hygiene	Permissive-license targets only for public case studies
Wins too small to matter	Cheap-wins-first ordering keeps cost-per-attempt low; ledger prevents re-grinding dead ends

12. Tech stack

Rust workspace (bench-runner, diff-test, ledger, report) · just · hyperfine-style timing + criterion hooks · perf, inferno flamegraphs, coz · proptest, cargo-fuzz, libFuzzer · ASan/UBSan/TSan, MIRI · podman/docker (no-net) · wrk2 + HdrHistogram, vegeta · SQLite · Python 3.11+ with Claude Agent SDK (agent/) — confirm current SDK package name and auth at https://docs.claude.com/en/docs/claude-code/overview.

13. Later / research fork (explicitly out of scope now)

• GPU kernel lane (where 2–10x gaps live): Triton/CUDA targets, correctness via reference-kernel differential testing, reward = measured kernel time.
• Learned optimization policies: ledger → RL; DELTA-over-program-graphs (ProGraML-style IR graphs) as the policy/ranking model. Prior art anchors: MLGO (learned inlining/regalloc in LLVM), AlphaDev (Nature 2023), Meta LLM Compiler (2024).

14. References

Mytkowicz et al., Producing Wrong Data Without Doing Anything Obviously Wrong (ASPLOS 2009) — read before building bench-runner · Curtsinger & Berger, Stabilizer and coz · Gil Tene on coordinated omission · MLGO (Trofin et al.) · ProGraML (Cummins et al.) · AlphaDev (Mankowitz et al., Nature 2023) · Meta LLM Compiler (Cummins et al., 2024).