mirror of https://github.com/bdbrown4/peltier.io.git synced 2026-11-07 10:58:28 +00:00

Files

T

Claude 4402775ed2 phase0: exit criteria met — tokei +10.4% median [8.5%, 12.0%] verified

Attempt 4 (phase0-tokei-002, class 3, ACCEPTED — first auto-accept):
every file was read through encoding_rs_io's DecodeReaderBytes via
generic read_to_end, which zero-fills every grown buffer chunk (~221MB
memset/run, 8.5% of Ir) and grows without a size hint. Patch: fs::read
with a BOM-gated fallback to the identical decode path (a UTF-8-BOM
file transcodes invalid sequences, so the fallback is required for
equivalence). Gates: golden byte-identical; differential fuzz old-vs-
new binaries 4,332 iterations (BOM grid, invalid UTF-8, binary junk,
randomized) zero divergence; 250 upstream tests green; ASan+LSan clean
with ASan-run output hashing to golden. Bench: +9.9% [8.0%, 12.7%].

Attempt 5 (phase0-tokei-003, ship tree, ACCEPTED): stacks fat LTO+cgu=1
(real +1.6% standalone but under threshold, phase0-tokei-001) on the
accepted patch; benched as a whole vs pristine: +10.4% [8.5%, 12.0%].
Median clears the Phase 0 exit bar; CI lower bound 8.5%.

Case study written (results/phase0/case-study-tokei.md), ledger at 5
attempts, CLAUDE.md phase line moved to Phase 1, next hypotheses
logged (tokei class-5 byte-skip table; comrak class-3 arena).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2

2026-07-06 19:57:05 +00:00

5.4 KiB

Raw Blame History

hotpath — profile-guided optimization agent

Working name. Rename freely; nothing depends on it.

Mission

An agent that profiles real binaries and services, isolates hot paths, proposes optimizations, and accepts a change only when (a) behavioral-equivalence gates pass and (b) the benchmark improvement is statistically significant on trusted infrastructure. Output: verified performance deltas translated into ROI reports nobody can dispute.

Full engineering spec: SPEC.md. Read it before starting any phase.

Non-negotiables (re-read every session)

No unverified performance claims. A change without passing equivalence gates AND a significant bench delta does not exist. Discard it. Log it.
Measure before optimizing. No patch is proposed until profile data identifies the hot path. Never trust a single benchmark run.
Cheap wins first, always. Build flags → LTO → PGO → allocator swap → then code changes. Check the ledger before re-attempting any class of change on a target.
Equivalence gates are hard gates. Changes touching floating-point ordering, concurrency primitives, or anything UB-adjacent are NEVER auto-accepted — verdict is needs-human-review.
Every attempt goes in the ledger, including failures and rejections. Failures are future training data.
The agent never modifies the trust layer. crates/, config/, corpora/, and upstream test suites are read-only to the agent. Patches may touch only allowlisted paths under targets/<name>/. See SPEC.md §10.
Sandbox everything. Target code runs only inside no-network containers. Never on the host.
Statistical bar: accept only if the bootstrap 95% CI lower bound of speedup ≥ threshold in config/accept.toml (default 2%), from interleaved A/B runs on the same pinned hardware.

Architecture map

crates/bench-runner — trust layer: containerized, CPU-pinned, interleaved A/B benchmarking with bootstrap confidence intervals and A/A self-tests
crates/diff-test — behavioral equivalence: upstream test suite + golden I/O replay + differential fuzzing of changed functions + sanitizers
crates/ledger — append-only SQLite record of every attempt: hypothesis, patch, gate results, bench deltas, verdict, cost
crates/report — ROI generator: cores saved × $/core-hr, latency percentile deltas, CIs, workload caveats printed on every number
agent/ — Claude Agent SDK (Python) loop, prompts, tool definitions
targets/ — vendored OSS targets for case studies (permissive licenses only)
playbook/ — optimization classes, ordered, with preconditions and known risks
results/ — generated reports and flamegraphs

Current phase

Phase 0 complete → Phase 1 — trust layer. Phase 0 exit met 2026-07-06: verified +10.4% median win on tokei (95% CI [+8.5%, +12.0%], ledger phase0-tokei-003) + two case studies in results/phase0/. Phase 1 gaps to close: dedicated no-net bench containers, perf-based profiling, automated A/A + regression-injection calibration, diff-test orchestration, agent↔harness IPC. (Update this line as phases complete; exit criteria in SPEC.md §5.)

Commands

Fill these in as the harness is built; keep them current:

just profile <target>       # perf record + flamegraph for a target's benchmark workload
just bench <target> <ref>   # interleaved A/B: pristine baseline vs working tree
just gates <target>         # test suite + golden replay + diff-fuzz + sanitizers
just verdict <target>       # runs gates + bench, writes ledger row, prints verdict
just report <run-id>        # ROI report from ledger
just aa <target>            # A/A calibration run (must show null result)

Built so far (command-level, pre-target-integration):

just build / test / lint    # trust-layer workspace: cargo build/test/clippy+fmt
just aa [cmd]               # A/A self-test of bench-runner on a shell command
just compare <a> <b>        # interleaved A/B of two shell commands, bootstrap CI
just pin-check <target>     # verify corpus MANIFEST.sha256

Definition of done — one optimization attempt

Hypothesis logged in ledger before patching
Patch touches only allowlisted paths under targets/<name>/
Upstream test suite green
Golden replay byte-identical (or within the target's explicit FP tolerance policy)
Differential fuzz on changed functions: no divergence (60s or 10k iterations minimum)
ASan + UBSan clean; TSan if the patch touches anything threaded
Interleaved A/B bench passes the CI-lower-bound threshold
Ledger row written with verdict and full evidence
If FP-ordering / concurrency / UB-adjacent: verdict needs-human-review, never auto-accept

An attempt that fails any gate is a valid, complete outcome — write the ledger row and move on. Do not iterate on a rejected patch more than twice without a new hypothesis.

Conventions

Rust workspace (edition 2021+) for all harness crates; just for task running
Python 3.11+ for agent/; Claude Agent SDK — verify current package name and API at https://docs.claude.com/en/docs/claude-code/overview before scaffolding
Every number shown to a human carries its confidence interval and workload description. No naked percentages, ever.
Commit style: phase0:, bench:, gates:, agent:, playbook: prefixes
When in doubt between shipping a feature and hardening measurement, harden measurement. The product is trust.

5.4 KiB Raw Blame History Unescape Escape