Two defects found and fixed in a hostile review of the whole package:
1. The ROI report generator trusted the ledger verdict field and would
mint a clean $16,886/year report for phase2-comrak-010 — the
memory-leaking patch the audit overturned (immutably 'accepted' in the
append-only ledger, sanitizers_clean:false). report_row now exposes
sanitizers_clean and the report treats accepted-but-not-sanitizer-clean
as NOT shippable, printing a warning before any dollar figure. Legit
wins (sanitizers_clean:true) still render clean.
2. bench-runner service --rate 0 / --count 0 could panic via
from_secs_f64(1/0); run_session now guards finite/positive rate and
non-zero count/workers.
Attacks correctly refused (verified, no change): ledger UPDATE/DELETE,
propose_patch ../ + .git escapes, read_target_source traversal. All 10
accepted rows satisfy the green-gates + CI>=1.02 invariant. Findings
recorded in results/adversarial-review.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
- bench-runner service mode (SPEC §3.1 mode c): a coordinated-omission-
correct, open-loop, fixed-rate load generator; latency measured from
the intended send time so queueing is counted, not hidden. Interleaved
A/B of two server binaries, exact percentiles (bounded sample, no HDR
bucketing error), bootstrap p50/p99 CIs via the existing ratio-CI
machine. service-calibrate: A/A false-positive + injected latency-
regression detection (calibration in progress).
- targets/cjson/service.c: minimal single-threaded HTTP server wrapping
the patched cJSON.c (trust-layer, outside the workspace allowlist);
env-toggled per-request busy-wait for the injection calibration only.
- report generator (crates/report bin): reads a ledger row +
optional service-latency JSON, renders throughput→cores→dollars and/or
latency percentiles, every figure with its CI, pricing source, and the
methodology printed inline (SPEC §9); flags any non-accepted row so a
rejected change never reads as a committed saving.
- ledger::report_row; justfile service/service-calibrate/report.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
- Profiles regenerated with callgrind --cache-sim --branch-sim for both
targets: per-function D1/LL miss and branch-mispredict columns unlock
playbook class-4 preconditions (comrak: syntect write_highlighted owns
31% of D1 read misses; tokei: parse_lines owns 56% of Ir).
- tokei workspace: accepted Phase 0 wins banked as a local commit so
stash-based baseline rebuilds attribute only the marginal patch to new
attempts (false-accept trap closed before grinding).
- read_ledger now returns per-attempt history (class, hotspot,
hypothesis, verdict): a class may be re-entered with a materially new
hypothesis, never a duplicate one. Task prompt updated to match.
- read_verdict long-polls the ledger up to 45s server-side (under the
60s MCP cap), cutting the agent's polling turn burn ~15x.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
The Claude Agent SDK loop (agent/hotpath_agent/loop.py, model
claude-fable-5) drives the seven harness tools over a standalone stdio
MCP server (mcp_server.py) and runs the full profile -> hypothesize ->
patch -> gated-verdict cycle on comrak with no human in the loop.
Three audited attempts, all honest rejections, ZERO false accepts:
- phase2-comrak-001 class 3: syntect buffer with_capacity, [0.971,1.007]
- phase2-comrak-002 class 5: gate dead VecDeque build, [0.982,1.026]
- phase2-comrak-003 class 6: hoist find_special_char check, [0.970,1.007]
The agent picked the cheapest untried class each time (3->5->6) and
TWICE refused to fabricate a verdict it couldn't read back, verifying
ledger integrity instead — the anti-reward-hacking behavior SPEC §10
targets.
Harness changes surfaced by the loop:
- run_verdict is now async: launches the multi-minute build+bench
detached (past the 60s MCP transport cap) and returns immediately;
new read_verdict op polls the append-only ledger row (ledger
verdict_summary). Fixes attempt 001's unreadable-verdict gap.
- read_target_source takes offset/limit line windows so 100KB+ files
page instead of overflowing — removes the agent's reason to reach for
a shell (attempt 003 paged a 2400-line file cleanly, no shell).
Honest boundary note (case study + code comments): in THIS dev
container the SDK tool allow/deny layer is defense-in-depth, not the
trust boundary — a nested CLI as root under a parent session doesn't
reliably gate built-ins. SPEC §10's boundary is OS process/user
isolation; that hardening is the remaining Phase 2 exit item
(docs/infra.md). What holds regardless: propose_patch is the only
mediated write path (path-allowlist + git apply) and the ledger is
append-only.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
Commit the spec and scaffold the system it describes:
- SPEC.md: the full v0.1 engineering spec
- crates/: trust-layer Rust workspace, unit-tested
- bench-runner: interleaved ABAB scheduling with discarded warm-ups,
deterministic bootstrap CIs over the ratio of medians,
CI-lower-bound accept rule, env fingerprinting, and a CLI with
compare + A/A self-test modes (A/A verified passing end-to-end)
- diff-test: gate outcome model, sha256 corpus pin verifier that
refuses on mismatch, per-target equivalence policy parser
(byte-identical default, explicit fp-tolerance opt-in)
- ledger: append-only SQLite attempt schema with UPDATE/DELETE
refused by triggers, anti-double-attempt query
- report: cores-saved and dollars/yr ROI math over speedup CIs,
pricing.toml inputs
- playbook/: classes 1-7 with preconditions, procedure, verification
notes, and known failure modes per class
- agent/: Python package skeleton — six-tool contract and prompting
spine, harness IPC deferred to Phase 1
- config/: accept.toml (2% threshold, 30 runs/side), pricing.toml
- targets/, corpora/, results/calibration/: layout and policy docs
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
Claude