agent: unattended Phase 2 loop — profile to gated verdict, zero false accepts

The Claude Agent SDK loop (agent/hotpath_agent/loop.py, model
claude-fable-5) drives the seven harness tools over a standalone stdio
MCP server (mcp_server.py) and runs the full profile -> hypothesize ->
patch -> gated-verdict cycle on comrak with no human in the loop.

Three audited attempts, all honest rejections, ZERO false accepts:
- phase2-comrak-001 class 3: syntect buffer with_capacity, [0.971,1.007]
- phase2-comrak-002 class 5: gate dead VecDeque build, [0.982,1.026]
- phase2-comrak-003 class 6: hoist find_special_char check, [0.970,1.007]
The agent picked the cheapest untried class each time (3->5->6) and
TWICE refused to fabricate a verdict it couldn't read back, verifying
ledger integrity instead — the anti-reward-hacking behavior SPEC §10
targets.

Harness changes surfaced by the loop:
- run_verdict is now async: launches the multi-minute build+bench
  detached (past the 60s MCP transport cap) and returns immediately;
  new read_verdict op polls the append-only ledger row (ledger
  verdict_summary). Fixes attempt 001's unreadable-verdict gap.
- read_target_source takes offset/limit line windows so 100KB+ files
  page instead of overflowing — removes the agent's reason to reach for
  a shell (attempt 003 paged a 2400-line file cleanly, no shell).

Honest boundary note (case study + code comments): in THIS dev
container the SDK tool allow/deny layer is defense-in-depth, not the
trust boundary — a nested CLI as root under a parent session doesn't
reliably gate built-ins. SPEC §10's boundary is OS process/user
isolation; that hardening is the remaining Phase 2 exit item
(docs/infra.md). What holds regardless: propose_patch is the only
mediated write path (path-allowlist + git apply) and the ledger is
append-only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
This commit is contained in:
Claude
2026-07-09 23:29:52 +00:00
parent 5be5cad0d3
commit 535821cd48
13 changed files with 487 additions and 45 deletions
@@ -0,0 +1,26 @@
[agent] Still need `find_special_char` — it's somewhere in lines 700–2400. Paging through.
[agent] The diff was refused — likely a context-matching nit. Let me retry with a full git-style header and the whole function replaced as one deletion block (fewer interior blank-line context anchors).
[agent] Retrying with a minimal hunk (single non-blank context anchors, fewer context lines to mismatch):
[agent] Pipeline running (gates + interleaved A/B bench take several minutes). Polling…
[agent] **Attempt complete.** Ledger row written for `phase2-comrak-003`.
## Verdict (verbatim)
| Field | Value |
|---|---|
| run_id | `phase2-comrak-003` |
| verdict | **`rejected-bench`** |
| speedup_ci | `[0.9695641714548212, 1.007035795621153]` |
| speedup_median | `0.9871171594946015` |
## Summary of the attempt
- **Hotspot:** `comrak::parser::inlines::Subject::parse_inline` (6.67% of Ir) — the `find_special_char` per-byte scan
- **Class:** 6 (SIMD/autovectorization enablement — cheapest untried; classes 1, 2, 3, 5 already in the ledger; class 4's cache-miss preconditions and class 7's contention preconditions were not supported by the profile)
- **Hypothesis:** hoisting the `^`-within-brackets exception out of the per-byte predicate would tighten the inner table-lookup scan for ~1–3% end-to-end
- **Result:** the hypothesis is refuted by measurement. The bootstrap 95% CI `[0.970, 1.007]` straddles 1.0 with a median of 0.987 (a slight *slowdown* if anything) — far from the required CI lower bound ≥ 1.02. Gates were not the blocker; the mechanism simply didn't materialize, plausibly because LLVM already hoisted the loop-invariant `within_brackets` check, or the restart-scan structure inhibited the previous straight-line codegen.
This is a clean, complete outcome: hypothesis recorded before patching, patch expressed the hypothesis faithfully (byte/integer-only, no FP/concurrency/unsafe taint, no gate-routing avoidance), and the ledger row now prevents re-attempting (parse_inline, class 6, branch-hoist) in future sessions. Per the two-iteration rule I'm stopping here rather than iterating on a rejected hypothesis without new evidence — the next session should move to the next class or hotspot (the allocator-dominated profile, ~26% in malloc/free, suggests the biggest remaining lever is allocation-volume reduction, if any class-5 hypothesis distinct from the ledger's prior ones exists).
One operational note: `propose_patch` rejects diffs lacking a trailing newline with the generic "does not apply cleanly" error — worth a friendlier message in `harnessd`.
[result] turns=41 cost_usd=6.470653