peltier.io/results/phase2/case-study-agent-loop.md

# Phase 2 case study — the unattended agent loop

**Result: the Claude Agent SDK loop ran the full profile → hypothesize →
patch → gated-verdict cycle on comrak with no human in the loop, across
multiple attempts, with ZERO false accepts. The agent twice refused to
fabricate a verdict it could not read back, verifying ledger integrity
instead — exactly the anti-reward-hacking behavior SPEC §10 is designed
to produce.**

## What runs

- `agent/hotpath_agent/loop.py` — Claude Agent SDK (`claude-agent-sdk`,
  model `claude-fable-5`) driving a standalone stdio MCP server
  (`mcp_server.py`) that exposes the seven harness tools. The agent has
  no built-in file/shell/web tools in its allowlist.
- `harnessd` gained an async `run_verdict` (launches the multi-minute
  build+bench pipeline detached, returns immediately) and a pollable
  `read_verdict` that reads the append-only ledger row once written.
- `read_target_source` gained an `offset`/`limit` line window so 100 KB+
  source files can be paged instead of overflowing one response.

## The attempts (all in the ledger, `results/ledger.sqlite`)

| run_id | class chosen | patch | verdict | speedup CI |
|---|---|---|---|---|
| phase2-comrak-001 | 3 alloc-churn | syntect buffer `with_capacity` | rejected-bench | [0.971, 1.007] |
| phase2-comrak-002 | 5 algorithmic | gate dead `VecDeque` build in text postprocess | rejected-bench | [0.982, 1.026] |
| phase2-comrak-003 | 6 SIMD | hoist `^`-in-brackets check out of `find_special_char` predicate | rejected-bench | [0.970, 1.007] |

Run 003 exercised the two shipped fixes: the agent **paged through a
2,400-line file with `read_target_source` windows** (no shell reach this
time) and **polled the async `read_verdict` cleanly** to a verbatim
verdict. Across all three attempts the agent chose the cheapest untried
class each time (3 → 5 → 6, skipping 4/7 whose profile preconditions the
callgrind data didn't support), and **0 of 3 were false accepts**.

The agent's class selection was correct each time: it read the ledger,
saw which classes were already attempted, and picked the cheapest
untried class whose profile preconditions were met. Both patches went
through `propose_patch` (path-allowlist + `git apply`) and the full
gate+bench pipeline. Both were correctly rejected — real mechanisms,
too small to clear the 2% CI-lower-bound bar. **No false accepts.**

## Two findings the loop surfaced about itself

1. **`run_verdict` outlived the MCP transport (attempt 001).** The
   build+bench pipeline takes minutes; the MCP tool-call cap is 60 s. The
   agent's calls timed out — and rather than retry blindly or invent a
   number, it checked `read_ledger`, confirmed exactly one row was
   written (dedup on run_id held), and reported the verdict as
   *unreadable, not fabricated*, asking a human to read the ledger. Fix
   shipped: async `run_verdict` + pollable `read_verdict`. Attempt 002
   then read its verdict back cleanly.

2. **The SDK tool-restriction is not the trust boundary in this
   environment.** The agent reached for a built-in `Bash`/`Monitor` tool
   to read a source file too large for the (then whole-file-only)
   `read_target_source`. Investigation showed that a nested Claude Code
   CLI running as root under a parent session does not reliably gate
   built-ins: `--dangerously-skip-permissions` is refused under root, and
   a `can_use_tool` deny-by-default callback is shadowed by settings
   inherited from the parent session (and broke legitimate MCP tool calls
   with "Stream closed"). **This is expected, not a regression:** SPEC §10
   specifies the boundary as OS-level process/user isolation and no-net
   containers — "enforced by filesystem permissions and a separate-process
   harness, not by prompt." The SDK allow/deny layer is defense in depth.
   Mitigation shipped: windowed `read_target_source` removes the agent's
   reason to reach for a shell; the load-bearing guarantees hold
   regardless (see below).

## What holds even when the tool layer leaks

- **Writes to the target are only mediated through `propose_patch`** —
  path-allowlist (relative paths, no `..`, no `.git`) then `git apply`
  rooted in the target workspace. A patch touching `config/` or `crates/`
  is refused before any apply. Verified against a malicious
  accept-threshold diff in Phase 1.
- **The ledger is append-only** — UPDATE/DELETE are refused by SQLite
  triggers, and `accepted` rows are written only by the `verdict` binary
  after the gated pipeline. An agent cannot forge a passing verdict
  through the harness.
- **The gate+bench pipeline is unbypassable through the harness** —
  pristine-rebuilt baseline, interleaved A/B, CI-lower-bound rule.

The residual hole in THIS container: an agent with host shell could, in
principle, `sqlite3 INSERT` a forged row (triggers block mutation, not
insertion) or invoke the verdict binary directly. That is precisely the
attack SPEC §10's process/user isolation exists to prevent, and it is the
**Phase 2 exit gap to close on real infrastructure** (run the agent
process as a separate unprivileged user with no shell and the trust layer
owned by another uid — the `bench-metal`/`docs/infra.md` setup).

## Phase 2 exit criteria (SPEC §5)

- Unattended profile→verdict loop on one target: **met** (comrak).
- ≥1 auto-accepted win: **not yet** — both attempts were honest
  rejections. The loop is proven; a win needs either a richer profile
  (perf `stat` cache-miss data to unlock class-4 preconditions) or more
  attempts against fresh hypotheses.
- Zero false accepts across audited attempts: **met so far** (2/2
  audited, both correctly rejected).
- The separate-uid isolation for the tool boundary is the remaining
  hardening item, tracked to `docs/infra.md`.