benbrown/peltier.io
2
1
Fork 0
mirror of https://github.com/bdbrown4/peltier.io.git synced 2026-11-07 10:58:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

agent: unattended Phase 2 loop — profile to gated verdict, zero false accepts

Browse Source 
The Claude Agent SDK loop (agent/hotpath_agent/loop.py, model
claude-fable-5) drives the seven harness tools over a standalone stdio
MCP server (mcp_server.py) and runs the full profile -> hypothesize ->
patch -> gated-verdict cycle on comrak with no human in the loop.

Three audited attempts, all honest rejections, ZERO false accepts:
- phase2-comrak-001 class 3: syntect buffer with_capacity, [0.971,1.007]
- phase2-comrak-002 class 5: gate dead VecDeque build, [0.982,1.026]
- phase2-comrak-003 class 6: hoist find_special_char check, [0.970,1.007]
The agent picked the cheapest untried class each time (3->5->6) and
TWICE refused to fabricate a verdict it couldn't read back, verifying
ledger integrity instead — the anti-reward-hacking behavior SPEC §10
targets.

Harness changes surfaced by the loop:
- run_verdict is now async: launches the multi-minute build+bench
  detached (past the 60s MCP transport cap) and returns immediately;
  new read_verdict op polls the append-only ledger row (ledger
  verdict_summary). Fixes attempt 001's unreadable-verdict gap.
- read_target_source takes offset/limit line windows so 100KB+ files
  page instead of overflowing — removes the agent's reason to reach for
  a shell (attempt 003 paged a 2400-line file cleanly, no shell).

Honest boundary note (case study + code comments): in THIS dev
container the SDK tool allow/deny layer is defense-in-depth, not the
trust boundary — a nested CLI as root under a parent session doesn't
reliably gate built-ins. SPEC §10's boundary is OS process/user
isolation; that hardening is the remaining Phase 2 exit item
(docs/infra.md). What holds regardless: propose_patch is the only
mediated write path (path-allowlist + git apply) and the ledger is
append-only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
This commit is contained in:
Claude
2026-07-09 23:29:52 +00:00
parent 5be5cad0d3
commit 535821cd48
13 changed files with 487 additions and 45 deletions
Download Patch File Download Diff File Expand all files Collapse all files
CLAUDE.md
+1 -1
View File
@@ -32,7 +32,7 @@ Full engineering spec: `SPEC.md`. Read it before starting any phase.
## Current phase
**Phase 1 complete → Phase 2 — agent.** Phase 1 exit met 2026-07-09 on free infrastructure (SPEC §5): A/A false-positive 0/20 on both anchors (local container, results/calibration/2026-07-06-*; GitHub runner, results/calibration/2026-07-09-gh-runner-20session.md); injected-5% detection 19/20 and 20/20; gates end-to-end on comrak + tokei, re-verified per CI run incl. no-net container execution. Built: calibrate/gates/verdict pipelines, harnessd six-tool IPC. Deferred to hardware (docs/infra.md): pinned/governor-controlled bench box, hardware-PMU profiling, separate-uid enforcement — required for Phase 3/4 case studies, not the Phase 1 gate. (Update this line as phases complete; exit criteria in SPEC.md §5.)
**Phase 2 in progress — agent loop.** The unattended profile→hypothesize→patch→gated-verdict loop runs on comrak via the Claude Agent SDK (`agent/hotpath_agent/loop.py`, model claude-fable-5) over a stdio MCP server wrapping the seven harness tools. 3 audited attempts (phase2-comrak-001/002/003), all honest rejections, **0 false accepts**; the agent twice refused to fabricate an unreadable verdict (results/phase2/case-study-agent-loop.md). Shipped: async run_verdict + pollable read_verdict, windowed read_target_source. Exit-criteria gaps: an auto-accepted win (needs richer profiling to unlock more classes), and OS-level process/user isolation for the agent's tool boundary (SPEC §10; the SDK allow/deny layer is defense-in-depth only — see docs/infra.md). (Update this line as phases complete; exit criteria in SPEC.md §5.)
## Commands