mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
An adversarial audit found the enforcement story ahead of the code: gates that were documented as hard gates were advisory, manual, or absent, and several human-facing claims outran the machine record. This makes the claims true. High findings - Differential fuzz was documentation, not enforcement: verdict hardcoded fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and it is inherently baseline-vs-candidate, so it runs on the accept path and reports Skipped in the standalone `just gates` flow. New hard rule: a machine `accepted` verdict is impossible unless the fuzz gate actually passed — otherwise it caps at needs-human-review. - The agent's system prompt claimed FP/concurrency/unsafe patches were "routed to human review automatically". No such mechanism existed. Added a conservative lexical risk classifier (crates/verdict/src/risk.rs) that forces needs-human-review on any would-be accept carrying those signals; fp-tolerance equivalence always signals. - The local verdict path ran agent-influenced code unsandboxed. It now runs under scripts/no-net.sh (network namespace) and fails closed. Upstream test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when present (scripts/pin-testsuite.sh generates them). Also fixed, found while fixing the above - The pipeline script's `set -e` was inert (brace group as the left operand of `&&`), so a failing candidate build did not abort the run and verdict could bench a stale binary. Now a scoped subshell whose status is branched on at top level. - A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path) stranded RUNNING.lock and left read_verdict polling forever. Failure accounting now lives outside the wrapper. - harnessd's patch allowlist ignored git rename/copy headers and --no-prefix diffs; the rename-escape case is now a committed regression test. - The workspace did not compile on Windows (ungated Unix APIs). cfg-gated; 71 portable tests now run on both platforms. - pin-check regenerated the corpus manifest instead of verifying it; it is now verify-only, with a deliberate `just pin-corpus` to re-pin. - Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics (compare --perf-stat), never part of the accept decision. Docs now match the code, including what is NOT yet enforced: no target ships a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not wired to the fp-tolerance gate — each is stated as mechanized-but-inert rather than done. Corrected the stale "33 ledger rows" (34), the playbook's class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation claim (five accepted rows carry sanitizers_clean=false), the unsupported "40,000 requests" figure, and the blanket "historical rows record fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
121 lines
5.3 KiB
Python
121 lines
5.3 KiB
Python
"""Phase 2 agent loop (SPEC §3.5): Claude Agent SDK driving the seven
|
|
harness tools over a standalone stdio MCP server. The agent gets NO
|
|
built-in tools — no shell, no file writes; every capability is one of
|
|
the seven IPC operations, and the trust layer's allowlist/gates remain
|
|
the hard boundary (SPEC §10).
|
|
|
|
Run one attempt:
|
|
python -m hotpath_agent.loop <target> --run-id <id> --repo-root <path>
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import argparse
|
|
import asyncio
|
|
import os
|
|
import sys
|
|
|
|
from claude_agent_sdk import (
|
|
AssistantMessage,
|
|
ClaudeAgentOptions,
|
|
ResultMessage,
|
|
TextBlock,
|
|
query,
|
|
)
|
|
|
|
from .prompts import SYSTEM_PROMPT
|
|
|
|
HARNESS_TOOLS = [
|
|
"mcp__harness__read_profile",
|
|
"mcp__harness__read_ledger",
|
|
"mcp__harness__read_playbook",
|
|
"mcp__harness__read_target_source",
|
|
"mcp__harness__propose_patch",
|
|
"mcp__harness__run_verdict",
|
|
"mcp__harness__read_verdict",
|
|
]
|
|
|
|
# Built-in tools we deny at the SDK layer. This is DEFENSE IN DEPTH, not the
|
|
# trust boundary. SPEC §10's real boundary is OS-level isolation, shipped in
|
|
# scripts/agent-isolated.sh: this whole process tree runs either inside a
|
|
# mount namespace with the repo read-only and CAP_SYS_ADMIN dropped, or as
|
|
# the unprivileged hpagent user — verified by `just isolation-check`. Even
|
|
# if the nested CLI reaches a shell despite this list (settings inherited
|
|
# from a parent session can shadow SDK gating), repo writes fail at the OS.
|
|
# What holds regardless: propose_patch is the only harness-mediated write
|
|
# path (path-allowlist + git apply), and the ledger is append-only.
|
|
BUILTIN_TOOLS_DENIED = [
|
|
"Bash", "Read", "Write", "Edit", "Glob", "Grep",
|
|
"WebSearch", "WebFetch", "Task", "TodoWrite",
|
|
]
|
|
|
|
|
|
async def run_attempt(target: str, run_id: str, max_turns: int, repo_root: str) -> None:
|
|
root = os.path.abspath(repo_root)
|
|
options = ClaudeAgentOptions(
|
|
model="claude-fable-5",
|
|
cwd=root,
|
|
mcp_servers={
|
|
"harness": {
|
|
"type": "stdio",
|
|
"command": sys.executable,
|
|
"args": ["-m", "hotpath_agent.mcp_server"],
|
|
"env": {**os.environ, "HOTPATH_REPO_ROOT": root, "PYTHONPATH": os.path.join(root, "agent")},
|
|
}
|
|
},
|
|
allowed_tools=HARNESS_TOOLS,
|
|
disallowed_tools=BUILTIN_TOOLS_DENIED, # defense in depth; see note above
|
|
system_prompt=SYSTEM_PROMPT,
|
|
# acceptEdits (not bypassPermissions — the CLI refuses
|
|
# --dangerously-skip-permissions under root) with the allowlist runs the
|
|
# loop headless. The real boundary is OS isolation, not this flag.
|
|
permission_mode="acceptEdits",
|
|
max_turns=max_turns,
|
|
)
|
|
task = (
|
|
f"Perform ONE optimization attempt on target '{target}'. You have NO shell, file, or "
|
|
f"web tools — only the seven mcp__harness__ tools. Do not attempt any other tool.\n"
|
|
f"1. read_ledger for the attempt history (class, hotspot, hypothesis, verdict per "
|
|
f"prior attempt); read_profile for hotspots (includes cache-miss and branch-miss "
|
|
f"columns from cache simulation).\n"
|
|
f"2. Pick the cheapest playbook class whose preconditions the profile satisfies AND "
|
|
f"whose (hotspot, class, hypothesis) would be materially NEW — an attempted class "
|
|
f"may be re-entered with a different hotspot or a genuinely different mechanism, "
|
|
f"but never re-submit a hypothesis equivalent to one already in the ledger; "
|
|
f"read_playbook for it.\n"
|
|
f"3. Read the relevant source with read_target_source — for large files pass "
|
|
f"offset/limit and page through in windows (do NOT attempt any shell). State your "
|
|
f"hypothesis, then propose_patch (unified diff, a/ b/ prefixes, paths relative to the "
|
|
f"target workspace root, trailing newline required). Each proposal is STANDALONE — "
|
|
f"the workspace resets to pristine before every apply, so always submit the complete "
|
|
f"patch, never an increment on a prior proposal.\n"
|
|
f"4. run_verdict with run_id '{run_id}', the class number, and the hotspot string. "
|
|
f"It launches a multi-minute pipeline and returns immediately.\n"
|
|
f"5. Poll read_verdict('{run_id}') until it returns a verdict (not a 'running' status). "
|
|
f"The pipeline takes several minutes; keep polling patiently.\n"
|
|
f"6. Report the verdict and speedup CI verbatim. A rejection with a clean ledger row is a "
|
|
f"successful outcome — do not iterate more than twice on a rejected hypothesis."
|
|
)
|
|
|
|
async for message in query(prompt=task, options=options):
|
|
if isinstance(message, AssistantMessage):
|
|
for block in message.content:
|
|
if isinstance(block, TextBlock):
|
|
print(f"[agent] {block.text}", flush=True)
|
|
elif isinstance(message, ResultMessage):
|
|
print(f"[result] turns={message.num_turns} cost_usd={message.total_cost_usd}", flush=True)
|
|
|
|
|
|
def main() -> None:
|
|
p = argparse.ArgumentParser()
|
|
p.add_argument("target")
|
|
p.add_argument("--run-id", required=True)
|
|
p.add_argument("--max-turns", type=int, default=40)
|
|
p.add_argument("--repo-root", default=".")
|
|
a = p.parse_args()
|
|
asyncio.run(run_attempt(a.target, a.run_id, a.max_turns, a.repo_root))
|
|
|
|
|
|
if __name__ == "__main__":
|
|
main()
|