mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
benbrownandClaude Opus 4.8
899d55f4b6
An adversarial audit found the enforcement story ahead of the code: gates that were documented as hard gates were advisory, manual, or absent, and several human-facing claims outran the machine record. This makes the claims true. High findings - Differential fuzz was documentation, not enforcement: verdict hardcoded fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and it is inherently baseline-vs-candidate, so it runs on the accept path and reports Skipped in the standalone `just gates` flow. New hard rule: a machine `accepted` verdict is impossible unless the fuzz gate actually passed — otherwise it caps at needs-human-review. - The agent's system prompt claimed FP/concurrency/unsafe patches were "routed to human review automatically". No such mechanism existed. Added a conservative lexical risk classifier (crates/verdict/src/risk.rs) that forces needs-human-review on any would-be accept carrying those signals; fp-tolerance equivalence always signals. - The local verdict path ran agent-influenced code unsandboxed. It now runs under scripts/no-net.sh (network namespace) and fails closed. Upstream test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when present (scripts/pin-testsuite.sh generates them). Also fixed, found while fixing the above - The pipeline script's `set -e` was inert (brace group as the left operand of `&&`), so a failing candidate build did not abort the run and verdict could bench a stale binary. Now a scoped subshell whose status is branched on at top level. - A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path) stranded RUNNING.lock and left read_verdict polling forever. Failure accounting now lives outside the wrapper. - harnessd's patch allowlist ignored git rename/copy headers and --no-prefix diffs; the rename-escape case is now a committed regression test. - The workspace did not compile on Windows (ungated Unix APIs). cfg-gated; 71 portable tests now run on both platforms. - pin-check regenerated the corpus manifest instead of verifying it; it is now verify-only, with a deliberate `just pin-corpus` to re-pin. - Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics (compare --perf-stat), never part of the accept decision. Docs now match the code, including what is NOT yet enforced: no target ships a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not wired to the fp-tolerance gate — each is stated as mechanized-but-inert rather than done. Corrected the stale "33 ledger rows" (34), the playbook's class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation claim (five accepted rows carry sanitizers_clean=false), the unsupported "40,000 requests" figure, and the blanket "historical rows record fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
79 lines
2.9 KiB
Python
79 lines
2.9 KiB
Python
"""Standalone MCP stdio server exposing the seven harness tools (SPEC §3.5).
|
|
|
|
Spawned as a subprocess by the agent loop; speaks MCP over stdio. Wraps
|
|
the harnessd IPC client — the trust-layer allowlist and gates remain the
|
|
hard boundary. Run standalone for debugging:
|
|
|
|
HOTPATH_REPO_ROOT=/path/to/repo python -m hotpath_agent.mcp_server
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
import os
|
|
|
|
from mcp.server.fastmcp import FastMCP
|
|
|
|
from . import tools as harness
|
|
|
|
REPO_ROOT = os.environ.get("HOTPATH_REPO_ROOT", ".")
|
|
harness.connect(REPO_ROOT)
|
|
|
|
mcp = FastMCP("harness")
|
|
|
|
|
|
@mcp.tool()
|
|
def read_profile(target: str) -> str:
|
|
"""Latest profile for a target: ranked hotspots with % of instructions."""
|
|
return str(harness.read_profile(target))
|
|
|
|
|
|
@mcp.tool()
|
|
def read_ledger(target: str) -> str:
|
|
"""Prior attempts against a target. Never re-attempt a playbook class listed here."""
|
|
return str(harness.read_ledger(target))
|
|
|
|
|
|
@mcp.tool()
|
|
def read_playbook(class_number: int) -> str:
|
|
"""Optimization playbook class 1-7: preconditions, procedure, verification, failure modes."""
|
|
return harness.read_playbook(class_number)
|
|
|
|
|
|
@mcp.tool()
|
|
def read_target_source(target: str, path: str, offset: int = 0, limit: int = 400) -> str:
|
|
"""Read one file from the target workspace (path relative to the workspace root),
|
|
as a line window: `offset` (0-based first line) and `limit` (max lines). Large
|
|
files (100KB+) MUST be read in windows — the response is truncated otherwise.
|
|
Returns numbered lines plus total_lines/returned_lines/truncated metadata."""
|
|
return str(harness.read_target_source(target, path, offset, limit))
|
|
|
|
|
|
@mcp.tool()
|
|
def propose_patch(target: str, diff: str, hypothesis: str) -> str:
|
|
"""Submit a unified git diff (a/ b/ prefixes, paths relative to the target
|
|
workspace, MUST end with a trailing newline) plus the hypothesis it tests.
|
|
Each proposal is STANDALONE: the workspace is reset to pristine before the
|
|
diff is applied, so always submit the complete patch — never an increment
|
|
on top of an earlier proposal. The harness applies it only after a
|
|
path-allowlist check. Returns a patch_id, or an error string if refused."""
|
|
return harness.propose_patch(target, diff, hypothesis)
|
|
|
|
|
|
@mcp.tool()
|
|
def run_verdict(patch_id: str, run_id: str, playbook_class: int, hotspot: str) -> str:
|
|
"""Launch the gate+bench pipeline on a proposed patch (runs detached, several
|
|
minutes) and return immediately. Poll read_verdict with the same run_id for the
|
|
result; the ledger row is written when the pipeline finishes."""
|
|
return str(harness.run_verdict(patch_id, run_id, playbook_class, hotspot))
|
|
|
|
|
|
@mcp.tool()
|
|
def read_verdict(run_id: str) -> str:
|
|
"""Poll for a completed verdict by run_id. Returns the verdict and speedup CI once
|
|
the ledger row is written, or a 'running' status while the pipeline is still going."""
|
|
return str(harness.read_verdict(run_id))
|
|
|
|
|
|
if __name__ == "__main__":
|
|
mcp.run()
|