mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 19:08:27 +00:00
benbrownandClaude Opus 4.8
899d55f4b6
An adversarial audit found the enforcement story ahead of the code: gates that were documented as hard gates were advisory, manual, or absent, and several human-facing claims outran the machine record. This makes the claims true. High findings - Differential fuzz was documentation, not enforcement: verdict hardcoded fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and it is inherently baseline-vs-candidate, so it runs on the accept path and reports Skipped in the standalone `just gates` flow. New hard rule: a machine `accepted` verdict is impossible unless the fuzz gate actually passed — otherwise it caps at needs-human-review. - The agent's system prompt claimed FP/concurrency/unsafe patches were "routed to human review automatically". No such mechanism existed. Added a conservative lexical risk classifier (crates/verdict/src/risk.rs) that forces needs-human-review on any would-be accept carrying those signals; fp-tolerance equivalence always signals. - The local verdict path ran agent-influenced code unsandboxed. It now runs under scripts/no-net.sh (network namespace) and fails closed. Upstream test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when present (scripts/pin-testsuite.sh generates them). Also fixed, found while fixing the above - The pipeline script's `set -e` was inert (brace group as the left operand of `&&`), so a failing candidate build did not abort the run and verdict could bench a stale binary. Now a scoped subshell whose status is branched on at top level. - A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path) stranded RUNNING.lock and left read_verdict polling forever. Failure accounting now lives outside the wrapper. - harnessd's patch allowlist ignored git rename/copy headers and --no-prefix diffs; the rename-escape case is now a committed regression test. - The workspace did not compile on Windows (ungated Unix APIs). cfg-gated; 71 portable tests now run on both platforms. - pin-check regenerated the corpus manifest instead of verifying it; it is now verify-only, with a deliberate `just pin-corpus` to re-pin. - Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics (compare --perf-stat), never part of the accept decision. Docs now match the code, including what is NOT yet enforced: no target ships a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not wired to the fp-tolerance gate — each is stated as mechanized-but-inert rather than done. Corrected the stale "33 ledger rows" (34), the playbook's class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation claim (five accepted rows carry sanitizers_clean=false), the unsupported "40,000 requests" figure, and the blanket "historical rows record fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
77 lines
2.7 KiB
Python
77 lines
2.7 KiB
Python
"""Tool surface exposed to the model (SPEC §3.5).
|
|
|
|
Exactly seven tools, all mediated by harnessd over line-delimited JSON —
|
|
the agent process has no direct filesystem or shell capability beyond
|
|
what these return. Deliberately NOT exposed: shell on the host, writes
|
|
outside targets/<name>/, any access to crates/, config/, corpora/.
|
|
|
|
Phase 2 wiring: each function becomes a Claude Agent SDK tool
|
|
definition. The bodies talk to the live harness; construct a client
|
|
with `connect(repo_root)` first.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from typing import Any
|
|
|
|
from .harness import HarnessClient
|
|
|
|
_client: HarnessClient | None = None
|
|
|
|
|
|
def connect(repo_root: str, binary: str = "target/debug/harnessd") -> None:
|
|
global _client
|
|
_client = HarnessClient(repo_root, binary)
|
|
|
|
|
|
def _c() -> HarnessClient:
|
|
if _client is None:
|
|
raise RuntimeError("call connect(repo_root) first")
|
|
return _client
|
|
|
|
|
|
def read_profile(target: str) -> dict[str, Any]:
|
|
"""Latest profile for a target: ranked hotspots. Read-only."""
|
|
return _c().call("read_profile", target=target)
|
|
|
|
|
|
def read_ledger(target: str) -> dict[str, Any]:
|
|
"""Prior attempts against this target — the anti-double-attempt
|
|
memory. Never re-grind a class that already has a verdict."""
|
|
return _c().call("read_ledger", target=target)
|
|
|
|
|
|
def read_playbook(class_number: int) -> str:
|
|
"""Playbook class markdown (1-7)."""
|
|
return _c().call("read_playbook", **{"class": class_number})["markdown"]
|
|
|
|
|
|
def read_target_source(target: str, path: str, offset: int = 0, limit: int = 400) -> dict[str, Any]:
|
|
"""Read-only line window of a target workspace file at the pinned commit."""
|
|
return _c().call("read_target_source", target=target, path=path, offset=offset, limit=limit)
|
|
|
|
|
|
def propose_patch(target: str, diff: str, hypothesis: str) -> str:
|
|
"""Submit a unified diff plus its hypothesis. The harness applies it
|
|
via `git apply` after a path-allowlist check (SPEC §10); anything
|
|
outside targets/<target>/workspace is rejected before any gate
|
|
runs. Returns a patch id."""
|
|
return _c().call("propose_patch", target=target, diff=diff, hypothesis=hypothesis)["patch_id"]
|
|
|
|
|
|
def run_verdict(patch_id: str, run_id: str, playbook_class: int, hotspot: str) -> dict[str, Any]:
|
|
"""Launch the gate + bench pipeline for a proposed patch (runs detached).
|
|
Returns immediately; poll run_verdict's result via read_verdict(run_id)."""
|
|
return _c().call(
|
|
"run_verdict",
|
|
patch_id=patch_id,
|
|
run_id=run_id,
|
|
playbook_class=str(playbook_class),
|
|
hotspot=hotspot,
|
|
)
|
|
|
|
|
|
def read_verdict(run_id: str) -> dict[str, Any]:
|
|
"""Poll for a completed verdict: verdict + speedup CI, or a running status."""
|
|
return _c().call("read_verdict", run_id=run_id)
|