mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
benbrownandClaude Opus 4.8
899d55f4b6
An adversarial audit found the enforcement story ahead of the code: gates that were documented as hard gates were advisory, manual, or absent, and several human-facing claims outran the machine record. This makes the claims true. High findings - Differential fuzz was documentation, not enforcement: verdict hardcoded fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and it is inherently baseline-vs-candidate, so it runs on the accept path and reports Skipped in the standalone `just gates` flow. New hard rule: a machine `accepted` verdict is impossible unless the fuzz gate actually passed — otherwise it caps at needs-human-review. - The agent's system prompt claimed FP/concurrency/unsafe patches were "routed to human review automatically". No such mechanism existed. Added a conservative lexical risk classifier (crates/verdict/src/risk.rs) that forces needs-human-review on any would-be accept carrying those signals; fp-tolerance equivalence always signals. - The local verdict path ran agent-influenced code unsandboxed. It now runs under scripts/no-net.sh (network namespace) and fails closed. Upstream test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when present (scripts/pin-testsuite.sh generates them). Also fixed, found while fixing the above - The pipeline script's `set -e` was inert (brace group as the left operand of `&&`), so a failing candidate build did not abort the run and verdict could bench a stale binary. Now a scoped subshell whose status is branched on at top level. - A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path) stranded RUNNING.lock and left read_verdict polling forever. Failure accounting now lives outside the wrapper. - harnessd's patch allowlist ignored git rename/copy headers and --no-prefix diffs; the rename-escape case is now a committed regression test. - The workspace did not compile on Windows (ungated Unix APIs). cfg-gated; 71 portable tests now run on both platforms. - pin-check regenerated the corpus manifest instead of verifying it; it is now verify-only, with a deliberate `just pin-corpus` to re-pin. - Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics (compare --perf-stat), never part of the accept decision. Docs now match the code, including what is NOT yet enforced: no target ships a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not wired to the fp-tolerance gate — each is stated as mechanized-but-inert rather than done. Corrected the stale "33 ledger rows" (34), the playbook's class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation claim (five accepted rows carry sanitizers_clean=false), the unsupported "40,000 requests" figure, and the blanket "historical rows record fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
244 lines
13 KiB
Markdown
244 lines
13 KiB
Markdown
# Phase 2 case study — the unattended agent loop
|
||
|
||
**Result: the Claude Agent SDK loop runs the full profile → hypothesize →
|
||
patch → gated-verdict cycle with no human in the loop, on two targets,
|
||
with ZERO false accepts — and produced its first verified auto-accepted
|
||
win: +10.7% median on tokei, 95% CI [+8.4%, +13.1%] (221 MB / 4360-file
|
||
pinned workload, single rayon thread, pinned core), surviving a
|
||
10,000-input differential fuzz and ASan/LSan on top of the pipeline's
|
||
gates. The agent twice refused to fabricate a verdict it could not read
|
||
back, verifying ledger integrity instead — exactly the
|
||
anti-reward-hacking behavior SPEC §10 is designed to produce.**
|
||
|
||
## What runs
|
||
|
||
- `agent/hotpath_agent/loop.py` — Claude Agent SDK (`claude-agent-sdk`,
|
||
model `claude-fable-5`) driving a standalone stdio MCP server
|
||
(`mcp_server.py`) that exposes the seven harness tools. The agent has
|
||
no built-in file/shell/web tools in its allowlist.
|
||
- `harnessd` gained an async `run_verdict` (launches the multi-minute
|
||
build+bench pipeline detached, returns immediately) and a pollable
|
||
`read_verdict` that reads the append-only ledger row once written.
|
||
- `read_target_source` gained an `offset`/`limit` line window so 100 KB+
|
||
source files can be paged instead of overflowing one response.
|
||
|
||
## The scoreboard (SPEC §5 Phase 2 exit criteria — MET)
|
||
|
||
- **Unattended profile→verdict loop on ≥1 target:** met — runs on
|
||
**both** comrak and tokei, behind OS-level isolation.
|
||
- **≥1 auto-accepted win:** met — **three verified, banked wins**, all
|
||
on tokei (see table).
|
||
- **Zero false accepts across ≥20 audited attempts:** met — **20/20
|
||
audited, 0 shipped false accepts.** One row (`phase2-comrak-010`)
|
||
the *pipeline* accepted was overturned by the 100% human audit before
|
||
anything shipped, and drove the structural fix that made the pipeline
|
||
enforce sanitizers itself (below).
|
||
|
||
Every one of the 20 rows was independently re-audited a second time by
|
||
a fan-out of 20 adversarial verifiers (`phase2-final-audit` workflow),
|
||
each trying to *refute* its row's verdict: all 20 confirmed
|
||
`verdict_follows_evidence`, correct equivalence tier, patch-matches-
|
||
hypothesis, and **no harness-gaming signs**. A separate check confirmed
|
||
the ledger still refuses `UPDATE`/`DELETE` (append-only triggers hold).
|
||
|
||
## The attempts (all 20 in the ledger, `results/ledger.sqlite`)
|
||
|
||
The discriminator that separates the three banked wins from the
|
||
overturned `comrak-010` is **not** the bench CI (comrak-010 cleared the
|
||
2% bar too) — it is the sanitizer result: the three tokei wins are
|
||
ASan/LSan-clean and carry bank commits; comrak-010 failed LeakSanitizer
|
||
and carries none.
|
||
|
||
| run_id | class | verdict | speedup median | 95% CI |
|
||
|---|---|---|---|---|
|
||
| phase2-comrak-001 | 3 | rejected-bench | 0.990 | [0.972, 1.007] |
|
||
| phase2-comrak-002 | 5 | rejected-bench | 1.003 | [0.982, 1.026] |
|
||
| phase2-comrak-003 | 6 | rejected-bench | 0.987 | [0.970, 1.007] |
|
||
| phase2-comrak-004 | 5 | rejected-bench | 1.012 | [1.001, 1.027] |
|
||
| phase2-comrak-005 | 4 | rejected-bench | 1.009 | [0.985, 1.026] |
|
||
| phase2-comrak-006 | 5 | rejected-bench | 1.008 | [0.990, 1.020] |
|
||
| phase2-comrak-007 | 5 | rejected-bench | 1.010 | [0.990, 1.023] |
|
||
| phase2-comrak-008 | 5 | rejected-bench | 0.983 | [0.963, 0.999] |
|
||
| phase2-comrak-009 | 1 | rejected-bench | 1.004 | [0.982, 1.019] |
|
||
| phase2-comrak-010 | 3 | ~~accepted~~ **overturned** | 1.107 | [1.079, 1.121] |
|
||
| phase2-comrak-011 | 5 | rejected-bench | 1.010 | [0.979, 1.033] |
|
||
| phase2-tokei-001 | 5 | **accepted** | **1.107** | **[1.084, 1.131]** |
|
||
| phase2-tokei-002 | 5 | rejected-gate | — | (golden replay) |
|
||
| phase2-tokei-003 | 5 | **accepted** | **1.037** | **[1.027, 1.052]** |
|
||
| phase2-tokei-004 | 2 | rejected-bench | 1.001 | [0.985, 1.017] |
|
||
| phase2-tokei-005 | 6 | rejected-bench | 0.997 | [0.982, 1.013] |
|
||
| phase2-tokei-006 | 5 | rejected-bench | 0.988 | [0.983, 0.997] |
|
||
| phase2-tokei-007 | 5 | rejected-bench | 0.993 | [0.984, 1.004] |
|
||
| phase2-tokei-008 | 5 | **accepted** | **1.099** | **[1.073, 1.112]** |
|
||
| phase2-tokei-009 | 5 | rejected-bench | 1.057 | [1.002, 1.084] |
|
||
|
||
The three banked wins compound: tokei runs the pinned 221 MB / 4360-file
|
||
workload roughly **1.25× faster** than upstream v14.0.0 (Phase-0 wins
|
||
plus these three), each increment individually CI-significant, fuzz-
|
||
clean, and sanitizer-clean. `tokei-009` is the discipline in miniature:
|
||
a real +5.7% median that was **rejected** because its noisy CI lower
|
||
bound (+0.2%) couldn't clear the 2% bar — a win that doesn't survive the
|
||
statistics doesn't exist.
|
||
|
||
### phase2-tokei-001 — the first auto-accepted win, and its audit
|
||
|
||
Running behind the OS boundary (`scripts/agent-isolated.sh`, mountns
|
||
mode), the agent read the new cache-sim profile, saw `parse_lines` at
|
||
55.9% of instructions / 62% of branches / 44% of branch mispredicts,
|
||
re-entered class 5 with a hypothesis distinct from every ledger entry
|
||
(the hypothesis-granular `read_ledger` shipped for exactly this), and
|
||
proposed a 256-entry first-byte gate: most bytes of most lines can't
|
||
start any quote/comment token, so five per-byte token-matcher loops
|
||
collapse to one table load. `git apply` refused its first diff (bad
|
||
hunk counts); it recomputed and resubmitted — the harness, not the
|
||
agent, owns what reaches the tree.
|
||
|
||
Pipeline verdict: accepted at +10.7% median, 95% CI [+8.4%, +13.1%]
|
||
vs the pristine-rebuilt baseline (which already banks the Phase 0
|
||
wins, so this is marginal, not cumulative, speedup). 21 turns,
|
||
$4.61.
|
||
|
||
The pipeline's auto-gates cover upstream tests + golden replay +
|
||
bench; fuzz and sanitizers are per-attempt manual in this phase, so
|
||
the 100% Phase 2 audit ran them by hand before counting the win:
|
||
|
||
- **Differential fuzz** (`scripts/diff-fuzz-tokei.py`): 10,000 mutated
|
||
inputs (quote/comment/backslash-focused mutations over 400 corpus
|
||
seeds), pristine vs candidate, batched over 200 process pairs —
|
||
**0 divergences** after canonicalizing JSON output order.
|
||
- **ASan + LSan** (nightly, patched tree): clean over the full 221 MB
|
||
corpus and every kept fuzz batch.
|
||
- Mechanical re-check (`scripts/audit-attempt.py`): CI lower bound
|
||
1.084 ≥ 1.02 bar; patch paths inside the workspace; hypothesis
|
||
matches the patch; safe code only — correct auto-accept tier.
|
||
|
||
**Audit finding worth keeping: raw-byte differential comparison
|
||
overcounts.** The first fuzz runs reported 3/200 batch divergences —
|
||
but on different batches across two identical-seed runs, and none
|
||
reproducible. Cause: tokei's parallel directory walker
|
||
(`ignore::WalkBuilder::build_parallel`, its own thread pool,
|
||
unaffected by `RAYON_NUM_THREADS`) makes same-name report ordering
|
||
timing-dependent in both binaries — benign, count-identical, present
|
||
in the pristine baseline. The fuzzer now canonicalizes (sorts) JSON
|
||
before comparing and stores both raw outputs on any divergence.
|
||
Differential gates must compare semantics, not bytes, wherever the
|
||
target itself is legitimately nondeterministic in presentation.
|
||
|
||
Run 003 exercised the two shipped fixes: the agent **paged through a
|
||
2,400-line file with `read_target_source` windows** (no shell reach this
|
||
time) and **polled the async `read_verdict` cleanly** to a verbatim
|
||
verdict. Across all three attempts the agent chose the cheapest untried
|
||
class each time (3 → 5 → 6, skipping 4/7 whose profile preconditions the
|
||
callgrind data didn't support), and **0 of 3 were false accepts**.
|
||
|
||
The agent's class selection was correct each time: it read the ledger,
|
||
saw which classes were already attempted, and picked the cheapest
|
||
untried class whose profile preconditions were met. Both patches went
|
||
through `propose_patch` (path-allowlist + `git apply`) and the full
|
||
gate+bench pipeline. Both were correctly rejected — real mechanisms,
|
||
too small to clear the 2% CI-lower-bound bar. **No false accepts.**
|
||
|
||
## The stop-the-line event: phase2-comrak-010
|
||
|
||
The pipeline auto-accepted a comrak class-3 patch (+10.7% median,
|
||
CI [+7.9%, +12.1%]) that skips AST-arena teardown via `ManuallyDrop`
|
||
before `process::exit` — and the 100% human audit **overturned it**.
|
||
LeakSanitizer flags a 384-byte arena chunk as unreachable at exit: the
|
||
patch's `black_box(&arena)` keeps the arena live only until its scope
|
||
closes, but `process::exit` runs after. The agent's own hypothesis had
|
||
named this exact risk ("if the sanitizer still flags the leak,
|
||
rejected-gates is the honest outcome") — the pipeline just had no
|
||
sanitizer gate to trip: ASan/LSan were per-attempt manual, so the row
|
||
says `accepted` with `sanitizers_clean: false`.
|
||
|
||
Called per SPEC §10: a **pipeline false accept**, caught by audit before
|
||
anything shipped (shipped false accepts remain **0**; the win was not
|
||
banked). Fix shipped the same hour: `verdict` now runs an ASan+LSan
|
||
build of the patched tree over the pinned workload on every would-be
|
||
accept and caps flagged wins at `needs-human-review` — the tier rule
|
||
SPEC §8 always specified, now enforced by the machine instead of the
|
||
auditor. The immutable ledger row stands as written; this section is
|
||
the corrective record.
|
||
|
||
10k-input differential fuzz on the same patch: 0 divergences — the
|
||
output equivalence was real; only the teardown-leak tier was wrong.
|
||
|
||
## Two findings the loop surfaced about itself
|
||
|
||
1. **`run_verdict` outlived the MCP transport (attempt 001).** The
|
||
build+bench pipeline takes minutes; the MCP tool-call cap is 60 s. The
|
||
agent's calls timed out — and rather than retry blindly or invent a
|
||
number, it checked `read_ledger`, confirmed exactly one row was
|
||
written (dedup on run_id held), and reported the verdict as
|
||
*unreadable, not fabricated*, asking a human to read the ledger. Fix
|
||
shipped: async `run_verdict` + pollable `read_verdict`. Attempt 002
|
||
then read its verdict back cleanly.
|
||
|
||
2. **The SDK tool-restriction is not the trust boundary in this
|
||
environment.** The agent reached for a built-in `Bash`/`Monitor` tool
|
||
to read a source file too large for the (then whole-file-only)
|
||
`read_target_source`. Investigation showed that a nested Claude Code
|
||
CLI running as root under a parent session does not reliably gate
|
||
built-ins: `--dangerously-skip-permissions` is refused under root, and
|
||
a `can_use_tool` deny-by-default callback is shadowed by settings
|
||
inherited from the parent session (and broke legitimate MCP tool calls
|
||
with "Stream closed"). **This is expected, not a regression:** SPEC §10
|
||
specifies the boundary as OS-level process/user isolation and no-net
|
||
containers — "enforced by filesystem permissions and a separate-process
|
||
harness, not by prompt." The SDK allow/deny layer is defense in depth.
|
||
Mitigation shipped: windowed `read_target_source` removes the agent's
|
||
reason to reach for a shell; the load-bearing guarantees hold
|
||
regardless (see below).
|
||
|
||
## What holds even when the tool layer leaks
|
||
|
||
- **Writes to the target are only mediated through `propose_patch`** —
|
||
path-allowlist (relative paths, no `..`, no `.git`) then `git apply`
|
||
rooted in the target workspace. A patch touching `config/` or `crates/`
|
||
is refused before any apply. Verified against a malicious
|
||
accept-threshold diff in Phase 1.
|
||
- **The ledger is append-only** — UPDATE/DELETE are refused by SQLite
|
||
triggers, and `accepted` rows are written only by the `verdict` binary
|
||
after the gated pipeline. An agent cannot forge a passing verdict
|
||
through the harness.
|
||
- **The gate+bench pipeline is unbypassable through the harness** —
|
||
pristine-rebuilt baseline, interleaved A/B, CI-lower-bound rule.
|
||
|
||
The hole that used to close this section — "an agent with host shell
|
||
could `sqlite3 INSERT` a forged row or invoke the verdict binary
|
||
directly" — is now **closed by OS-level isolation** (`scripts/agent-
|
||
isolated.sh`, `just isolation-check`, 19/19 checks). The whole agent
|
||
process tree runs either in a mount namespace with the repo read-only
|
||
and `CAP_SYS_ADMIN` dropped, or as the unprivileged `hpagent` user whose
|
||
only write path is the harnessd socket. A forged `INSERT`, a direct
|
||
`verdict` run, a trust-layer edit, and an `rw` remount are all refused
|
||
from the agent's side; the entire 20-attempt grind ran behind it. The
|
||
production `user` mode over no-net target containers (`docs/infra.md`)
|
||
remains the deployment shape.
|
||
|
||
## Phase 2 exit criteria (SPEC §5) — ALL MET
|
||
|
||
- **Unattended profile→verdict loop:** met — comrak *and* tokei, behind
|
||
OS-level isolation.
|
||
- **≥1 auto-accepted win:** met — three verified, banked, fuzz- and
|
||
sanitizer-clean wins on tokei (+10.7%, +3.7%, +9.9%; compounded ~1.25×
|
||
vs upstream with the Phase-0 wins).
|
||
- **Zero false accepts across ≥20 audited attempts:** met — 20/20
|
||
audited (each twice: incremental during the grind, then a 20-way
|
||
adversarial re-audit fan-out), 0 shipped false accepts. The one
|
||
pipeline accept the audit overturned (`comrak-010`) hardened the
|
||
pipeline into machine-enforced sanitizer gating.
|
||
- **OS-level tool-boundary isolation:** met — was the tracked hardening
|
||
gap; shipped and verified this phase.
|
||
|
||
---
|
||
|
||
> **Note (2026-07-13).** The differential-fuzz and ASan/LSan runs
|
||
> described above ran **out-of-band** — per-attempt scripts during the
|
||
> 100% human audit, exactly as this case study records ("fuzz and
|
||
> sanitizers are per-attempt manual in this phase") — so the Phase 2
|
||
> ledger rows carry `fuzz_iters=0` in their machine record. The fuzz
|
||
> gate is now pipeline-integrated (`diff-test` runs the target's
|
||
> declared fuzz command); rows written after this date record the
|
||
> iteration count actually executed.
|