diff --git a/CLAUDE.md b/CLAUDE.md index cde5ecd..9ebaa12 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -32,7 +32,7 @@ Full engineering spec: `SPEC.md`. Read it before starting any phase. ## Current phase -**Phase 2 in progress — agent loop.** The unattended profile→hypothesize→patch→gated-verdict loop runs on comrak via the Claude Agent SDK (`agent/hotpath_agent/loop.py`, model claude-fable-5) over a stdio MCP server wrapping the seven harness tools. 3 audited attempts (phase2-comrak-001/002/003), all honest rejections, **0 false accepts**; the agent twice refused to fabricate an unreadable verdict (results/phase2/case-study-agent-loop.md). Shipped: async run_verdict + pollable read_verdict, windowed read_target_source. Exit-criteria gaps: an auto-accepted win (needs richer profiling to unlock more classes), and OS-level process/user isolation for the agent's tool boundary (SPEC §10; the SDK allow/deny layer is defense-in-depth only — see docs/infra.md). (Update this line as phases complete; exit criteria in SPEC.md §5.) +**Phase 2 COMPLETE — all SPEC §5 exit criteria met.** The unattended profile→hypothesize→patch→gated-verdict loop ran on comrak *and* tokei via the Claude Agent SDK (`agent/hotpath_agent/loop.py`, model claude-fable-5) over a stdio MCP server wrapping the seven harness tools, behind OS-level isolation (`scripts/agent-isolated.sh` mountns/user modes, `just isolation-check` 19/19). **20 audited attempts, 0 shipped false accepts, 3 verified auto-accepted wins** — all tokei, class 5 (phase2-tokei-001 +10.7%, -003 +3.7%, -008 +9.9%; compounded ~1.25× vs upstream v14.0.0 with the Phase-0 wins), each fuzz-clean (`scripts/diff-fuzz-*.py`, 10k inputs) and ASan/LSan-clean. One pipeline accept (phase2-comrak-010, a `ManuallyDrop` teardown skip) was **overturned by the 100% human audit** on a LeakSanitizer flag before shipping; the fix made `verdict` run ASan/LSan on every would-be accept and cap flagged wins at needs-human-review (SPEC §8). Every row re-audited twice — incrementally during the grind, then a 20-way adversarial refute-the-verdict fan-out (`phase2-final-audit` workflow): all confirmed, no gaming, ledger append-only intact. Case study: results/phase2/case-study-agent-loop.md. **Next: Phase 3** (playbook classes 3–7, coz, C/C++ targets, 3–5 public case studies — SPEC.md §5). ## Commands diff --git a/crates/verdict/src/main.rs b/crates/verdict/src/main.rs index 3392ed9..fc37c97 100644 --- a/crates/verdict/src/main.rs +++ b/crates/verdict/src/main.rs @@ -296,12 +296,22 @@ fn main() -> Result<()> { Some(p) => std::fs::read_to_string(p)?, None => "(no source patch)".into(), }; + // Derive the phase from the run_id namespace (phaseN-*) so the column + // matches the run_id prefix; the historical hardcoded `1` left every + // verdict-written row tagged phase 1 regardless of namespace (found in + // the phase2-final-audit sweep). Falls back to 1 for un-prefixed ids. + let phase = cli + .run_id + .strip_prefix("phase") + .and_then(|r| r.split('-').next()) + .and_then(|n| n.parse::().ok()) + .unwrap_or(1); let attempt = Attempt { run_id: cli.run_id.clone(), timestamp: now_utc(), target: cli.target.clone(), target_commit: spec.source.commit.clone(), - phase: 1, + phase, hotspot: cli.hotspot, playbook_class: cli.playbook_class, hypothesis: cli.hypothesis, diff --git a/results/phase2/case-study-agent-loop.md b/results/phase2/case-study-agent-loop.md index fdb4cbb..093b2b3 100644 --- a/results/phase2/case-study-agent-loop.md +++ b/results/phase2/case-study-agent-loop.md @@ -22,14 +22,63 @@ anti-reward-hacking behavior SPEC §10 is designed to produce.** - `read_target_source` gained an `offset`/`limit` line window so 100 KB+ source files can be paged instead of overflowing one response. -## The attempts (all in the ledger, `results/ledger.sqlite`) +## The scoreboard (SPEC §5 Phase 2 exit criteria — MET) -| run_id | class chosen | patch | verdict | speedup CI | +- **Unattended profile→verdict loop on ≥1 target:** met — runs on + **both** comrak and tokei, behind OS-level isolation. +- **≥1 auto-accepted win:** met — **three verified, banked wins**, all + on tokei (see table). +- **Zero false accepts across ≥20 audited attempts:** met — **20/20 + audited, 0 shipped false accepts.** One row (`phase2-comrak-010`) + the *pipeline* accepted was overturned by the 100% human audit before + anything shipped, and drove the structural fix that made the pipeline + enforce sanitizers itself (below). + +Every one of the 20 rows was independently re-audited a second time by +a fan-out of 20 adversarial verifiers (`phase2-final-audit` workflow), +each trying to *refute* its row's verdict: all 20 confirmed +`verdict_follows_evidence`, correct equivalence tier, patch-matches- +hypothesis, and **no harness-gaming signs**. A separate check confirmed +the ledger still refuses `UPDATE`/`DELETE` (append-only triggers hold). + +## The attempts (all 20 in the ledger, `results/ledger.sqlite`) + +The discriminator that separates the three banked wins from the +overturned `comrak-010` is **not** the bench CI (comrak-010 cleared the +2% bar too) — it is the sanitizer result: the three tokei wins are +ASan/LSan-clean and carry bank commits; comrak-010 failed LeakSanitizer +and carries none. + +| run_id | class | verdict | speedup median | 95% CI | |---|---|---|---|---| -| phase2-comrak-001 | 3 alloc-churn | syntect buffer `with_capacity` | rejected-bench | [0.971, 1.007] | -| phase2-comrak-002 | 5 algorithmic | gate dead `VecDeque` build in text postprocess | rejected-bench | [0.982, 1.026] | -| phase2-comrak-003 | 6 SIMD | hoist `^`-in-brackets check out of `find_special_char` predicate | rejected-bench | [0.970, 1.007] | -| phase2-tokei-001 | 5 algorithmic | first-byte gate table for per-byte token matchers in `perform_multi_line_analysis` | **accepted** | **[1.084, 1.131]** | +| phase2-comrak-001 | 3 | rejected-bench | 0.990 | [0.972, 1.007] | +| phase2-comrak-002 | 5 | rejected-bench | 1.003 | [0.982, 1.026] | +| phase2-comrak-003 | 6 | rejected-bench | 0.987 | [0.970, 1.007] | +| phase2-comrak-004 | 5 | rejected-bench | 1.012 | [1.001, 1.027] | +| phase2-comrak-005 | 4 | rejected-bench | 1.009 | [0.985, 1.026] | +| phase2-comrak-006 | 5 | rejected-bench | 1.008 | [0.990, 1.020] | +| phase2-comrak-007 | 5 | rejected-bench | 1.010 | [0.990, 1.023] | +| phase2-comrak-008 | 5 | rejected-bench | 0.983 | [0.963, 0.999] | +| phase2-comrak-009 | 1 | rejected-bench | 1.004 | [0.982, 1.019] | +| phase2-comrak-010 | 3 | ~~accepted~~ **overturned** | 1.107 | [1.079, 1.121] | +| phase2-comrak-011 | 5 | rejected-bench | 1.010 | [0.979, 1.033] | +| phase2-tokei-001 | 5 | **accepted** | **1.107** | **[1.084, 1.131]** | +| phase2-tokei-002 | 5 | rejected-gate | — | (golden replay) | +| phase2-tokei-003 | 5 | **accepted** | **1.037** | **[1.027, 1.052]** | +| phase2-tokei-004 | 2 | rejected-bench | 1.001 | [0.985, 1.017] | +| phase2-tokei-005 | 6 | rejected-bench | 0.997 | [0.982, 1.013] | +| phase2-tokei-006 | 5 | rejected-bench | 0.988 | [0.983, 0.997] | +| phase2-tokei-007 | 5 | rejected-bench | 0.993 | [0.984, 1.004] | +| phase2-tokei-008 | 5 | **accepted** | **1.099** | **[1.073, 1.112]** | +| phase2-tokei-009 | 5 | rejected-bench | 1.057 | [1.002, 1.084] | + +The three banked wins compound: tokei runs the pinned 221 MB / 4360-file +workload roughly **1.25× faster** than upstream v14.0.0 (Phase-0 wins +plus these three), each increment individually CI-significant, fuzz- +clean, and sanitizer-clean. `tokei-009` is the discipline in miniature: +a real +5.7% median that was **rejected** because its noisy CI lower +bound (+0.2%) couldn't clear the 2% bar — a win that doesn't survive the +statistics doesn't exist. ### phase2-tokei-001 — the first auto-accepted win, and its audit @@ -92,7 +141,7 @@ too small to clear the 2% CI-lower-bound bar. **No false accepts.** ## The stop-the-line event: phase2-comrak-010 The pipeline auto-accepted a comrak class-3 patch (+10.7% median, -CI [+7.9%, +12.2%]) that skips AST-arena teardown via `ManuallyDrop` +CI [+7.9%, +12.1%]) that skips AST-arena teardown via `ManuallyDrop` before `process::exit` — and the 100% human audit **overturned it**. LeakSanitizer flags a 384-byte arena chunk as unreachable at exit: the patch's `black_box(&arena)` keeps the arena live only until its scope @@ -155,22 +204,29 @@ output equivalence was real; only the teardown-leak tier was wrong. - **The gate+bench pipeline is unbypassable through the harness** — pristine-rebuilt baseline, interleaved A/B, CI-lower-bound rule. -The residual hole in THIS container: an agent with host shell could, in -principle, `sqlite3 INSERT` a forged row (triggers block mutation, not -insertion) or invoke the verdict binary directly. That is precisely the -attack SPEC §10's process/user isolation exists to prevent, and it is the -**Phase 2 exit gap to close on real infrastructure** (run the agent -process as a separate unprivileged user with no shell and the trust layer -owned by another uid — the `bench-metal`/`docs/infra.md` setup). +The hole that used to close this section — "an agent with host shell +could `sqlite3 INSERT` a forged row or invoke the verdict binary +directly" — is now **closed by OS-level isolation** (`scripts/agent- +isolated.sh`, `just isolation-check`, 19/19 checks). The whole agent +process tree runs either in a mount namespace with the repo read-only +and `CAP_SYS_ADMIN` dropped, or as the unprivileged `hpagent` user whose +only write path is the harnessd socket. A forged `INSERT`, a direct +`verdict` run, a trust-layer edit, and an `rw` remount are all refused +from the agent's side; the entire 20-attempt grind ran behind it. The +production `user` mode over no-net target containers (`docs/infra.md`) +remains the deployment shape. -## Phase 2 exit criteria (SPEC §5) +## Phase 2 exit criteria (SPEC §5) — ALL MET -- Unattended profile→verdict loop on one target: **met** (comrak). -- ≥1 auto-accepted win: **not yet** — both attempts were honest - rejections. The loop is proven; a win needs either a richer profile - (perf `stat` cache-miss data to unlock class-4 preconditions) or more - attempts against fresh hypotheses. -- Zero false accepts across audited attempts: **met so far** (2/2 - audited, both correctly rejected). -- The separate-uid isolation for the tool boundary is the remaining - hardening item, tracked to `docs/infra.md`. +- **Unattended profile→verdict loop:** met — comrak *and* tokei, behind + OS-level isolation. +- **≥1 auto-accepted win:** met — three verified, banked, fuzz- and + sanitizer-clean wins on tokei (+10.7%, +3.7%, +9.9%; compounded ~1.25× + vs upstream with the Phase-0 wins). +- **Zero false accepts across ≥20 audited attempts:** met — 20/20 + audited (each twice: incremental during the grind, then a 20-way + adversarial re-audit fan-out), 0 shipped false accepts. The one + pipeline accept the audit overturned (`comrak-010`) hardened the + pipeline into machine-enforced sanitizer gating. +- **OS-level tool-boundary isolation:** met — was the tracked hardening + gap; shipped and verified this phase.