Commit Graph
6 Commits
Author SHA1 Message Date
Claude 393ca7117f agent: case study — phase2-tokei-001 verified win + audit methodology finding
Also refreshes tokei's cache-sim profile against the new banked-wins
baseline (parse_lines share drops after the first-byte gate landed).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
2026-07-10 05:11:22 +00:00
Claude abaed3c49d agent: phase2-tokei-001 — first auto-accepted win from the unattended loop
Ledger row: class 5 (algorithmic) on tokei, first-byte gate table for
the per-byte token matchers in perform_multi_line_analysis.
Speedup median 1.107, 95% CI [1.084, 1.131] vs the banked-wins baseline,
workload: 221 MB / 4360 files, single rayon thread, pinned core.

Manual audit gates (pipeline covers tests+golden+bench only):
- Differential fuzz, 10k mutated inputs across 200 batched runs:
  0 divergences after canonicalizing tokei's benign parallel-walker
  output ordering (present in pristine baseline; counts identical).
  Fuzzer added as scripts/diff-fuzz-tokei.py.
- ASan run recorded in the case study (next commit).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
2026-07-10 05:08:16 +00:00
Claude 189939fe58 agent: transcript progress snapshot
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
2026-07-10 04:58:29 +00:00
Claude d49169084f agent: phase2-tokei-001 transcript (in progress)
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
2026-07-10 04:57:10 +00:00
Claude 5b198b7091 agent: per-attempt audit script for the Phase 2 grind
Mechanically re-checks a ledger row against the accept rule (CI lower
bound vs threshold, gate greenness, patch path allowlist) and prints
the full evidence for the human part of the 100% Phase 2 audit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
2026-07-10 04:52:43 +00:00
Claude 535821cd48 agent: unattended Phase 2 loop — profile to gated verdict, zero false accepts
The Claude Agent SDK loop (agent/hotpath_agent/loop.py, model
claude-fable-5) drives the seven harness tools over a standalone stdio
MCP server (mcp_server.py) and runs the full profile -> hypothesize ->
patch -> gated-verdict cycle on comrak with no human in the loop.

Three audited attempts, all honest rejections, ZERO false accepts:
- phase2-comrak-001 class 3: syntect buffer with_capacity, [0.971,1.007]
- phase2-comrak-002 class 5: gate dead VecDeque build, [0.982,1.026]
- phase2-comrak-003 class 6: hoist find_special_char check, [0.970,1.007]
The agent picked the cheapest untried class each time (3->5->6) and
TWICE refused to fabricate a verdict it couldn't read back, verifying
ledger integrity instead — the anti-reward-hacking behavior SPEC §10
targets.

Harness changes surfaced by the loop:
- run_verdict is now async: launches the multi-minute build+bench
  detached (past the 60s MCP transport cap) and returns immediately;
  new read_verdict op polls the append-only ledger row (ledger
  verdict_summary). Fixes attempt 001's unreadable-verdict gap.
- read_target_source takes offset/limit line windows so 100KB+ files
  page instead of overflowing — removes the agent's reason to reach for
  a shell (attempt 003 paged a 2400-line file cleanly, no shell).

Honest boundary note (case study + code comments): in THIS dev
container the SDK tool allow/deny layer is defense-in-depth, not the
trust boundary — a nested CLI as root under a parent session doesn't
reliably gate built-ins. SPEC §10's boundary is OS process/user
isolation; that hardening is the remaining Phase 2 exit item
(docs/infra.md). What holds regardless: propose_patch is the only
mediated write path (path-allowlist + git apply) and the ledger is
append-only.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
2026-07-09 23:29:52 +00:00