benbrown/peltier.io
2
1
Fork 0
mirror of https://github.com/bdbrown4/peltier.io.git synced 2026-11-07 10:58:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

phase2: close out — all SPEC §5 exit criteria met

Browse Source 
20 audited attempts, 0 shipped false accepts, 3 verified auto-accepted
wins (tokei class-5: +10.7%, +3.7%, +9.9%; ~1.25x vs upstream with
Phase-0 wins). Independent 20-way adversarial re-audit (phase2-final-audit
workflow) confirmed every verdict follows its evidence, correct
equivalence tier, no harness-gaming, ledger append-only intact.

Closeout changes:
- case study: full 20-row attempt table, scoreboard, corrected
  comrak-010 CI upper bound (+12.1%, was +12.2%), exit-criteria section
  flipped to ALL MET, OS-isolation gap marked closed.
- CLAUDE.md Current-phase line: Phase 2 complete, next Phase 3.
- verdict: derive the ledger 'phase' column from the run_id namespace
  (phaseN-*) instead of the hardcoded 1 the re-audit flagged; historical
  rows are immutable and stay as written (run_id prefix is authoritative).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
This commit is contained in:
Claude
2026-07-11 17:47:37 +00:00
parent 0c4de51a1a
commit cabac5268d
3 changed files with 92 additions and 26 deletions
Download Patch File Download Diff File Expand all files Collapse all files
results/phase2/case-study-agent-loop.md
+80 -24
View File
@@ -22,14 +22,63 @@ anti-reward-hacking behavior SPEC §10 is designed to produce.**
- `read_target_source` gained an `offset`/`limit` line window so 100 KB+
source files can be paged instead of overflowing one response.
## The attempts (all in the ledger, `results/ledger.sqlite`)
## The scoreboard (SPEC §5 Phase 2 exit criteria — MET)
| run_id | class chosen | patch | verdict | speedup CI |
- **Unattended profile→verdict loop on ≥1 target:** met — runs on
**both** comrak and tokei, behind OS-level isolation.
- **≥1 auto-accepted win:** met — **three verified, banked wins**, all
on tokei (see table).
- **Zero false accepts across ≥20 audited attempts:** met — **20/20
audited, 0 shipped false accepts.** One row (`phase2-comrak-010`)
the *pipeline* accepted was overturned by the 100% human audit before
anything shipped, and drove the structural fix that made the pipeline
enforce sanitizers itself (below).
Every one of the 20 rows was independently re-audited a second time by
a fan-out of 20 adversarial verifiers (`phase2-final-audit` workflow),
each trying to *refute* its row's verdict: all 20 confirmed
`verdict_follows_evidence`, correct equivalence tier, patch-matches-
hypothesis, and **no harness-gaming signs**. A separate check confirmed
the ledger still refuses `UPDATE`/`DELETE` (append-only triggers hold).
## The attempts (all 20 in the ledger, `results/ledger.sqlite`)
The discriminator that separates the three banked wins from the
overturned `comrak-010` is **not** the bench CI (comrak-010 cleared the
2% bar too) — it is the sanitizer result: the three tokei wins are
ASan/LSan-clean and carry bank commits; comrak-010 failed LeakSanitizer
and carries none.
| run_id | class | verdict | speedup median | 95% CI |
|---|---|---|---|---|
| phase2-comrak-001 | 3 alloc-churn | syntect buffer `with_capacity` | rejected-bench | [0.971, 1.007] |
| phase2-comrak-002 | 5 algorithmic | gate dead `VecDeque` build in text postprocess | rejected-bench | [0.982, 1.026] |
| phase2-comrak-003 | 6 SIMD | hoist `^`-in-brackets check out of `find_special_char` predicate | rejected-bench | [0.970, 1.007] |
| phase2-tokei-001 | 5 algorithmic | first-byte gate table for per-byte token matchers in `perform_multi_line_analysis` | **accepted** | **[1.084, 1.131]** |
| phase2-comrak-001 | 3 | rejected-bench | 0.990 | [0.972, 1.007] |
| phase2-comrak-002 | 5 | rejected-bench | 1.003 | [0.982, 1.026] |
| phase2-comrak-003 | 6 | rejected-bench | 0.987 | [0.970, 1.007] |
| phase2-comrak-004 | 5 | rejected-bench | 1.012 | [1.001, 1.027] |
| phase2-comrak-005 | 4 | rejected-bench | 1.009 | [0.985, 1.026] |
| phase2-comrak-006 | 5 | rejected-bench | 1.008 | [0.990, 1.020] |
| phase2-comrak-007 | 5 | rejected-bench | 1.010 | [0.990, 1.023] |
| phase2-comrak-008 | 5 | rejected-bench | 0.983 | [0.963, 0.999] |
| phase2-comrak-009 | 1 | rejected-bench | 1.004 | [0.982, 1.019] |
| phase2-comrak-010 | 3 | ~~accepted~~ **overturned** | 1.107 | [1.079, 1.121] |
| phase2-comrak-011 | 5 | rejected-bench | 1.010 | [0.979, 1.033] |
| phase2-tokei-001 | 5 | **accepted** | **1.107** | **[1.084, 1.131]** |
| phase2-tokei-002 | 5 | rejected-gate | — | (golden replay) |
| phase2-tokei-003 | 5 | **accepted** | **1.037** | **[1.027, 1.052]** |
| phase2-tokei-004 | 2 | rejected-bench | 1.001 | [0.985, 1.017] |
| phase2-tokei-005 | 6 | rejected-bench | 0.997 | [0.982, 1.013] |
| phase2-tokei-006 | 5 | rejected-bench | 0.988 | [0.983, 0.997] |
| phase2-tokei-007 | 5 | rejected-bench | 0.993 | [0.984, 1.004] |
| phase2-tokei-008 | 5 | **accepted** | **1.099** | **[1.073, 1.112]** |
| phase2-tokei-009 | 5 | rejected-bench | 1.057 | [1.002, 1.084] |
The three banked wins compound: tokei runs the pinned 221 MB / 4360-file
workload roughly **1.25× faster** than upstream v14.0.0 (Phase-0 wins
plus these three), each increment individually CI-significant, fuzz-
clean, and sanitizer-clean. `tokei-009` is the discipline in miniature:
a real +5.7% median that was **rejected** because its noisy CI lower
bound (+0.2%) couldn't clear the 2% bar — a win that doesn't survive the
statistics doesn't exist.
### phase2-tokei-001 — the first auto-accepted win, and its audit
@@ -92,7 +141,7 @@ too small to clear the 2% CI-lower-bound bar. **No false accepts.**
## The stop-the-line event: phase2-comrak-010
The pipeline auto-accepted a comrak class-3 patch (+10.7% median,
CI [+7.9%, +12.2%]) that skips AST-arena teardown via `ManuallyDrop`
CI [+7.9%, +12.1%]) that skips AST-arena teardown via `ManuallyDrop`
before `process::exit` — and the 100% human audit **overturned it**.
LeakSanitizer flags a 384-byte arena chunk as unreachable at exit: the
patch's `black_box(&arena)` keeps the arena live only until its scope
@@ -155,22 +204,29 @@ output equivalence was real; only the teardown-leak tier was wrong.
- **The gate+bench pipeline is unbypassable through the harness** —
pristine-rebuilt baseline, interleaved A/B, CI-lower-bound rule.
The residual hole in THIS container: an agent with host shell could, in
principle, `sqlite3 INSERT` a forged row (triggers block mutation, not
insertion) or invoke the verdict binary directly. That is precisely the
attack SPEC §10's process/user isolation exists to prevent, and it is the
**Phase 2 exit gap to close on real infrastructure** (run the agent
process as a separate unprivileged user with no shell and the trust layer
owned by another uid — the `bench-metal`/`docs/infra.md` setup).
The hole that used to close this section — "an agent with host shell
could `sqlite3 INSERT` a forged row or invoke the verdict binary
directly" — is now **closed by OS-level isolation** (`scripts/agent-
isolated.sh`, `just isolation-check`, 19/19 checks). The whole agent
process tree runs either in a mount namespace with the repo read-only
and `CAP_SYS_ADMIN` dropped, or as the unprivileged `hpagent` user whose
only write path is the harnessd socket. A forged `INSERT`, a direct
`verdict` run, a trust-layer edit, and an `rw` remount are all refused
from the agent's side; the entire 20-attempt grind ran behind it. The
production `user` mode over no-net target containers (`docs/infra.md`)
remains the deployment shape.
## Phase 2 exit criteria (SPEC §5)
## Phase 2 exit criteria (SPEC §5) — ALL MET
- Unattended profile→verdict loop on one target: **met** (comrak).
- ≥1 auto-accepted win: **not yet** — both attempts were honest
rejections. The loop is proven; a win needs either a richer profile
(perf `stat` cache-miss data to unlock class-4 preconditions) or more
attempts against fresh hypotheses.
- Zero false accepts across audited attempts: **met so far** (2/2
audited, both correctly rejected).
- The separate-uid isolation for the tool boundary is the remaining
hardening item, tracked to `docs/infra.md`.
- **Unattended profile→verdict loop:** met — comrak *and* tokei, behind
OS-level isolation.
- **≥1 auto-accepted win:** met — three verified, banked, fuzz- and
sanitizer-clean wins on tokei (+10.7%, +3.7%, +9.9%; compounded ~1.25×
vs upstream with the Phase-0 wins).
- **Zero false accepts across ≥20 audited attempts:** met — 20/20
audited (each twice: incremental during the grind, then a 20-way
adversarial re-audit fan-out), 0 shipped false accepts. The one
pipeline accept the audit overturned (`comrak-010`) hardened the
pipeline into machine-enforced sanitizer gating.
- **OS-level tool-boundary isolation:** met — was the tracked hardening
gap; shipped and verified this phase.