Commit Graph
3 Commits
Author SHA1 Message Date
Claude 73763321c6 bench: automated A/A + regression-injection calibration (Phase 1)
Adds 'bench-runner calibrate': N A/A sessions counted against a <5%
false-positive bar, and N sessions with a synthetic slowdown injected
as a real busy-wait inside the candidate's timed window, counted
against a >=95% detection bar (detected = ratio CI entirely below 1.0).
Bootstrap seed varies per session so resampling noise is not
correlated across the very statistic being validated. JSON evidence
includes per-session CIs, injected wait, acceptance verdict, and env
fingerprint.

First run against the real comrak workload (20+20 sessions, pinned
core): false-positive rate 0.000, detection rate 0.950 -> PASS.
Evidence: results/calibration/2026-07-06-comrak-full-calibration.json.
Closes the SPEC 3.1 acceptance-criteria gap; remaining Phase 1 gaps:
no-net containers, perf profiling, diff-test orchestration, agent IPC.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
2026-07-06 20:20:40 +00:00
Claude 4402775ed2 phase0: exit criteria met — tokei +10.4% median [8.5%, 12.0%] verified
Attempt 4 (phase0-tokei-002, class 3, ACCEPTED — first auto-accept):
every file was read through encoding_rs_io's DecodeReaderBytes via
generic read_to_end, which zero-fills every grown buffer chunk (~221MB
memset/run, 8.5% of Ir) and grows without a size hint. Patch: fs::read
with a BOM-gated fallback to the identical decode path (a UTF-8-BOM
file transcodes invalid sequences, so the fallback is required for
equivalence). Gates: golden byte-identical; differential fuzz old-vs-
new binaries 4,332 iterations (BOM grid, invalid UTF-8, binary junk,
randomized) zero divergence; 250 upstream tests green; ASan+LSan clean
with ASan-run output hashing to golden. Bench: +9.9% [8.0%, 12.7%].

Attempt 5 (phase0-tokei-003, ship tree, ACCEPTED): stacks fat LTO+cgu=1
(real +1.6% standalone but under threshold, phase0-tokei-001) on the
accepted patch; benched as a whole vs pristine: +10.4% [8.5%, 12.0%].
Median clears the Phase 0 exit bar; CI lower bound 8.5%.

Case study written (results/phase0/case-study-tokei.md), ledger at 5
attempts, CLAUDE.md phase line moved to Phase 1, next hypotheses
logged (tokei class-5 byte-skip table; comrak class-3 arena).

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
2026-07-06 19:57:05 +00:00
Claude ad0c6b6213 phase0: add CLAUDE.md project charter
Session-level non-negotiables, architecture map, phase tracker,
command surface, and definition of done for one optimization attempt.
Commands section notes the currently built command-level harness
entries alongside the planned per-target ones.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
2026-07-06 18:37:00 +00:00