phase0: dry run — comrak & tokei vendored, 3 attempts in ledger

Target selection (SPEC §6): comrak 0.53.0 (45c1995, BSD-2-Clause) and
tokei 14.0.0 (fa44e51, MIT/Apache-2.0); both CPU-bound with pinned
corpora (progit book 11MB; 8x replicated pinned trees, 221MB) and
committed MANIFEST.sha256 + golden-output pins + regen scripts.

Calibration: A/A null verdicts on both real workloads; taskset -c 2
pinning tightens the A/A CI from [0.938, 1.008] to [0.993, 1.008],
making the 2% accept threshold meaningful in this container.

Profiles (callgrind; perf unavailable): comrak allocator symbols 26.7%
of Ir; tokei perform_multi_line_analysis 31.7%.

Attempts (ledger + results/phase0/):
- comrak class-1 target-cpu=native: null [0.983, 1.011], rejected-bench
- comrak class-2 mimalloc swap: +4.6% [1.033, 1.058], 848 upstream
  tests green, golden byte-identical, ASan-proper clean; LSan flags
  484B in onig (reachability false-positive analysis in case study) ->
  needs-human-review per SPEC §8
- tokei class-1 fat LTO+cgu=1: +1.6% [1.003, 1.026], real but under
  the 2% bar, rejected-bench

Also: ledger-cli for manual rows, targets/ excluded from trust-layer
workspace, corpora/build artifacts gitignored, case study written.
Phase 0 exit criteria (>=10% verified win) not yet met; next
hypotheses logged in results/phase0/attempts.md.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
This commit is contained in:
Claude
2026-07-06 19:28:50 +00:00
parent 1ef2bb4fdb
commit 7c3c59f5f8
26 changed files with 1001 additions and 2 deletions
+87
View File
@@ -0,0 +1,87 @@
# Phase 0 case study — comrak (CommonMark/GFM renderer)
**Result: +4.6% median wall-time speedup (95% CI [+3.3%, +5.8%]) from a
2-line allocator swap, byte-identical output, upstream tests green,
verdict `needs-human-review` on a LeakSanitizer reachability flag.**
## Target
- comrak v0.53.0, commit `45c1995f`, BSD-2-Clause, built `--release`
(upstream profile already ships `lto = true`, `codegen-units = 1`).
- Workload: the target's own benchmark (benches/bench.sh) — the Pro Git
book, all translations, 11.0 MB concatenated markdown, single stream
through stdin, HTML to /dev/null. Corpus pinned:
`corpora/comrak/MANIFEST.sha256` (progit @ `61833a52`).
## Environment
Remote 4-core container, Intel Xeon @ 2.10GHz, kernel 6.18.5, governor
and turbo state unavailable/uncontrollable, SMT off, ASLR system
default, benches pinned with `taskset -c 2`, 30 measured + 3 warm-up
runs per side, interleaved ABAB, bootstrap 95% CI (10k resamples).
**A/A calibration, same session, pinned** (results/calibration/):
median 1.0006, 95% CI [0.9932, 1.0080] — null, as required. Unpinned
A/A for contrast: CI [0.9376, 1.0080] — 5x noisier; pinning is what
makes a 2% threshold meaningful in this container.
## Loop
1. **Profile** (callgrind; perf unavailable in container — Ir counts,
not cycles): allocator symbols total **26.7% of instructions**
(`_int_malloc` 9.99%, `_int_free` 6.85%, `malloc` 3.92%,
`malloc_consolidate` 3.48%, `free` 2.49%). Parser frames
(`parse_inline` 6.7%, `process_line` 6.5%) come *after* the
allocator. Full ranking: results/comrak/hotspots.txt.
2. **Attempt 1 — class 1, build config** (`target-cpu=native`):
hypothesis logged first; golden replay byte-identical; bench median
0.9965, 95% CI [0.9826, 1.0110] → **rejected-bench** (null). Ledger
`phase0-comrak-001`.
3. **Attempt 2 — class 2, allocator swap** (mimalloc as
`#[global_allocator]` in the CLI binary only; library consumers
unaffected; patch: results/phase0/comrak-mimalloc.patch):
- Golden replay: stdout sha256 identical to pinned golden.
- Upstream test suite on the patched tree: green (see ledger row).
- Bench, interleaved A/B vs pristine-built baseline: **median
speedup 1.0459, 95% CI [1.0333, 1.0577]** — CI lower bound 3.3%
clears the 2% accept threshold.
- Sanitizers (nightly ASan on the full corpus run):
- pristine binary: clean, output hash = golden.
- patched binary: ASan-proper clean with `detect_leaks=0`;
**LeakSanitizer reports 484 B / 3 allocations in
`onig_new`/`onig_compile`** (oniguruma, syntect's C regex
engine). Analysis: these allocations exist identically in the
pristine binary (LSan exit 0 there) — the report appears because
LSan cannot scan mimalloc-owned memory for roots, so a
still-reachable global compiled regex looks unreachable. Almost
certainly a false positive of the LSan × custom-allocator
interaction, **not** a leak introduced by the patch.
- Per SPEC §8 ("anything the sanitizers flag, even as warnings"):
verdict **needs-human-review**, not auto-accept. Ledger
`phase0-comrak-002`.
## ROI (illustrative, config/pricing.toml rates)
For a hypothetical fleet spending 100 cores 24/7 on this workload shape,
at $0.04/core-hr (public-cloud example rate — replace per engagement):
- cores returned: median **4.4**, 95% CI [3.2, 5.5]
(`fleet × (1 − 1/speedup)`)
- dollars/yr: median **$1,540**, 95% CI [$1,130, $1,910]
Caveats printed with the number, per policy: single workload shape
(large-document batch rendering); Ir-based profile, not cycles;
container hardware without governor control; speedup CI is wall-time on
pinned single-core execution; ISA = generic x86-64 (no target-cpu
narrowing — attempt 1 showed it buys nothing here anyway).
## Reproduction
```
just pin-check comrak # corpus manifest verify
cargo run -p bench-runner -- --config config/accept.toml compare \
--baseline "taskset -c 2 sh -c '<pristine comrak> < corpora/comrak/progit.md > /dev/null'" \
--candidate "taskset -c 2 sh -c '<patched comrak> < corpora/comrak/progit.md > /dev/null'"
```
Patch: `results/phase0/comrak-mimalloc.patch` applied at `45c1995f`.