phase0: exit criteria met — tokei +10.4% median [8.5%, 12.0%] verified

Attempt 4 (phase0-tokei-002, class 3, ACCEPTED — first auto-accept): every file was read through encoding_rs_io's DecodeReaderBytes via generic read_to_end, which zero-fills every grown buffer chunk (~221MB memset/run, 8.5% of Ir) and grows without a size hint. Patch: fs::read with a BOM-gated fallback to the identical decode path (a UTF-8-BOM file transcodes invalid sequences, so the fallback is required for equivalence). Gates: golden byte-identical; differential fuzz old-vs- new binaries 4,332 iterations (BOM grid, invalid UTF-8, binary junk, randomized) zero divergence; 250 upstream tests green; ASan+LSan clean with ASan-run output hashing to golden. Bench: +9.9% [8.0%, 12.7%]. Attempt 5 (phase0-tokei-003, ship tree, ACCEPTED): stacks fat LTO+cgu=1 (real +1.6% standalone but under threshold, phase0-tokei-001) on the accepted patch; benched as a whole vs pristine: +10.4% [8.5%, 12.0%]. Median clears the Phase 0 exit bar; CI lower bound 8.5%. Case study written (results/phase0/case-study-tokei.md), ledger at 5 attempts, CLAUDE.md phase line moved to Phase 1, next hypotheses logged (tokei class-5 byte-skip table; comrak class-3 arena). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
2026-11-07 10:58:28 +00:00 · 2026-07-06 19:57:05 +00:00
parent 7c3c59f5f8
commit 4402775ed2
10 changed files with 263 additions and 14 deletions
@@ -0,0 +1,64 @@
+diff --git a/Cargo.toml b/Cargo.toml
+index 8b19522..b64e2b4 100644
+--- a/Cargo.toml
+++ b/Cargo.toml
+@@ -30,7 +30,8 @@ default = ["cli"]
+ yaml = ["dep:serde_yaml"]
+ 
+ [profile.release]
+-lto = "thin"
+lto = true
+codegen-units = 1
+ panic = "abort"
+ 
+ [[bin]]
+diff --git a/src/language/language_type.rs b/src/language/language_type.rs
+index bffdf7a..86c9f94 100644
+--- a/src/language/language_type.rs
+++ b/src/language/language_type.rs
+@@ -1,6 +1,6 @@
+ use std::{
+     borrow::Cow,
+-    fmt,
+    fmt, fs,
+     fs::File,
+     io::{self, Read},
+     path::{Path, PathBuf},
+@@ -38,17 +38,29 @@ impl LanguageType {
+     /// on success and giving back ownership of [`PathBuf`] on error.
+     pub fn parse(self, path: PathBuf, config: &Config) -> Result<Report, (io::Error, PathBuf)> {
+         let text = {
+-            let f = match File::open(&path) {
+-                Ok(f) => f,
+            // fs::read pre-sizes from metadata and skips buffer zeroing;
+            // the decoding reader only matters when a BOM is present, so
+            // BOM-less files (the overwhelming majority) take the raw path.
+            // encoding_rs_io passes BOM-less input through unchanged, so
+            // both paths produce identical bytes.
+            let raw = match fs::read(&path) {
+                Ok(raw) => raw,
+                 Err(e) => return Err((e, path)),
+             };
+-            let mut s = Vec::new();
+-            let mut reader = DecodeReaderBytesBuilder::new().build(f);
+-
+-            if let Err(e) = reader.read_to_end(&mut s) {
+-                return Err((e, path));
+            let has_bom = raw.starts_with(&[0xEF, 0xBB, 0xBF])
+                || raw.starts_with(&[0xFF, 0xFE])
+                || raw.starts_with(&[0xFE, 0xFF]);
+            if has_bom {
+                let mut s = Vec::new();
+                let mut reader =
+                    DecodeReaderBytesBuilder::new().build(io::Cursor::new(raw));
+                if let Err(e) = reader.read_to_end(&mut s) {
+                    return Err((e, path));
+                }
+                s
+            } else {
+                raw
+             }
+-            s
+         };
+ 
+         let mut stats = Report::new(path);