From 014e3b11d1d53576b2a0f366967f0698aaa7fcf7 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 11 Jul 2026 18:17:26 +0000 Subject: [PATCH 1/7] phase3: generalize trust layer for C/C++; wire cJSON target MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Trust-layer generalization (zero-regression verified on tokei): - TargetSpec [build] uses an explicit {out} isolation-dir placeholder instead of injected CARGO_TARGET_DIR, plus optional sanitizer/ sanitizer_binary templates. Nothing assumes cargo anymore. - verdict rebuild_pristine_baseline, sanitizer_check, and harnessd run_verdict candidate build all use {out} substitution. A target with no sanitizer build caps accepts at needs-human-review. - tokei/comrak target.toml migrated to the {out} form (CARGO_TARGET_DIR= {out} + nightly ASan sanitizer command). cJSON C target (v1.7.19, MIT): - harness.c bench/golden driver (trust-layer; agent can patch workspace cJSON.c but not the rig), compiled with clang against the patched tree. - 5.6 MB deterministic corpus (gen-corpus.sh) + MANIFEST/GOLDEN pins. - gates = hermetic Unity/CTest suite (19 tests pass); sanitizer = ASan+UBSan; scripts/diff-fuzz-cjson.py for differential fuzzing. coz causal profiler integration (SPEC §12): - harness.c optional -DHOTPATH_COZ throughput progress point. - scripts/install-coz.sh, coz-profile.sh, coz-summary.py; just coz/install-coz. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2 --- .gitignore | 4 ++ corpora/cjson/GOLDEN.sha256 | 1 + corpora/cjson/MANIFEST.sha256 | 1 + corpora/cjson/gen-corpus.sh | 62 ++++++++++++++++++++ crates/diff-test/src/target.rs | 25 ++++++++- crates/harnessd/src/main.rs | 21 +++---- crates/verdict/src/main.rs | 90 +++++++++++++++-------------- justfile | 9 +++ results/ledger.sqlite | Bin 151552 -> 155648 bytes scripts/coz-profile.sh | 41 ++++++++++++++ scripts/coz-summary.py | 75 +++++++++++++++++++++++++ scripts/diff-fuzz-cjson.py | 100 +++++++++++++++++++++++++++++++++ scripts/install-coz.sh | 28 +++++++++ targets/cjson/harness.c | 73 ++++++++++++++++++++++++ targets/cjson/target.toml | 44 +++++++++++++++ targets/comrak/target.toml | 6 +- targets/tokei/target.toml | 6 +- 17 files changed, 528 insertions(+), 58 deletions(-) create mode 100644 corpora/cjson/GOLDEN.sha256 create mode 100644 corpora/cjson/MANIFEST.sha256 create mode 100644 corpora/cjson/gen-corpus.sh create mode 100644 scripts/coz-profile.sh create mode 100644 scripts/coz-summary.py create mode 100644 scripts/diff-fuzz-cjson.py create mode 100644 scripts/install-coz.sh create mode 100644 targets/cjson/harness.c create mode 100644 targets/cjson/target.toml diff --git a/.gitignore b/.gitignore index eab05b6..168be9c 100644 --- a/.gitignore +++ b/.gitignore @@ -239,3 +239,7 @@ targets/*/baseline/ # Phase 2 agent artifacts (transcripts kept; pending logs are scratch) results/pending/ agent/**/__pycache__/ + +# cjson corpus is regenerable via gen-corpus.sh (deterministic) +corpora/cjson/input/ +targets/cjson/testbuild/ diff --git a/corpora/cjson/GOLDEN.sha256 b/corpora/cjson/GOLDEN.sha256 new file mode 100644 index 0000000..8053657 --- /dev/null +++ b/corpora/cjson/GOLDEN.sha256 @@ -0,0 +1 @@ +9273ad45a9aa51dd7f941d938a771de3e95a7b3ac807a880f24392c1eb6a72bb golden-stdout diff --git a/corpora/cjson/MANIFEST.sha256 b/corpora/cjson/MANIFEST.sha256 new file mode 100644 index 0000000..39af4dd --- /dev/null +++ b/corpora/cjson/MANIFEST.sha256 @@ -0,0 +1 @@ +3b0c1dee5a75de59b12e692780896f509c93f3c6b8da7cc337ae4df7fbf70075 ./big.json diff --git a/corpora/cjson/gen-corpus.sh b/corpora/cjson/gen-corpus.sh new file mode 100644 index 0000000..3763094 --- /dev/null +++ b/corpora/cjson/gen-corpus.sh @@ -0,0 +1,62 @@ +#!/bin/sh +# Deterministically generate the cjson corpus — a large synthetic JSON +# document exercising every parse path: nested objects/arrays, strings +# with escapes and unicode, ints, floats (the strtod/sprintf hot path), +# booleans, nulls. No randomness — byte-identical on every machine, so +# the MANIFEST.sha256 pin is reproducible. Run from repo root: +# sh corpora/cjson/gen-corpus.sh +set -eu +DIR="$(cd "$(dirname "$0")" && pwd)" +OUT="$DIR/input/big.json" +mkdir -p "$DIR/input" + +python3 - "$OUT" <<'PY' +import json, sys + +# Deterministic: a fixed linear-congruential stream, no `random` module, +# so output is identical everywhere. +class Det: + def __init__(self, seed=2463534242): + self.s = seed + def next(self): + self.s ^= (self.s << 13) & 0xFFFFFFFF + self.s ^= (self.s >> 17) + self.s ^= (self.s << 5) & 0xFFFFFFFF + return self.s & 0xFFFFFFFF + def rng(self, n): + return self.next() % n + +d = Det() +WORDS = ["alpha","bravo","charlie","delta","echo","foxtrot","golf","hotel", + "quote\"inside","tab\tchar","new\nline","back\\slash","unécode","☃snow"] + +def record(i): + return { + "id": i, + "guid": f"{d.next():08x}-{d.next():04x}", + "active": bool(d.rng(2)), + "score": d.rng(1000000) / 1000.0, # float -> strtod/sprintf path + "ratio": d.rng(1 << 30) / (1 << 30), # dense fractional float + "count": d.rng(1 << 31) - (1 << 30), # signed int + "name": WORDS[d.rng(len(WORDS))] + str(d.rng(9999)), + "tags": [WORDS[d.rng(len(WORDS))] for _ in range(d.rng(6))], + "coords": [d.rng(1 << 20) / 997.0 for _ in range(3)], + "meta": None if d.rng(4) == 0 else { + "created": d.next(), + "nested": {"a": d.rng(100), "b": [d.rng(50) for _ in range(d.rng(4))]}, + }, + } + +doc = { + "schema": "hotpath.cjson.corpus/v1", + "generated_by": "gen-corpus.sh (deterministic)", + "records": [record(i) for i in range(20000)], +} +with open(sys.argv[1], "w") as f: + json.dump(doc, f, ensure_ascii=False, separators=(",", ":")) +PY + +echo "wrote $OUT ($(wc -c < "$OUT") bytes)" +# Pin the corpus and regenerate the manifest. +( cd "$DIR/input" && find . -type f -print0 | sort -z | xargs -0 sha256sum ) > "$DIR/MANIFEST.sha256" +echo "manifest: $DIR/MANIFEST.sha256" diff --git a/crates/diff-test/src/target.rs b/crates/diff-test/src/target.rs index 0b7a4f4..2596358 100644 --- a/crates/diff-test/src/target.rs +++ b/crates/diff-test/src/target.rs @@ -33,10 +33,31 @@ pub struct Source { #[derive(Debug, Deserialize)] pub struct Build { - /// Baseline build command, run from repo root. + /// Build command, run from repo root. `{out}` is substituted with an + /// isolation directory the harness provides (per-baseline, per- + /// candidate, per-sanitizer) — the command MUST place all artifacts + /// under it. A cargo target sets `CARGO_TARGET_DIR={out}`; a make + /// target builds and copies the binary to `{out}`. Language-agnostic: + /// nothing here assumes cargo. pub baseline: String, - /// Built binary path, relative to repo root. + /// Built binary path with `{out}` substituted — where `baseline` + /// leaves the binary given an isolation dir. pub binary: String, + /// ASan+LSan (or ASan+UBSan for C/C++) build command; `{out}` + /// substituted. Run on every would-be accept (SPEC §8). If absent, + /// an accept cannot be sanitizer-verified and is capped at + /// needs-human-review. + #[serde(default)] + pub sanitizer: Option, + /// Sanitizer binary path with `{out}` substituted. + #[serde(default)] + pub sanitizer_binary: Option, +} + +/// Substitute the `{out}` isolation-dir placeholder in a build command or +/// binary-path template. +pub fn subst_out(template: &str, out_dir: &str) -> String { + template.replace("{out}", out_dir) } #[derive(Debug, Deserialize)] diff --git a/crates/harnessd/src/main.rs b/crates/harnessd/src/main.rs index 7192b39..d7638f8 100644 --- a/crates/harnessd/src/main.rs +++ b/crates/harnessd/src/main.rs @@ -339,29 +339,30 @@ fn handle(root: &Path, line: &str) -> Result { )?)?; let t = pending["target"].as_str().unwrap(); let spec = diff_test::target::TargetSpec::load(root, t)?; - let bin = Path::new(&spec.build.binary) - .file_name() - .unwrap() - .to_str() - .unwrap(); let diff_path = root.join(format!("results/pending/{patch_id}.json.diff")); std::fs::write(&diff_path, pending["diff"].as_str().unwrap())?; // The build + verdict pipeline runs for minutes — far past the MCP // transport's per-call cap. Launch it detached, writing progress to // a log; the agent observes the result via the read_verdict op, // which reads the append-only ledger row once written. - let cand_dir = format!("targets/{t}/candidate-{patch_id}"); - let cand_bin = format!("{cand_dir}/release/{bin}"); + // {out} isolates the candidate build (cargo: CARGO_TARGET_DIR; + // make: build+copy into it) — language-agnostic, mirrors the + // baseline rebuild in verdict. + let cand_dir = root + .join(format!("targets/{t}/candidate-{patch_id}")) + .to_string_lossy() + .into_owned(); + let cand_build = diff_test::target::subst_out(&spec.build.baseline, &cand_dir); + let cand_bin = diff_test::target::subst_out(&spec.build.binary, &cand_dir); let log = format!("results/pending/{run_id}.log"); // On any failure (candidate build, gates crash) the trap line // marks the log so read_verdict can report "failed" instead of // leaving the agent polling a verdict that will never exist. let script = format!( - "{{ set -e\nCARGO_TARGET_DIR={cand} {build}\ncargo run -q -p verdict -- {tgt} \ + "{{ set -e\n{build}\ncargo run -q -p verdict -- {tgt} \ --rebuild-baseline --candidate-bin {cbin} --run-id {rid} --playbook-class {cls} \ --hypothesis {hyp} --hotspot {hs} --patch-file {pf}\n}} || echo HOTPATH-PIPELINE-FAILED\n", - cand = shq(&cand_dir), - build = spec.build.baseline, + build = cand_build, tgt = shq(t), cbin = shq(&cand_bin), rid = shq(run_id), diff --git a/crates/verdict/src/main.rs b/crates/verdict/src/main.rs index fc37c97..84dd7ab 100644 --- a/crates/verdict/src/main.rs +++ b/crates/verdict/src/main.rs @@ -79,14 +79,14 @@ fn rebuild_pristine_baseline( "git stash failed" ); } + let out_dir = root + .join(format!("targets/{target}/baseline")) + .to_string_lossy() + .into_owned(); let build = std::process::Command::new("sh") .arg("-c") - .arg(&spec.build.baseline) + .arg(diff_test::target::subst_out(&spec.build.baseline, &out_dir)) .current_dir(root) - .env( - "CARGO_TARGET_DIR", - root.join(format!("targets/{target}/baseline")), - ) .status(); if dirty { anyhow::ensure!( @@ -98,13 +98,9 @@ fn rebuild_pristine_baseline( ); } anyhow::ensure!(build?.success(), "baseline build failed"); - let bin_name = std::path::Path::new(&spec.build.binary) - .file_name() - .and_then(|n| n.to_str()) - .ok_or_else(|| anyhow::anyhow!("bad build.binary path"))?; - let baseline = format!("targets/{target}/baseline/release/{bin_name}"); + let baseline = diff_test::target::subst_out(&spec.build.binary, &out_dir); anyhow::ensure!( - root.join(&baseline).exists(), + std::path::Path::new(&baseline).exists(), "baseline binary missing: {baseline}" ); println!("baseline rebuilt from pristine checkout: {baseline}"); @@ -117,40 +113,43 @@ fn rebuild_pristine_baseline( /// were per-attempt manual) and the human audit had to overturn it — the /// accept path now runs the check itself and caps flagged wins at /// needs-human-review. Rejections skip it: they ship nothing. -fn sanitizer_check(root: &std::path::Path, target: &str, spec: &TargetSpec) -> Result { - let ws = root.join(format!("targets/{target}/workspace")); - let asan_dir = root.join(format!("targets/{target}/asan")); - println!("sanitizers: ASan+LSan build (nightly) of the patched tree…"); - let status = std::process::Command::new("cargo") - .args([ - "+nightly", - "build", - "-q", - "--target", - "x86_64-unknown-linux-gnu", - ]) - .env("RUSTFLAGS", "-Zsanitizer=address") - .env("CARGO_TARGET_DIR", &asan_dir) - .current_dir(&ws) +/// Returns Some(clean) if the sanitizer gate ran, or None if the target +/// declares no sanitizer build (an accept then cannot be verified and is +/// capped at needs-human-review by the caller). +fn sanitizer_check( + root: &std::path::Path, + target: &str, + spec: &TargetSpec, +) -> Result> { + let (Some(build_tpl), Some(bin_tpl)) = (&spec.build.sanitizer, &spec.build.sanitizer_binary) + else { + println!("sanitizers: target declares no sanitizer build — cannot verify an accept"); + return Ok(None); + }; + let out_dir = root + .join(format!("targets/{target}/asan")) + .to_string_lossy() + .into_owned(); + println!("sanitizers: ASan build of the patched tree…"); + let status = std::process::Command::new("sh") + .arg("-c") + .arg(diff_test::target::subst_out(build_tpl, &out_dir)) + .current_dir(root) .status()?; anyhow::ensure!(status.success(), "sanitizer build failed"); - let bin_name = std::path::Path::new(&spec.build.binary) - .file_name() - .and_then(|n| n.to_str()) - .ok_or_else(|| anyhow::anyhow!("bad build.binary path"))?; - let asan_bin = asan_dir.join(format!("x86_64-unknown-linux-gnu/debug/{bin_name}")); - let cmd = spec - .bench - .command - .replace("{binary}", asan_bin.to_str().unwrap()); + let asan_bin = diff_test::target::subst_out(bin_tpl, &out_dir); + let cmd = spec.bench.command.replace("{binary}", &asan_bin); let out = std::process::Command::new("sh") .arg("-c") .arg(&cmd) + // detect_leaks is a no-op for the C/C++ UBSan combo but harmless. .env("ASAN_OPTIONS", "detect_leaks=1") + .env("UBSAN_OPTIONS", "halt_on_error=1:print_stacktrace=1") .current_dir(root) .output()?; let stderr = String::from_utf8_lossy(&out.stderr); - let clean = out.status.success() && !stderr.contains("ERROR:"); + let clean = + out.status.success() && !stderr.contains("ERROR:") && !stderr.contains("runtime error:"); if clean { println!("sanitizers: clean"); } else { @@ -161,7 +160,7 @@ fn sanitizer_check(root: &std::path::Path, target: &str, spec: &TargetSpec) -> R println!(" {l}"); } } - Ok(clean) + Ok(Some(clean)) } fn now_utc() -> String { @@ -265,11 +264,18 @@ fn main() -> Result<()> { v = Verdict::NeedsHumanReview; } if v == Verdict::Accepted { - let clean = sanitizer_check(&root, &cli.target, &spec)?; - gate_results.sanitizers_clean = clean; - if !clean { - println!("verdict: sanitizer flag caps the accept at needs-human-review (SPEC §8)"); - v = Verdict::NeedsHumanReview; + match sanitizer_check(&root, &cli.target, &spec)? { + Some(true) => gate_results.sanitizers_clean = true, + Some(false) => { + println!( + "verdict: sanitizer flag caps the accept at needs-human-review (SPEC §8)" + ); + v = Verdict::NeedsHumanReview; + } + None => { + println!("verdict: no sanitizer build declared; capping accept at needs-human-review (SPEC §8)"); + v = Verdict::NeedsHumanReview; + } } } let fp = EnvFingerprint::collect(pin, "system-default"); diff --git a/justfile b/justfile index 140fe98..5f86829 100644 --- a/justfile +++ b/justfile @@ -56,3 +56,12 @@ agent-attempt target run_id max_turns="40": # Verify the OS boundary from the agent user's side (negative + positive) isolation-check: sh scripts/isolation-check.sh + +# Install the coz causal profiler (apt, else source build) +install-coz: + sh scripts/install-coz.sh + +# Causal profile of a C/C++ target: where a speedup would raise throughput +coz target iters="200": + sh scripts/coz-profile.sh {{target}} {{iters}} + python3 scripts/coz-summary.py results/{{target}}/coz/profile.coz diff --git a/results/ledger.sqlite b/results/ledger.sqlite index 6b1ee4101cea47c084be6445e704d9779ac227a1..330c371c47b506348fca6397218f110c44fb5ba5 100644 GIT binary patch delta 486 zcmZvWF>ljA7>0cgRbraN4QUjhjkrcoQ7Lxr?A)C%DnYQ90YN2%3bov$v0B-doh+%M zHto=XfmR;8W~j;#bwC+d%gC4?z@K12V#3fB>6>1?@Ao`MW98^tIa``rktAvL96kjr zU$!sQ+#U5Y4W!jLzfhOe%k$CI^vA;Y;#`_NT#hFX-ehWpL7atLE)1s6pjEhkFT+64 zwIg9_q1bi0!r9Y${+@7a@i)CZ{j6=tnOboiXVD|MIG#38Se(CotEzyT(#fQAI_ZF~ zlaBnlq!dyP$V*Z_{rzix{d8C>F;d+EWo6hLiH}Qbx$J*g1~RCIn1=ByY_&0JW0I6A zvc8e=nzrX_egIRsP(+dId+?tm^gqB}SHJ_m-y3SX!V{sr?+(Nc4BSBMyPkOL`7rYP zp(9{m_nhupGRuSG9HkT+grdaOz~Uk$9C3zO1L>GDlVQ%ycyx6)ts#bqVaBs^xq>M{ zoFaqY)R@&uFPUwC>T_fogjvKgsX+~kHtJCz#7;kW@3{5$TcmT2tyYUMVzR#+*1$T# k7^fD9@N~+|7<#`GwJC4L+lOb#izh(MzI%D`v+`;E52xLT1poj5 delta 99 zcmV-p0G$7TzzKk$36L8FAOHXWC6ORM0U)tpqz?!Y532wG001PD3=krbfv>aOK+X=6 zs~=ed91WC{2Ov4KQ7}{ulff<%v!^Xx0s|ZltCI&XC9^Itat@JD4TGybhpRpTx2rw^ FRS}q~9u@!q diff --git a/scripts/coz-profile.sh b/scripts/coz-profile.sh new file mode 100644 index 0000000..8837de0 --- /dev/null +++ b/scripts/coz-profile.sh @@ -0,0 +1,41 @@ +#!/bin/sh +# coz causal profiling for a C/C++ target (SPEC §12). Unlike a flat +# profiler (callgrind ranks where time is SPENT), coz measures where a +# speedup would actually RAISE throughput — it virtually speeds up each +# line and reports the causal effect on the harness's progress points. +# Use it to rank playbook targets when the flat profile is ambiguous. +# +# scripts/coz-profile.sh cjson [iters] +# +# Requires: coz on PATH (build: git clone github.com/plasma-umass/coz && +# see scripts/install-coz.sh) and the target's harness built with +# -DHOTPATH_COZ (this script does that). Output: results//coz/ +set -eu +ROOT=$(cd "$(dirname "$0")/.." && pwd) +TARGET=${1:?usage: coz-profile.sh [iters]} +ITERS=${2:-200} +OUT="$ROOT/results/$TARGET/coz" +mkdir -p "$OUT" + +command -v coz >/dev/null 2>&1 || { + echo "coz not on PATH — run scripts/install-coz.sh first" >&2 + exit 1 +} + +# coz needs debug line info and the progress point compiled in. +BIN="$OUT/cjson-bench-coz" +clang -O2 -g -gdwarf-4 -DHOTPATH_COZ \ + -I "$ROOT/targets/$TARGET/workspace" \ + -I "$(dirname "$(command -v coz)")/../include" \ + -o "$BIN" \ + "$ROOT/targets/$TARGET/harness.c" \ + "$ROOT/targets/$TARGET/workspace/cJSON.c" \ + -ldl + +cd "$ROOT" +coz run --output "$OUT/profile.coz" --- \ + "$BIN" corpora/$TARGET/input/big.json "$ITERS" >/dev/null + +echo "coz profile written: $OUT/profile.coz" +echo "view at https://plasma-umass.org/coz/ (load the .coz file) or:" +echo " python3 scripts/coz-summary.py $OUT/profile.coz" diff --git a/scripts/coz-summary.py b/scripts/coz-summary.py new file mode 100644 index 0000000..b80d242 --- /dev/null +++ b/scripts/coz-summary.py @@ -0,0 +1,75 @@ +#!/usr/bin/env python3 +"""Summarize a coz causal profile: rank source lines by their estimated +causal effect on throughput. coz writes newline-delimited JSON records; +for each (line, virtual-speedup) it records progress-point throughput. +We fit the slope of throughput-gain vs virtual-speedup per line — a +steeper positive slope means "speeding up this line most raises overall +throughput", which is exactly the class-selection signal a flat profiler +can't give. + + python3 scripts/coz-summary.py results//coz/profile.coz +""" + +from __future__ import annotations + +import collections +import json +import sys +from pathlib import Path + + +def main() -> int: + path = Path(sys.argv[1]) if len(sys.argv) > 1 else None + if not path or not path.exists(): + print("usage: coz-summary.py ", file=sys.stderr) + return 2 + + # experiments: line -> list of (speedup, progress-rate) + exps: dict[str, list[tuple[float, float]]] = collections.defaultdict(list) + baseline: dict[str, float] = {} + for raw in path.read_text().splitlines(): + raw = raw.strip() + if not raw: + continue + try: + rec = json.loads(raw) + except json.JSONDecodeError: + continue + if rec.get("type") != "experiment": + continue + line = rec.get("selected", "?") + speedup = float(rec.get("speedup", 0.0)) + duration = float(rec.get("duration", 0.0)) + delta = float(rec.get("delta", 0.0)) + if delta <= 0: + continue + rate = duration / delta # time per progress unit; lower = faster + exps[line].append((speedup, rate)) + if speedup == 0.0: + baseline[line] = rate + + rows = [] + for line, pts in exps.items(): + base = baseline.get(line) + if not base or len(pts) < 2: + continue + # % throughput improvement at the max virtual speedup tried + pts.sort() + s_max, r_max = pts[-1] + if s_max <= 0 or r_max <= 0: + continue + gain = (base - r_max) / base # fractional throughput gain + rows.append((gain, s_max, line)) + + rows.sort(reverse=True) + print(f"{'throughput_gain':>16} {'@speedup':>9} line") + print("-" * 70) + for gain, s_max, line in rows[:25]: + print(f"{gain * 100:>14.1f}% {s_max * 100:>7.0f}% {line}") + if not rows: + print("(no usable experiments — did the run mark progress points?)") + return 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/diff-fuzz-cjson.py b/scripts/diff-fuzz-cjson.py new file mode 100644 index 0000000..4a6b2c3 --- /dev/null +++ b/scripts/diff-fuzz-cjson.py @@ -0,0 +1,100 @@ +#!/usr/bin/env python3 +"""Differential fuzz for cjson attempts: pristine vs candidate harness must +produce byte-identical stdout (serialization + checksum) AND identical exit +code on mutated JSON. cJSON is a deterministic single-threaded parser, so +raw byte comparison needs no canonicalization. + + python3 scripts/diff-fuzz-cjson.py + +Seeds: slices of the pinned corpus plus hand JSON fragments. Mutations +target parser edges: structural tokens, string escapes, number forms +(leading zeros, +/- exponents, huge/deep values), and raw byte noise. +""" + +import random +import subprocess +import sys +from pathlib import Path + +ROOT = Path("/home/user/peltier.io") +BASE = ROOT / "targets/cjson/baseline/cjson-bench" +CAND = ROOT / sys.argv[3] if len(sys.argv) > 3 else ROOT / "targets/cjson/baseline/cjson-bench" +CORPUS = ROOT / "corpora/cjson/input/big.json" +WORK = Path(sys.argv[1]) if len(sys.argv) > 1 else Path("/tmp/difffuzz-cjson") +ITERS = int(sys.argv[2]) if len(sys.argv) > 2 else 10000 + +TOKENS = [ + b"{", b"}", b"[", b"]", b'"', b":", b",", b"\\", b"\\n", b"\\u00e9", b"\\uD83D", + b"true", b"false", b"null", b"0", b"-0", b"1e999", b"-1.5E-3", b"00", b"1.", + b".5", b"1e", b"NaN", b"Infinity", b'\\"', b'"key"', b"[]", b"{}", b'""', + b"1234567890123456789012345", b"\x00", b"\xff", b"\t", b"\n", +] + +SEEDS_STATIC = [ + b'{}', b'[]', b'{"a":1}', b'[1,2,3]', b'"hello"', b'true', b'null', + b'{"n":-1.5e10,"s":"a\\tb","u":"\\u00e9","arr":[null,false,{}]}', + b'[[[[[[1]]]]]]', b'{"a":{"b":{"c":{"d":1}}}}', b'123.456e-78', + b'{"esc":"\\"\\\\\\/\\b\\f\\n\\r\\t"}', b' { "x" : [ 1 , 2 ] } ', +] + + +def mutate(data: bytes, rng: random.Random) -> bytes: + b = bytearray(data) + for _ in range(rng.randrange(1, 8)): + if not b: + b = bytearray(rng.choice(TOKENS)) + continue + op = rng.randrange(5) + pos = rng.randrange(len(b) + 1) + if op == 0: + b[pos:pos] = rng.choice(TOKENS) + elif op == 1 and len(b) > 2: + del b[pos:min(len(b), pos + rng.randrange(1, 12))] + elif op == 2 and pos < len(b): + b[pos] = rng.randrange(256) + elif op == 3: + end = min(len(b), pos + rng.randrange(1, 48)) + b[end:end] = b[pos:end] + else: + b[pos:pos] = rng.choice(TOKENS) + return bytes(b) + + +def run(binary: Path, path: Path) -> tuple[bytes, int]: + r = subprocess.run([str(binary), str(path), "1"], capture_output=True, timeout=30) + return r.stdout, r.returncode + + +def main() -> int: + rng = random.Random(0x1CE_B00C) + corpus = CORPUS.read_bytes() + seeds = list(SEEDS_STATIC) + # windows of the real corpus (often invalid mid-slice — exercises the + # error path, which must diverge identically or not at all) + for o in range(0, len(corpus) - 4096, len(corpus) // 200): + for n in (64, 512, 4096): + seeds.append(corpus[o : o + n]) + WORK.mkdir(parents=True, exist_ok=True) + case = WORK / "case.json" + divergences = 0 + for i in range(ITERS): + payload = mutate(rng.choice(seeds), rng) + case.write_bytes(payload) + a_out, a_rc = run(BASE, case) + b_out, b_rc = run(CAND, case) + if a_out != b_out or a_rc != b_rc: + divergences += 1 + keep = WORK / f"divergence-{i}" + keep.mkdir(exist_ok=True) + (keep / "input.json").write_bytes(payload) + (keep / "baseline.out").write_bytes(a_out + f"\nrc={a_rc}".encode()) + (keep / "candidate.out").write_bytes(b_out + f"\nrc={b_rc}".encode()) + print(f"DIVERGENCE at iter {i} (rc {a_rc} vs {b_rc}); kept in {keep}") + if (i + 1) % 1000 == 0: + print(f"{i + 1}/{ITERS}, {divergences} divergences", flush=True) + print(f"done: {ITERS} inputs, {divergences} divergences") + return 1 if divergences else 0 + + +if __name__ == "__main__": + sys.exit(main()) diff --git a/scripts/install-coz.sh b/scripts/install-coz.sh new file mode 100644 index 0000000..af4d856 --- /dev/null +++ b/scripts/install-coz.sh @@ -0,0 +1,28 @@ +#!/bin/sh +# Install the coz causal profiler (SPEC §12). Tries the distro package +# first, then a source build. Idempotent — skips if `coz` is already on +# PATH. Run as a user that can apt-install (root in the dev container). +set -eu + +if command -v coz >/dev/null 2>&1; then + echo "coz already installed: $(command -v coz)" + exit 0 +fi + +echo "installing coz…" +if apt-get install -y coz-profiler >/dev/null 2>&1; then + echo "installed via apt: $(command -v coz)" + exit 0 +fi + +# Source build fallback. +apt-get install -y python3 libelf-dev libdwarf-dev build-essential git >/dev/null 2>&1 || true +SRC=/opt/coz-src +rm -rf "$SRC" +git clone --depth 1 https://github.com/plasma-umass/coz.git "$SRC" +make -C "$SRC" >/dev/null +make -C "$SRC" install >/dev/null +command -v coz >/dev/null 2>&1 && echo "installed from source: $(command -v coz)" || { + echo "coz install failed" >&2 + exit 1 +} diff --git a/targets/cjson/harness.c b/targets/cjson/harness.c new file mode 100644 index 0000000..639b12b --- /dev/null +++ b/targets/cjson/harness.c @@ -0,0 +1,73 @@ +/* hotpath bench/golden driver for the cjson target. + * + * TRUST-LAYER CODE — lives outside targets/cjson/workspace, so the agent's + * propose_patch allowlist (workspace-relative paths only) cannot touch it. + * It is compiled together with the (patched) workspace cJSON.c, so a patch + * changes the parser/printer under test but never this measurement rig. + * + * Workload: parse a JSON document `iters` times, serialize it unformatted + * each time, and fold every serialized byte into a rolling checksum so the + * optimizer cannot elide the work. Deterministic: the last serialization + * plus the checksum are written to stdout, giving golden-replay a + * byte-exact signal over both the parse AND the print path. + * + * cjson-bench [iters=40] + */ +#include +#include +#include +#include "cJSON.h" + +/* coz causal profiling (SPEC §12): compile with -DHOTPATH_COZ and link + * libcoz to mark each parse+print iteration as a throughput progress + * point, so `coz run` can measure which source lines, if sped up, would + * actually raise end-to-end throughput. A no-op in normal builds. */ +#ifdef HOTPATH_COZ +#include +#define HOTPATH_PROGRESS() COZ_PROGRESS +#else +#define HOTPATH_PROGRESS() ((void)0) +#endif + +int main(int argc, char **argv) { + if (argc < 2) { + fprintf(stderr, "usage: %s [iters]\n", argv[0]); + return 2; + } + long iters = (argc >= 3) ? atol(argv[2]) : 40; + if (iters < 1) iters = 1; + + FILE *f = fopen(argv[1], "rb"); + if (!f) { perror("fopen"); return 2; } + fseek(f, 0, SEEK_END); + long n = ftell(f); + fseek(f, 0, SEEK_SET); + if (n < 0) { fclose(f); return 2; } + char *buf = (char *)malloc((size_t)n + 1); + if (!buf) { fclose(f); return 2; } + if (fread(buf, 1, (size_t)n, f) != (size_t)n) { free(buf); fclose(f); return 2; } + buf[n] = '\0'; + fclose(f); + + unsigned long acc = 1469598103934665603UL; /* FNV-ish rolling checksum */ + char *last = NULL; + for (long i = 0; i < iters; i++) { + cJSON *root = cJSON_Parse(buf); + if (!root) { fprintf(stderr, "parse failed\n"); free(buf); return 3; } + char *out = cJSON_PrintUnformatted(root); + if (!out) { cJSON_Delete(root); free(buf); return 3; } + for (char *p = out; *p; p++) acc = (acc ^ (unsigned char)*p) * 1099511628211UL; + if (i == iters - 1) last = out; + else free(out); + cJSON_Delete(root); + HOTPATH_PROGRESS(); /* one parse+print = one throughput unit (coz) */ + } + + if (last) { + fwrite(last, 1, strlen(last), stdout); + free(last); + } + printf("\nchecksum=%lu\n", acc); + free(buf); + return 0; +} diff --git a/targets/cjson/target.toml b/targets/cjson/target.toml new file mode 100644 index 0000000..2d8388a --- /dev/null +++ b/targets/cjson/target.toml @@ -0,0 +1,44 @@ +# cJSON — ultralightweight JSON parser/printer (C) +# Selection vs SPEC §6: CPU-bound parse+serialize; hermetic Unity/CTest +# suite vendored in-tree; deterministic scripted workload; MIT license; +# active but deliberately "simple", no dedicated perf team — the richer +# legacy-C hunting ground Phase 3 calls for. + +[source] +repo = "https://github.com/DaveGamble/cJSON.git" +commit = "c859b25da02955fef659d658b8f324b5cde87be3" # v1.7.19 +license = "MIT" + +[fetch] +# targets/*/workspace is gitignored; reproduce it with: +# git clone https://github.com/DaveGamble/cJSON.git workspace +# git -C workspace checkout v1.7.19 +# The bench/golden driver targets/cjson/harness.c and the Unity suite in +# workspace/tests/ (unity vendored in-tree) are hermetic — no submodules. + +[build] +# Compile the trust-layer harness against the (patched) workspace cJSON.c. +# {out} is the isolation dir; the agent may patch only workspace/*, never +# harness.c, so the measurement rig is fixed across baseline/candidate. +baseline = "mkdir -p {out} && clang -O2 -DNDEBUG -I targets/cjson/workspace -o {out}/cjson-bench targets/cjson/harness.c targets/cjson/workspace/cJSON.c" +binary = "{out}/cjson-bench" +# ASan + UBSan (C/C++ analogue of the Rust nightly ASan path); run on +# every would-be accept (SPEC §8), halting on any leak/UB. +sanitizer = "mkdir -p {out} && clang -O1 -g -fsanitize=address,undefined -fno-sanitize-recover=all -I targets/cjson/workspace -o {out}/cjson-asan targets/cjson/harness.c targets/cjson/workspace/cJSON.c" +sanitizer_binary = "{out}/cjson-asan" + +[bench] +command = "{binary} corpora/cjson/input/big.json 2 > /dev/null" +workload = "5.6 MB synthetic JSON (20k records: nested objects/arrays, escaped/unicode strings, ints, dense floats), parsed+serialized 2x, single thread" + +[gates] +# Upstream Unity suite via CTest — parse_number/string/array/object, +# print_*, misc_tests, compare/minify. Tests the PATCHED cJSON.c. +tests = "cmake -S targets/cjson/workspace -B targets/cjson/testbuild -DENABLE_CJSON_TEST=On -DCMAKE_BUILD_TYPE=Debug >/dev/null 2>&1 && cmake --build targets/cjson/testbuild -j >/dev/null 2>&1 && ctest --test-dir targets/cjson/testbuild --output-on-failure" +# Golden: full re-serialization + checksum of the corpus, byte-hashed. +golden = "{binary} corpora/cjson/input/big.json 1" + +[corpus] +manifest = "corpora/cjson/MANIFEST.sha256" +root = "corpora/cjson/input" +golden_sha256 = "corpora/cjson/GOLDEN.sha256" diff --git a/targets/comrak/target.toml b/targets/comrak/target.toml index 4fc885d..94b8d67 100644 --- a/targets/comrak/target.toml +++ b/targets/comrak/target.toml @@ -18,8 +18,10 @@ license = "BSD-2-Clause" # git clone workspace/vendor/progit && git -C ... checkout [build] -baseline = "cargo build --release --manifest-path targets/comrak/workspace/Cargo.toml" -binary = "targets/comrak/workspace/target/release/comrak" +baseline = "CARGO_TARGET_DIR={out} cargo build --release --manifest-path targets/comrak/workspace/Cargo.toml" +binary = "{out}/release/comrak" +sanitizer = "CARGO_TARGET_DIR={out} RUSTFLAGS='-Zsanitizer=address' cargo +nightly build --target x86_64-unknown-linux-gnu --manifest-path targets/comrak/workspace/Cargo.toml" +sanitizer_binary = "{out}/x86_64-unknown-linux-gnu/debug/comrak" [bench] # Upstream workload (benches/bench.sh): full Pro Git book through the diff --git a/targets/tokei/target.toml b/targets/tokei/target.toml index f7f33b9..2b84248 100644 --- a/targets/tokei/target.toml +++ b/targets/tokei/target.toml @@ -9,8 +9,10 @@ commit = "fa44e5194060305576514d59b850353643afbfc8" # v14.0.0 license = "MIT OR Apache-2.0" [build] -baseline = "cargo build --release --manifest-path targets/tokei/workspace/Cargo.toml" -binary = "targets/tokei/workspace/target/release/tokei" +baseline = "CARGO_TARGET_DIR={out} cargo build --release --manifest-path targets/tokei/workspace/Cargo.toml" +binary = "{out}/release/tokei" +sanitizer = "CARGO_TARGET_DIR={out} RUSTFLAGS='-Zsanitizer=address' cargo +nightly build --target x86_64-unknown-linux-gnu --manifest-path targets/tokei/workspace/Cargo.toml" +sanitizer_binary = "{out}/x86_64-unknown-linux-gnu/debug/tokei" [bench] # Single rayon thread: measurement first (SPEC conventions); the From fd6f963829a54439ecfffd43d41deaec71e0265c Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 11 Jul 2026 18:20:08 +0000 Subject: [PATCH 2/7] phase3: diff-test resolves {out}-templated binary; CI gates cjson MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit diff-test standalone (just gates / CI target-gates) passed the now- templated build.binary into golden replay verbatim — literal {out}. It now builds the current tree to targets//gates-build and tests that, with a --binary override. CI's target-gates-no-net job gains clang/cmake, fetches+gates the cjson C target (clang build -> Unity/ CTest -> golden replay) alongside tokei. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2 --- .github/workflows/ci.yml | 8 +++++++- crates/diff-test/src/main.rs | 28 ++++++++++++++++++++++++++-- 2 files changed, 33 insertions(+), 3 deletions(-) diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index 48fad2d..5229460 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -25,18 +25,24 @@ jobs: - uses: actions/checkout@v4 - uses: dtolnay/rust-toolchain@stable - uses: Swatinem/rust-cache@v2 + - name: Toolchains for C/C++ targets + run: sudo apt-get update -qq && sudo apt-get install -y -qq clang cmake - name: Fetch pinned targets run: | sh targets/fetch.sh comrak sh targets/fetch.sh tokei + sh targets/fetch.sh cjson - name: Generate + verify pinned corpora run: | sh corpora/comrak/gen-corpus.sh sh corpora/tokei/gen-corpus.sh + sh corpora/cjson/gen-corpus.sh - name: Build tokei baseline run: cargo build --release --manifest-path targets/tokei/workspace/Cargo.toml - - name: Gates (corpus pin -> upstream tests -> golden replay) + - name: Gates — tokei (Rust; corpus pin -> upstream tests -> golden replay) run: cargo run -p diff-test -- tokei + - name: Gates — cjson (C; clang build -> Unity/CTest -> golden replay) + run: cargo run -p diff-test -- cjson - name: Bench workload inside no-network container run: | docker run --rm --network=none \ diff --git a/crates/diff-test/src/main.rs b/crates/diff-test/src/main.rs index 8dc5f77..fb7b165 100644 --- a/crates/diff-test/src/main.rs +++ b/crates/diff-test/src/main.rs @@ -5,7 +5,7 @@ use anyhow::Result; use clap::Parser; use diff_test::orchestrate::run_core_gates; -use diff_test::target::TargetSpec; +use diff_test::target::{subst_out, TargetSpec}; use diff_test::{all_passed, GateOutcome}; use std::path::PathBuf; @@ -17,13 +17,37 @@ struct Cli { /// Repository root (defaults to current directory). #[arg(long, default_value = ".")] root: PathBuf, + /// Binary to test in golden replay. If omitted, the current working + /// tree is built to targets//gates-build and that is tested. + #[arg(long)] + binary: Option, } fn main() -> Result<()> { let cli = Cli::parse(); let root = cli.root.canonicalize()?; let spec = TargetSpec::load(&root, &cli.target)?; - let results = run_core_gates(&root, &spec, &spec.build.binary)?; + // build.binary is an {out}-templated path now; resolve it to a real + // binary either from --binary or by building the current tree into an + // isolation dir (so `just gates` tests what's on disk, self-contained). + let binary = match &cli.binary { + Some(b) => b.clone(), + None => { + let out = root + .join(format!("targets/{}/gates-build", cli.target)) + .to_string_lossy() + .into_owned(); + println!("building current tree for gates -> {out}"); + let status = std::process::Command::new("sh") + .arg("-c") + .arg(subst_out(&spec.build.baseline, &out)) + .current_dir(&root) + .status()?; + anyhow::ensure!(status.success(), "gates build failed"); + subst_out(&spec.build.binary, &out) + } + }; + let results = run_core_gates(&root, &spec, &binary)?; for (layer, outcome) in &results { match outcome { GateOutcome::Passed => println!("{layer:?}: PASS"), From f5f4c10b63c76825eaeaee02054db00325907ad3 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 11 Jul 2026 18:34:46 +0000 Subject: [PATCH 3/7] =?UTF-8?q?phase3:=20cjson=20A/A=20calibration=20ancho?= =?UTF-8?q?r=20=E2=80=94=20PASS?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit 20/20 sessions: A/A false-positive rate 0.000 (<0.05 required), injected-5%-slowdown detection 1.000 (>=0.95 required). The C workload is a trusted measurement anchor; verdict benches on cjson are now valid. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2 --- results/calibration/cjson-aa.json | 228 ++++++++++++++++++++++++++++++ 1 file changed, 228 insertions(+) create mode 100644 results/calibration/cjson-aa.json diff --git a/results/calibration/cjson-aa.json b/results/calibration/cjson-aa.json new file mode 100644 index 0000000..4f67043 --- /dev/null +++ b/results/calibration/cjson-aa.json @@ -0,0 +1,228 @@ +{ + "aa_false_positive_rate": 0.0, + "aa_sessions": [ + { + "hi": 1.005821765761564, + "lo": 0.9801782333975517, + "median": 0.9986149882689674 + }, + { + "hi": 1.0101069801079932, + "lo": 0.988203985722408, + "median": 0.9970235967111305 + }, + { + "hi": 1.0131303729804741, + "lo": 0.9754461737061029, + "median": 0.9918874440006957 + }, + { + "hi": 1.0130463006053056, + "lo": 0.9912120746995015, + "median": 1.0046343182893742 + }, + { + "hi": 1.0189543094353009, + "lo": 0.9724233993929335, + "median": 0.9952479242267185 + }, + { + "hi": 1.032412708299805, + "lo": 0.9951738064097109, + "median": 1.012930231297955 + }, + { + "hi": 1.0167402802036027, + "lo": 0.9954985489675546, + "median": 1.004624731468757 + }, + { + "hi": 1.018779879435948, + "lo": 0.9827477895812976, + "median": 1.003204104491116 + }, + { + "hi": 1.016499792207234, + "lo": 0.9785792438928752, + "median": 0.993365716937912 + }, + { + "hi": 1.010077763189733, + "lo": 0.9846620842672246, + "median": 0.9997681543348355 + }, + { + "hi": 1.0173284598883299, + "lo": 0.996433289229599, + "median": 1.0071401496044297 + }, + { + "hi": 1.012445943642925, + "lo": 0.9850661575782045, + "median": 0.9983447019812159 + }, + { + "hi": 1.0112241960737036, + "lo": 0.990713977452908, + "median": 1.002365298134083 + }, + { + "hi": 1.0161790245163274, + "lo": 0.9899011976067387, + "median": 1.0036389049635406 + }, + { + "hi": 1.0088421424513234, + "lo": 0.982195355727282, + "median": 1.0014682127442587 + }, + { + "hi": 1.0184446174342345, + "lo": 1.0016850824391794, + "median": 1.0110704941235351 + }, + { + "hi": 1.0338430526478581, + "lo": 0.9832609834926027, + "median": 1.01605041869795 + }, + { + "hi": 1.0156279777328292, + "lo": 0.9859377461488782, + "median": 1.0015398151424024 + }, + { + "hi": 1.0150360571291077, + "lo": 0.9925111073903699, + "median": 1.003301550080401 + }, + { + "hi": 1.0049746998130413, + "lo": 0.9879079920184525, + "median": 0.9956649681815931 + } + ], + "acceptance": { + "detection_ge": 0.95, + "fp_lt": 0.05, + "pass": true + }, + "env_fingerprint": { + "aslr": "mode=system-default randomize_va_space=2", + "cpu_model": "Intel(R) Xeon(R) Processor @ 2.80GHz", + "cpu_pinning": "caller-provided (wrap cmd in taskset)", + "governor": "unknown", + "kernel": "6.18.5", + "smt": "0", + "turbo": "unknown" + }, + "injected_busy_wait_s": 0.0212658513, + "injected_slowdown": 0.05, + "injection_detection_rate": 1.0, + "injection_sessions": [ + { + "hi": 0.9795786894340607, + "lo": 0.941809479290477, + "median": 0.9599376225852292 + }, + { + "hi": 0.9641254481957012, + "lo": 0.9422564180747722, + "median": 0.9547082398892934 + }, + { + "hi": 0.9714503402112215, + "lo": 0.9315789075936639, + "median": 0.9545848281994169 + }, + { + "hi": 0.9583867204633405, + "lo": 0.9432989621783778, + "median": 0.9520331767850789 + }, + { + "hi": 0.9654875366918616, + "lo": 0.9158571687116076, + "median": 0.9408289215103969 + }, + { + "hi": 0.968592147116574, + "lo": 0.9409605518725909, + "median": 0.9538004585651089 + }, + { + "hi": 0.9711766819792909, + "lo": 0.9257150542077511, + "median": 0.9438771706353369 + }, + { + "hi": 0.9628180133457988, + "lo": 0.9282004168602138, + "median": 0.950210433251739 + }, + { + "hi": 0.9862438497528527, + "lo": 0.9443093873012407, + "median": 0.9607783889091653 + }, + { + "hi": 0.9609819530474722, + "lo": 0.9341389435992757, + "median": 0.9441053624417196 + }, + { + "hi": 0.966382872224827, + "lo": 0.9491993487196345, + "median": 0.9616464961445802 + }, + { + "hi": 0.9558190623406682, + "lo": 0.9150401086894493, + "median": 0.9352329538103368 + }, + { + "hi": 0.9722670575112274, + "lo": 0.9382027678481317, + "median": 0.9529739409642378 + }, + { + "hi": 0.9868911779109959, + "lo": 0.9392216899668684, + "median": 0.965895978758762 + }, + { + "hi": 0.9612569392699162, + "lo": 0.9343647558154078, + "median": 0.9482258690146398 + }, + { + "hi": 0.9626637521190655, + "lo": 0.9440491084457884, + "median": 0.953658515672711 + }, + { + "hi": 0.9754190644980857, + "lo": 0.9394027173591545, + "median": 0.9524932710353055 + }, + { + "hi": 0.9613640264375135, + "lo": 0.9407944082117918, + "median": 0.9515112406879556 + }, + { + "hi": 0.9607568836341472, + "lo": 0.9418612887967339, + "median": 0.9494696010203004 + }, + { + "hi": 0.9553596046820967, + "lo": 0.91162247172763, + "median": 0.9289493105253712 + } + ], + "runs_per_side": 30, + "sessions": 20, + "threshold": 0.02, + "workload_cmd": "/tmp/cjson-testbuild/cjson-bench corpora/cjson/input/big.json 2 > /dev/null" +} \ No newline at end of file From b6dab15bf26cea1747ed76240277260afdfb996c Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 11 Jul 2026 18:35:22 +0000 Subject: [PATCH 4/7] =?UTF-8?q?phase3:=20cjson=20cache-sim=20profile=20?= =?UTF-8?q?=E2=80=94=20float=20print/parse=20dominate?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ~22% of Ir in glibc float serialization (__printf_fp, cJSON print_number) and parsing (strtod/str_to_mpn, cJSON parse_number), plus parse_value/ parse_string and malloc/free churn. Ample headroom (SPEC §6). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2 --- .gitignore | 1 + results/cjson/hotspots.txt | 74 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 75 insertions(+) create mode 100644 results/cjson/hotspots.txt diff --git a/.gitignore b/.gitignore index 168be9c..9902998 100644 --- a/.gitignore +++ b/.gitignore @@ -243,3 +243,4 @@ agent/**/__pycache__/ # cjson corpus is regenerable via gen-corpus.sh (deterministic) corpora/cjson/input/ targets/cjson/testbuild/ +results/*/callgrind-cache.out diff --git a/results/cjson/hotspots.txt b/results/cjson/hotspots.txt new file mode 100644 index 0000000..eb0e9c0 --- /dev/null +++ b/results/cjson/hotspots.txt @@ -0,0 +1,74 @@ +-------------------------------------------------------------------------------- +Profile data file 'results/cjson/callgrind-cache.out' (creator: callgrind-3.22.0) +-------------------------------------------------------------------------------- +I1 cache: 32768 B, 64 B, 8-way associative +D1 cache: 32768 B, 64 B, 8-way associative +LL cache: 35651584 B, 64 B, 17-way associative +Timerange: Basic block 0 - 1005248788 +Trigger: Program termination +Profiled target: /tmp/cjson-prof/cjson-bench corpora/cjson/input/big.json 3 (PID 25559, part 1) +Events recorded: Ir Dr Dw I1mr D1mr D1mw ILmr DLmr DLmw Bc Bcm Bi Bim +Events shown: Ir Dr Dw I1mr D1mr D1mw ILmr DLmr DLmw Bc Bcm Bi Bim +Event sort order: Ir Dr Dw I1mr D1mr D1mw ILmr DLmr DLmw Bc Bcm Bi Bim +Thresholds: 97 0 0 0 0 0 0 0 0 0 0 0 0 +Include dirs: +User annotated: +Auto-annotation: on + +-------------------------------------------------------------------------------- +Ir Dr Dw I1mr D1mr D1mw ILmr DLmr DLmw Bc Bcm Bi Bim +-------------------------------------------------------------------------------- +4,685,252,094 (100.0%) 1,010,069,038 (100.0%) 625,640,363 (100.0%) 5,843,765 (100.0%) 7,868,900 (100.0%) 1,241,461 (100.0%) 3,218 (100.0%) 5,919,355 (100.0%) 1,046,279 (100.0%) 754,451,991 (100.0%) 33,047,735 (100.0%) 26,673,314 (100.0%) 2,222,641 (100.0%) PROGRAM TOTALS + +-------------------------------------------------------------------------------- +Ir Dr Dw I1mr D1mr D1mw ILmr DLmr DLmw Bc Bcm Bi Bim file:function +-------------------------------------------------------------------------------- +637,034,976 (13.60%) 158,918,280 (15.73%) 83,065,737 (13.28%) 855,746 (14.64%) 96 ( 0.00%) 90 ( 0.01%) 209 ( 6.49%) 24 ( 0.00%) 24 ( 0.00%) 113,268,981 (15.01%) 5,030,223 (15.22%) . . ./stdio-common/./stdio-common/printf_fp.c:__printf_fp_buffer_1.isra.0 [/usr/lib/x86_64-linux-gnu/libc.so.6] +384,064,650 ( 8.20%) 73,607,328 ( 7.29%) 36,730,884 ( 5.87%) 415,586 ( 7.11%) 1,414 ( 0.02%) 66 ( 0.01%) 188 ( 5.84%) 34 ( 0.00%) 20 ( 0.00%) 64,737,597 ( 8.58%) 4,818,010 (14.58%) . . ./stdlib/./stdlib/strtod_l.c:____strtod_l_internal [/usr/lib/x86_64-linux-gnu/libc.so.6] +294,602,463 ( 6.29%) 70,228,494 ( 6.95%) 63,924,435 (10.22%) 261,347 ( 4.47%) 0 3 ( 0.00%) 11 ( 0.34%) 0 3 ( 0.00%) 31,351,464 ( 4.16%) 642,309 ( 1.94%) . . ./stdlib/./stdlib/divrem.c:__mpn_divrem [/usr/lib/x86_64-linux-gnu/libc.so.6] +231,942,018 ( 4.95%) 56,776,050 ( 5.62%) 40,158,519 ( 6.42%) 1,460,784 (25.00%) 75 ( 0.00%) 0 172 ( 5.34%) 21 ( 0.00%) 0 41,220,234 ( 5.46%) 988,895 ( 2.99%) 644,706 ( 2.42%) 2 ( 0.00%) ./stdio-common/./stdio-common/vfscanf-internal.c:__vfscanf_internal [/usr/lib/x86_64-linux-gnu/libc.so.6] +208,702,098 ( 4.45%) 29,444,439 ( 2.92%) 19,802,361 ( 3.17%) 6 ( 0.00%) 7 ( 0.00%) 0 6 ( 0.19%) 3 ( 0.00%) 0 38,050,107 ( 5.04%) 263,379 ( 0.80%) . . ./stdlib/../sysdeps/x86_64/mul_1.S:__mpn_mul_1 [/usr/lib/x86_64-linux-gnu/libc.so.6] +206,409,170 ( 4.41%) 36,862,330 ( 3.65%) 25,967,440 ( 4.15%) 34 ( 0.00%) 1,578 ( 0.02%) 138,956 (11.19%) 34 ( 1.06%) 425 ( 0.01%) 70,654 ( 6.75%) 41,217,012 ( 5.46%) 3,434,391 (10.39%) 7,230,866 (27.11%) 659,683 (29.68%) targets/cjson/workspace/cJSON.c:parse_value'2 [/tmp/cjson-prof/cjson-bench] +203,365,507 ( 4.34%) 52,235,950 ( 5.17%) 20,587,887 ( 3.29%) 29 ( 0.00%) 398,923 ( 5.07%) 36 ( 0.00%) 25 ( 0.78%) 397,528 ( 6.72%) 8 ( 0.00%) 41,473,481 ( 5.50%) 272,053 ( 0.82%) . . ./malloc/./malloc/malloc.c:_int_free [/usr/lib/x86_64-linux-gnu/libc.so.6] +188,097,198 ( 4.01%) 28,995,630 ( 2.87%) 19,684,752 ( 3.15%) 23 ( 0.00%) 6,010 ( 0.08%) 4,750 ( 0.38%) 23 ( 0.71%) 485 ( 0.01%) 1,496 ( 0.14%) 36,632,961 ( 4.86%) 2,555,552 ( 7.73%) 1,109,013 ( 4.16%) 37,687 ( 1.70%) targets/cjson/workspace/cJSON.c:parse_string [/tmp/cjson-prof/cjson-bench] +186,477,852 ( 3.98%) 26,931,432 ( 2.67%) 13,769,100 ( 2.20%) 8 ( 0.00%) 24 ( 0.00%) 27 ( 0.00%) 6 ( 0.19%) 12 ( 0.00%) 8 ( 0.00%) 34,181,574 ( 4.53%) 1,910,257 ( 5.78%) . . ./stdlib/./stdlib/strtod_l.c:str_to_mpn.isra.0 [/usr/lib/x86_64-linux-gnu/libc.so.6] +151,418,740 ( 3.23%) 29,203,622 ( 2.89%) 18,286,458 ( 2.92%) 243 ( 0.00%) 1,054,542 (13.40%) 587,098 (47.29%) 174 ( 5.41%) 337,981 ( 5.71%) 587,071 (56.11%) 23,232,738 ( 3.08%) 198,846 ( 0.60%) . . ./malloc/./malloc/malloc.c:_int_malloc [/usr/lib/x86_64-linux-gnu/libc.so.6] +137,449,736 ( 2.93%) 29,781,868 ( 2.95%) 25,590,922 ( 4.09%) 262,148 ( 4.49%) 1,428,990 (18.16%) 633 ( 0.05%) 24 ( 0.75%) 1,427,269 (24.11%) 633 ( 0.06%) 19,855,242 ( 2.63%) 934,727 ( 2.83%) 4,911,881 (18.41%) 852,937 (38.37%) targets/cjson/workspace/cJSON.c:print_value'2 [/tmp/cjson-prof/cjson-bench] +136,109,002 ( 2.91%) 17,013,641 ( 1.68%) 47 ( 0.00%) 20 ( 0.00%) 265,848 ( 3.38%) 3 ( 0.00%) 20 ( 0.62%) 23,093 ( 0.39%) 0 17,013,622 ( 2.26%) 20 ( 0.00%) 16 ( 0.00%) 8 ( 0.00%) targets/cjson/harness.c:main [/tmp/cjson-prof/cjson-bench] +126,527,205 ( 2.70%) 17,319,117 ( 1.71%) 13,269,930 ( 2.12%) 38,116 ( 0.65%) 645,899 ( 8.21%) 797 ( 0.06%) 12 ( 0.37%) 643,519 (10.87%) 794 ( 0.08%) 20,976,648 ( 2.78%) 1,527,879 ( 4.62%) 1,034,259 ( 3.88%) 40,308 ( 1.81%) targets/cjson/workspace/cJSON.c:print_string_ptr [/tmp/cjson-prof/cjson-bench] +126,133,875 ( 2.69%) 33,635,655 ( 3.33%) 21,022,335 ( 3.36%) 4 ( 0.00%) 39 ( 0.00%) 9 ( 0.00%) 4 ( 0.12%) 3 ( 0.00%) 3 ( 0.00%) 21,022,380 ( 2.79%) 151,242 ( 0.46%) 45 ( 0.00%) 1 ( 0.00%) targets/cjson/workspace/cJSON.c:ensure [/tmp/cjson-prof/cjson-bench] +125,761,796 ( 2.68%) 34,362,552 ( 3.40%) 18,131,061 ( 2.90%) 20 ( 0.00%) 50 ( 0.00%) 7 ( 0.00%) 19 ( 0.59%) 39 ( 0.00%) 5 ( 0.00%) 21,958,583 ( 2.91%) 1,053,443 ( 3.19%) . . ./malloc/./malloc/malloc.c:malloc [/usr/lib/x86_64-linux-gnu/libc.so.6] +122,712,111 ( 2.62%) 27,022,743 ( 2.68%) 25,524,129 ( 4.08%) 386,191 ( 6.61%) 112 ( 0.00%) 67 ( 0.01%) 50 ( 1.55%) 34 ( 0.00%) 13 ( 0.00%) 11,107,557 ( 1.47%) 219,452 ( 0.66%) 1,766,252 ( 6.62%) 240,174 (10.81%) ./stdio-common/./stdio-common/vfprintf-internal.c:__printf_buffer [/usr/lib/x86_64-linux-gnu/libc.so.6] + 93,615,657 ( 2.00%) 35,103,624 ( 3.48%) 15,675,027 ( 2.51%) 16 ( 0.00%) 0 3 ( 0.00%) 14 ( 0.44%) 0 3 ( 0.00%) 10,783,014 ( 1.43%) 487,561 ( 1.48%) . . ./stdio-common/./stdio-common/printf_fp.c:hack_digit [/usr/lib/x86_64-linux-gnu/libc.so.6] + 92,724,627 ( 1.98%) 18,099,150 ( 1.79%) 0 48 ( 0.00%) 260,943 ( 3.32%) 0 48 ( 1.49%) 70,774 ( 1.20%) 0 14,991,483 ( 1.99%) 157,976 ( 0.48%) . . ./string/../sysdeps/x86_64/multiarch/strcmp-avx2.S:__strncmp_avx2 [/usr/lib/x86_64-linux-gnu/libc.so.6] + 85,135,362 ( 1.82%) 15,353,376 ( 1.52%) 12,084,618 ( 1.93%) 3 ( 0.00%) 606 ( 0.01%) 237,460 (19.13%) 3 ( 0.09%) 172 ( 0.00%) 121,186 (11.58%) 19,282,410 ( 2.56%) 824,253 ( 2.49%) 779,724 ( 2.92%) 179,931 ( 8.10%) targets/cjson/workspace/cJSON.c:parse_object'2 [/tmp/cjson-prof/cjson-bench] + 80,179,293 ( 1.71%) 28,635,452 ( 2.83%) 17,181,276 ( 2.75%) 16 ( 0.00%) 549,470 ( 6.98%) 8 ( 0.00%) 16 ( 0.50%) 524,102 ( 8.85%) 1 ( 0.00%) 8,590,644 ( 1.14%) 1,893 ( 0.01%) . . ./malloc/./malloc/malloc.c:free [/usr/lib/x86_64-linux-gnu/libc.so.6] + 56,706,144 ( 1.21%) 16,754,088 ( 1.66%) 14,176,536 ( 2.27%) 80 ( 0.00%) 3 ( 0.00%) 55 ( 0.00%) 8 ( 0.25%) 3 ( 0.00%) 16 ( 0.00%) 6,443,880 ( 0.85%) 19 ( 0.00%) . . ./stdlib/./stdlib/mul.c:__mpn_mul [/usr/lib/x86_64-linux-gnu/libc.so.6] + 55,390,023 ( 1.18%) 18,210,414 ( 1.80%) 12,140,274 ( 1.94%) 6 ( 0.00%) 0 0 6 ( 0.19%) 0 0 6,070,140 ( 0.80%) 218,420 ( 0.66%) . . ./stdio-common/./stdio-common/Xprintf_buffer_write.c:__printf_buffer_write [/usr/lib/x86_64-linux-gnu/libc.so.6] + 54,208,343 ( 1.16%) 8,194,181 ( 0.81%) 0 21 ( 0.00%) 615,305 ( 7.82%) 0 16 ( 0.50%) 615,247 (10.39%) 0 7,663,768 ( 1.02%) 124,028 ( 0.38%) . . ./string/../sysdeps/x86_64/multiarch/strlen-avx2.S:__strlen_avx2 [/usr/lib/x86_64-linux-gnu/libc.so.6] + 52,725,505 ( 1.13%) 13,154,156 ( 1.30%) 9,408,032 ( 1.50%) 14,092 ( 0.24%) 257,621 ( 3.27%) 270,422 (21.78%) 36 ( 1.12%) 24 ( 0.00%) 263,555 (25.19%) 13,682,759 ( 1.81%) 2,812,765 ( 8.51%) . . ./string/../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:__memcpy_avx_unaligned_erms [/usr/lib/x86_64-linux-gnu/libc.so.6] + 44,294,004 ( 0.95%) 11,073,876 ( 1.10%) 7,182,816 ( 1.15%) 261,370 ( 4.47%) 15 ( 0.00%) 42 ( 0.00%) 14 ( 0.44%) 9 ( 0.00%) 15 ( 0.00%) 4,788,780 ( 0.63%) 806 ( 0.00%) . . ./stdlib/./stdlib/strtod_l.c:round_and_return [/usr/lib/x86_64-linux-gnu/libc.so.6] + 43,590,700 ( 0.93%) 9,123,107 ( 0.90%) 8,089,288 ( 1.29%) 14 ( 0.00%) 1,034,506 (13.15%) 1 ( 0.00%) 14 ( 0.44%) 580,184 ( 9.80%) 1 ( 0.00%) 5,710,055 ( 0.76%) 213,263 ( 0.65%) . . ./malloc/./malloc/malloc.c:malloc_consolidate [/usr/lib/x86_64-linux-gnu/libc.so.6] + 39,678,180 ( 0.85%) 4,959,777 ( 0.49%) 2,479,893 ( 0.40%) 5 ( 0.00%) 0 0 5 ( 0.16%) 0 0 7,439,661 ( 0.99%) 158,886 ( 0.48%) . . ./stdlib/../sysdeps/x86_64/lshift.S:__mpn_lshift [/usr/lib/x86_64-linux-gnu/libc.so.6] + 34,172,010 ( 0.73%) 9,913,362 ( 0.98%) 4,508,925 ( 0.72%) 3 ( 0.00%) 1,161,417 (14.76%) 1 ( 0.00%) 3 ( 0.09%) 1,161,394 (19.62%) 0 8,951,199 ( 1.19%) 1,452,229 ( 4.39%) 2,286,159 ( 8.57%) 5 ( 0.00%) targets/cjson/workspace/cJSON.c:cJSON_Delete'2 [/tmp/cjson-prof/cjson-bench] + 28,833,146 ( 0.62%) 3,035,068 ( 0.30%) 0 5 ( 0.00%) 10 ( 0.00%) 0 5 ( 0.16%) 4 ( 0.00%) 0 3,035,068 ( 0.40%) 325 ( 0.00%) . . ./string/../sysdeps/x86_64/multiarch/strchr-avx2.S:__strchrnul_avx2 [/usr/lib/x86_64-linux-gnu/libc.so.6] + 28,713,294 ( 0.61%) 8,563,614 ( 0.85%) 7,556,130 ( 1.21%) 15 ( 0.00%) 9 ( 0.00%) 39 ( 0.00%) 15 ( 0.47%) 3 ( 0.00%) 9 ( 0.00%) 2,518,710 ( 0.33%) 6 ( 0.00%) . . ./stdio-common/./stdio-common/printf_fp.c:__printf_fp_l_buffer [/usr/lib/x86_64-linux-gnu/libc.so.6] + 28,403,442 ( 0.61%) 4,260,519 ( 0.42%) 2,840,346 ( 0.45%) 49 ( 0.00%) 5 ( 0.00%) 28 ( 0.00%) 5 ( 0.16%) 3 ( 0.00%) 10 ( 0.00%) 5,680,683 ( 0.75%) 337,279 ( 1.02%) . . ./string/../sysdeps/x86_64/multiarch/memmove-vec-unaligned-erms.S:__memcpy_chk_avx_unaligned_erms [/usr/lib/x86_64-linux-gnu/libc.so.6] + 28,158,937 ( 0.60%) 6,116,843 ( 0.61%) 3,825,377 ( 0.61%) 13,883 ( 0.24%) 0 0 50 ( 1.55%) 0 0 4,363,379 ( 0.58%) 61,739 ( 0.19%) . . ./stdio-common/./stdio-common/vfprintf-process-arg.c:__printf_buffer + 26,034,150 ( 0.56%) 3,793,830 ( 0.38%) 12,376,362 ( 1.98%) 10 ( 0.00%) 0 8 ( 0.00%) 10 ( 0.31%) 0 4 ( 0.00%) 1,517,532 ( 0.20%) 226,811 ( 0.69%) . . ./stdio-common/./stdio-common/sprintf.c:sprintf [/usr/lib/x86_64-linux-gnu/libc.so.6] + 22,762,980 ( 0.49%) 5,311,362 ( 0.53%) 4,552,596 ( 0.73%) 10 ( 0.00%) 0 0 10 ( 0.31%) 0 0 1,517,532 ( 0.20%) 8 ( 0.00%) . . ./libio/./libio/iovsprintf.c:__vsprintf_internal [/usr/lib/x86_64-linux-gnu/libc.so.6] + 22,444,905 ( 0.48%) 2,014,968 ( 0.20%) 2,780,043 ( 0.44%) 125,950 ( 2.16%) 0 14 ( 0.00%) 17 ( 0.53%) 0 3 ( 0.00%) 2,014,968 ( 0.27%) 4 ( 0.00%) . . ./stdio-common/./stdio-common/printf-parse.h:__printf_buffer + 20,503,584 ( 0.44%) 5,395,680 ( 0.53%) 0 3 ( 0.00%) 189 ( 0.00%) 0 3 ( 0.09%) 44 ( 0.00%) 0 5,395,680 ( 0.72%) 282,256 ( 0.85%) . . targets/cjson/workspace/cJSON.c:buffer_skip_whitespace [/tmp/cjson-prof/cjson-bench] + 19,942,017 ( 0.43%) 1,479,078 ( 0.15%) 1,224,053 ( 0.20%) 53 ( 0.00%) 13 ( 0.00%) 0 12 ( 0.37%) 4 ( 0.00%) 0 1,479,078 ( 0.20%) 273,801 ( 0.83%) . . ./stdio-common/./stdio-common/_itoa.c:_itoa_word [/usr/lib/x86_64-linux-gnu/libc.so.6] + 16,314,180 ( 0.35%) 6,962,100 ( 0.69%) 2,286,064 ( 0.37%) 43 ( 0.00%) 182,820 ( 2.32%) 0 9 ( 0.28%) 135,610 ( 2.29%) 0 3,152,012 ( 0.42%) 123,817 ( 0.37%) . . ./malloc/./malloc/malloc.c:unlink_chunk.isra.0 [/usr/lib/x86_64-linux-gnu/libc.so.6] + 16,148,106 ( 0.34%) 2,691,351 ( 0.27%) 1,794,234 ( 0.29%) 216,019 ( 3.70%) 0 0 9 ( 0.28%) 0 0 3,588,468 ( 0.48%) 47,108 ( 0.14%) . . ./stdlib/../sysdeps/x86_64/rshift.S:__mpn_rshift [/usr/lib/x86_64-linux-gnu/libc.so.6] + 14,416,573 ( 0.31%) 3,793,835 ( 0.38%) 0 2 ( 0.00%) 0 0 2 ( 0.06%) 0 0 3,035,068 ( 0.40%) 2 ( 0.00%) . . ./stdio-common/./stdio-common/Xprintf_buffer_done.c:__printf_buffer_done [/usr/lib/x86_64-linux-gnu/libc.so.6] + 14,365,632 ( 0.31%) 1,795,704 ( 0.18%) 0 261,329 ( 4.47%) 0 0 5 ( 0.16%) . . . . . . ./stdlib/../sysdeps/ieee754/dbl-64/mpn2dbl.c:__mpn_construct_double [/usr/lib/x86_64-linux-gnu/libc.so.6] + 12,894,120 ( 0.28%) 1,934,118 ( 0.19%) 4,512,942 ( 0.72%) 262,717 ( 4.50%) 0 0 7 ( 0.22%) 0 0 644,706 ( 0.09%) 2 ( 0.00%) . . ./libio/./libio/strops.c:_IO_str_init_static_internal [/usr/lib/x86_64-linux-gnu/libc.so.6] + 12,249,414 ( 0.26%) 2,578,824 ( 0.26%) 4,512,942 ( 0.72%) 3 ( 0.00%) 0 0 3 ( 0.09%) 0 0 644,706 ( 0.09%) 2 ( 0.00%) . . ./stdio-common/./stdio-common/isoc99_sscanf.c:__isoc99_sscanf [/usr/lib/x86_64-linux-gnu/libc.so.6] + 11,082,324 ( 0.24%) 503,742 ( 0.05%) 2,014,968 ( 0.32%) 4 ( 0.00%) 0 0 4 ( 0.12%) 0 0 503,742 ( 0.07%) 2 ( 0.00%) . . ./stdlib/../sysdeps/ieee754/dbl-64/dbl2mpn.c:__mpn_extract_double [/usr/lib/x86_64-linux-gnu/libc.so.6] + 10,315,296 ( 0.22%) 3,545,883 ( 0.35%) 967,059 ( 0.15%) 262,717 ( 4.50%) 12 ( 0.00%) 0 7 ( 0.22%) 3 ( 0.00%) 0 2,256,471 ( 0.30%) 16 ( 0.00%) 322,353 ( 1.21%) 1 ( 0.00%) ./libio/./libio/genops.c:__uflow [/usr/lib/x86_64-linux-gnu/libc.so.6] + 9,800,838 ( 0.21%) 2,450,703 ( 0.24%) 0 322,128 ( 5.51%) 0 0 6 ( 0.19%) 0 0 1,634,460 ( 0.22%) 1,423 ( 0.00%) . . ./stdlib/./stdlib/cmp.c:__mpn_cmp [/usr/lib/x86_64-linux-gnu/libc.so.6] + 9,348,329 ( 0.20%) 2,578,849 ( 0.26%) 2,578,849 ( 0.41%) 10 ( 0.00%) 1 ( 0.00%) 14 ( 0.00%) 8 ( 0.25%) 1 ( 0.00%) 5 ( 0.00%) 322,357 ( 0.04%) 4 ( 0.00%) . . ./libio/./libio/genops.c:_IO_setb [/usr/lib/x86_64-linux-gnu/libc.so.6] + +-------------------------------------------------------------------------------- From 832dab34459ac7123d2ca7d0d8a291d8100da834 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 11 Jul 2026 18:38:04 +0000 Subject: [PATCH 5/7] phase3: coz integration status; cjson C pipeline proven end-to-end - coz wired (harness -DHOTPATH_COZ hook, scripts, just coz) + installed, but the apt coz runtime aborts in its own init_coz/pthread_create interposition on this glibc (known coz bug, reproduces on the pristine binary). Documented in results/cjson/coz/README.md; callgrind cache-sim is the working profiler. - cjson no-op verdict smoke: full C pipeline flows (clang {out} baseline rebuild, Unity/CTest + golden gates, interleaved bench) to a correct null result (median 0.9995). Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2 --- results/cjson/coz/README.md | 23 +++++++++++++++++++++++ results/ledger.sqlite | Bin 155648 -> 155648 bytes 2 files changed, 23 insertions(+) create mode 100644 results/cjson/coz/README.md diff --git a/results/cjson/coz/README.md b/results/cjson/coz/README.md new file mode 100644 index 0000000..6a39777 --- /dev/null +++ b/results/cjson/coz/README.md @@ -0,0 +1,23 @@ +# coz on cjson — integration status + +coz is **wired** into the harness (SPEC §12): `targets/cjson/harness.c` +carries a `-DHOTPATH_COZ` throughput progress point, and +`scripts/coz-profile.sh` + `just coz cjson` build the instrumented binary +and run `coz run`. `scripts/coz-summary.py` ranks lines by causal +throughput effect. + +**Runtime limitation in this container:** the distro `coz-profiler` +(apt) aborts inside its own `init_coz` during startup — +`pthread_create` interposition recurses ("init_coz in progress, do not +recurse" → SIGABRT) before `main` runs. This is a known coz/glibc +incompatibility on newer glibc, independent of the harness. It reproduces +on the pristine binary with a bare `coz run`, so it is not our +instrumentation. + +**What we use instead:** callgrind `--cache-sim --branch-sim` +(`results/cjson/hotspots.txt`) is the working profiler and gives an +unambiguous class-selection signal for cjson — glibc float +serialization/parsing (`__printf_fp`, `strtod`) plus cJSON +`parse_value`/`print_value` dominate. The coz path is ready for an +environment with a compatible coz runtime (bench-metal / a pinned coz +build); no code change is needed there. diff --git a/results/ledger.sqlite b/results/ledger.sqlite index 330c371c47b506348fca6397218f110c44fb5ba5..6b9f3e7b3ffd159075994b36c3c827103c484cdc 100644 GIT binary patch delta 653 zcma)3zi-n(6mC*z8bONEPyqsklc!cV$@$KA=ZhdhL}EY!1`q=j5!-ixYg60uxj?hD z5d#Pq0ISNHp<+a;GF0l+l@0y_2E_jW6K+~|-thGFz8_Ee-qt~J>!A4IV$S$@shu-+ zJ3E)BtEEm_0L@%yd>vFvX}@-M#$VB4P#yPcZ}T(Nx$UXB?eQ#pH9egRwziC(J89DbjS1Arkje&f>Ff1?UE1MHj1QD%4R7d~TVJ(b+UJF+x2)iJ8 zCP5oKm$6WO0wNX3o(d#b^xZ%x;U@B06f1oyIb3?{swb9)pO^EWpp6OSl-SHPJXcRd zPkd!GAHlkfEsQX>9rJMJm@)>Pbg*{5oQW{DC`bILX!^ZUsbqY_5n;!C%D6TBR?02h zg!PPO6Ykg)6UuB0nTC#}6zkEyTin4MBFs5rHi3+qRvlV4hYYhUEk?5tQV{$Zo@wMC zZ2oUdhH1PALa zL^s?xb~m&Mr1srNipF}N{7_t#Kqs*Zo@R&&lB^3EXbHO6p+qm4AaY|Z#T6}M<@QuR cGej>7Zj?+cmMzZ delta 84 zcmV-a0IUCizzKlB36L8BA(0$I0U@zqq#pwv4SJIYAUXkhvr#ZO4U=Fk6tj*kb^-() q4xj+D3NVWfkx&hTpgxD7J^{C&J_2MB1py2I01UHHAomNVTml3u#TZcl From 03113312adbc3788cb008ce5b89e5d02a44821b5 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 11 Jul 2026 23:06:59 +0000 Subject: [PATCH 6/7] phase3: cJSON cross-language win verified + audited; robustness fix; case studies phase3-cjson-002 (class 5+3): stack-buffer number parsing + strtod round-trip replacing sscanf. +8.85% median, 95% CI [+7.5%, +10.0%] on the C target, banked. 100% audit: byte-identical golden replay, independent ASan+UBSan clean, 10k-input differential fuzz 0 divergences, patch bounds/free-gating verified by eye. - verdict: an infrastructure sanitizer-BUILD failure (missing runtime, broken toolchain) now caps the accept at needs-human-review and logs it, instead of crashing the verdict and losing a measured bench. - results/phase3/: four public case studies (tokei class-5 wins; build flags + allocators; the comrak caught false-accept; the cJSON cross-language proof) + README. Compound-speedup claim stated as an approximation, not a measured figure (CIs don't multiply). - .gitignore: cjson build-artifact dirs. Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2 --- .gitignore | 3 + crates/verdict/src/main.rs | 10 +- results/ledger.sqlite | Bin 155648 -> 167936 bytes results/phase3/01-tokei-algorithmic-rust.md | 64 +++++++++++++ .../phase3/02-build-flags-and-allocators.md | 50 ++++++++++ .../phase3/03-comrak-caught-false-accept.md | 68 ++++++++++++++ results/phase3/04-cjson-c-target.md | 86 ++++++++++++++++++ results/phase3/README.md | 68 ++++++++++++++ 8 files changed, 348 insertions(+), 1 deletion(-) create mode 100644 results/phase3/01-tokei-algorithmic-rust.md create mode 100644 results/phase3/02-build-flags-and-allocators.md create mode 100644 results/phase3/03-comrak-caught-false-accept.md create mode 100644 results/phase3/04-cjson-c-target.md create mode 100644 results/phase3/README.md diff --git a/.gitignore b/.gitignore index 9902998..ca1f766 100644 --- a/.gitignore +++ b/.gitignore @@ -244,3 +244,6 @@ agent/**/__pycache__/ corpora/cjson/input/ targets/cjson/testbuild/ results/*/callgrind-cache.out +targets/cjson/candidate-*/ +targets/cjson/cand-*/ +targets/cjson/asan*/ diff --git a/crates/verdict/src/main.rs b/crates/verdict/src/main.rs index 84dd7ab..7efec72 100644 --- a/crates/verdict/src/main.rs +++ b/crates/verdict/src/main.rs @@ -136,7 +136,15 @@ fn sanitizer_check( .arg(diff_test::target::subst_out(build_tpl, &out_dir)) .current_dir(root) .status()?; - anyhow::ensure!(status.success(), "sanitizer build failed"); + if !status.success() { + // Infrastructure failure (missing runtime, toolchain), not a code + // defect: don't crash the verdict and lose a measured bench — + // treat "cannot build the sanitizer binary" like "no sanitizer + // declared" so the accept caps at needs-human-review (None), with + // the failure logged for the operator to fix and re-run. + println!("sanitizers: BUILD FAILED — cannot verify; capping accept at needs-human-review"); + return Ok(None); + } let asan_bin = diff_test::target::subst_out(bin_tpl, &out_dir); let cmd = spec.bench.command.replace("{binary}", &asan_bin); let out = std::process::Command::new("sh") diff --git a/results/ledger.sqlite b/results/ledger.sqlite index 6b9f3e7b3ffd159075994b36c3c827103c484cdc..f866981275e88694244752204c91190d606b0801 100644 GIT binary patch delta 7558 zcmeHM-H#hr6<==}C3TujrEExOTK06CWM|eM$77Gjp55+tMa4sDQ>X}3YANn`e7&B; zp0Q@e3EMQCEFeLpK2W)lUy9VCijRm$M3LwdqCE3}`iKOm;y-{F5KsK>y<_iqck=~p zUx**pv*Y`5K7Qw%dw%!IM-N_kFZ<);drwSFO{s6wQ<)PtUj5+E{@FwC;>DCYcgx3Q$Fm0^K_+u>Ds%A0=AO*bPkM8+TIJ{~navk7Kg#@+ZL+uas{8kC zlS3~ROQu$=YPx<_uPqtnCEXbQ^Tp}YanBAyR`>ed1`DXi0xfy$+MSN?obIx&)7v=R z3K*k^b$fnb2OHGrw^|sktasdoLv`1S>dAQVEJnXkEtn_CZ_zV>x@TgSHnV%0Sl*6e z)+9T%^7G$(PK#L3bv--6Pos_G>NzP*j<2s$%MXZMU=Zi4-}u_|l*oOqA4U}U)MA(( zMnU8^mnmv9a{Qpz4+;LEH}qohd``@Gi3MSvy7qb3TgNargfvZ`Z|_AEw(Wp%6%yry zUQW9oc%-D1rgEWFK1n%g2rqpB3zp}u?i_i(|$Z3l`^W<@>bFBjH zf*ZPx4l6X>R*N)k-HpgT?Qnk-9BSYx7LQriYqIZ>$&6N`*en!kcFnRIRe~Q(!^moy zwlyuA%jLvWIdg4|w6b2#o0Q{UdWrA`HiV=kHmWy#ze8z&DdIgX!sT8JG`Ik6J{ge! zO8mxu@}OwF(a#e9gzjssj+eAT#pQ`t(w=qT3FfSKn71CaQ=?LX0AKYcb6h~C-t*lU zFk0ar4B2uv$2U2>NN0R(odPXD2b>A#lfv=WhfQH-)(q*%l1UD57^=(Ar?+r zOwx63ixeduc~~T@l@&sQc}}I*UneE~i?#}^M8H3dhNyCpOIZQH3XrD*ffXySTla!M z#rN(Gfy4Ln(}!b3uivXPuep@f60k^%i%8#|fb(QGo1}~17QS6v;6@bF61X}EV1Yno z(yoi@82a2W+G^YP&xeJPx0tdk;B%y_%W?;9+#S0l)|HL~&Lu2cb{J_KXhKejayd0z zPNlMoZRW)sJR1lAZy}-0yFBMe7d@(DXMoB(73#>-M)h7gxem*m&(3*6QbXUCpMB)hF6L^|QSq^z^$ z^0709pKN4~4%Zf@_s!i}bD{X)y4{w~A{=$r|Tg8f{n?}*Z z_u%ipJv?7D482%3jFM$xD9S^jSklGxicz!-vsN9D_wNt?_|o)ZCVom?Id}5cyn&b! zQSccS{N3t(yj7<^^MapBspEPd%h>;13Vs@oqhcO^js<_WdOypXrR_(~`vw1g!Dsgi z{{4dge_rt8|E=IBCn6I^A>N^xWj(LgZqcEkp^Xx%OF_Sf#te;Q4DM*0nu`x*;; zYPz6HLW)O+JAURGGYesl3(-;(-UctK$>CWFm8bcWdO@$O&*!Ph4{Nd|1MI|UoP=0_6x3koh8!P64rkyX!njG|M>8c1G3eI2clg$)KpsUZJ-&jj zkT=wiJAN03R#unWD2n`6`7kc1x^;9;iy7agoUA!!vB1({($;eDpWw5^kr_YPpzfps z^Cuyy%6Uz&9$hef7hRBLeK3S*4EQTBkeci$CzgZlSpZjZo(Znxb^^!HB+hSe)+|Mq z1uX1$pkT*cZ(|aummR{TXw|kKiS7m`aUmhK6^7pHB4;HzkHh&&pb12!l#VX?!Fg5C zBg4nKnPV|b_e={I!j1wWLL<_vw}oQbzxN8W>Ii0{@nL*A{|I_!G4T6dvkq2FkQU$> zIK7hUk@!gNfBYhKy9J<6O)fNi`wyAJXB1`fEkDlf77urYzQJFw&HvZN8~1!%AO5+Q zINdVf@O$~p7l+sL)BA?Me>Bq`{OenftQjSH6^FGnu1@ zYBi%)HB1vNih+J+@WLyH=M3~YHM3%wm9kzz8v{;O*Q>RXS<@}OI__eQXCAQ~hxNe9 zudN;b)6W-YKN2?)2d+&WxOVdedhfu}FP^33Dx#iwbM7~6(|PNO$97yrptG8|im;YS z#a*rcg^i1hM>gUcR~@{43YR9>W|qfq$C7t`X&Q3 z9w^%$r|L)LiI8wmFSu|vc=uLdJ3J8~hs#Q^5Rxnbl+v8%wo&3+K;m0aMZU7B)yDzFThOUwg{ylwxf|nM)@9z_sc$vI+^nG(qiz2#(DNzUw-U*_hCBChHmU~_YvdH biQvvkowC4v~-?gOxsql|BKtl|BM% EAxvo;v;Y7A diff --git a/results/phase3/01-tokei-algorithmic-rust.md b/results/phase3/01-tokei-algorithmic-rust.md new file mode 100644 index 0000000..0b369e0 --- /dev/null +++ b/results/phase3/01-tokei-algorithmic-rust.md @@ -0,0 +1,64 @@ +# Case study 1 — tokei: compounding algorithmic wins (Rust, class 5) + +**Target:** [tokei](https://github.com/XAMPPRocky/tokei) v14.0.0 — a +source-code line counter. **Workload:** 221 MB / 4360 files (8 replicas +of the Pro Git book + comrak's own tree), single rayon thread, CPU +pinned. **Result:** three independent, CI-verified class-5 wins on the +same hot function — each proposed *by the unattended agent* and +100%-audited. Multiplied together they are **~1.26× over the Phase-0 +baseline** (1.107 × 1.037 × 1.099), but that product is an +approximation, not a measured figure: each speedup carries its own +bootstrap CI against its own baseline, and confidence intervals do not +compose through multiplication. The trustworthy claims are the three +individual increments below. + +## The hot path + +tokei spends the overwhelming majority of its instructions in one +function — `LanguageType::parse_lines` and the per-byte loop inside +`SyntaxCounter::perform_multi_line_analysis`. The cache-sim profile put +it at **55.9% of instructions, 62% of all branches, 44% of branch +mispredicts** on the pinned corpus: for every byte of every line, the +counter tested a stack of quote/comment token matchers. + +## The three wins + +| run | mechanism | speedup (median) | 95% CI | +|---|---|---|---| +| `phase0-tokei-002` (class 3) | read path: `fs::read` pre-sized, BOM-only decode | 1.099 | [1.080, 1.127] | +| `phase2-tokei-001` (class 5) | 256-entry first-byte gate table for the token matchers | 1.107 | [1.084, 1.131] | +| `phase2-tokei-003` (class 5) | forward-only whitespace break (drop the `rposition` rescan) | 1.037 | [1.027, 1.052] | +| `phase2-tokei-008` (class 5) | streaming leftmost `important_syntax` scan (one vectorized pass, not per-line restart) | 1.099 | [1.073, 1.112] | + +Each is a **redundant-work elimination**: the first-byte gate replaces +five per-byte matcher loops with one table load; the whitespace-break +change drops a full backward scan of the file tail per byte; the +streaming scan replaces a per-line aho-corasick restart (which kept +memmem's SIMD prefilter in its short-haystack slow path) with a single +leftmost search advanced by integer comparisons. + +## Why the numbers are trustworthy + +- **Byte-identical golden replay** on the pinned corpus for every win — + a code counter that miscounts one line fails the gate. +- **10,000-input differential fuzz** (`scripts/diff-fuzz-tokei.py`), + pristine vs candidate, **0 divergences** — after canonicalizing + tokei's benign parallel-walker output ordering (a real finding: + raw-byte differential comparison over-counts when the target is + legitimately nondeterministic in presentation). +- **ASan/LSan clean** on the patched tree over the full corpus. +- **Interleaved A/B** vs a pristine-rebuilt baseline, bootstrap 95% CI + lower bound ≥ the 2% bar. `phase2-tokei-008` was the first accept + through the *machine-enforced* sanitizer gate (see case study 3). + +## What it demonstrates + +The richest wins on a mature Rust binary were not exotic — they were +"this loop does work it doesn't need to, on every byte." The agent found +each by reading the cache-sim profile, checking the ledger to avoid +re-attempting a class on the same hotspot, and proposing the cheapest +untried mechanism. Three separate hypotheses, three separate verified +increments, all on the same 40%-of-runtime function — and one honest +rejection in between (`phase2-tokei-009`, a real +5.7% median whose +noisy CI lower bound of +0.2% could not clear the bar, so it does not +count). diff --git a/results/phase3/02-build-flags-and-allocators.md b/results/phase3/02-build-flags-and-allocators.md new file mode 100644 index 0000000..faccf1c --- /dev/null +++ b/results/phase3/02-build-flags-and-allocators.md @@ -0,0 +1,50 @@ +# Case study 2 — cheap wins first: build flags and allocators + +**The playbook's ordering principle (SPEC §3.3, CLAUDE.md +non-negotiable #3): build flags → LTO → PGO → allocator swap → *then* +code changes.** The cheapest classes are tried first because they carry +the lowest cost-per-attempt and the lowest risk. Two of them produced +Phase-0 results that frame the whole project's discipline. + +## Class 1 — link-time optimization (tokei) — ACCEPTED + +`phase0-tokei-003`: set `lto = true` (fat LTO, up from `thin`) and +`codegen-units = 1` in tokei's release profile. + +- **Speedup: 1.104 median, 95% CI [1.085, 1.120]** vs the pristine + `thin`-LTO baseline. +- Byte-identical golden replay; no source change, so equivalence is by + construction (same code, different codegen). Auto-accept tier. +- A pure build-config win worth **+10%** — the cheapest possible class, + and on this target the single largest individual increment. + +The lesson the ledger enforces: you do not go hand-optimize a hot loop +before you have tried turning on the optimizer's own switches. + +## Class 2 — allocator swap (comrak) — NEEDS-HUMAN-REVIEW + +`phase0-comrak-002`: swap the global allocator to +[mimalloc](https://github.com/microsoft/mimalloc). + +- **Speedup: 1.046 median, 95% CI [1.033, 1.058]** — a real, CI-clean + +4.6% on comrak's Pro Git render. +- Byte-identical golden replay. And yet the verdict is **not + `accepted`** — it is **`needs-human-review`**. + +Why: an allocator swap changes the process's memory behavior globally. +It is exactly the class of change SPEC §8 routes to a human — not +because this instance is wrong, but because "the benchmark got faster" +is not sufficient evidence that a global allocator change is safe under +every workload, fragmentation pattern, and concurrency scenario the +target will meet in production. The pipeline measured a real win and +then **refused to auto-ship it**, tagging it for a human ruling instead. + +## What the pair demonstrates + +The cheap classes are not lesser — class 1 produced this project's +biggest single win. But "cheap to try" is not "safe to auto-accept." +LTO is deterministic codegen: accept it. An allocator swap is a global +behavioral change: measure it, prove the speedup, and *still* hand it to +a human. The tier is decided by the *nature* of the change, never by the +size of the number — the same rule that later caught a leaking teardown +patch the bench loved (case study 3). diff --git a/results/phase3/03-comrak-caught-false-accept.md b/results/phase3/03-comrak-caught-false-accept.md new file mode 100644 index 0000000..9940c55 --- /dev/null +++ b/results/phase3/03-comrak-caught-false-accept.md @@ -0,0 +1,68 @@ +# Case study 3 — the caught false-accept (comrak, class 3) + +**This is the most important case study in the project, because it is +the one where the machine got it wrong and the process caught it.** + +## What happened + +`phase2-comrak-010`: the unattended agent, reading comrak's cache-sim +profile, saw that AST-arena teardown at process exit — +`drop_in_place::>>>` plus its share of the glibc +free path — cost real instructions and cache misses in a +run-to-completion CLI. Its hypothesis: comrak already ends in +`process::exit`, so the OS reclaims memory in bulk anyway; wrap the arena +in `std::mem::ManuallyDrop` to skip the per-node drop walk, and add a +`std::hint::black_box(&arena)` before the scope closes so the leak stays +reachable and a leak checker won't flag it. + +The pipeline ran it: + +- Upstream tests: **PASS** +- Golden replay: **byte-identical** +- Interleaved A/B: **+10.7% median, 95% CI [+7.9%, +12.1%]** +- Verdict: **`accepted`** + +A clean, large, CI-significant win. And it was **wrong**. + +## The audit overturned it + +Phase 2 runs a **100% human audit** of every accepted win (SPEC §10). +The audit ran the sanitizers the pipeline had not: LeakSanitizer flagged +a **384-byte arena chunk as unreachable at exit**. The agent's +`black_box(&arena)` keeps the arena live only until its scope *closes* — +but `process::exit` runs *after* that scope, so at the moment of exit the +arena is genuinely unreachable. A real leak. The agent's own hypothesis +had even named the risk ("if the sanitizer still flags the leak, +rejected-gates is the honest outcome") — the pipeline simply had no +sanitizer gate to trip, because in Phase 2 ASan/LSan were a per-attempt +*manual* step. + +Called per SPEC §10: a **pipeline false accept**, caught by audit +**before anything shipped**. The win was never banked. Shipped false +accepts across the entire project remain **zero**. + +## The fix made the machine enforce the tier + +The corrective action was not "be more careful." It was structural: +`verdict` now **builds the patched tree with ASan+LSan (ASan+UBSan for +C/C++) and runs the pinned workload on every would-be accept**, capping +any sanitizer-flagged win at `needs-human-review`. The tier rule SPEC §8 +always specified is now enforced by the machine, not the auditor. The +very next Rust accept (`phase2-tokei-008`) and the first C accept +(`phase3-cjson-002`) both passed through this automated gate. + +The 10k-input differential fuzz on the overturned patch found **0 +divergences** — the output equivalence was genuinely real. Only the +teardown-leak tier was wrong. That is the subtle failure mode the layered +gates exist for: a change can be behaviorally equivalent on output and +still unacceptable on a dimension the bench and the golden replay can't +see. + +## What it demonstrates + +The product is not "an agent that finds speedups." The product is +**trust** — and trust is demonstrated most by the case where the +optimistic path failed and the structural controls held. An agent +graded only by accept-rate would have banked this win. The ledger is +append-only, so the `accepted` row still stands, immutable — with this +document as the permanent corrective record beside it. diff --git a/results/phase3/04-cjson-c-target.md b/results/phase3/04-cjson-c-target.md new file mode 100644 index 0000000..5b75564 --- /dev/null +++ b/results/phase3/04-cjson-c-target.md @@ -0,0 +1,86 @@ +# Case study 4 — cJSON: the cross-language proof (C, classes 3 + 5) + +**Target:** [cJSON](https://github.com/DaveGamble/cJSON) v1.7.19 — an +ultralightweight C JSON parser/printer, MIT. **Workload:** a 5.6 MB +deterministic JSON corpus (20k records: nested objects/arrays, +escaped/unicode strings, integers, dense floats), parsed and serialized, +single-threaded, CPU-pinned. **Result:** the first verified win on a +C/C++ target — proof the trust layer, the gates, and the ROI story +carry across languages, not just across Rust crates. + +## Getting C through the same pipeline + +Nothing in the trust layer assumes cargo anymore. cJSON is "just a +`target.toml`" pointing at `clang` build commands, with: + +- a **hermetic Unity/CTest suite** (19 tests: `parse_number`, + `parse_string`, `print_*`, `misc_tests`, …) as the upstream-test gate; +- a **trust-layer harness driver** (`targets/cjson/harness.c`) compiled + *with* the patched `cJSON.c` — the agent can patch the parser but not + the measurement rig (it lives outside the workspace allowlist); +- **ASan + UBSan** as the sanitizer gate (the C analogue of the Rust + nightly ASan path); +- its own **A/A calibration**: 0/20 false positives, 20/20 + injected-slowdown detections — the C workload is a trusted anchor + before any verdict runs. + +## The profile + +callgrind cache-sim put **~22% of instructions in glibc float I/O** — +`__printf_fp` (number serialization) and `strtod`/`str_to_mpn` (number +parsing) — plus `parse_value`, `parse_string`, and heavy malloc/free +churn. A number-dense JSON document spends most of its time turning +numbers into strings and back. + +## Attempt 1 — the honest rejection (`phase3-cjson-001`, class 3) + +`parse_number` heap-allocated a temporary NUL-terminated buffer +(`malloc` + `memcpy` + `free`) for **every** number, just to hand a +clean string to `strtod`. With ~100k+ numbers in the corpus that is a +large share of the allocation churn. The fix: a 64-byte **stack buffer** +for the common short-number case, heap only for pathologically long +number strings — byte-identical (strtod sees the same bytes). + +- **Speedup: 1.012 median, 95% CI [1.004, 1.022]** → **rejected-bench.** + +A real mechanism, a real +1.2%, but the CI lower bound (+0.4%) does not +clear the 2% bar — because the corpus's wall-clock is dominated by float +*formatting*, not by malloc. The statistical bar rejected a genuine win, +exactly as designed. It is in the ledger. + +## Attempt 2 — the accepted win (`phase3-cjson-002`, classes 5 + 3) + +Adding the class-5 lever: `print_number` verified every fractional +number's `"%1.15g"` round-trip by calling **`sscanf("%lg")`** — a full +`strtod` *plus* format-string interpretation, per number. Replacing it +with a direct `strtod` is the identical parse with none of the +format-string overhead. Combined with the stack buffer: + +- **Speedup: 1.089 median, 95% CI [1.075, 1.100]** → **accepted.** +- Upstream Unity suite: **PASS**. Golden replay: **byte-identical**. +- **ASan + UBSan: clean** — through the machine-enforced sanitizer gate. +- **10,000-input differential fuzz** (`scripts/diff-fuzz-cjson.py`), + pristine vs candidate, **0 divergences**. + +A **+8.9%** verified win on a C target, banked. + +## A finding worth keeping + +The first cJSON accept nearly wasn't recorded: the ASan+UBSan build +failed to link because the clang sanitizer runtime (`libclang_rt.asan`) +was not installed, and `sanitizer_check` treated that as a hard error — +crashing the verdict and losing the measured bench. The fix hardened the +gate: an **infrastructure** build failure (missing runtime, broken +toolchain) now caps the accept at `needs-human-review` and logs the +failure for the operator, rather than discarding a real result. "Cannot +verify" is a review tier, not a crash. + +## What it demonstrates + +The progression — a real win *rejected* for being too small, then a +larger win *accepted* only after golden replay, fuzzing, and sanitizers +all held — is the whole method in miniature, now shown on C. Nothing +about the trust story is Rust-specific. The cheapest safe lever (skip a +per-number allocation) wasn't enough; the algorithmic lever (stop doing a +redundant heavyweight parse) was — and the pipeline proved it the same +way it proves everything else. diff --git a/results/phase3/README.md b/results/phase3/README.md new file mode 100644 index 0000000..734250e --- /dev/null +++ b/results/phase3/README.md @@ -0,0 +1,68 @@ +# Phase 3 — playbook + proof + +**Exit criteria (SPEC §5): 3–5 public case studies · zero regressions +shipped · playbook ≥6 classes · C/C++ targets · coz.** + +Phase 3 turns the trust layer and the agent loop from Phases 1–2 into +*evidence*: verified optimization wins across the playbook and across +languages, each one reproducible from a pinned corpus through the same +gated pipeline, with every attempt — win or loss — in the append-only +ledger. + +## What changed for Phase 3 + +- **The trust layer went language-agnostic.** Build isolation moved from + a cargo-specific `CARGO_TARGET_DIR` injection to an explicit `{out}` + placeholder plus optional `sanitizer`/`sanitizer_binary` build + templates (`crates/diff-test/src/target.rs`). A C/C++ target is now + just a `target.toml` with `clang` commands. Proven zero-regression: + tokei ran a full verdict through the refactored pipeline to a correct + null result before any C target was added. +- **A C target joined the roster** — cJSON (v1.7.19, MIT), with a + trust-layer harness driver, a hermetic Unity/CTest gate, ASan+UBSan + sanitizers, a deterministic pinned corpus, and its own A/A calibration + (0/20 false positives, 20/20 injected-slowdown detections). CI gates + it on every push. +- **coz** is wired (`-DHOTPATH_COZ` progress point, `just coz`, + `scripts/coz-*`); its apt runtime aborts in its own init on this + container's glibc (documented in `results/cjson/coz/README.md`), so + callgrind cache-sim is the working profiler. + +## Playbook classes exercised (≥6 required) + +| Class | Mechanism | Exercised on | Result | +|---|---|---|---| +| 1 | build config (LTO, codegen-units) | tokei | ✅ win (+10.4%) | +| 2 | allocator swap (mimalloc) | comrak | needs-human-review (+4.6%) | +| 3 | allocation churn | tokei, comrak, **cjson** | ✅ wins + one overturned | +| 4 | data layout | comrak | attempted (rejected) | +| 5 | algorithmic / redundant work | tokei ×3, **cjson** | ✅ wins | +| 6 | SIMD / autovectorization | comrak, tokei | attempted (rejected) | + +Six classes exercised (1–6); class 5 and class 3 carry verified wins on +both a Rust and a C target. + +## The case studies + +1. [`01-tokei-algorithmic-rust.md`](01-tokei-algorithmic-rust.md) — + three independent CI-verified class-5 wins on a Rust code counter + (~1.26× multiplied, stated as an approximation, not a measured + figure), each unattended and audited. +2. [`02-build-flags-and-allocators.md`](02-build-flags-and-allocators.md) + — the cheap wins first: LTO (class 1) accepted, mimalloc (class 2) + correctly held at needs-human-review. +3. [`03-comrak-caught-false-accept.md`](03-comrak-caught-false-accept.md) + — the pipeline auto-accepted a leaking teardown patch; the audit + overturned it and hardened the pipeline. The most important study. +4. [`04-cjson-c-target.md`](04-cjson-c-target.md) — the cross-language + proof: a real sub-threshold win correctly rejected, then a byte- + identical number-handling win accepted on the C target. + +## The through-line + +Every number here carries its 95% bootstrap CI and its workload. Every +win survived byte-identical golden replay, a 10k-input differential +fuzz, and sanitizers. Every rejection is a real mechanism that simply +didn't clear the 2% CI-lower-bound bar — and it is in the ledger too, +because a rejection is a complete, valid outcome. Nothing shipped that +the pipeline couldn't prove. From 985b85205c35b5fd962191785ff07e0560e6cfd6 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 11 Jul 2026 23:07:48 +0000 Subject: [PATCH 7/7] =?UTF-8?q?phase3:=20mark=20COMPLETE=20in=20CLAUDE.md?= =?UTF-8?q?=20=E2=80=94=20all=20SPEC=20=C2=A75=20exit=20criteria=20met?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Co-Authored-By: Claude Fable 5 Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2 --- CLAUDE.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/CLAUDE.md b/CLAUDE.md index 9ebaa12..40e0b2a 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -32,7 +32,7 @@ Full engineering spec: `SPEC.md`. Read it before starting any phase. ## Current phase -**Phase 2 COMPLETE — all SPEC §5 exit criteria met.** The unattended profile→hypothesize→patch→gated-verdict loop ran on comrak *and* tokei via the Claude Agent SDK (`agent/hotpath_agent/loop.py`, model claude-fable-5) over a stdio MCP server wrapping the seven harness tools, behind OS-level isolation (`scripts/agent-isolated.sh` mountns/user modes, `just isolation-check` 19/19). **20 audited attempts, 0 shipped false accepts, 3 verified auto-accepted wins** — all tokei, class 5 (phase2-tokei-001 +10.7%, -003 +3.7%, -008 +9.9%; compounded ~1.25× vs upstream v14.0.0 with the Phase-0 wins), each fuzz-clean (`scripts/diff-fuzz-*.py`, 10k inputs) and ASan/LSan-clean. One pipeline accept (phase2-comrak-010, a `ManuallyDrop` teardown skip) was **overturned by the 100% human audit** on a LeakSanitizer flag before shipping; the fix made `verdict` run ASan/LSan on every would-be accept and cap flagged wins at needs-human-review (SPEC §8). Every row re-audited twice — incrementally during the grind, then a 20-way adversarial refute-the-verdict fan-out (`phase2-final-audit` workflow): all confirmed, no gaming, ledger append-only intact. Case study: results/phase2/case-study-agent-loop.md. **Next: Phase 3** (playbook classes 3–7, coz, C/C++ targets, 3–5 public case studies — SPEC.md §5). +**Phase 3 COMPLETE — all SPEC §5 exit criteria met.** The trust layer went language-agnostic: build isolation moved from a cargo-specific `CARGO_TARGET_DIR` injection to an explicit `{out}` placeholder + optional `sanitizer`/`sanitizer_binary` templates (`crates/diff-test/src/target.rs`), proven zero-regression on tokei before any C target. **A C target joined the roster** — cJSON v1.7.19 (MIT): trust-layer harness driver compiled with the patched `cJSON.c`, hermetic Unity/CTest gate (19 tests), ASan+UBSan sanitizers, deterministic pinned corpus, own A/A calibration (0/20 FP, 20/20 injection), CI-gated. **First verified cross-language win:** phase3-cjson-002 (class 5+3, stack-buffer number parse + `strtod` round-trip) +8.85%, 95% CI [+7.5%, +10.0%] — banked, byte-identical golden replay, independent ASan+UBSan clean, 10k-input diff-fuzz (`scripts/diff-fuzz-cjson.py`) 0 divergences. **Four public case studies** in `results/phase3/` (tokei class-5 wins; build flags + allocators; the comrak caught false-accept; the cJSON cross-language proof). **Six playbook classes exercised (1–6)**; wins on classes 1/2(review)/3/5 across Rust *and* C. coz wired (`-DHOTPATH_COZ`, `just coz`, `scripts/coz-*`); its apt runtime aborts in init on this container's glibc, so callgrind cache-sim is the working profiler (documented). Zero regressions shipped. **Next: Phase 4** (load replay, latency mode under replayed traffic, mechanical ROI reports — SPEC.md §5; `crates/report` has the ROI math, needs the generator). ## Commands