An adversarial audit found the enforcement story ahead of the code: gates
that were documented as hard gates were advisory, manual, or absent, and
several human-facing claims outran the machine record. This makes the
claims true.
High findings
- Differential fuzz was documentation, not enforcement: verdict hardcoded
fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case
studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real
pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and
it is inherently baseline-vs-candidate, so it runs on the accept path and
reports Skipped in the standalone `just gates` flow. New hard rule: a
machine `accepted` verdict is impossible unless the fuzz gate actually
passed — otherwise it caps at needs-human-review.
- The agent's system prompt claimed FP/concurrency/unsafe patches were
"routed to human review automatically". No such mechanism existed. Added
a conservative lexical risk classifier (crates/verdict/src/risk.rs) that
forces needs-human-review on any would-be accept carrying those signals;
fp-tolerance equivalence always signals.
- The local verdict path ran agent-influenced code unsandboxed. It now runs
under scripts/no-net.sh (network namespace) and fails closed. Upstream
test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when
present (scripts/pin-testsuite.sh generates them).
Also fixed, found while fixing the above
- The pipeline script's `set -e` was inert (brace group as the left operand
of `&&`), so a failing candidate build did not abort the run and verdict
could bench a stale binary. Now a scoped subshell whose status is branched
on at top level.
- A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path)
stranded RUNNING.lock and left read_verdict polling forever. Failure
accounting now lives outside the wrapper.
- harnessd's patch allowlist ignored git rename/copy headers and --no-prefix
diffs; the rename-escape case is now a committed regression test.
- The workspace did not compile on Windows (ungated Unix APIs). cfg-gated;
71 portable tests now run on both platforms.
- pin-check regenerated the corpus manifest instead of verifying it; it is
now verify-only, with a deliberate `just pin-corpus` to re-pin.
- Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics
(compare --perf-stat), never part of the accept decision.
Docs now match the code, including what is NOT yet enforced: no target ships
a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not
wired to the fp-tolerance gate — each is stated as mechanized-but-inert
rather than done. Corrected the stale "33 ledger rows" (34), the playbook's
class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation
claim (five accepted rows carry sanitizers_clean=false), the unsupported
"40,000 requests" figure, and the blanket "historical rows record
fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The kernel lane (PR #14) added a second binary to the diff-test crate
(src/bin/fp-compare.rs), which made `cargo run -p diff-test -- <target>`
— the exact invocation the target-gates-no-net CI job uses to run a target's
equivalence gate — ambiguous:
error: `cargo run` could not determine which binary to run.
available binaries: diff-test, fp-compare
Naming the primary binary via `default-run` restores the single-binary
behavior without touching the CI workflow or the gate command. Regression
was latent on main since #14 merged; this PR's CI re-run surfaced it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
The project ships as peltier.io; "hotpath" was the placeholder working name
(CLAUDE.md: "Rename freely; nothing depends on it"). Replaces the product
name in the README front page and every mdBook source chapter — book title,
nav header, and prose ("Peltier profiles…", "Peltier sells…") — plus the
theme CSS comment.
The metaphor is unchanged (a Peltier device is a thermoelectric cooler:
"spot cooling for hot paths"), and the literal noun phrase "hot paths"
stays. Two code identifiers are intentionally left as-is because they are
not prose: the `agent/hotpath_agent/` Python module path and the
`HOTPATH_ISOLATION` env var — renaming those is a code change, not a docs
one.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
Phase 5 write-up for the two SPEC §13 forks now that both are built:
- results/phase5/README.md — the full argument for each fork, with the
kernel-lane demo and learned-policy ranking tables and reproduce commands.
- results/phase5/learned-policy.txt — captured `cargo run -p policy` output
over the 34-row ledger.
- docs/src/research-forks.md — new mdBook chapter, wired into SUMMARY.md.
- docs/src/roadmap.md — Phase 5 row added; the Frontier section rewritten
from "out of scope" to "built, both routed to human review", with the
honest note that the actual GPU run needs GPU hardware absent here.
- CLAUDE.md — Phase 5 COMPLETE status paragraph.
No trust-layer or gate change; this is documentation of work already
committed (learned policy b7297ad, kernel lane 5abeb60).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
The parser/service targets only ever needed byte-identical golden replay.
The kernel lane is the case that breaks it: a matmul optimization
(transpose-B for cache locality + eight-accumulator ILP) that REORDERS the
floating-point reduction, so its low bits differ by construction —
byte-identical is the wrong gate.
Adds:
- EquivalencePolicy::compare + Divergence (diff-test): byte-identical is
exact; fp-tolerance tokenizes and compares numeric tokens within
abs + rel*|baseline|, with explicit NaN handling. Non-numeric tokens and
token-count mismatches still fail exactly.
- fp-compare bin: applies a target's equivalence.toml to two output files.
- targets/matmul/kernel.c: matmul_ref (single-accumulator oracle) vs
matmul_opt (reordered, faster), with emit/check/run/bench modes.
- targets/matmul/equivalence.toml: fp-tolerance abs 1e-4 + rel 1e-3.
- scripts/kernel-lane-demo.sh + results/phase5/kernel-lane.txt: byte-identical
FAILs, fp-tolerance is EQUIVALENT, a +0.5 perturbation is still REJECTED,
speedup 3.23× measured with the same interleaved A/B + bootstrap CI.
Ledger row phase5-matmul-opt = needs-human-review: using the FP-tolerance
tier at all is a human-review signal (SPEC §8), same posture as mimalloc.
The trust machinery is unchanged; only the timer and the gate differ, which
is exactly the shape a future GPU kernel would slot into.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
crates/policy: reads the append-only ledger and ranks playbook classes by
the Wilson score lower bound of their machine-verified win rate — the same
'quote the lower bound' discipline the rest of the project uses. A class
that won 1/1 does not outrank one that won 8/10 (wider interval); untried
classes fall back to the cheapest-first prior; tried-and-failed classes
rank last. Advisory only — the gates still decide every verdict.
On the current 33-row ledger it ranks class 5 (algorithmic) first, then
class 3, then class 1 — the proven winners — with class 7 (untried) above
the evidence-against classes. Wins count only machine-sanitizer-verified
accepts (conservatively excluding pre-gate accepts and the overturned
comrak-010): the policy trusts the ledger's machine record, not narrative.
4 unit tests (Wilson monotonicity, small-sample penalty, ranking order).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
- docs/: an mdBook documentation site — thesis, architecture, measurement
discipline, equivalence gates, agent loop, isolation/anti-reward-hacking,
playbook, ROI, reproduce, roadmap, and the five case studies (pulled from
their single source in results/ via {{#include}}). Builds clean (21
pages).
- .github/workflows/docs.yml: builds mdBook and deploys to GitHub Pages on
push to main (one-time setup: Settings → Pages → Source = GitHub Actions).
- README.md: rewritten as the project front page — the thesis, the verified
results table, layout, quick start, how it stays honest, links to the
book. Replaces the stale Phase-0-era text.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
Human ruling on the needs-human-review allocator swap (+4.6%, CI
[+3.3%,+5.8%]): accepted as a real win for the measured comrak CLI
single-stream markdown-render workload, explicitly not a blanket
mimalloc mandate. Ledger row stays needs-human-review (append-only);
the ruling is documented in results/rulings/phase0-comrak-002.md and
reflected in the case study.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
Two defects found and fixed in a hostile review of the whole package:
1. The ROI report generator trusted the ledger verdict field and would
mint a clean $16,886/year report for phase2-comrak-010 — the
memory-leaking patch the audit overturned (immutably 'accepted' in the
append-only ledger, sanitizers_clean:false). report_row now exposes
sanitizers_clean and the report treats accepted-but-not-sanitizer-clean
as NOT shippable, printing a warning before any dollar figure. Legit
wins (sanitizers_clean:true) still render clean.
2. bench-runner service --rate 0 / --count 0 could panic via
from_secs_f64(1/0); run_session now guards finite/positive rate and
non-zero count/workers.
Attacks correctly refused (verified, no change): ledger UPDATE/DELETE,
propose_patch ../ + .git escapes, read_target_source traversal. All 10
accepted rows satisfy the green-gates + CI>=1.02 invariant. Findings
recorded in results/adversarial-review.md.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
phase4-cjson-service (ledger): the accepted batch win phase3-cjson-002
measured under a 150 rps coordinated-omission-correct replay (20 rounds,
40k requests, 0 drops). p50 latency +6.2%, 95% CI [+5.8%, +7.2%] —
accepted; p99 CI [0.07, 4.97] too noisy (single-worker loopback tail),
harness correctly rejected. ROI (500-core fleet, $0.04/core-hr): 27.5
cores / $9,621/year at the CI lower bound, report generated mechanically
(results/phase4/cjson-service-roi.md).
- results/phase4/: case study + README + the generated ROI report.
- CLAUDE.md: Phase 4 complete; Phases 0-4 all done.
All SPEC §5 exit criteria met across every phase; zero shipped false
accepts across the whole project.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
- bench-runner service mode (SPEC §3.1 mode c): a coordinated-omission-
correct, open-loop, fixed-rate load generator; latency measured from
the intended send time so queueing is counted, not hidden. Interleaved
A/B of two server binaries, exact percentiles (bounded sample, no HDR
bucketing error), bootstrap p50/p99 CIs via the existing ratio-CI
machine. service-calibrate: A/A false-positive + injected latency-
regression detection (calibration in progress).
- targets/cjson/service.c: minimal single-threaded HTTP server wrapping
the patched cJSON.c (trust-layer, outside the workspace allowlist);
env-toggled per-request busy-wait for the injection calibration only.
- report generator (crates/report bin): reads a ledger row +
optional service-latency JSON, renders throughput→cores→dollars and/or
latency percentiles, every figure with its CI, pricing source, and the
methodology printed inline (SPEC §9); flags any non-accepted row so a
rejected change never reads as a committed saving.
- ledger::report_row; justfile service/service-calibrate/report.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
phase3-cjson-002 (class 5+3): stack-buffer number parsing + strtod
round-trip replacing sscanf. +8.85% median, 95% CI [+7.5%, +10.0%] on
the C target, banked. 100% audit: byte-identical golden replay,
independent ASan+UBSan clean, 10k-input differential fuzz 0 divergences,
patch bounds/free-gating verified by eye.
- verdict: an infrastructure sanitizer-BUILD failure (missing runtime,
broken toolchain) now caps the accept at needs-human-review and logs
it, instead of crashing the verdict and losing a measured bench.
- results/phase3/: four public case studies (tokei class-5 wins; build
flags + allocators; the comrak caught false-accept; the cJSON
cross-language proof) + README. Compound-speedup claim stated as an
approximation, not a measured figure (CIs don't multiply).
- .gitignore: cjson build-artifact dirs.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
- coz wired (harness -DHOTPATH_COZ hook, scripts, just coz) + installed,
but the apt coz runtime aborts in its own init_coz/pthread_create
interposition on this glibc (known coz bug, reproduces on the pristine
binary). Documented in results/cjson/coz/README.md; callgrind cache-sim
is the working profiler.
- cjson no-op verdict smoke: full C pipeline flows (clang {out} baseline
rebuild, Unity/CTest + golden gates, interleaved bench) to a correct
null result (median 0.9995).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
~22% of Ir in glibc float serialization (__printf_fp, cJSON print_number)
and parsing (strtod/str_to_mpn, cJSON parse_number), plus parse_value/
parse_string and malloc/free churn. Ample headroom (SPEC §6).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
20/20 sessions: A/A false-positive rate 0.000 (<0.05 required),
injected-5%-slowdown detection 1.000 (>=0.95 required). The C workload
is a trusted measurement anchor; verdict benches on cjson are now valid.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
diff-test standalone (just gates / CI target-gates) passed the now-
templated build.binary into golden replay verbatim — literal {out}.
It now builds the current tree to targets/<t>/gates-build and tests
that, with a --binary override. CI's target-gates-no-net job gains
clang/cmake, fetches+gates the cjson C target (clang build -> Unity/
CTest -> golden replay) alongside tokei.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
20 audited attempts, 0 shipped false accepts, 3 verified auto-accepted
wins (tokei class-5: +10.7%, +3.7%, +9.9%; ~1.25x vs upstream with
Phase-0 wins). Independent 20-way adversarial re-audit (phase2-final-audit
workflow) confirmed every verdict follows its evidence, correct
equivalence tier, no harness-gaming, ledger append-only intact.
Closeout changes:
- case study: full 20-row attempt table, scoreboard, corrected
comrak-010 CI upper bound (+12.1%, was +12.2%), exit-criteria section
flipped to ALL MET, OS-isolation gap marked closed.
- CLAUDE.md Current-phase line: Phase 2 complete, next Phase 3.
- verdict: derive the ledger 'phase' column from the run_id namespace
(phaseN-*) instead of the hardcoded 1 the re-audit flagged; historical
rows are immutable and stay as written (run_id prefix is authoritative).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
The pipeline auto-accepted an LSan-flagged teardown patch (+10.7%
median) because ASan/LSan were per-attempt manual; the 100% audit
overturned it (black_box liveness ends at scope close, process::exit
runs after — 384-byte arena chunk unreachable at exit). verdict now
builds the patched tree with ASan+LSan and runs the pinned workload on
every would-be accept, capping flagged wins at needs-human-review per
SPEC §8. Shipped false accepts remain 0; the win was not banked; case
study carries the corrective record.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
comrak-010's patch added an unresolvable dependency version; the
candidate build failed before the verdict binary ran, no ledger row was
ever written, and the agent polled read_verdict into its turn cap. The
detached pipeline now traps any failure into a log marker, and
read_verdict reports status=failed with the log tail instead of
'running' forever. Attempt relaunched.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
Class 5 on tokei: forward-only all-whitespace break test replaces
trim().is_empty() (drops the redundant rposition scan per processed
byte). Median 1.0367, 95% CI [1.027, 1.052]. Equivalence eyeball done
(same AsciiExt::is_whitespace predicate, empty-window case identical);
diff-fuzz + ASan running before the win is counted.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
- propose_patch allows the /dev/null headers git emits for new/deleted
files (previously every new-file diff was refused as an absolute path).
- read_target_source serves HEAD (pristine) content instead of the
working tree, matching propose_patch's standalone-vs-pristine base so
post-proposal reads can't drift a later diff's context lines.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
A stale socket file survived a container idle window while the daemon
died; the [ -S ] check skipped the restart and the nested agent's MCP
server crashed at connect. The agent refused cleanly (no ledger row, no
fabrication). Liveness now means answering read_playbook, not existing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
benbrownandClaude Opus 4.8
copilot-swe-agent[bot]andGitHub