mirror of https://github.com/bdbrown4/peltier.io.git synced 2026-11-07 10:58:28 +00:00

Files

T

benbrownandClaude Opus 4.8 899d55f4b6 gates: close the audit findings — enforce the gates the docs claimed

An adversarial audit found the enforcement story ahead of the code: gates
that were documented as hard gates were advisory, manual, or absent, and
several human-facing claims outran the machine record. This makes the
claims true.

High findings

- Differential fuzz was documentation, not enforcement: verdict hardcoded
  fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case
  studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real
  pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and
  it is inherently baseline-vs-candidate, so it runs on the accept path and
  reports Skipped in the standalone `just gates` flow. New hard rule: a
  machine `accepted` verdict is impossible unless the fuzz gate actually
  passed — otherwise it caps at needs-human-review.
- The agent's system prompt claimed FP/concurrency/unsafe patches were
  "routed to human review automatically". No such mechanism existed. Added
  a conservative lexical risk classifier (crates/verdict/src/risk.rs) that
  forces needs-human-review on any would-be accept carrying those signals;
  fp-tolerance equivalence always signals.
- The local verdict path ran agent-influenced code unsandboxed. It now runs
  under scripts/no-net.sh (network namespace) and fails closed. Upstream
  test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when
  present (scripts/pin-testsuite.sh generates them).

Also fixed, found while fixing the above

- The pipeline script's `set -e` was inert (brace group as the left operand
  of `&&`), so a failing candidate build did not abort the run and verdict
  could bench a stale binary. Now a scoped subshell whose status is branched
  on at top level.
- A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path)
  stranded RUNNING.lock and left read_verdict polling forever. Failure
  accounting now lives outside the wrapper.
- harnessd's patch allowlist ignored git rename/copy headers and --no-prefix
  diffs; the rename-escape case is now a committed regression test.
- The workspace did not compile on Windows (ungated Unix APIs). cfg-gated;
  71 portable tests now run on both platforms.
- pin-check regenerated the corpus manifest instead of verifying it; it is
  now verify-only, with a deliberate `just pin-corpus` to re-pin.
- Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics
  (compare --perf-stat), never part of the accept decision.

Docs now match the code, including what is NOT yet enforced: no target ships
a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not
wired to the fp-tolerance gate — each is stated as mechanized-but-inert
rather than done. Corrected the stale "33 ledger rows" (34), the playbook's
class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation
claim (five accepted rows carry sanitizers_clean=false), the unsupported
"40,000 requests" figure, and the blanket "historical rows record
fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-07-14 13:10:14 -04:00

3.8 KiB

Raw Blame History

Case study 1 — tokei: compounding algorithmic wins (Rust, class 5)

Target: tokei v14.0.0 — a source-code line counter. Workload: 221 MB / 4360 files (8 replicas of the Pro Git book + comrak's own tree), single rayon thread, CPU pinned. Result: three independent, CI-verified class-5 wins on the same hot function — each proposed by the unattended agent and 100%-audited. Multiplied together they are ~1.26× over the Phase-0 baseline (1.107 × 1.037 × 1.099), but that product is an approximation, not a measured figure: each speedup carries its own bootstrap CI against its own baseline, and confidence intervals do not compose through multiplication. The trustworthy claims are the three individual increments below.

The hot path

tokei spends the overwhelming majority of its instructions in one function — LanguageType::parse_lines and the per-byte loop inside SyntaxCounter::perform_multi_line_analysis. The cache-sim profile put it at 55.9% of instructions, 62% of all branches, 44% of branch mispredicts on the pinned corpus: for every byte of every line, the counter tested a stack of quote/comment token matchers.

The three wins

run	mechanism	speedup (median)	95% CI
`phase0-tokei-002` (class 3)	read path: `fs::read` pre-sized, BOM-only decode	1.099	[1.080, 1.127]
`phase2-tokei-001` (class 5)	256-entry first-byte gate table for the token matchers	1.107	[1.084, 1.131]
`phase2-tokei-003` (class 5)	forward-only whitespace break (drop the `rposition` rescan)	1.037	[1.027, 1.052]
`phase2-tokei-008` (class 5)	streaming leftmost `important_syntax` scan (one vectorized pass, not per-line restart)	1.099	[1.073, 1.112]

Each is a redundant-work elimination: the first-byte gate replaces five per-byte matcher loops with one table load; the whitespace-break change drops a full backward scan of the file tail per byte; the streaming scan replaces a per-line aho-corasick restart (which kept memmem's SIMD prefilter in its short-haystack slow path) with a single leftmost search advanced by integer comparisons.

Why the numbers are trustworthy

Byte-identical golden replay on the pinned corpus for every win — a code counter that miscounts one line fails the gate.
10,000-input differential fuzz (scripts/diff-fuzz-tokei.py), pristine vs candidate, 0 divergences — after canonicalizing tokei's benign parallel-walker output ordering (a real finding: raw-byte differential comparison over-counts when the target is legitimately nondeterministic in presentation).
ASan/LSan clean on the patched tree over the full corpus.
Interleaved A/B vs a pristine-rebuilt baseline, bootstrap 95% CI lower bound ≥ the 2% bar. phase2-tokei-008 was the first accept through the machine-enforced sanitizer gate (see case study 3).

What it demonstrates

The richest wins on a mature Rust binary were not exotic — they were "this loop does work it doesn't need to, on every byte." The agent found each by reading the cache-sim profile, checking the ledger to avoid re-attempting a class on the same hotspot, and proposing the cheapest untried mechanism. Three separate hypotheses, three separate verified increments, all on the same 40%-of-runtime function — and one honest rejection in between (phase2-tokei-009, a real +5.7% median whose noisy CI lower bound of +0.2% could not clear the bar, so it does not count).

Note (2026-07-13). The 10,000-input differential fuzz cited above ran out-of-band (scripts/diff-fuzz-tokei.py, per-attempt), not inside the verdict pipeline — so these ledger rows record fuzz_iters=0 in their machine record. The fuzz gate is now pipeline-integrated; rows written after this date record the iteration count actually executed.

3.8 KiB Raw Blame History Unescape Escape