An adversarial audit found the enforcement story ahead of the code: gates
that were documented as hard gates were advisory, manual, or absent, and
several human-facing claims outran the machine record. This makes the
claims true.
High findings
- Differential fuzz was documentation, not enforcement: verdict hardcoded
fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case
studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real
pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and
it is inherently baseline-vs-candidate, so it runs on the accept path and
reports Skipped in the standalone `just gates` flow. New hard rule: a
machine `accepted` verdict is impossible unless the fuzz gate actually
passed — otherwise it caps at needs-human-review.
- The agent's system prompt claimed FP/concurrency/unsafe patches were
"routed to human review automatically". No such mechanism existed. Added
a conservative lexical risk classifier (crates/verdict/src/risk.rs) that
forces needs-human-review on any would-be accept carrying those signals;
fp-tolerance equivalence always signals.
- The local verdict path ran agent-influenced code unsandboxed. It now runs
under scripts/no-net.sh (network namespace) and fails closed. Upstream
test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when
present (scripts/pin-testsuite.sh generates them).
Also fixed, found while fixing the above
- The pipeline script's `set -e` was inert (brace group as the left operand
of `&&`), so a failing candidate build did not abort the run and verdict
could bench a stale binary. Now a scoped subshell whose status is branched
on at top level.
- A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path)
stranded RUNNING.lock and left read_verdict polling forever. Failure
accounting now lives outside the wrapper.
- harnessd's patch allowlist ignored git rename/copy headers and --no-prefix
diffs; the rename-escape case is now a committed regression test.
- The workspace did not compile on Windows (ungated Unix APIs). cfg-gated;
71 portable tests now run on both platforms.
- pin-check regenerated the corpus manifest instead of verifying it; it is
now verify-only, with a deliberate `just pin-corpus` to re-pin.
- Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics
(compare --perf-stat), never part of the accept decision.
Docs now match the code, including what is NOT yet enforced: no target ships
a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not
wired to the fp-tolerance gate — each is stated as mechanized-but-inert
rather than done. Corrected the stale "33 ledger rows" (34), the playbook's
class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation
claim (five accepted rows carry sanitizers_clean=false), the unsupported
"40,000 requests" figure, and the blanket "historical rows record
fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332).
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The parser/service targets only ever needed byte-identical golden replay.
The kernel lane is the case that breaks it: a matmul optimization
(transpose-B for cache locality + eight-accumulator ILP) that REORDERS the
floating-point reduction, so its low bits differ by construction —
byte-identical is the wrong gate.
Adds:
- EquivalencePolicy::compare + Divergence (diff-test): byte-identical is
exact; fp-tolerance tokenizes and compares numeric tokens within
abs + rel*|baseline|, with explicit NaN handling. Non-numeric tokens and
token-count mismatches still fail exactly.
- fp-compare bin: applies a target's equivalence.toml to two output files.
- targets/matmul/kernel.c: matmul_ref (single-accumulator oracle) vs
matmul_opt (reordered, faster), with emit/check/run/bench modes.
- targets/matmul/equivalence.toml: fp-tolerance abs 1e-4 + rel 1e-3.
- scripts/kernel-lane-demo.sh + results/phase5/kernel-lane.txt: byte-identical
FAILs, fp-tolerance is EQUIVALENT, a +0.5 perturbation is still REJECTED,
speedup 3.23× measured with the same interleaved A/B + bootstrap CI.
Ledger row phase5-matmul-opt = needs-human-review: using the FP-tolerance
tier at all is a human-review signal (SPEC §8), same posture as mimalloc.
The trust machinery is unchanged; only the timer and the gate differ, which
is exactly the shape a future GPU kernel would slot into.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
A stale socket file survived a container idle window while the daemon
died; the [ -S ] check skipped the restart and the nested agent's MCP
server crashed at connect. The agent refused cleanly (no ledger row, no
fabrication). Liveness now means answering read_playbook, not existing.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
Ledger row: class 5 (algorithmic) on tokei, first-byte gate table for
the per-byte token matchers in perform_multi_line_analysis.
Speedup median 1.107, 95% CI [1.084, 1.131] vs the banked-wins baseline,
workload: 221 MB / 4360 files, single rayon thread, pinned core.
Manual audit gates (pipeline covers tests+golden+bench only):
- Differential fuzz, 10k mutated inputs across 200 batched runs:
0 divergences after canonicalizing tokei's benign parallel-walker
output ordering (present in pristine baseline; counts identical).
Fuzzer added as scripts/diff-fuzz-tokei.py.
- ASan run recorded in the case study (next commit).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
Mechanically re-checks a ledger row against the accept rule (CI lower
bound vs threshold, gate greenness, patch path allowlist) and prints
the full evidence for the human part of the 100% Phase 2 audit.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
harnessd gains --socket mode (Unix socket, trusted uid, mutating ops
serialized); the harness client speaks socket when
HOTPATH_HARNESS_SOCKET is set. scripts/agent-isolated.sh launches an
attempt with the agent tree confined — mountns mode (repo bind-mounted
read-only, CAP_SYS_ADMIN/CAP_SYS_PTRACE dropped; no credential copying)
or user mode (unprivileged hpagent, bench-metal shape).
scripts/isolation-check.sh + just isolation-check verify the boundary
from the agent's side: 19 checks, both modes, all passing — repo
writes, forged ledger INSERTs, trust-binary replacement, rw remounts,
and userns re-entry all refused; the harnessd socket door works.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01EcKPWQ3MQD4NdEHwar96u2
benbrownandClaude Opus 4.8