mirror of
https://github.com/bdbrown4/peltier.io.git
synced 2026-11-07 10:58:28 +00:00
benbrownandClaude Opus 4.8
899d55f4b6
An adversarial audit found the enforcement story ahead of the code: gates that were documented as hard gates were advisory, manual, or absent, and several human-facing claims outran the machine record. This makes the claims true. High findings - Differential fuzz was documentation, not enforcement: verdict hardcoded fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and it is inherently baseline-vs-candidate, so it runs on the accept path and reports Skipped in the standalone `just gates` flow. New hard rule: a machine `accepted` verdict is impossible unless the fuzz gate actually passed — otherwise it caps at needs-human-review. - The agent's system prompt claimed FP/concurrency/unsafe patches were "routed to human review automatically". No such mechanism existed. Added a conservative lexical risk classifier (crates/verdict/src/risk.rs) that forces needs-human-review on any would-be accept carrying those signals; fp-tolerance equivalence always signals. - The local verdict path ran agent-influenced code unsandboxed. It now runs under scripts/no-net.sh (network namespace) and fails closed. Upstream test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when present (scripts/pin-testsuite.sh generates them). Also fixed, found while fixing the above - The pipeline script's `set -e` was inert (brace group as the left operand of `&&`), so a failing candidate build did not abort the run and verdict could bench a stale binary. Now a scoped subshell whose status is branched on at top level. - A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path) stranded RUNNING.lock and left read_verdict polling forever. Failure accounting now lives outside the wrapper. - harnessd's patch allowlist ignored git rename/copy headers and --no-prefix diffs; the rename-escape case is now a committed regression test. - The workspace did not compile on Windows (ungated Unix APIs). cfg-gated; 71 portable tests now run on both platforms. - pin-check regenerated the corpus manifest instead of verifying it; it is now verify-only, with a deliberate `just pin-corpus` to re-pin. - Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics (compare --perf-stat), never part of the accept decision. Docs now match the code, including what is NOT yet enforced: no target ships a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not wired to the fp-tolerance gate — each is stated as mechanized-but-inert rather than done. Corrected the stale "33 ledger rows" (34), the playbook's class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation claim (five accepted rows carry sanitizers_clean=false), the unsupported "40,000 requests" figure, and the blanket "historical rows record fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
98 lines
4.1 KiB
Makefile
98 lines
4.1 KiB
Makefile
# hotpath task runner. `just --list` for a summary.
|
|
|
|
# Build the trust-layer workspace
|
|
build:
|
|
cargo build --workspace
|
|
|
|
# Run all trust-layer unit tests
|
|
test:
|
|
cargo test --workspace
|
|
|
|
# Lint: warnings are errors in the trust layer
|
|
lint:
|
|
cargo clippy --workspace --all-targets -- -D warnings
|
|
cargo fmt --all --check
|
|
|
|
# A/A self-test of the bench harness against a sample command.
|
|
# Must print a null verdict; an "accept" is a calibration failure.
|
|
aa cmd="sleep 0.05":
|
|
cargo run -p bench-runner -- --config config/accept.toml aa --cmd "{{cmd}}"
|
|
|
|
# Interleaved A/B comparison of two shell commands
|
|
compare baseline candidate:
|
|
cargo run -p bench-runner -- --config config/accept.toml compare \
|
|
--baseline "{{baseline}}" --candidate "{{candidate}}"
|
|
|
|
# Verify a target corpus against its pinned manifest (never rewrites it)
|
|
pin-check target:
|
|
cd corpora/{{target}} && sh gen-corpus.sh --check
|
|
|
|
# Deliberately re-pin a target corpus manifest — a trust-layer change;
|
|
# review and commit the new MANIFEST.sha256 with justification
|
|
pin-corpus target:
|
|
@echo "WARNING: rewriting corpora/{{target}}/MANIFEST.sha256 — deliberate human action (corpora/README.md)"
|
|
cd corpora/{{target}} && sh gen-corpus.sh --pin
|
|
|
|
# Automated calibration: N A/A sessions (<5% false-positive required) +
|
|
# N injected-slowdown sessions (>=95% detection required), JSON evidence
|
|
calibrate cmd out sessions="20":
|
|
cargo run -p bench-runner -- --config config/accept.toml calibrate \
|
|
--cmd "{{cmd}}" --sessions {{sessions}} --out "{{out}}"
|
|
|
|
# Equivalence gates for a target: corpus + test-suite pins (refuse on
|
|
# mismatch) -> upstream tests -> golden replay. Differential fuzz needs a
|
|
# pristine baseline to differ against, so it reports Skipped here and runs
|
|
# for real on the accept path; sanitizers likewise run in verdict.
|
|
gates target:
|
|
cargo run -p diff-test -- {{target}}
|
|
|
|
# One-shot attempt verdict: gates on candidate -> interleaved A/B vs a
|
|
# pristine-rebuilt baseline -> ledger row. Extra flags pass through
|
|
# (--patch-file, --needs-human-review, --baseline-bin).
|
|
verdict target candidate run_id class hypothesis hotspot *flags:
|
|
cargo run -p verdict -- {{target}} --rebuild-baseline \
|
|
--candidate-bin "{{candidate}}" --run-id "{{run_id}}" \
|
|
--playbook-class {{class}} --hypothesis "{{hypothesis}}" \
|
|
--hotspot "{{hotspot}}" {{flags}}
|
|
|
|
# One unattended agent attempt behind the SPEC §10 OS boundary:
|
|
# harnessd (trusted uid) on a Unix socket, agent loop as user hpagent.
|
|
agent-attempt target run_id max_turns="40":
|
|
sh scripts/agent-isolated.sh {{target}} {{run_id}} {{max_turns}}
|
|
|
|
# Verify the OS boundary from the agent user's side (negative + positive)
|
|
isolation-check:
|
|
sh scripts/isolation-check.sh
|
|
|
|
# Install the coz causal profiler (apt, else source build)
|
|
install-coz:
|
|
sh scripts/install-coz.sh
|
|
|
|
# Causal profile of a C/C++ target: where a speedup would raise throughput
|
|
coz target iters="200":
|
|
sh scripts/coz-profile.sh {{target}} {{iters}}
|
|
python3 scripts/coz-summary.py results/{{target}}/coz/profile.coz
|
|
|
|
# Service mode (Phase 4): interleaved A/B latency bench of two server
|
|
# binaries under coordinated-omission-correct open-loop load. p50/p99 CIs.
|
|
service baseline candidate doc *flags:
|
|
cargo run -p bench-runner -- --config config/accept.toml service \
|
|
--baseline-bin "{{baseline}}" --candidate-bin "{{candidate}}" \
|
|
--doc "{{doc}}" --pin "taskset -c 2" {{flags}}
|
|
|
|
# Service-mode calibration: A/A false-positive + injected latency-regression
|
|
# detection (SPEC §3.1); writes JSON evidence.
|
|
service-calibrate server doc out *flags:
|
|
cargo run -p bench-runner -- --config config/accept.toml service-calibrate \
|
|
--server-bin "{{server}}" --doc "{{doc}}" --pin "taskset -c 2" \
|
|
--out "{{out}}" {{flags}}
|
|
|
|
# Mechanical ROI report from a ledger row (SPEC §9): throughput→cores→
|
|
# dollars and/or latency percentiles, CIs + methodology printed inline.
|
|
report run_id *flags:
|
|
cargo run -p report -- --run-id "{{run_id}}" {{flags}}
|
|
|
|
# Learned playbook-class ranking from the ledger (SPEC §13 research fork)
|
|
policy *flags:
|
|
cargo run -p policy -- {{flags}}
|