mirror of https://github.com/bdbrown4/peltier.io.git synced 2026-11-07 10:58:28 +00:00

Files

T

benbrownandClaude Opus 4.8 899d55f4b6 gates: close the audit findings — enforce the gates the docs claimed

An adversarial audit found the enforcement story ahead of the code: gates
that were documented as hard gates were advisory, manual, or absent, and
several human-facing claims outran the machine record. This makes the
claims true.

High findings

- Differential fuzz was documentation, not enforcement: verdict hardcoded
  fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case
  studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real
  pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and
  it is inherently baseline-vs-candidate, so it runs on the accept path and
  reports Skipped in the standalone `just gates` flow. New hard rule: a
  machine `accepted` verdict is impossible unless the fuzz gate actually
  passed — otherwise it caps at needs-human-review.
- The agent's system prompt claimed FP/concurrency/unsafe patches were
  "routed to human review automatically". No such mechanism existed. Added
  a conservative lexical risk classifier (crates/verdict/src/risk.rs) that
  forces needs-human-review on any would-be accept carrying those signals;
  fp-tolerance equivalence always signals.
- The local verdict path ran agent-influenced code unsandboxed. It now runs
  under scripts/no-net.sh (network namespace) and fails closed. Upstream
  test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when
  present (scripts/pin-testsuite.sh generates them).

Also fixed, found while fixing the above

- The pipeline script's `set -e` was inert (brace group as the left operand
  of `&&`), so a failing candidate build did not abort the run and verdict
  could bench a stale binary. Now a scoped subshell whose status is branched
  on at top level.
- A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path)
  stranded RUNNING.lock and left read_verdict polling forever. Failure
  accounting now lives outside the wrapper.
- harnessd's patch allowlist ignored git rename/copy headers and --no-prefix
  diffs; the rename-escape case is now a committed regression test.
- The workspace did not compile on Windows (ungated Unix APIs). cfg-gated;
  71 portable tests now run on both platforms.
- pin-check regenerated the corpus manifest instead of verifying it; it is
  now verify-only, with a deliberate `just pin-corpus` to re-pin.
- Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics
  (compare --perf-stat), never part of the accept decision.

Docs now match the code, including what is NOT yet enforced: no target ships
a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not
wired to the fp-tolerance gate — each is stated as mechanized-but-inert
rather than done. Corrected the stale "33 ledger rows" (34), the playbook's
class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation
claim (five accepted rows carry sanitizers_clean=false), the unsupported
"40,000 requests" figure, and the blanket "historical rows record
fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

2026-07-14 13:10:14 -04:00

4.3 KiB

Raw Blame History

Isolation & anti-reward-hacking

The guarantees only matter if the proposer cannot cheat. Peltier's defense is structural: the agent is denied the capabilities it would need to manufacture a false win, at the OS level, not by prompt.

What holds even if the proposer is adversarial

Writes go through one door. propose_patch is the only harness- mediated write path: a diff is path-allowlisted (relative paths only, no .., no .git, nothing outside targets/<name>/workspace) and then git apply-ed rooted in that workspace. A patch aimed at config/ or crates/ is refused before any apply. Verified against a malicious accept-threshold diff.
The ledger is append-only. UPDATE and DELETE are refused by SQLite triggers; accepted rows are written only by the verdict binary after the gated pipeline. The agent cannot forge a passing verdict through the harness.
The baseline is unbypassable. It is rebuilt from a pristine checkout; the agent's workspace never becomes the comparison.

OS-level isolation (SPEC §10)

The SDK's tool allow/deny list is defense-in-depth, not the boundary — scripts/agent-isolated.sh provides the real one, verified by just isolation-check (19 checks, both modes):

mountns mode — the whole agent process tree runs in a private mount namespace with the repo bind-mounted read-only and CAP_SYS_ADMIN/ CAP_SYS_PTRACE dropped from the bounding set. Repo writes fail with EROFS; a forged INSERT, a direct verdict invocation, a mount -o remount,rw, and even userns re-entry to undo the mount are all refused.
user mode — the agent runs as an unprivileged user whose only write path into the repo is the harnessd Unix socket; the trust layer is owned by another uid. This is the production shape (docs/infra.md).

Network isolation on the verdict path

Target code should not be able to phone home during a measurement. What is actually enforced, stated honestly:

The agent path is wrapped. When harnessd launches the verdict pipeline, it execs through scripts/no-net.sh, which runs it inside an unprivileged network namespace (unshare --net --map-current-user). HOTPATH_VERDICT_WRAPPER overrides the wrapper.
It fails closed. Where namespaces are unavailable the wrapper exits 97 without running the pipeline — it does not degrade silently. The one bypass, HOTPATH_ALLOW_UNISOLATED=1, runs the pipeline with full network access; it warns loudly on stderr and the resulting ledger row records the isolation note no-net.sh (HOTPATH_ALLOW_UNISOLATED=1 — network NOT isolated). A run cannot claim an isolation it did not have.
In CI, the bench workload runs under docker run --network=none.

Two limits, stated plainly rather than buried:

Only the harnessd (agent) path is wrapped. A human who invokes just verdict directly runs it unwrapped on the host; those rows are stamped isolation: "unwrapped-host" in the ledger. The wrapper is a property of the agent pipeline, not of the verdict binary.
Full-container isolation is still an open gap. The namespace isolates the network — not the filesystem, not the syscall surface. The fully containerized, seccomp-restricted bench container of SPEC §10 is not built. The agent-side OS boundary above is a separate mechanism and is shipped.

The two times the pipeline was wrong

Anti-reward-hacking is not "it never fails" — it is "when it fails, the audit catches it and the system gets harder." That happened twice, and both are documented, not hidden:

A leaking teardown patch the bench loved was auto-accepted and then overturned by the 100% human audit's LeakSanitizer run. The fix made the sanitizer gate machine-enforced on every accept.
During the adversarial review, the ROI report generator was found to mint a clean dollar figure for that same overturned row (it trusted the immutable ledger's historical accepted verdict). The fix made the report treat an accepted-but-not- sanitizer-clean row as not shippable.

Both defects were found by attacking the system's own output, fixed, and turned into permanent gates. Zero false accepts shipped.

4.3 KiB Raw Blame History