benbrown/peltier.io
2
1
Fork 0
mirror of https://github.com/bdbrown4/peltier.io.git synced 2026-11-07 10:58:28 +00:00
Code Issues Packages Projects Releases Wiki Activity

gates: close the audit findings — enforce the gates the docs claimed

Browse Source 
An adversarial audit found the enforcement story ahead of the code: gates
that were documented as hard gates were advisory, manual, or absent, and
several human-facing claims outran the machine record. This makes the
claims true.

High findings

- Differential fuzz was documentation, not enforcement: verdict hardcoded
  fuzz_iters=0 and diff-test hardcoded the gate to Skipped, while the case
  studies claimed 10k-iteration fuzz on the same wins. Fuzz is now a real
  pipeline gate (strict FUZZ-RESULT contract, executed count recorded), and
  it is inherently baseline-vs-candidate, so it runs on the accept path and
  reports Skipped in the standalone `just gates` flow. New hard rule: a
  machine `accepted` verdict is impossible unless the fuzz gate actually
  passed — otherwise it caps at needs-human-review.
- The agent's system prompt claimed FP/concurrency/unsafe patches were
  "routed to human review automatically". No such mechanism existed. Added
  a conservative lexical risk classifier (crates/verdict/src/risk.rs) that
  forces needs-human-review on any would-be accept carrying those signals;
  fp-tolerance equivalence always signals.
- The local verdict path ran agent-influenced code unsandboxed. It now runs
  under scripts/no-net.sh (network namespace) and fails closed. Upstream
  test suites are hash-verified against corpora/<t>/TESTSUITE.sha256 when
  present (scripts/pin-testsuite.sh generates them).

Also fixed, found while fixing the above

- The pipeline script's `set -e` was inert (brace group as the left operand
  of `&&`), so a failing candidate build did not abort the run and verdict
  could bench a stale binary. Now a scoped subshell whose status is branched
  on at top level.
- A wrapper that exits before exec'ing (no-net.sh refusing, bad wrapper path)
  stranded RUNNING.lock and left read_verdict polling forever. Failure
  accounting now lives outside the wrapper.
- harnessd's patch allowlist ignored git rename/copy headers and --no-prefix
  diffs; the rename-escape case is now a committed regression test.
- The workspace did not compile on Windows (ungated Unix APIs). cfg-gated;
  71 portable tests now run on both platforms.
- pin-check regenerated the corpus manifest instead of verifying it; it is
  now verify-only, with a deliberate `just pin-corpus` to re-pin.
- Max RSS is captured per run on Unix; PMU counters are opt-in diagnostics
  (compare --perf-stat), never part of the accept decision.

Docs now match the code, including what is NOT yet enforced: no target ships
a TESTSUITE.sha256 pin, no target declares [build].tsan, and matmul is not
wired to the fp-tolerance gate — each is stated as mechanized-but-inert
rather than done. Corrected the stale "33 ledger rows" (34), the playbook's
class-3 cJSON win (rejected), adversarial-review.md's sanitizer-separation
claim (five accepted rows carry sanitizers_clean=false), the unsupported
"40,000 requests" figure, and the blanket "historical rows record
fuzz_iters=0" (8 of 10; phase0-tokei-002/-003 record 4,332).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This commit is contained in:
benbrown
2026-07-14 13:10:14 -04:00
co-authored by Claude Opus 4.8
parent 21352efe6c
commit 899d55f4b6
57 changed files with 3308 additions and 692 deletions
Download Patch File Download Diff File Expand all files Collapse all files
crates/harnessd/src/main.rs
+22 -431
View File
@@ -1,438 +1,29 @@
//! harnessd — the only door between the agent and the trust layer
//! (SPEC §3.5, §10). One JSON request per stdin line, one JSON response
//! per stdout line. The agent gets exactly six operations; everything
//! else (shell, trust-layer writes, patches outside the target
//! workspace) simply has no code path here. Phase 1 caveat, recorded:
//! same-uid filesystem read-only enforcement still requires the
//! separate-user/container setup (Phase 1 infra gap).
//! per stdout line. The agent gets exactly seven operations (the six
//! synchronous ops plus the async read_verdict poll); everything else
//! (shell, trust-layer writes, patches outside the target workspace)
//! simply has no code path here. Phase 1 caveat, recorded: same-uid
//! filesystem read-only enforcement still requires the separate-user/
//! container setup (Phase 1 infra gap).
//!
//! The daemon itself is POSIX-only (Unix sockets, setsid); the diff
//! allowlist is a portable module so its escape-regression tests run on
//! every development platform.
use anyhow::{anyhow, ensure, Result};
use serde_json::{json, Value};
use sha2::{Digest, Sha256};
use std::io::{BufRead, Write};
use std::os::unix::net::UnixListener;
use std::path::{Component, Path};
use std::process::Command;
use std::sync::Mutex;
// On non-Unix only the allowlist tests compile against this module.
#[cfg_attr(not(unix), allow(dead_code))]
mod allowlist;
#[cfg(unix)]
mod daemon;
/// Serializes mutating ops (git apply, pending writes) across socket
/// connections; reads don't need it but the cost is nil at one agent.
static HANDLE_LOCK: Mutex<()> = Mutex::new(());
fn main() -> Result<()> {
let root = std::env::current_dir()?.canonicalize()?;
let args: Vec<String> = std::env::args().collect();
if let Some(i) = args.iter().position(|a| a == "--socket") {
let path = args
.get(i + 1)
.ok_or_else(|| anyhow!("--socket requires a path"))?;
return serve_socket(&root, Path::new(path));
}
let stdin = std::io::stdin();
let mut stdout = std::io::stdout().lock();
for line in stdin.lock().lines() {
let line = line?;
if line.trim().is_empty() {
continue;
}
let resp = respond(&root, &line);
writeln!(stdout, "{resp}")?;
stdout.flush()?;
}
Ok(())
#[cfg(unix)]
fn main() -> anyhow::Result<()> {
daemon::run()
}
/// SPEC §10 socket mode: harnessd runs as the trusted uid and listens on a
/// Unix socket; the agent process runs as an unprivileged user whose only
/// write path into the repo is this daemon. Socket permissions (owner/group/
/// mode) are set by the supervisor script that starts us.
fn serve_socket(root: &Path, sock: &Path) -> Result<()> {
if let Some(dir) = sock.parent() {
std::fs::create_dir_all(dir)?;
}
let _ = std::fs::remove_file(sock);
let listener = UnixListener::bind(sock)?;
eprintln!("harnessd listening on {}", sock.display());
for conn in listener.incoming() {
let stream = match conn {
Ok(s) => s,
Err(e) => {
eprintln!("accept error: {e}");
continue;
}
};
let root = root.to_path_buf();
std::thread::spawn(move || {
let reader = match stream.try_clone() {
Ok(s) => std::io::BufReader::new(s),
Err(_) => return,
};
let mut writer = stream;
for line in reader.lines() {
let Ok(line) = line else { break };
if line.trim().is_empty() {
continue;
}
let resp = respond(&root, &line);
if writeln!(writer, "{resp}").is_err() {
break;
}
}
});
}
Ok(())
}
fn respond(root: &Path, line: &str) -> Value {
// read_verdict long-polls the ledger (read-only) and must not hold
// the lock that serializes mutating ops while it sleeps.
let is_read_verdict = serde_json::from_str::<Value>(line)
.ok()
.and_then(|v| {
v.get("op")
.and_then(Value::as_str)
.map(|o| o == "read_verdict")
})
.unwrap_or(false);
let _guard = if is_read_verdict {
None
} else {
Some(HANDLE_LOCK.lock().unwrap_or_else(|p| p.into_inner()))
};
match handle(root, line) {
Ok(v) => json!({"ok": true, "result": v}),
Err(e) => json!({"ok": false, "error": e.to_string()}),
}
}
/// Single-quote a string for safe embedding in a generated shell script.
fn shq(s: &str) -> String {
format!("'{}'", s.replace('\'', "'\\''"))
}
fn field<'a>(req: &'a Value, key: &str) -> Result<&'a str> {
req.get(key)
.and_then(Value::as_str)
.ok_or_else(|| anyhow!("missing string field: {key}"))
}
/// A target name must be a plain directory name — no separators, no dots.
fn check_target(name: &str) -> Result<()> {
ensure!(
!name.is_empty()
&& name
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_'),
"invalid target name"
);
Ok(())
}
/// Relative paths only; no parent components, no .git, no absolutes.
fn check_rel_path(p: &str) -> Result<()> {
let path = Path::new(p);
ensure!(!path.is_absolute(), "absolute paths forbidden");
for c in path.components() {
match c {
Component::Normal(seg) => ensure!(seg != ".git", ".git access forbidden"),
Component::CurDir => {}
_ => anyhow::bail!("path escapes workspace: {p}"),
}
}
Ok(())
}
fn handle(root: &Path, line: &str) -> Result<Value> {
let req: Value = serde_json::from_str(line)?;
match field(&req, "op")? {
"read_profile" => {
let t = field(&req, "target")?;
check_target(t)?;
let text = std::fs::read_to_string(root.join(format!("results/{t}/hotspots.txt")))?;
Ok(json!({"hotspots_text": text}))
}
"read_ledger" => {
let t = field(&req, "target")?;
check_target(t)?;
// Read-only surface over the append-only DB.
let ledger = ledger::Ledger::open(&root.join("results/ledger.sqlite"))?;
let classes = ledger.attempted_classes(t)?;
let history = ledger.attempt_history(t)?;
Ok(json!({
"attempted_playbook_classes": classes,
"attempts": history,
"total_attempts": ledger.count()?,
"note": "a class may be re-entered with a materially NEW hypothesis; \
never duplicate a (hotspot, class, hypothesis) that has a verdict",
}))
}
"read_playbook" => {
let class = match req.get("class") {
Some(Value::Number(n)) => n.as_u64().ok_or_else(|| anyhow!("bad class"))? as u8,
Some(Value::String(s)) => s.parse()?,
_ => anyhow::bail!("missing class"),
};
ensure!((1..=7).contains(&class), "playbook class must be 1-7");
let dir = root.join("playbook");
let entry = std::fs::read_dir(&dir)?
.filter_map(|e| e.ok())
.find(|e| {
e.file_name()
.to_string_lossy()
.starts_with(&format!("{class:02}-"))
})
.ok_or_else(|| anyhow!("no playbook file for class {class}"))?;
Ok(json!({"markdown": std::fs::read_to_string(entry.path())?}))
}
"read_target_source" => {
let t = field(&req, "target")?;
let p = field(&req, "path")?;
check_target(t)?;
check_rel_path(p)?;
// Serve the PRISTINE (HEAD) content, not the working tree:
// propose_patch applies each proposal against a reset
// workspace, so reads must show the same base the diff needs
// — otherwise reads after a proposal show the patched tree
// and the next diff's context lines drift (phase2-comrak-005
// burned turns rediscovering this).
let ws = root.join(format!("targets/{t}/workspace"));
let shown = Command::new("git")
.args(["-C", ws.to_str().unwrap(), "show", &format!("HEAD:{p}")])
.output()?;
ensure!(
shown.status.success(),
"cannot read {p} at HEAD: {}",
String::from_utf8_lossy(&shown.stderr).trim()
);
let text = String::from_utf8_lossy(&shown.stdout).into_owned();
// Optional line window so large source files (100KB+) can be read in
// chunks instead of overflowing one response — otherwise the agent
// is tempted to reach for a shell it must not have.
let lines: Vec<&str> = text.lines().collect();
let total = lines.len();
let offset = req.get("offset").and_then(Value::as_u64).unwrap_or(0) as usize;
let limit = req
.get("limit")
.and_then(Value::as_u64)
.map(|v| v as usize)
.unwrap_or(usize::MAX);
let end = offset.saturating_add(limit).min(total);
let start = offset.min(total);
let slice = lines[start..end]
.iter()
.enumerate()
.map(|(i, l)| format!("{}\t{}", start + i + 1, l))
.collect::<Vec<_>>()
.join("\n");
Ok(json!({
"content": slice,
"total_lines": total,
"returned_lines": [start + 1, end],
"truncated": end < total,
}))
}
"propose_patch" => {
let t = field(&req, "target")?;
let diff = field(&req, "diff")?;
let hypothesis = field(&req, "hypothesis")?;
check_target(t)?;
// git rejects a diff without a final newline with an opaque
// "corrupt patch" — say it plainly (phase2-comrak-004 burned
// several turns rediscovering this).
ensure!(
diff.ends_with('\n'),
"diff must end with a trailing newline (git would report 'corrupt patch' at EOF)"
);
// Allowlist: every path named by the diff must be a safe
// relative path (the git -C below roots them in the target
// workspace; nothing outside it is reachable). `/dev/null`
// is the one absolute path git legitimately emits, for
// new/deleted files.
for l in diff.lines() {
if let Some(p) = l
.strip_prefix("--- a/")
.or_else(|| l.strip_prefix("+++ b/"))
{
check_rel_path(p.trim())?;
}
ensure!(
(!l.starts_with("--- /") && !l.starts_with("+++ /"))
|| l == "--- /dev/null"
|| l == "+++ /dev/null",
"absolute diff paths forbidden (only /dev/null is allowed, for new/deleted files)"
);
}
let ws = root.join(format!("targets/{t}/workspace"));
// Every proposal stands alone: reset the workspace to HEAD
// first so probe/iteration patches cannot accumulate into a
// tested tree the ledger's patch field doesn't describe
// (phase2-comrak-004 audit finding).
ensure!(
Command::new("git")
.args(["-C", ws.to_str().unwrap(), "checkout", "--", "."])
.status()?
.success(),
"workspace reset failed"
);
ensure!(
Command::new("git")
.args(["-C", ws.to_str().unwrap(), "clean", "-fdq"])
.status()?
.success(),
"workspace clean failed"
);
let run_git_apply = |check: bool| -> Result<std::process::Output> {
let mut args = vec!["-C", ws.to_str().unwrap(), "apply"];
if check {
args.push("--check");
}
args.push("-");
let mut child = Command::new("git")
.args(&args)
.stdin(std::process::Stdio::piped())
.stderr(std::process::Stdio::piped())
.spawn()?;
child.stdin.take().unwrap().write_all(diff.as_bytes())?;
Ok(child.wait_with_output()?)
};
let checked = run_git_apply(true)?;
ensure!(
checked.status.success(),
"diff does not apply cleanly against the pristine workspace \
(proposals are standalone; prior proposals were reset): {}",
String::from_utf8_lossy(&checked.stderr).trim()
);
let applied = run_git_apply(false)?;
ensure!(
applied.status.success(),
"git apply failed: {}",
String::from_utf8_lossy(&applied.stderr).trim()
);
let patch_id = format!("{:x}", Sha256::digest(diff.as_bytes()))[..12].to_string();
let pending = root.join("results/pending");
std::fs::create_dir_all(&pending)?;
std::fs::write(
pending.join(format!("{patch_id}.json")),
serde_json::to_string_pretty(
&json!({"target": t, "hypothesis": hypothesis, "diff": diff}),
)?,
)?;
Ok(json!({"patch_id": patch_id}))
}
"run_verdict" => {
let patch_id = field(&req, "patch_id")?;
ensure!(
patch_id.chars().all(|c| c.is_ascii_hexdigit()) && patch_id.len() == 12,
"bad patch id"
);
let run_id = field(&req, "run_id")?;
ensure!(
run_id
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_'),
"bad run id"
);
let class = field(&req, "playbook_class")?;
let hotspot = field(&req, "hotspot")?;
let pending: Value = serde_json::from_str(&std::fs::read_to_string(
root.join(format!("results/pending/{patch_id}.json")),
)?)?;
let t = pending["target"].as_str().unwrap();
let spec = diff_test::target::TargetSpec::load(root, t)?;
let diff_path = root.join(format!("results/pending/{patch_id}.json.diff"));
std::fs::write(&diff_path, pending["diff"].as_str().unwrap())?;
// The build + verdict pipeline runs for minutes — far past the MCP
// transport's per-call cap. Launch it detached, writing progress to
// a log; the agent observes the result via the read_verdict op,
// which reads the append-only ledger row once written.
// {out} isolates the candidate build (cargo: CARGO_TARGET_DIR;
// make: build+copy into it) — language-agnostic, mirrors the
// baseline rebuild in verdict.
let cand_dir = root
.join(format!("targets/{t}/candidate-{patch_id}"))
.to_string_lossy()
.into_owned();
let cand_build = diff_test::target::subst_out(&spec.build.baseline, &cand_dir);
let cand_bin = diff_test::target::subst_out(&spec.build.binary, &cand_dir);
let log = format!("results/pending/{run_id}.log");
// On any failure (candidate build, gates crash) the trap line
// marks the log so read_verdict can report "failed" instead of
// leaving the agent polling a verdict that will never exist.
let script = format!(
"{{ set -e\n{build}\ncargo run -q -p verdict -- {tgt} \
--rebuild-baseline --candidate-bin {cbin} --run-id {rid} --playbook-class {cls} \
--hypothesis {hyp} --hotspot {hs} --patch-file {pf}\n}} || echo HOTPATH-PIPELINE-FAILED\n",
build = cand_build,
tgt = shq(t),
cbin = shq(&cand_bin),
rid = shq(run_id),
cls = shq(class),
hyp = shq(pending["hypothesis"].as_str().unwrap()),
hs = shq(hotspot),
pf = shq(diff_path.to_str().unwrap()),
);
let logf = std::fs::File::create(root.join(&log))?;
let errf = logf.try_clone()?;
Command::new("setsid")
.arg("sh")
.arg("-c")
.arg(&script)
.current_dir(root)
.stdout(std::process::Stdio::from(logf))
.stderr(std::process::Stdio::from(errf))
.spawn()?;
Ok(json!({"status": "started", "run_id": run_id, "log": log,
"note": "pipeline runs detached; poll read_verdict with this run_id"}))
}
"read_verdict" => {
let run_id = field(&req, "run_id")?;
// Long-poll: the pipeline runs for minutes and the MCP transport
// caps a tool call at 60s. Waiting ~45s server-side per poll cuts
// the agent's turn burn ~15x versus instant "running" replies.
let deadline = std::time::Instant::now() + std::time::Duration::from_secs(45);
loop {
let ledger = ledger::Ledger::open(&root.join("results/ledger.sqlite"))?;
if let Some(v) = ledger.verdict_summary(run_id)? {
return Ok(v);
}
drop(ledger);
// A pipeline that died before the verdict binary ran never
// writes a row; its trap line marks the log instead.
if run_id
.chars()
.all(|c| c.is_ascii_alphanumeric() || c == '-' || c == '_')
{
let log = root.join(format!("results/pending/{run_id}.log"));
if let Ok(text) = std::fs::read_to_string(&log) {
if text.contains("HOTPATH-PIPELINE-FAILED") {
let tail: String = text
.lines()
.rev()
.take(12)
.collect::<Vec<_>>()
.into_iter()
.rev()
.collect::<Vec<_>>()
.join("\n");
return Ok(json!({
"status": "failed",
"note": "the pipeline died before producing a verdict (usually a \
candidate build failure); no ledger row was or will be \
written for this run_id. Failure log tail follows.",
"log_tail": tail,
}));
}
}
}
if std::time::Instant::now() >= deadline {
return Ok(json!({"status": "running",
"note": "no ledger row yet; the pipeline is still building/benching — poll again"}));
}
std::thread::sleep(std::time::Duration::from_secs(3));
}
}
other => Err(anyhow!(
"unknown op: {other} (seven ops, SPEC §3.5 + async read_verdict)"
)),
}
#[cfg(not(unix))]
fn main() {
eprintln!("harnessd requires a Unix host (Unix sockets, setsid)");
std::process::exit(1);
}