Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 10 additions & 2 deletions docs/evaluation/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -234,8 +234,16 @@ Each batch creates `jobs/evaluations/<unique-run>/` or a new `--output` director
| `logs/` | Asset preparation and evaluation logs |
| Harbor result paths recorded in `run.json` | Scores, verifier health, trajectories and artifacts |

`completed` means the launcher exited successfully; check Harbor reports for
scores and verifier health. Ctrl-C/SIGTERM cancels active runs and lets storage
`completed` requires a successful launch, a finished Harbor trial and verifier,
and a primary `reward` between 0 and 1 inclusive. A graded zero counts as completed, including when the
agent timed out and the verifier scored its work. A setup failure or missing
trial result marks the task `failed`. Batch runs and regrades use the same verifier
diagnostics. A verifier timeout, failed answer or trajectory judging, or invalid
resource evidence marks the task `invalid-measurement`. A completed audit may
reject the submission and award a valid zero. Failed and invalid measurements
give the batch a nonzero exit code.

Ctrl-C/SIGTERM cancels active runs and lets storage
teardown finish. Cleanup failure stops the batch, cancels other active runs,
records `cleanup-error` and leaves queued tasks `not-started`; follow the recovery
instructions before another batch. Tasks with
Expand Down
39 changes: 39 additions & 0 deletions docs/evaluation/runtime.md
Original file line number Diff line number Diff line change
Expand Up @@ -22,6 +22,45 @@ checks. Task libraries run in the published CPU/GPU
[base images](../../docker/README.md); ordinary runs pull these images.
See [data and models](assets.md) to plan space for task copies, cache and outputs.

## Linux cgroups

The resource checks use the current container's CPU and memory controllers.
This lets the same task budgets apply across different CPU and GPU models.

| Task | cgroup v1 requirements |
| --- | --- |
| `constrained-molecular-search` | CPU CFS quota and a finite aggregate RAM limit |
| `budgeted-candidate-reranking` | A finite RAM limit and the kernel's `oom_kill` counter for the container's memory cgroup |
| `memory-constrained-dense-retrieval`, `training-error-tracing` | A finite RAM limit, a container-scoped `oom_kill` counter and no active host swap |
| `budgeted-bimetric-retrieval`, `large-scale-formula-retrieval` | A finite RAM limit and no active host swap. The launcher subscribes to the kernel’s OOM notifications for the task and its parent. |

On v1, the effective RAM cap includes ancestor memory and combined RAM+swap
limits reported by the kernel, even when the container has a private cgroup
namespace. The v1 OOM-kill checks use a leaf memory cgroup, the normal layout for a
single Docker task container. The CPU affinity requirements and GPU
requirements still apply. A missing diagnostic memory peak is recorded as
`null`. Missing required evidence produces a verifier
`cgroup-status.json` with `measurement_valid: false` and the failing check.
For Formula, the agent environment also checks for the required evidence
before starting the coding agent. If that health check fails, the batch records
a failed trial with the Harbor exception type.

For Bimetric and Formula, the launcher registers v1 notifications on the host
and exposes a read-only bind mount with a private socket to the trusted task
process. This distinguishes successful page-cache reclaim from an OOM, including
an OOM without a kill. Bimetric checks the interval after verifier startup.
Formula also checks the full development container lifetime. If notifications
are unavailable, untouched RAM and combined RAM+swap limits can still establish
a clean run. Formula also recognizes a kernel-recorded OOM kill. Unattributable
pressure produces an invalid measurement.

The probes were verified on Linux 6.8 in a Docker v2 container and an OCI
container on a v1 guest, including successful reclaim and a kernel OOM kill.

See the [Linux cgroup v1 memory controller](https://docs.kernel.org/admin-guide/cgroup-v1/memory.html),
[cgroup v2 interface](https://docs.kernel.org/admin-guide/cgroup-v2.html)
and [CPU bandwidth control](https://docs.kernel.org/scheduler/sched-bwc.html).

## Select a task

Use `search-swe list` to discover `tasks/*/task.toml`. Read the chosen package:
Expand Down
215 changes: 215 additions & 0 deletions environments/cgroups/cgroup_resources.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,215 @@
"""Read Linux cgroup counters for the current process and its container.

Canonical source is environments/cgroups/cgroup_resources.py. Task image
contexts contain identical copies so they can be built independently.
"""

from dataclasses import dataclass
from functools import cache
import json
import os
from pathlib import Path
import re


class CgroupError(RuntimeError):
"""The environment cannot establish the task's required resource budget."""


def _read(path):
try:
return path.read_text(encoding="ascii")
except (OSError, UnicodeError) as error:
raise CgroupError(f"Cannot read cgroup evidence at {path}: {error}") from error


def _read_proc(path):
# Proc path fields contain filesystem bytes, including non-UTF-8 names.
try:
return path.read_bytes()
except OSError as error:
raise CgroupError(f"Cannot read cgroup evidence at {path}: {error}") from error


def _integer(value, path, minimum=0):
try:
result = int(value)
if result >= minimum:
return result
except ValueError:
pass
raise CgroupError(f"Invalid cgroup value in {path}: {value!r}")


def _mount_path(value):
unescaped = re.sub(rb"\\([0-3][0-7]{2})", lambda m: bytes([int(m[1], 8)]), value)
return Path(os.fsdecode(unescaped))


@cache
def controller(name):
"""Resolve a controller using this process's membership and visible mounts."""
return resolve_controller(name, Path("/proc/self/cgroup"))


def resolve_controller(name, membership_file):
"""Resolve a process's membership against mounts visible to the reader."""
key_name = name.encode("ascii")
memberships = {}
for line in _read_proc(membership_file).rstrip(b"\n").split(b"\n"):
fields = line.split(b":", 2)
if len(fields) != 3:
raise CgroupError(f"Invalid cgroup membership: {line!r}")
_, controllers, location = fields
for key in controllers.split(b","):
memberships[key] = Path(os.fsdecode(location))
# A hybrid host can mount v2 alongside active v1 controllers.
version = 1 if key_name in memberships else 2
member = memberships.get(key_name if version == 1 else b"")
if member is None or not member.is_absolute() or ".." in member.parts:
raise CgroupError(f"No visible cgroup membership for the {name} controller")
candidates = []
for line in _read_proc(Path("/proc/self/mountinfo")).split(b"\n"):
before, _, after = line.partition(b" - ")
mount, filesystem = before.split(b" "), after.split(b" ")
if len(mount) < 6 or len(filesystem) < 3:
continue
if version == 1:
if filesystem[0] != b"cgroup" or key_name not in filesystem[2].split(b","):
continue
elif filesystem[0] != b"cgroup2":
continue
root, target = map(_mount_path, mount[3:5])
if root.is_absolute() and ".." not in root.parts and member.is_relative_to(root):
candidates.append((len(root.parts), target / member.relative_to(root)))
if not candidates:
raise CgroupError(f"No cgroup v{version} mount covers {name} membership {member}")
_, directory = max(candidates)
return Controller(version, directory)


@dataclass(frozen=True)
class Controller:
version: int
path: Path

def _text(self, name):
return _read(self.path / name)

def _number(self, name, minimum=0):
return _integer(self._text(name).strip(), self.path / name, minimum)

def cpu_quota(self):
"""Return (quota in microseconds or None for unlimited, period in microseconds)."""
if self.version == 1:
quota = self._number("cpu.cfs_quota_us", -1)
period = self._number("cpu.cfs_period_us", 1)
if quota == 0:
raise CgroupError(f"Invalid zero CPU quota in {self.path / 'cpu.cfs_quota_us'}")
return (None if quota == -1 else quota), period
path = self.path / "cpu.max"
fields = _read(path).split()
if len(fields) != 2:
raise CgroupError(f"Expected quota and period in {path}")
quota, period = fields
return (None if quota == "max" else _integer(quota, path, 1)), _integer(period, path, 1)

def memory_limit(self):
if self.version == 2:
value = self._text("memory.max").strip()
return None if value == "max" else _integer(value, self.path / "memory.max", 1)
# memory.stat includes the tightest ancestor even in a private cgroup namespace.
path = self.path / "memory.stat"
bounds = {}
for line in _read(path).splitlines():
fields = line.split()
if fields and fields[0] in ("hierarchical_memory_limit", "hierarchical_memsw_limit"):
if len(fields) != 2 or fields[0] in bounds:
raise CgroupError(f"Malformed {fields[0]} in {path}")
bounds[fields[0]] = _integer(fields[1], path)
if "hierarchical_memory_limit" not in bounds:
raise CgroupError(f"Missing hierarchical_memory_limit in {path}")
if "hierarchical_memsw_limit" not in bounds and (self.path / "memory.memsw.limit_in_bytes").exists():
raise CgroupError(f"Missing hierarchical_memsw_limit in {path}")
limit = min(self._number("memory.limit_in_bytes"), *bounds.values())
# v1 uses page-rounded LONG_MAX to represent an unbounded limit.
page_size = os.sysconf("SC_PAGE_SIZE")
unlimited = (2**63 - 1) // page_size * page_size
return None if limit >= unlimited else limit

def swap_limit(self):
"""Return a proven swap-only byte limit, or None when v2 allows 'max'."""
if self.version == 2:
name = "memory.swap.max"
value = self._text(name).strip()
return None if value == "max" else _integer(value, self.path / name)
# The v1 combined RAM+swap limit does not prove zero swapped pages.
# Linux cannot swap when the host has no active swap devices or files.
swaps = _read(Path("/proc/swaps")).splitlines()
if swaps and swaps[0].split() == ["Filename", "Type", "Size", "Used", "Priority"] and not any(
line.strip() for line in swaps[1:]):
return 0
raise CgroupError("cgroup v1 cannot verify zero swap with active or unreadable host swap")

def memory_charge(self):
name = "memory.usage_in_bytes" if self.version == 1 else "memory.current"
return self._number(name)

def memory_peak(self):
"""Peak is a report-only kernel counter."""
name = "memory.max_usage_in_bytes" if self.version == 1 else "memory.peak"
try:
return self._number(name)
except CgroupError:
return None

def cpu_stat(self):
return self._text("cpu.stat")

@cache
def _require_v1_leaf(self):
# A v1 counter can be local. A leaf has the same process scope as
# the hierarchical v2 counter, including forked submission processes.
try:
if any(path.is_dir() for path in self.path.iterdir()):
raise CgroupError(f"Cannot establish container-wide v1 OOM counts in {self.path} with child cgroups")
except OSError as error:
raise CgroupError(f"Cannot inspect the v1 memory cgroup {self.path}: {error}") from error

def v1_limit_hits(self):
# v1 may reject a combined memory+swap charge before charging RAM.
# Kernels without swap accounting omit the memsw counter entirely.
try:
combined = (self.path / "memory.memsw.failcnt").read_text(encoding="ascii")
except FileNotFoundError:
combined = None
except (OSError, UnicodeError) as error:
raise CgroupError(f"Cannot read memory.memsw.failcnt: {error}") from error
return [self._number("memory.failcnt"),
None if combined is None else _integer(combined.strip(), self.path / "memory.memsw.failcnt")]

def memory_events(self, *required):
name = "memory.oom_control" if self.version == 1 else "memory.events"
if self.version == 1:
if "oom" in required:
raise CgroupError(f"{self.path / name} does not expose a cumulative oom event count")
self._require_v1_leaf()
raw = self._text(name)
counters = {}
for line in raw.splitlines():
fields = line.split()
if len(fields) != 2 or fields[0] in counters:
raise CgroupError(f"Malformed cgroup events in {self.path / name}")
key, value = fields
counters[key] = _integer(value, self.path / name)
for key in required:
if key not in counters:
raise CgroupError(f"Missing {key} counter in {self.path / name}")
return raw, counters


def write_resource_error(directory, error):
"""Record why a verifier could not measure a resource-constrained task."""
path = Path(directory) / "cgroup-status.json"
path.write_text(json.dumps({"measurement_valid": False, "failure_kind": "cgroup_environment",
"error": str(error)}) + "\n")
92 changes: 92 additions & 0 deletions environments/cgroups/v1_oom.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,92 @@
"""Container-side OOM evidence shared by Formula and Bimetric on cgroup v1."""

import json
import socket

if __package__:
from .cgroup_resources import CgroupError
else: # Independent task image.
from cgroup_resources import CgroupError


SOCKET = "/run/search-swe-memory/oom.sock"


def snapshot(memory):
"""Sample the limit counters first so the notification check covers that interval."""
hits = memory.v1_limit_hits()
_, counters = memory.memory_events("oom_kill")
try:
with socket.socket(socket.AF_UNIX, socket.SOCK_STREAM) as client:
client.settimeout(5)
client.connect(SOCKET)
with client.makefile("rb") as stream:
watch = json.loads(stream.readline(4096))
except (FileNotFoundError, PermissionError, ConnectionError, TimeoutError):
watch = None # Raw cgroup counters can still establish some outcomes.
except (OSError, ValueError) as error:
raise CgroupError(f"Cannot read v1 OOM observer: {error}") from error
if watch is not None:
try:
info = memory.path.stat()
except OSError as error:
raise CgroupError(f"Cannot identify the task memory cgroup: {error}") from error
if not isinstance(watch, dict) or watch.get("cgroup") != [info.st_dev, info.st_ino]:
raise CgroupError("Host OOM observer and task see different memory cgroups")
return {"limit_hits": hits, "oom_kill": counters["oom_kill"], "observer": watch}


def _counter(value):
return type(value) is int and value >= 0


def _limit_hits(value):
return (isinstance(value, list) and len(value) == 2 and _counter(value[0])
and (value[1] is None or _counter(value[1])))


def _validate(sample):
if (not isinstance(sample, dict) or not _limit_hits(sample.get("limit_hits"))
or not _counter(sample.get("oom_kill")) or "observer" not in sample):
raise CgroupError("Invalid v1 memory counters")
watch = sample["observer"]
if watch is not None and (not isinstance(watch, dict) or not isinstance(watch.get("id"), str)
or not watch["id"] or not _counter(watch.get("events")) or not _counter(watch.get("parent_events"))
or not _limit_hits(watch.get("initial_hits"))):
raise CgroupError("Invalid v1 OOM notification evidence")


def violation(sample, baseline=None, *, include_kills):
"""Formula checks the container lifetime. Bimetric supplies a verifier baseline."""
_validate(sample)
hits = sample["limit_hits"]
old_hits = [0, None if hits[1] is None else 0]
old_kills = 0
old_watch = None
if baseline is not None:
_validate(baseline)
old_hits, old_kills, old_watch = baseline["limit_hits"], baseline["oom_kill"], baseline["observer"]
if (sample["oom_kill"] < old_kills
or any((a is None) != (b is None) or (a is not None and a < b) for a, b in zip(hits, old_hits))):
raise CgroupError("Reset v1 memory counters")
if include_kills and sample["oom_kill"] > old_kills:
return True
watch = sample["observer"]
if watch and (baseline is None or (old_watch and old_watch["id"] == watch["id"])):
start = old_watch["events"] if old_watch else 0
parent_start = old_watch["parent_events"] if old_watch else 0
if watch["events"] < start or watch["parent_events"] < parent_start:
raise CgroupError("Reset v1 OOM notifications")
# Ancestor OOMs notify descendants too. A plain eventfd cannot
# attribute a notification if an ancestor was also under pressure.
if watch["parent_events"] == parent_start:
if watch["events"] > start:
return True
if old_watch is not None or not any(watch["initial_hits"]):
return False
# With stable caps, local memcg OOM first requires a failed RAM or memsw
# charge. Successful reclaim also increments these counters.
if hits == old_hits:
return False
raise CgroupError("The v1 memory limit was reached, but local OOM could not be distinguished "
"from reclaim or ancestor pressure in this measurement")
Loading
Loading