Skip to content
validate

Declare effects and guard uncertain writes in agent plans #91

Sign in to view logs

Declare effects and guard uncertain writes in agent plans

Declare effects and guard uncertain writes in agent plans #91

Workflow file for this run

.github/workflows/validate.yml at 17ce3db
# Learning OS v3 — CI validation (added 2026-08-03, ADR-003 S5; sharded
# 2026-10-24, C3). Runs the same checks the local `make system-check` gate
# runs — the complete test suite, static gates, and the paired UI check —
# split over parallel shard jobs whose union is provably the whole suite.
# Weekly run adds --online (URL-rot audit over the sources registry — the
# job local runs skip).
#
# Shard invariant (why this is not a coverage shrink): every
# `tests/test_*.py` file appears in exactly one leg of the shard manifest
# (.github/shard-manifest.json) — except files under `node_shards`, which
# a stable node-id hash splits over exactly the listed legs, each test
# still running exactly once. tests/test_workflows.py pins the manifest
# against the tree; every leg deselects exactly `live_install`; and the
# gate job proves that the summed shard results equal a fresh full
# collection before running the static gates. When the balance drifts,
# regenerate the manifest from a CI-measured durations artifact.
#
# The witness population (SPEC §3.3 A9, §3.6 A19) crosses the process
# boundary explicitly, from every suite leg: each leg uploads its session
# witness file, attribution sidecar and audit dir, and the zz-* legs merge
# all of them (tests/conftest.py's premerged-mode envs) and run the two
# `test_scenarios_zz_*` population files — `test_scenarios_zz_refusal_
# sites.py` node-sharded, because its 799 parametrized site checks are
# the suite's slowest single file (~40 min unsplit). No leg list decides
# whose witnesses count: a test in any leg that writes to the session
# witness file is in the merged population by construction.
name: validate
on:
push:
branches: ['**']
pull_request: {}
schedule:
- cron: "0 6 * * 1" # Mondays 06:00 UTC ≈ 08:00 Berlin
workflow_dispatch: {}
# Read-only: this workflow checks out both repositories and runs the local
# paired gate's contents. It never pushes, tags, releases, or otherwise
# needs write access to anything — narrowing this is free safety, not a
# functional change.
permissions:
contents: read
jobs:
pytest-shards:
runs-on: ubuntu-latest
# The unsharded suite took 2h01m of pytest; legs are balanced to well
# under 45 minutes, and a leg that hangs must die rather than queue.
timeout-minutes: 45
permissions:
contents: read
strategy:
fail-fast: false
matrix:
shard: [witness-0, witness-1, rest-0, rest-1, rest-2]
defaults:
run:
working-directory: repository
steps:
- name: Check out Core
uses: actions/checkout@v4
with:
path: repository
fetch-depth: 0
# The public manifest has a separately released UI consumer. Checking
# its mirror only in the UI repository recreates the failure topology
# the producer-owned manifest contract removed: Core can pass while
# shipping a shape the UI cannot decode.
#
# Prefer a same-named consumer branch when one exists, otherwise use
# main. This makes a two-repository change test the pair being
# reviewed without reintroducing a permanent branch pin.
- name: Resolve the matching consumer branch
id: consumer-ref
env:
REQUESTED_REF: ${{ github.head_ref || github.ref_name }}
run: |
if git ls-remote --exit-code --heads \
https://github.com/Aram164/learningos-obsidian-ui.git \
"refs/heads/$REQUESTED_REF" >/dev/null 2>&1; then
echo "ref=$REQUESTED_REF" >> "$GITHUB_OUTPUT"
else
echo "ref=main" >> "$GITHUB_OUTPUT"
fi
# The UI checkout also serves the paired tests inside the suite
# (test_ui_gateway_recovery drives the real harness in
# obsidian-ui/tests/; the contract-mirror tests read the sibling).
- name: Check out the paired UI consumer
uses: actions/checkout@v4
with:
repository: Aram164/learningos-obsidian-ui
ref: ${{ steps.consumer-ref.outputs.ref }}
path: obsidian-ui
- name: Print the resolved UI ref and commit
working-directory: obsidian-ui
env:
RESOLVED_UI_REF: ${{ steps.consumer-ref.outputs.ref }}
run: |
printf 'UI ref resolved: %s\n' "$RESOLVED_UI_REF"
printf 'UI commit checked out: %s\n' "$(git rev-parse HEAD)"
- uses: actions/setup-python@v5
with:
python-version: "3.14.4"
- uses: actions/setup-node@v4
with:
node-version: 22
# The documented project environment is the repository venv: parser
# diagnostics are characterized bytes pinned to this interpreter, and
# the procedure transcripts invoke $LOS_REPO/.venv/bin/python.
- name: Install Core development dependencies
run: |
python -m venv .venv
.venv/bin/python -m pip install -e ".[dev]"
- name: Install UI dependencies
working-directory: obsidian-ui
run: npm ci --ignore-scripts
# The procedure transcripts (tests/test_procedures.py, PLAN D11) run
# under Scrut v0.5.0. The release archive is pinned by GitHub's
# published SHA-256 and checked before use; the binary stays outside
# the checkout, clear of the tree-contract test, and reaches the gate
# as LOS_SCRUT_BIN.
- name: Install PDF text extractor
run: |
sudo apt-get update
sudo apt-get install -y --no-install-recommends poppler-utils
pdftotext -v
- name: Install Scrut v0.5.0 (pinned digest)
run: |
set -eu
dir="$RUNNER_TEMP/scrut"
mkdir -p "$dir"
curl --proto '=https' --tlsv1.2 -fsSL -o "$dir/scrut.tar.gz" \
https://github.com/facebookincubator/scrut/releases/download/v0.5.0/scrut-v0.5.0-linux-x86_64.tar.gz
echo "5c373d077a73dc4a5d4b43f3474e0f443dc72fb42f89b312ef55f934e6e6ba15 $dir/scrut.tar.gz" | sha256sum -c -
tar -xzf "$dir/scrut.tar.gz" -C "$dir"
bin="$(find "$dir" -type f -name scrut | head -n 1)"
test -n "$bin"
chmod 0755 "$bin"
test "$("$bin" --version)" = "scrut 0.5.0"
echo "LOS_SCRUT_BIN=$bin" >> "$GITHUB_ENV"
- name: Resolve this shard's file list from the manifest
id: pick
env:
SHARD: ${{ matrix.shard }}
run: |
set -eu
files="$(python3 -c '
import json, os, sys
manifest = json.load(open(".github/shard-manifest.json"))["legs"]
shard = sys.argv[1]
try:
files = manifest[shard]
except KeyError:
raise SystemExit(
f"shard {shard!r} is not in .github/shard-manifest.json")
missing = sorted(f for f in files if not os.path.isfile(f))
assert not missing, f"shard {shard}: files missing from tests/: {missing}"
print("\n".join(files))
' "$SHARD")"
count="$(printf '%s' "$files" | grep -c .)"
printf 'shard %s: %s files\n' "$SHARD" "$count"
{
echo "files<<SHARD_FILES_EOF"
printf '%s\n' "$files"
echo "SHARD_FILES_EOF"
} >> "$GITHUB_OUTPUT"
# Same selection the unsharded gate pinned: deselect exactly the
# `live_install` marker, nothing broader. The file list comes only
# from the manifest; a leg never adds its own -k or -m filters.
# --basetemp pins the session witness file and audit dir under
# pytesttmp/ so this leg can upload them for the zz merge below.
- name: Run this shard of the suite
env:
PYTEST_ADDOPTS: '-m "not live_install"'
LOS_RATCHET_BASE: origin/main
SHARD: ${{ matrix.shard }}
PICKED: ${{ steps.pick.outputs.files }}
run: |
set -eu
mkdir -p ../junit-out
# shellcheck disable=SC2086 # manifest paths contain no spaces
.venv/bin/python -m pytest -q --basetemp=../pytesttmp \
--junitxml="../junit-out/junit-$SHARD.xml" $PICKED
- name: Upload this shard's JUnit results
if: always()
uses: actions/upload-artifact@v4
with:
name: junit-${{ matrix.shard }}
path: junit-out/
if-no-files-found: error
# The A9/A19 population artifacts, from every suite leg — not just
# the legs that historically produce witnesses: the session witness
# file, its attribution sidecar, and the audit dir with its
# cases.txt index (tests/conftest.py creates all three for every
# session, so a leg that produced nothing still uploads the empty
# triple). The globs name the session fixtures' own basetemp
# locations, because private witness files some tests create under
# their per-test tmp dirs must never enter the population.
- name: Upload this leg's witness population
if: always()
uses: actions/upload-artifact@v4
with:
name: population-${{ matrix.shard }}
path: |
pytesttmp/site-witness*/sites.jsonl
pytesttmp/site-witness*/sites.jsonl.attribution.json
pytesttmp/operational-audit*
if-no-files-found: error
# Intent: never follow symlinks. Empirically the upload still
# materializes pytest's <name>current aliases beside the
# session dirs, so the merge below collapses the byte-identical
# copies — the dedupe there is load-bearing, not this flag.
follow-symbolic-links: false
# The zz population files, over the merged witness population of every
# suite leg. test_scenarios_zz_refusal_sites.py (~40 min unsplit, the
# slowest file in the suite) is hash-split over the four legs;
# test_scenarios_zz_operational_owners.py runs whole in zz-0.
zz-shards:
needs: [pytest-shards]
runs-on: ubuntu-latest
timeout-minutes: 45
permissions:
contents: read
strategy:
fail-fast: false
matrix:
shard: [zz-0, zz-1, zz-2, zz-3]
defaults:
run:
working-directory: repository
steps:
- name: Check out Core
uses: actions/checkout@v4
with:
path: repository
fetch-depth: 0
- uses: actions/setup-python@v5
with:
python-version: "3.14.4"
- name: Install Core development dependencies
run: |
python -m venv .venv
.venv/bin/python -m pip install -e ".[dev]"
- name: Install PDF text extractor
run: |
sudo apt-get update
sudo apt-get install -y --no-install-recommends poppler-utils
pdftotext -v
- name: Install Scrut v0.5.0 (pinned digest)
run: |
set -eu
dir="$RUNNER_TEMP/scrut"
mkdir -p "$dir"
curl --proto '=https' --tlsv1.2 -fsSL -o "$dir/scrut.tar.gz" \
https://github.com/facebookincubator/scrut/releases/download/v0.5.0/scrut-v0.5.0-linux-x86_64.tar.gz
echo "5c373d077a73dc4a5d4b43f3474e0f443dc72fb42f89b312ef55f934e6e6ba15 $dir/scrut.tar.gz" | sha256sum -c -
tar -xzf "$dir/scrut.tar.gz" -C "$dir"
bin="$(find "$dir" -type f -name scrut | head -n 1)"
test -n "$bin"
chmod 0755 "$bin"
test "$("$bin" --version)" = "scrut 0.5.0"
echo "LOS_SCRUT_BIN=$bin" >> "$GITHUB_ENV"
- name: Download the witness population
uses: actions/download-artifact@v4
with:
pattern: population-*
path: population-in
# Deterministic merge over every suite leg's artifacts (leg order =
# artifact name order): the witness lines concatenate, the
# attribution sidecars re-key onto the merged file, the audit logs
# union, and cases.txt keeps its entry set. Every leg the manifest
# assigns to the suite phase (all legs but this job's zz-*) must
# have contributed exactly one witness file, one sidecar and one
# audit dir — a lost or surplus population artifact is loud, never
# silent.
- name: Merge the witness population
run: |
set -eu
python3 - <<'PY'
import json
import shutil
from pathlib import Path
src = Path("../population-in")
def _sole_identical(paths, what, leg):
"""All copies must be byte-identical; return one of them."""
assert paths, (
f"population artifact of leg {leg!r} has no {what}")
first = paths[0].read_bytes()
for other in paths[1:]:
assert other.read_bytes() == first, (
f"population artifact of leg {leg!r} holds more "
f"than one different {what}: "
f"{sorted(str(p) for p in paths)}")
return paths[0]
parts = {}
for artifact in sorted(p for p in src.glob("population-*")
if p.is_dir()):
leg = artifact.name.removeprefix("population-")
# pytest leaves a <name>current symlink beside each real
# session dir; if the upload materialized it anyway, it is
# a byte-identical second copy — identical copies collapse
# into one here, and anything that differs is a private
# artifact that must never enter the population.
witness = _sole_identical(
sorted({p.resolve() for p in artifact.rglob("sites.jsonl")}),
"session witness file", leg)
sidecar = _sole_identical(
sorted({p.resolve() for p in
artifact.rglob("sites.jsonl.attribution.json")}),
"attribution sidecar", leg)
audits = sorted({p.resolve() for p in
artifact.rglob("operational-audit*")
if p.is_dir()})
assert audits, (
f"population artifact of leg {leg!r} has no audit dir")
snapshots = [{str(p.relative_to(d)): p.read_bytes()
for p in sorted(d.rglob("*")) if p.is_file()}
for d in audits]
for snapshot in snapshots[1:]:
assert snapshot == snapshots[0], (
f"population artifact of leg {leg!r} holds more "
f"than one different audit dir: "
f"{sorted(str(d) for d in audits)}")
parts[leg] = (witness, sidecar, audits[0])
manifest = json.loads(
Path(".github/shard-manifest.json").read_text(encoding="utf-8"))
suite_legs = sorted(leg for leg in manifest["legs"]
if not leg.startswith("zz-"))
assert sorted(parts) == suite_legs, (
f"population artifacts {sorted(parts)} do not match the "
f"manifest's suite legs {suite_legs} — a leg's witness "
f"population would be silently lost")
merged = Path("../merged")
audit = merged / "audit"
audit.mkdir(parents=True)
with open(merged / "sites.jsonl", "wb") as out:
for leg in sorted(parts):
out.write(parts[leg][0].read_bytes())
attribution = {}
for leg in sorted(parts):
for key, pairs in json.loads(
parts[leg][1].read_text(encoding="utf-8")).items():
attribution.setdefault(key, []).extend(pairs)
(merged / "attr.json").write_text(
json.dumps(attribution, sort_keys=True), encoding="utf-8")
cases = []
for leg in sorted(parts):
for log in sorted(parts[leg][2].glob("*.jsonl")):
shutil.copy(log, audit / log.name)
index = parts[leg][2] / "cases.txt"
if index.is_file():
cases.extend(index.read_text(encoding="utf-8").split())
(audit / "cases.txt").write_text(
"\n".join(dict.fromkeys(cases)) + "\n", encoding="utf-8")
print(f"merged {len(parts)} suite legs: "
f"{sum(1 for _ in open(merged / 'sites.jsonl'))} witness "
f"lines, {len(list(audit.glob('*.jsonl')))} audit logs")
PY
merged="$(cd ../merged && pwd)"
{
echo "LOS_PREMERGED_WITNESS=$merged/sites.jsonl"
echo "LOS_PREMERGED_ATTRIBUTION=$merged/attr.json"
echo "LOS_PREMERGED_AUDIT_DIR=$merged/audit"
} >> "$GITHUB_ENV"
- name: Resolve this leg's node shard from the manifest
env:
SHARD: ${{ matrix.shard }}
run: |
set -eu
spec="$(python3 -c '
import json, os
manifest = json.load(open(".github/shard-manifest.json"))["node_shards"]
shard = os.environ["SHARD"]
for fname, spec in manifest.items():
legs = spec["legs"]
if shard in legs:
stem = fname.rsplit("/", 1)[-1]
print(stem + "=" + str(legs.index(shard)) + "/"
+ str(spec["count"]))
break
else:
raise SystemExit("zz leg " + shard + " has no node_shards entry")
')"
printf 'node shard: %s\n' "$spec"
echo "LOS_NODE_SHARDS=$spec" >> "$GITHUB_ENV"
- name: Resolve this shard's file list from the manifest
id: pick
env:
SHARD: ${{ matrix.shard }}
run: |
set -eu
files="$(python3 -c '
import json, os, sys
manifest = json.load(open(".github/shard-manifest.json"))["legs"]
shard = sys.argv[1]
try:
files = manifest[shard]
except KeyError:
raise SystemExit(
f"shard {shard!r} is not in .github/shard-manifest.json")
missing = sorted(f for f in files if not os.path.isfile(f))
assert not missing, f"shard {shard}: files missing from tests/: {missing}"
print("\n".join(files))
' "$SHARD")"
count="$(printf '%s' "$files" | grep -c .)"
printf 'shard %s: %s files\n' "$SHARD" "$count"
{
echo "files<<SHARD_FILES_EOF"
printf '%s\n' "$files"
echo "SHARD_FILES_EOF"
} >> "$GITHUB_OUTPUT"
- name: Run this zz shard of the suite
env:
PYTEST_ADDOPTS: '-m "not live_install"'
LOS_RATCHET_BASE: origin/main
SHARD: ${{ matrix.shard }}
PICKED: ${{ steps.pick.outputs.files }}
run: |
set -eu
mkdir -p ../junit-out
# shellcheck disable=SC2086 # manifest paths contain no spaces
.venv/bin/python -m pytest -q --basetemp=../pytesttmp \
--junitxml="../junit-out/junit-$SHARD.xml" $PICKED
- name: Upload this shard's JUnit results
if: always()
uses: actions/upload-artifact@v4
with:
name: junit-${{ matrix.shard }}
path: junit-out/
if-no-files-found: error
# The UI's own complete check (build, typecheck, contract gate, fixture
# suites, and the real cross-process Gateway recovery harness against
# this Core) — the same `npm run check` the local paired gate runs last.
ui-check:
runs-on: ubuntu-latest
timeout-minutes: 15
permissions:
contents: read
steps:
- name: Check out Core
uses: actions/checkout@v4
with:
path: repository
fetch-depth: 0
- name: Resolve the matching consumer branch
id: consumer-ref
env:
REQUESTED_REF: ${{ github.head_ref || github.ref_name }}
run: |
if git ls-remote --exit-code --heads \
https://github.com/Aram164/learningos-obsidian-ui.git \
"refs/heads/$REQUESTED_REF" >/dev/null 2>&1; then
echo "ref=$REQUESTED_REF" >> "$GITHUB_OUTPUT"
else
echo "ref=main" >> "$GITHUB_OUTPUT"
fi
- name: Check out the paired UI consumer
uses: actions/checkout@v4
with:
repository: Aram164/learningos-obsidian-ui
ref: ${{ steps.consumer-ref.outputs.ref }}
path: obsidian-ui
- uses: actions/setup-node@v4
with:
node-version: 22
# The UI check's contract:check-production step imports the real
# learning_os package from the sibling Core checkout (PYTHON, .venv,
# python3, python are probed, in that order) — the same project venv
# the paired gate builds before its own npm run check.
- uses: actions/setup-python@v5
with:
python-version: "3.14.4"
- name: Install Core development dependencies
working-directory: repository
run: |
python -m venv .venv
.venv/bin/python -m pip install -e ".[dev]"
- name: Install UI dependencies
working-directory: obsidian-ui
run: npm ci --ignore-scripts
- name: UI check (npm run check)
working-directory: obsidian-ui
run: npm run check
# The join point: nothing is green until the shards are proven to be
# exactly the whole suite, the static gates pass, and every leg passed.
gate:
needs: [pytest-shards, zz-shards, ui-check]
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
defaults:
run:
working-directory: repository
steps:
- name: Check out Core
uses: actions/checkout@v4
with:
path: repository
fetch-depth: 0
- uses: actions/setup-python@v5
with:
python-version: "3.14.4"
- name: Install Core development dependencies
run: |
python -m venv .venv
.venv/bin/python -m pip install -e ".[dev]"
- name: Download every shard's JUnit results
uses: actions/download-artifact@v4
with:
pattern: junit-*
merge-multiple: true
path: junit-in
# The completeness proof, run on every trigger: collect the whole
# suite fresh, count <testcase> elements across all shard results,
# and require set equality — no test lost, none duplicated, none
# invented. Then the static gates the local paired gate runs before
# pytest: lint, typecheck, validate, warning baseline.
- name: Prove the shards are exactly the suite, then run static gates
run: |
set -eu
.venv/bin/python - <<'PY'
import json
import subprocess
import sys
import xml.etree.ElementTree as ET
from pathlib import Path
manifest = json.loads(
Path(".github/shard-manifest.json").read_text(encoding="utf-8"))
disk = sorted(f"tests/{p.name}"
for p in Path("tests").glob("test_*.py"))
from collections import Counter
covered = Counter(f for leg in manifest["legs"].values()
for f in leg)
for name, count in sorted(covered.items()):
expected = (manifest.get("node_shards", {}).get(name, {})
.get("count", 1))
assert count == expected, (
f"{name} appears in {count} legs, expected {expected}")
assert sorted(covered) == disk, (
f".github/shard-manifest.json does not partition tests/: "
f"missing={sorted(set(disk) - set(covered))} "
f"extra={sorted(set(covered) - set(disk))}")
out = subprocess.run(
[".venv/bin/python", "-m", "pytest", "--collect-only", "-q",
"-m", "not live_install"],
capture_output=True, text=True, check=True)
tail = out.stdout.strip().splitlines()[-1]
# "8520/8521 tests collected (1 deselected) in 7.26s"
collected = int(tail.split()[0].split("/")[0])
ran = 0
skipped = 0
for path in sorted(Path("../junit-in").glob("junit-*.xml")):
root = ET.parse(path).getroot()
for case in root.iter("testcase"):
ran += 1
if case.find("skipped") is not None:
skipped += 1
if (case.find("failure") is not None
or case.find("error") is not None):
sys.exit(f"{path.name}: failing testcase "
f"{case.get('classname')}::{case.get('name')}")
assert ran == collected, (
f"shards ran {ran} tests; the full collection has {collected} "
"(deselecting live_install) — a test was lost, duplicated "
"or added")
print(f"gate: {ran} tests across shards == full collection "
f"({skipped} skipped, visible, never blocking)")
PY
.venv/bin/python -m ruff check tools tests
.venv/bin/python tools/code_reachability.py
.venv/bin/python -m mypy
.venv/bin/python tools/validate.py --no-report
.venv/bin/python tools/warning_baseline.py --check
- name: URL-rot audit (weekly only)
if: github.event_name == 'schedule'
run: .venv/bin/python tools/validate.py --online