Skip to content

[do-not-mirror][do-not-merge] chore(fork): document the SSH stall that blocks fetches and forkctl - #25

Closed
jdsika wants to merge 7 commits into
integration/3.0from
chore/wsl-network-note
Closed

jdsika wants to merge 7 commits into
integration/3.0from
chore/wsl-network-note

Conversation

@jdsika

@jdsika jdsika commented Nov 12, 2026

Copy link
Copy Markdown

What

Adds a troubleshooting section to CONTRIBUTING.md for the case where a forkctl command hangs with no output at all.

Why

forkctl doctor, status and mirror contact GitHub before they print their first line. When that call blocks, the tool looks hung rather than failed, and there is nothing on screen to diagnose from.

This is easy to hit on Windows. The make targets need a POSIX shell, so the natural habit is to run everything inside WSL — but WSL frequently cannot reach GitHub over SSH even on a machine where Windows reaches it in a couple of seconds, and git fetch / git ls-remote then block indefinitely instead of timing out. Encountered exactly this while running forkctl status during the last round of stack work.

Notes

The section gives the symptom, a one-line test that tells the two sides apart (git ls-remote origin -h refs/heads/integration/3.0, run from each), and the split that works: network-touching commands from Windows, offline checks (check-commit, check-stack, track) and make from WSL. HTTPS inside WSL is mentioned as the single-shell alternative.

No behaviour change — documentation only.

Validation

  • forkctl check-commit --chore — PASS (one commit, well-formed chore(fork): subject, markers intact, track B)
  • pre-commit run --all-files — passed (all 5 hooks)

The simulator suites were not run and are not applicable: the diff is one Markdown file.

Upstream readiness

Not applicable — chore(fork):, exempt from Gate 2 and never mirrored.

Disposition

Per .github/copilot-instructions.md §3.3 this is a review vehicle, not a delivery mechanism: once it is green and reviewed it gets folded into the base chore(fork): commit and closed unmerged.

ASCS-eV/PufferDrive is consumed as a git submodule internally while feeding
features back to Emerge-Lab/PufferDrive. This commit carries everything that
exists only in the fork, and is the single place fork-only content lives.

Branch model:
  3.0              verbatim, fast-forward-only mirror of upstream/3.0
  integration/3.0  default branch; this commit plus a linear stack of
                   single-commit features; what internal consumers pin
  feature/<slug>   one feature, exactly one commit
  mirror/<slug>    cherry-picked onto pure upstream/3.0 for upstream PRs

integration/3.0 is the default branch because GitHub reads the PR template,
the contributing guidelines, the agent instructions and scheduled workflows
only from the default branch, and none of those files may exist on 3.0
without destroying the mirror. A clone checks out the default branch, so the
copy of the instructions here is the process as far as every agent and every
submodule consumer is concerned; a rule that lives only in an open PR has no
effect.

Nothing reaches integration/3.0 until three gates pass:
  1. CI green on the final commit.
  2. Adversarial review by someone other than the author, with the findings
     written down -- including the probes that found nothing.
  3. Upstream-readiness proven rather than asserted: the commit cherry-picks
     onto a branch cut from pure upstream/3.0 and produces an upstream-clean
     diff. We stack only commits we could contribute today.

chore(fork): commits are exempt from gate 3, since fork-only infrastructure
can never be upstream-ready. They carry two orthogonal markers: do-not-mirror
(must never reach upstream) and do-not-merge (the merge button is never
pressed -- merging would create a second chore(fork): commit and break the
single-commit invariant that every sync asserts). Such a PR is folded into
this commit by fixup and closed unmerged.

Contents:
- .github/copilot-instructions.md: normative process for coding agents
- CONTRIBUTING.md: human orientation, same rules
- .github/pull_request_template.md: testing and stack-hygiene gates
- .github/workflows/sync-mirror.yml: fast-forward-only mirror sync that
  refuses any non-fast-forward instead of rewriting the upstream base
- AGENTS.md / CLAUDE.md: fenced pointer so agents find the fork process
  without colliding with upstream-owned content
- ci.yml / install.yml: run CI for PRs targeting integration/3.0, which
  upstream's branch filters would otherwise skip

A standing draft PR from integration/3.0 into 3.0 gives a continuous CI
verdict on the combined stack; it must never be merged.

Everything here is fork-only and must never be cherry-picked upstream.
The Linux CI jobs fail while installing the package:

    [Errno 122] Disk quota exceeded

Errno 122 is EDQUOT, hit while unpacking the CUDA torch stack that
`pip install -e .` resolves by default. torch alone is 554.6 MB and the
accompanying nvidia-* wheels add another ~2.4 GB, which no longer fits
the runner's quota. The failure occurs during "Installing build
dependencies", i.e. the isolated PEP 517 build environment is already
enough to exceed it on its own.

The trigger is a runner image change, not a code change:

  * upstream 11_bernhard_dev, 2026-10-20, Image: ubuntu-24.04 -> success
  * same workflows, 2026-10-23, Image: ubuntu-26.04 -> failure

Both resolve an identical wheel set (torch 2.14.1, 554.6 MB). The
`ubuntu-latest` label began migrating to Ubuntu 26 on 2026-10-19
(actions/runner-images#14748), between those two runs. Both workflows
already carry "Free up disk space" steps, so these jobs were running
close to the limit before the migration.

Fix: install CPU-only torch from download.pytorch.org, then build with
--no-build-isolation so the build reuses it instead of resolving torch
a second time. --no-build-isolation needs the build dependencies present
up front, hence the explicit setuptools/wheel/numpy install; the
`setuptools<81` bound matches the pin already in install_requires.

This loses no coverage: BUILD_CUDA_EXT keys off CUDA_HOME/ROCM_HOME,
which is unset on these runners, so the CUDA sources were never compiled
here. tests/smoke_tests/Dockerfile already installs torch this way.

macOS keeps the plain `pip install -e .` unchanged: its PyPI wheels are
already CPU-only, and leaving it alone keeps the default PEP 517
build-isolation path under test somewhere in the matrix.
The shipped puffer_drive.yaml sets policy.fused_slot_encoder=true, and
DriveBackbone raises ImportError when that is set without triton. Two CI
jobs build a policy straight from the shipped config, so Unit Tests and
Notebook Execution have been failing on every pull request:

  ImportError: fused_slot_encoder=true requires triton

The smoke and eval suites already pin the flag off for exactly this reason;
these two call sites were simply missed. test_action_selection only exercises
the discrete-to-continuous action table, which the fused kernel has no part
in, so it pins the flag off outright. The inference notebook keeps the kernel
whenever it has a CUDA device and drops it otherwise, which is also what a
reader running the notebook on a laptop needs.

Neither change touches the training path or the kernel itself: with triton
and a GPU present, both call sites behave exactly as before.
The two instruction files are byte-identical except for one line, and that line is a contradiction rather than a Claude-specific tweak. AGENTS.md asked for comments that explain invariants; CLAUDE.md banned them by default with a hard one-line cap and an explicit list of banned forms.

Agents load one file or the other depending on the tool, so the same codebase was being written to two different standards, and neither copy said the other existed. This adopts the stricter rule in both, because it is the one written for this codebase and the one the recent code follows.
Selecting the shipped carla_cosim benchmark crashed with UnboundLocalError:
the cosim branch of load_benchmark_config appended to resolved_benchmarks,
which is not assigned until after the loop it sits in. The parsed cosim
entries now accumulate in their own list and are appended to the normalized
replay benchmarks, so a selection may mix both kinds.

The bug is the kind a linter finds for free, so the ruff hook that has been
commented out since it was added is enabled, scoped to exactly that class of
defect: names that do not resolve and imports that are never used. The repo
has 104 other violations under ruff's full default set, most of them E402
import placement, and fixing those is neither this change nor obviously
desirable; a deliberately narrow select keeps the hook green today and lets
the set widen later alongside the fixes a wider set would demand.

Enabling it removed eight genuinely unused imports and required noqa markers
on the four import-availability probes that exist precisely to be unused.
tests/unit_tests/test_terminate_on_goal.py called pytest.main without
importing pytest, so its __main__ entry point had never worked.
A README or docstring change currently runs the full matrix -- unit, C, notebooks, a 30-minute Docker smoke train and the perf run -- none of which can observe a Markdown edit. On a fork whose process documentation changes often this is the single largest source of runner time, and it is pure waste.

A changes job diffs the PR against the merge base and votes code=false only when every changed path is recognised documentation (*.md, docs/, .github/ISSUE_TEMPLATE/). Anything unrecognised, and every push build, votes 	rue, so the failure direction is always 'run the tests'. The five heavy jobs gain
eeds: changes and an if:.

Job-level if: rather than workflow-level paths-ignore is deliberate: a workflow skipped by path filtering leaves its required checks pending forever and blocks the merge, whereas a skipped job is reported to branch protection as passing. No third-party action is used; the detection is five lines of git.
@jdsika jdsika added do-not-mirror Fork-only change; must never be sent to Emerge-Lab/PufferDrive do-not-merge Never use the merge button; applied by fixup (3.3) or never merged at all labels Nov 12, 2026
Every documented workflow opens with git fetch origin && git fetch upstream, and forkctl status, check-view and delete-mirror reach the network themselves. When the SSH path to GitHub is blocked these block indefinitely instead of failing, so the stall reads as a hung tool. It is easy to hit on Windows, where the make targets push people into WSL and WSL often cannot reach GitHub over SSH even though Windows can.

Add a troubleshooting section naming the symptom, the one-line test that tells the two sides apart, and which subcommands sit on each side of the split.
@jdsika
jdsika force-pushed the chore/wsl-network-note branch from 1ccb477 to 9aaccf3 Compare November 12, 2026 09:17
@jdsika jdsika changed the title [do-not-mirror][do-not-merge] chore(fork): document the WSL network hang that stalls forkctl [do-not-mirror][do-not-merge] chore(fork): document the SSH stall that blocks fetches and forkctl Nov 12, 2026
@jdsika

jdsika commented Nov 12, 2026

Copy link
Copy Markdown
Author

Review — Track B (independent reviewer, fresh context)

Verdict: request changes — one blocking factual error; the section described behaviour forkctl does not have.

Track B confirmations

  1. Scope matches the stated idea — yes. One file, one commit, CONTRIBUTING.md +19 lines, nothing else.
  2. Claimed commands were really run — yes. forkctl check-commit --chore reproduces PASS; pre-commit green in CI (32s).
  3. Really not sim-affecting — yes. forkctl track reports B; Detect code changes correctly skipped the simulator suites.

Gate 1 — all required checks green on 1ccb477.

Blocking — the opening paragraph

"doctor, status and mirror all talk to GitHub before they print anything"

Three inaccuracies, each checked against .github/fork/forkctl.py:

  • mirror makes no network call at all. cmd_mirror (L1161-1249) works purely from already-fetched local refs: require_refs, cherry-pick, git branch -f. There is no push.
  • doctor's only network call is gh pr list (L1544) — HTTPS through gh, not SSH. A blocked SSH path, which the section names as the cause, would not hang it.
  • Neither hangs silently. Report.__init__ prints == <title> on construction (L207-208); status prints the whole divergence report and stack listing before its first ls-remote (L1442-1461).

What genuinely blocks on SSH is the git fetch origin && git fetch upstream that the process mandates before doctor (§4.1) and before mirror (§6), plus status's ls-remote and check-view / delete-mirror.

Non-blocking

  • The closing enumeration omitted check-view and delete-mirror, which do touch the network (L1305-1309, L1347). If the section enumerates, it should be complete.
  • Tooling gap, raised once rather than annotated: nothing in forkctl tells you which subcommands reach the network — the doc exists to compensate. Marking it in each subparser's help=, or printing contacting GitHub… before the first gh/ls-remote, would make the section largely redundant and would not drift the way a hand-maintained list does. Not a condition of this PR.

Checked and fine

  • Placement in the three-document split — correct. Human environment orientation, not process law and not a procedure: it does not belong in auto-loaded .github/copilot-instructions.md, and is too situational for PROCESS.md. Directly after the first-clone forkctl doctor block is the right anchor.
  • Diagnostic command — git ls-remote origin -h refs/heads/integration/3.0 is valid (-h = --heads), ran clean on git 2.39.1.windows.1 returning 351531d8… refs/heads/integration/3.0, exit 0. origin is an SSH remote, so it genuinely exercises the path the section blames, and it does distinguish the two shells.
  • Offline-commands claim — accurate. check-commit, check-stack, track contain no fetch/push/ls-remote/gh call anywhere in their paths.
  • HTTPS alternative — sound for the git transport (does not help gh, which is already HTTPS and unaffected either way).
  • PR shape — exactly one commit; [do-not-mirror][do-not-merge] title prefix; both labels; still a draft; base integration/3.0; §3.3 disposition stated in the body.
  • Gate 2 — correctly declared not applicable; Fork gates passed with the chore(fork): exemption.
  • No secrets or internal identifiers; Conventional Commit subject, 61 chars, no authorship trailer.

Once the opening paragraph describes what forkctl actually does, this is a useful addition.

@jdsika

jdsika commented Nov 12, 2026

Copy link
Copy Markdown
Author

Blocking finding fixed in 9aaccf3, and I confirmed the reviewer's reading of the source first: cmd_mirror ends at L1249 with no network call, and upstream_prs_for (L1252) is a separate helper used by status/delete-mirror.

The section now says that the stall comes from the mandated git fetch plus status/check-view/delete-mirror, notes that status stalls after its local report rather than before it, puts mirror on the offline side, and states that doctor reaches GitHub over HTTPS through gh and therefore survives a broken SSH path. The non-blocking enumeration gap is fixed in the same edit. The title changed to match.

The tooling suggestion is a good one and deliberately not taken here: a contacting GitHub… marker touches forkctl.py and its test suite, which is a different idea from a troubleshooting note. Worth its own chore PR.

@jdsika

jdsika commented Nov 12, 2026

Copy link
Copy Markdown
Author

Re-review — Track B (independent reviewer, fresh context) — 9aaccf346

Verdict: approve.

Blocking finding — resolved

Every false claim is gone and each replacement checks out against .github/fork/forkctl.py (unchanged by this PR; the diff is CONTRIBUTING.md only, +28):

  • Cause correctly reattributed. git fetch origin && git fetch upstream really is the opening line of the "Working on a feature" block, and status, check-view, delete-mirror really do reach the network themselves — status L1461/L1472, check-view L1347/L1368, delete-mirror L1305-1309 plus upstream_prs_for L1258.
  • Partial-output symptom is right, and the clue is well chosen. A live run matches exactly: status prints the gap counts, stack listing and stale-commit verdict, then calls ls-remote; the mirror branches on origin: block (L1494) is the only thing after it. "Absence of the trailing mirror section" is genuinely diagnostic, not a hand-wave.
  • mirror correctly moved to the offline side. cmd_mirror (L1161-1248) touches only already-fetched refs and ends in git branch -f; there is no push anywhere in it, and --no-publish only suppresses moving the local branch.
  • doctor paragraph is accurate. Its sole network call is gh pr list (L1544) over the HTTPS API, so an SSH-only block leaves it working. Giving it its own sentence rather than forcing it into either list is the right call — it is genuinely neither.

Non-blocking enumeration gap — resolved

The network list is now exactly the set that exists in the tool, and the offline list is complete: verify_stack sits at L921 and every network call site in the file is at or below L1258, so check-commit, check-stack, track, fork-only and mirror are all provably offline.

Deferral of the tooling suggestion — accepted

Correct application of one-idea-per-commit. A help= annotation or a contacting GitHub… marker touches forkctl.py and test_forkctl.py and is its own chore. It was never a condition of this PR.

Re-checked and fine

  • Heading now matches the real symptom and is the phrase someone would actually search for.
  • Diagnostic command unchanged and still correct: git ls-remote origin -h refs/heads/integration/3.0 exits 0 and returns the ref, and origin is an SSH remote, so it genuinely exercises the path being blamed.
  • PR shape — exactly one commit, one file; [do-not-mirror][do-not-merge] title prefix; both labels; still a draft; subject 67 chars, well-formed Conventional Commit; no authorship trailer.
  • Gate 1 green on the final commit: all six Python matrix jobs, pre-commit, Fork gates and Detect code changes pass; the simulator jobs correctly skipped as documentation-only. Fork gates passing on 9aaccf346 is stronger evidence than a local check-commit --chore, and agrees with it.
  • Track still B, placement still right, no secrets or internal identifiers.

Two nuances checked and explicitly not raised as findings: copilot-instructions.md §4.1 runs forkctl doctor before its fetch, whereas this section says the fetch comes "just before" — within CONTRIBUTING.md's own reading order that is true, and either way the experience is identical (doctor succeeds, the fetch stalls). And the partial-output tell relies on forkctl writing to a terminal; piped into a file, Python block-buffers and nothing appears. Neither changes the advice.

Ready for the §3.3 fold.

@jdsika

jdsika commented Nov 12, 2026

Copy link
Copy Markdown
Author

Folded into the base chore(fork): commit per §3.3 and closed unmerged.

  • fork base: c509ccd52 → 6919dc3b9
  • integration/3.0: 351531d89 → aacc53de7
  • fork-only/3.0 republished to the new base in the same atomic push

git diff --name-only 351531d89 HEAD returns only CONTRIBUTING.md, so the product tree is byte-identical to the tip CI was already green on. forkctl check-stack passes all 13 invariants after the fold, including every feature commit on the stack is upstream-clean (5 checked).

@jdsika jdsika closed this Nov 12, 2026
@jdsika
jdsika deleted the chore/wsl-network-note branch November 12, 2026 09:24
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do-not-merge Never use the merge button; applied by fixup (3.3) or never merged at all do-not-mirror Fork-only change; must never be sent to Emerge-Lab/PufferDrive

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant