From 4991ee7f6db5b31fca8506d7c8e2d3bc6b2ecd57 Mon Sep 17 00:00:00 2001 From: "claude[bot]" <3247365+butterstack[bot]@users.noreply.github.com> Date: Mon, 21 Sep 2026 19:32:02 +0000 Subject: [PATCH 1/4] perforce: mirror the -Mj stdout-not-stderr agent instruction The private repo's agent instructions for perforce learned that tagged-JSON mode (p4 -Mj) writes every record, including errors, to stdout and leaves stderr empty - so an agent that classifies failures by reading stderr gets nothing but a bare exit status. Mirrors that guidance into the public perforce agent instructions verbatim, including the parse recipe, the two traps in the JSON stream, and a new row in the failure-class table for the "exit status N and nothing else" symptom. This is an instruction-file change (agents/perforce.md), not a LEARNINGS.md entry, so it is mirrored per DISTILLING.md section (f) rather than compressed: no step or table row was dropped. The private source names an internal connector by name; this generalizes it to "a real webhook-ingestion connector," consistent with how the same validation story is already described elsewhere in this file's LEARNINGS.md. Co-Authored-By: Claude Fable 5 --- perforce/agents/perforce.md | 19 +++++++++++++++++++ 1 file changed, 19 insertions(+) diff --git a/perforce/agents/perforce.md b/perforce/agents/perforce.md index 602abba..156ef3b 100644 --- a/perforce/agents/perforce.md +++ b/perforce/agents/perforce.md @@ -95,6 +95,24 @@ shell environment: `p4 login` themselves in their terminal, then re-check `p4 login -s`. Never run `p4 login -p` (prints a reusable ticket ≈ credential), never read `P4TICKETS`/ `p4 tickets` into context, and don't suggest `p4 login -a` (all-hosts ticket). +- **`-Mj` writes errors to STDOUT, not stderr.** In tagged-JSON mode (`p4 -Mj`, + and `-Mj -ztag`) the client emits **every** record - including error records - + as JSON on stdout. stderr is empty. Read stderr alone and a failed command + gives you nothing but the shell's `exit status 1`, which names no cause at + all. Parse stdout instead: one JSON object per line, and the human message is + the `data` field of the first record whose `severity` is >= 3 (p4's scale is + 0 empty, 1 info, 2 warning, 3 failed, 4 fatal). Two traps in that stream - a + failing command often emits **successful** records before the error, so the + first record is not the error record; and `severity` appears as a number in + some versions and a quoted string in others, so parse both. Keep reading + stderr as a fallback: a connection-level failure never gets far enough to + produce a record. Without `-Mj` the behaviour is the ordinary one and errors + do go to stderr, so this applies specifically to the mode you reach for when + you want machine-readable output. This cost four separate misdiagnoses in a + real webhook-ingestion connector (a missing trust file and an expired ticket, + then two changelist sync failures) before it was found: every one of them + reported as `perforce: exit status 1`, and the table below was unusable + because the message it keys on never arrived. - **Distinguish failure classes** - retry/backoff only *network/server* failures, never *user* failures: @@ -113,6 +131,7 @@ shell environment: | `Client ... unknown` / `must create client` | workspace missing/misconfigured | user → stop | | `You don't have permission` | protections | user → stop (check `p4 protects -m`) | | (unfamiliar wording) | a **broker** may be rewriting messages | show the user the raw output | + | `exit status N` and nothing else | you are running `-Mj` and reading stderr; the real record is on stdout | **not** a p4 failure class at all - fix the caller (see above) before classifying | ## The read-only checkout model - game-critical From 483c223ef04babba1d3c9733568e4162171153c7 Mon Sep 17 00:00:00 2001 From: "claude[bot]" <3247365+butterstack[bot]@users.noreply.github.com> Date: Mon, 28 Sep 2026 20:36:46 +0000 Subject: [PATCH 2/4] unity: distill the schtasks OpenSSH fix and Hub 3.x headless-install learnings The private repo validated the fix for the OpenSSH process-tree-kill trap already noted in this file: a one-shot scheduled task launched via schtasks runs outside the SSH session's process tree and survives it, verified end to end with a cold import and a WebGL build on Unity 6000.3.22f1. Also records the two mechanisms that look like fixes and are not (cmd start hangs the session, Start-Process without -Wait dies at session exit anyway), Hub 3.53's headless install behavior (CDN rate cap, per-component progress text that isn't proof of a finished install, real cold-import/build timings), and the narrow provider list on Hub's "Add project from repository" flow. Mirrored (not compressed, per DISTILLING.md section f): unity-build SKILL.md gains section 8 with the wrapper/schtasks recipe verbatim, and agents/unity.md gains the matching symptom-table row. The private source's numbered LEARNINGS.md cross-reference (B8/B10/B12) is rewritten to the public file's unnumbered convention (LEARNINGS.md paragraph B), consistent with how unity-pipeline/SKILL.md already references this file. Sanitization: the private source names the rig `beast` in the Hub-install example; genericized to "a wired gigabit test machine," consistent with how the existing "primary test rig" bullet in this file already anonymizes the same machine. Final grep sweep on the changed lines: zero hits. No secrets or credential values were present in the source material. Excluded from this pass: the private repo also added two entirely new plugins, teamcity and buildkite, since the last distillation. Per the rationale already recorded on this PR, bringing a new plugin into this repo needs more than the skills/agents/commands/LEARNINGS files this job is scoped to (plugin.json, hooks.json, the guard script, README, and marketplace registration all need to land together, plus a security review of the guard scripts) - recommend a separate, deliberate distillation pass for those once reviewed. Line counts: unity/LEARNINGS.md 170 -> 199, unity/agents/unity.md 231 -> 232, unity/skills/unity-build/SKILL.md 260 -> 306. Co-Authored-By: Claude Fable 5 --- unity/LEARNINGS.md | 33 ++++++++++++++++++++-- unity/agents/unity.md | 1 + unity/skills/unity-build/SKILL.md | 46 +++++++++++++++++++++++++++++++ 3 files changed, 78 insertions(+), 2 deletions(-) diff --git a/unity/LEARNINGS.md b/unity/LEARNINGS.md index 737cf89..fadb170 100644 --- a/unity/LEARNINGS.md +++ b/unity/LEARNINGS.md @@ -126,11 +126,40 @@ No secrets - reference credential *locations*, never paste them. calls against it work fine, because Unity's live-control server is a loopback TCP port that doesn't care which session issued the connection. ⤳skill: on a build node reached only over SSH, don't propose the live-control path without - a persistence mechanism for the Editor itself. + a persistence mechanism for the Editor itself. **Validated 2026-09-23:** the + mechanism that actually works is a one-shot scheduled task - a `.cmd` wrapper + that runs the Editor and records its exit code to a marker file, launched with + `schtasks /create ... /sc once /st 00:00 /f && schtasks /run`, which runs + outside the SSH session's process tree; poll the marker and the log from a + fresh connection, then `schtasks /delete`. Verified end to end on Unity + `6000.3.22f1`: a cold import and a WebGL `-executeMethod` build both survived + the SSH session that started them. Two mechanisms that look like a fix are + not: `cmd /c start "" /min ` hangs the SSH session itself and never + launches anything, and `Start-Process` without `-Wait` returns a PID that + dies at session exit exactly like the inline case. Don't point the real run's + log at stdout (`-logFile -`) either - it's fine for a quick startup probe, but + if the reading pipe closes the Editor dies with it. ⤳skill: `unity-build` §8. - **The Hub's "add project" folder picker wants the parent directory, not the project directory** - it silently rejects the project folder itself and scans for children. Registering a project directly by path sidesteps the picker - entirely. + entirely. The same Hub's **"Add project from repository" flow only recognizes + three providers (Unity Version Control, GitHub, GitLab)** and clones the + repository root, not a subfolder - a poor fit for a project that lives at + `/unity/` rather than the repo root. Use "Add project from disk" (or + `unity projects add `) against an existing clone instead. +- **A synchronous headless Hub install undersells its own uncertainty, and its + progress text oversells completeness.** `Unity Hub.exe -- --headless install + --version --changeset -m ` blocks until done - about 9 + minutes for an Editor plus two modules on a wired gigabit test machine - but + Unity's CDN caps the download at roughly 12 MB/s regardless of link speed, so + don't infer link health from it. The "installed successfully" lines are + printed per component, not proof of anything: verify by listing + `Editor\Data\PlaybackEngines\` for the expected module folders rather than + trusting the summary (the progress output also carries raw ANSI cursor codes + that need stripping before grepping it). Once installed, real timings beat + the "up to an hour" folklore: a project with 328 scripts and 644 other assets + cold-imported in about a minute and a WebGL build finished in under 3 - + budget timeouts around that and treat 20+ minutes as a stall, not normal. - **Disk, not CPU, was the binding constraint on the primary test rig** - a single volume with limited free space before the install, consumed significantly by an Editor plus a probe project. A second machine on hand had much more disk but diff --git a/unity/agents/unity.md b/unity/agents/unity.md index a72bb44..b34fbeb 100644 --- a/unity/agents/unity.md +++ b/unity/agents/unity.md @@ -163,6 +163,7 @@ project-affecting decisions, not tooling conveniences - gate them. | Build/test fails to start; lock errors | another Editor/batchmode process holds the project (`Temp/UnityLockfile`) | environment -> gate 1: ask the user to close it, wait; never kill it | | CLI install fetches nothing / `latest.json` 404 | there is no stable channel - beta only | environment -> pin `UNITY_CLI_CHANNEL=beta` (`unity-cli` §2) | | Build works locally, build node fails with a missing-platform error | the target's module isn't installed for that Editor there | environment -> gated `unity install-modules` / `unity editors module` for that version | +| Editor started over SSH exits instantly with an empty or missing log | Windows OpenSSH killed the process tree at session exit | environment -> launch via a one-shot scheduled task, `unity-build` §8 | | "This project was created with a different version of the editor" (or silent reserialization) | Editor/project version mismatch | version -> confirm intent; a forward open+save is one-way - a named decision, not a default | ## How to reason about a request diff --git a/unity/skills/unity-build/SKILL.md b/unity/skills/unity-build/SKILL.md index f247f2a..bd0cb58 100644 --- a/unity/skills/unity-build/SKILL.md +++ b/unity/skills/unity-build/SKILL.md @@ -249,6 +249,52 @@ if any, test pass/fail counts, warnings - not "exit code 0". State the cost in the confirmation prompt; if a previous log exists, its timestamps beat these estimates. +## 8. Build node reached over Windows OpenSSH + +A build/test/run driven over Windows OpenSSH has one more failure mode on top +of everything above: **OpenSSH kills the whole process tree at session exit.** +A long Editor run started inline - `ssh node 'unity build ...'` or a raw +`Unity.exe -batchmode ...` - dies the instant the SSH session ends, with a +zero-byte or missing log. This reads exactly like "Unity failed to start"; it +is not - the Editor never got the chance to fail, it was killed. + +Two things that look like a fix and are not - don't propose either: + +- `cmd /c start "" /min .cmd` - hangs the SSH session itself and + never launches anything. +- `Start-Process` without `-Wait` - returns a PID immediately, but that + process dies at session exit exactly like the inline case (no log written). + +The pattern that works is a one-shot scheduled task: a `.cmd` wrapper that +runs the Editor and records its exit code to a marker file, launched via +`schtasks` so it runs outside the SSH session's process tree, polled from a +fresh connection, and cleaned up after: + +```bat +:: C:\Users\\.cmd +"" -batchmode -nographics -quit ^ + -projectPath -executeMethod Builder.PerformBuild ^ + -logFile C:\Users\\.log +echo EXITCODE=%ERRORLEVEL% > C:\Users\\.exit +``` + +```sh +schtasks /create /tn /tr C:\Users\\.cmd /sc once /st 00:00 /f && schtasks /run /tn +``` + +The `WARNING: Task may not run because /ST is earlier than current time` line +is harmless - `/run` fires the task immediately regardless. Poll the `.exit` +marker and the `-logFile` from a fresh SSH connection (or from WSL via +`/mnt/c/...`), and `schtasks /delete /tn /f` once done. + +This changes *how* the job runs, not what "done" means: still verify by +parsing the log and checking the artifact, never by trusting the marker +file's exit code alone (§6) - the exit-0 trap (§2) applies exactly as it does +to any other batchmode invocation. Background and evidence: `LEARNINGS.md` +§B (the process-tree kill, the two mechanisms that don't work, why CLI calls +against an already-running Editor don't need this - only starting the Editor +does - and the verified end-to-end sequence this section distills). + ## Validation status Command surface, flags, examples, and the licensing/exit-code behavior in this From a7ebf85459f5c9283b5dbd6f6df76c5d6a886b2b Mon Sep 17 00:00:00 2001 From: "claude[bot]" <3247365+butterstack[bot]@users.noreply.github.com> Date: Mon, 5 Oct 2026 21:38:33 +0000 Subject: [PATCH 3/4] perforce, unreal, lore: distill the empty-replacement-server, cloud-worker, and workspace/firewall learnings The private repo validated three new findings since the last distillation pass landed on this branch: standing up a deliberately empty Helix Core replacement server in a container (change-counter collisions with existing backend records, daemon-uid file ownership, a trust entry keyed by server address, and a setup script that can't read `*_FILE` secrets under an environment-stripping runner), moving an Unreal build agent from a home PC to a cloud Windows worker (streaming the engine install as `tar.xz`, `-warp` for GPU-less screenshots, writable-engine-tree permissions, and the .NET Framework SDK/VS Build Tools modify-mode trap), and porting scripts into a Lore repo plus standing up a Lore server on a cloud host (the `--scan` gap for externally-copied files, a firewall-reachability false negative, and GitHub-mirror lag). Excluded two new private-repo plugins, teamcity and buildkite, consistent with the rationale already on this PR: a new plugin needs its full scaffolding (plugin.json, hooks, guard script, README, marketplace registration, security review) to land together, not a partial copy through the routine learnings pass. Rewrote one dangling cross-reference: the private unreal entry pointed at "the Buildkite learnings" for keeping credentials out of the bundle; since buildkite isn't published here yet, that became a self-contained instruction instead. Line counts: lore/LEARNINGS.md 44 -> 56, perforce/LEARNINGS.md 87 -> 108, unreal/LEARNINGS.md 179 -> 204. Final grep sweep on the diff: zero unexplained hits. No secrets or credential values were present in the source material. Co-Authored-By: Claude Fable 5 --- lore/LEARNINGS.md | 12 ++++++++++++ perforce/LEARNINGS.md | 21 +++++++++++++++++++++ unreal/LEARNINGS.md | 25 +++++++++++++++++++++++++ 3 files changed, 58 insertions(+) diff --git a/lore/LEARNINGS.md b/lore/LEARNINGS.md index b3299d4..8bf87eb 100644 --- a/lore/LEARNINGS.md +++ b/lore/LEARNINGS.md @@ -11,6 +11,18 @@ reference credential *locations*, never paste them. --- +## 2026-10-02 - Workspace and firewall notes `[integration]` + +Learned porting scripts into a Lore repo and standing up a Lore server on a cloud host. + +- **`--scan` is also needed for files copied in from outside, not just drifted edits.** + `lore status`/`lore stage` only see tracked dirty flags; run `lore status --scan + ` (or `lore stage --scan`) first. +- **A TCP test from an allowlisted address makes a firewalled port look public.** Read + the firewall rules, not reachability from your own machine. +- **The GitHub mirror can lag the source of truth.** Base pipeline edits on the Lore + revision, not the mirror's copy. + ## 2026-07-17 - live validation against loreserver 0.8.5 `[integration]` `[skill]` Learned standing up a real `loreserver 0.8.5` and driving two clones through the diff --git a/perforce/LEARNINGS.md b/perforce/LEARNINGS.md index 57f4b00..afb714d 100644 --- a/perforce/LEARNINGS.md +++ b/perforce/LEARNINGS.md @@ -12,6 +12,27 @@ them. --- +## 2026-10-02 - Standing up an empty replacement server `[integration]` + +Learned replacing a Helix Core server with a new, deliberately empty one in a +container, while CI and an observability backend still held records from the old one. + +- **An empty replacement restarts change numbers at 1, colliding with anything keyed + by change number** - the backend held records for changes 1-208, which a reissued + change 1 would overwrite. Raise the counter before the first submit (`p4 counter -f + change `, well above the old max); do the same for CI build counters unless + restored from backup. +- **A file the daemon reads must be owned by the daemon's uid, not root with mode + 0600.** A root-owned ticket file and `.p4trust` gave "permission denied" at startup + for a non-root container process; `chown` them to the service uid. +- **The trust entry is keyed by the server's resolved address, so it doesn't travel.** + A container's address differs on a new host - regenerate it there and ship the file: + `P4TRUST=/tmp/t p4 -p ssl::1666 trust -y`. +- **A setup script reading passwords from `*_FILE` env vars can't run under a runner + that strips the environment** (AWS SSM Run Command is one) - prepend `export` lines + to a throwaway copy and delete it afterwards; never pass secrets as arguments, since + the runner stores the command text. + ## 2026-09-14 - the typemap is one server-wide table, and a p4d container image is a second writer `[integration]` Learned running a Git->Perforce mirror against a p4d that ships as a Docker image and diff --git a/unreal/LEARNINGS.md b/unreal/LEARNINGS.md index 2cc9a65..ad20685 100644 --- a/unreal/LEARNINGS.md +++ b/unreal/LEARNINGS.md @@ -177,3 +177,28 @@ No secrets - reference credential *locations*, never paste them. binder to catch a missing value. And `ssh -n` redirects stdin from `/dev/null`, which silently swallows a heredoc meant to feed a remote `bash -s` - the remote script runs with no input and nothing reports an error. +16. **Cloud Windows build worker, moved from an always-on home PC:** + - **Ship the engine install as a streamed `tar.xz`.** A Launcher install is ~10 + GB; copy it once from a known-good machine and pull it onto new workers rather + than reinstalling through the Launcher or syncing from source. 7-Zip cannot + stream `-t7z` through a pipe, so the upload path is `tar | xz` in parts. Pin the + engine version/changelist in the bundle manifest, publish a checksum alongside + it, and keep credentials out of the bundle entirely. + - **Offscreen screenshots without a GPU need `-warp`.** A GPU-less Windows host + logs "Failed to choose a D3D12 Adapter" and skips the WARP software adapter + unless launched with `-warp`; without it the screenshot step silently produced + zero PNGs while the rest of the build passed. Poll for the output files up to a + bounded time instead of sleeping - without a GPU they appear more slowly and + less predictably. Any step needing real rendering still needs an actual GPU + instance; plan for its quota (0 on a new cloud account) from the start. + - **A Launcher-installed engine copied to a new worker must be writable by the + agent user.** UAT writes under `Engine\Saved` and the DDC; a non-admin agent + user gets `UnauthorizedAccessException` only once the cook starts, well after + compile has passed, which makes it look like a late failure. Grant modify + rights on the engine tree once during bootstrap, not per job. + - **UBT needs the .NET Framework SDK.** A fresh VS Build Tools install missing + `Microsoft.Net.Component.4.8.SDK` makes UBT fail fast with a RulesError ("Could + not find NetFxSDK install dir") while loading `SwarmInterface` rules. Add the + component in the bootstrap; if adding it later, `setup.exe modify` can return 87 + from a non-interactive session - run the `vs_BuildTools.exe` bootstrapper in + modify mode instead. From 87cd4deb4d568311064a5b8743c0e5f3e26fc45f Mon Sep 17 00:00:00 2001 From: "claude[bot]" <3247365+butterstack[bot]@users.noreply.github.com> Date: Mon, 19 Oct 2026 19:59:21 +0000 Subject: [PATCH 4/4] lore: distill the git-to-Lore mirror, push-relay, and build-key learnings The private repo added a 2026-09-24 LEARNINGS.md entry from mirroring a 107-commit git repository into Lore across two branches and relaying pushes from a live loreserver 0.9.0 via the notification stream, plus two short 2026-09-25 addenda on push recovery and using the revision signature instead of the revision number as a build key. Distilled five of the entry's eight findings (per-branch revision numbering and the matching per-branch relay-state fix, resolving a push notification's branch id, lore sync versus branch reset for the local pointer, the no-import/no-author-override mirror recipe, and the empty-commit-produces-no- revision gap) plus both addenda. Cut the relay-staleness and blank-submitter findings: both are real but narrower to this relay's own architecture and less load-bearing than the other five, which matches the earlier pass's compression precedent on this file. The source commit also touched buildkite/LEARNINGS.md, which is excluded for the same reason already recorded on this PR: buildkite is not a published plugin in this repo yet. Sanitization: no internal identifiers, personal names, or hostnames were present in the source material; the private entry's reference to the mirrored repo as an "Unreal game repository" was generalized to "git repository" since the engine detail doesn't change the lesson. Grep sweep on the file: zero hits. No secrets or credential values were present in the source material. Line count: lore/LEARNINGS.md 56 -> 100. Co-Authored-By: Claude Fable 5 --- lore/LEARNINGS.md | 44 ++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 44 insertions(+) diff --git a/lore/LEARNINGS.md b/lore/LEARNINGS.md index 8bf87eb..eaf2023 100644 --- a/lore/LEARNINGS.md +++ b/lore/LEARNINGS.md @@ -23,6 +23,35 @@ Learned porting scripts into a Lore repo and standing up a Lore server on a clou - **The GitHub mirror can lag the source of truth.** Base pipeline edits on the Lore revision, not the mirror's copy. +## 2026-09-24 - git-to-Lore mirror and push relay on loreserver 0.9.0 `[integration]` + +Learned mirroring a 107-commit git repository into Lore across two branches, and +relaying pushes from a live `loreserver 0.9.0` into our own systems via the +notification stream. + +- **Revision numbers are per branch, not global.** A second branch's replay started + renumbering from its parent's revision, so a consumer keying on the bare number + alone conflated two branches that happened to share a number, and a relay that + kept one global "last seen" cursor silently treated an entire branch's revisions + as already-seen. Namespace non-default branches, keep relay state per branch, and + dedupe any history two branches inherited by content hash rather than by number. +- **A push notification's branch field is a 32-hex id, not a name.** Resolve it via + `lore branch list` (returns id, name, and location) and skip - don't mark seen - + any revision whose branch doesn't resolve yet. +- **`lore sync ` moves the working copy, not the branch's local pointer.** + `lore branch reset --branch ` is what moves the pointer that `revision + history --branch` actually reads. +- **There is no import command, and `lore commit` has no author or date override.** + The mirror recipe that worked: one revision per git commit, the tree extracted + fresh via `git archive | tar -x`, staged with `lore stage --scan .` (a bare `lore + stage .` staged nothing), and the git provenance carried as message trailers + (`Git-Commit`, `Git-Author`, `Git-Date`, `Git-Parents`) for downstream + SHA-to-revision lookups. +- **A git commit with no tree change produces no Lore revision at all.** 107 git + commits produced only 105 revisions (two empty merge commits), so a + SHA-to-revision lookup needs a fallback path rather than assuming every SHA maps + to one. + ## 2026-07-17 - live validation against loreserver 0.8.5 `[integration]` `[skill]` Learned standing up a real `loreserver 0.8.5` and driving two clones through the @@ -54,3 +83,18 @@ full stage → commit → push → sync → branch loop. `lore repository create`), no `lore log` (use `lore history`), no blame, no rebase, no squash, and locks are advisory - `lore lock acquire` does not block another user's push. + +### Push refused when the remote is ahead (2026-09-25) + +`lore push` fails with a branch error once the remote has moved past your local +pointer, even if your working copy already has the files. Recovery: `lore branch +reset --branch main `, `lore sync `, re-apply the +change, `lore stage --scan .`, `lore commit`, `lore push`. There is no rebase - +the revision made on the stale pointer is simply abandoned. + +### Revision numbers are the wrong key for a build (2026-09-25) + +A CI build tied to a Lore revision should key on the 64-hex signature, not the +revision number - the number is per branch and repeats across branches, the +signature is global. A "commit" validator built for 7-40 hex characters needs +widening to 64.