(Fav-164880) Allow running unit tests and e2e tests in parallel to enable agents to run more autonomously - #40
Open
OleksandrChekhovskyi wants to merge 7 commits into
Open
(Fav-164880) Allow running unit tests and e2e tests in parallel to enable agents to run more autonomously#40OleksandrChekhovskyi wants to merge 7 commits into
OleksandrChekhovskyi wants to merge 7 commits into
Conversation
A linker cache key is "<prefix>_<suffix>", and only the newest suffix per prefix is ever read. The write path removed older entries by passing a "<prefix>_*.cache" pattern to files.rm_recursive, but since 3.3 that call stats the string instead of expanding it, so nothing was removed and a checkout rebuilt all day accumulated gigabytes of dead entries. List the directory and unlink the superseded entries explicitly instead.
_loadLocalPackage decided in one long branch whether the isopack saved on disk could be used, loaded it if so, and compiled and saved it if not. Move that into _buildOrLoadLocalPackage, so that a later change can put the work under a lock without the diff mixing the move with the locking. No behavior change.
Replace the directory-based .npm lock with SQLite's exclusive transaction: advisory locking that the kernel releases when the process ends, so a killed build leaves nothing to detect, clean up or remove wrongly. Locks are named after the identity of the directory they stand for rather than the path that reached it, and live away from the resource they guard. Take such a lock around isopacket building, so applications sharing a warehouse build isopackets once between them, and give each checkout its own isopacket directory, recorded in the buildinfo, since isopackets are built from a checkout's own sources. The isopack cache takes one lock per package around deciding whether the entry on disk can be used and replacing it; each build holds one lock at a time, so builds cannot wait on each other.
renameDirAlmostAtomically renames the directory it replaces aside and deletes that copy afterwards, so a build killed in between strands it, one copy per interrupted build. Collect the leftover copies of a directory before renaming another one aside, and remove them along with it. Removal is best effort, since another build replacing the same directory may be removing the same copy.
The cache keeps one entry per package name, so two builds that disagree about its configuration take turns rebuilding the same entry on every build, and nothing says so. Report once per build which package was rebuilt and why, only for the reasons a build does not expect; an entry that is missing or whose sources changed stays quiet.
An isopack's npm directories are almost all of its weight, and every application using a plugin kept its own copy of the same directories. An isopack in a local cache is rebuilt from sources beside it and never moves, so symlink its npm directories to what it was built from instead of copying them, which takes almost all of that weight out of the cache. An isopack that is published or unpacked elsewhere still carries its own copy. An entry that is itself a symlink to outside the directory, like a file: dependency living in its own checkout, is still dereferenced and copied: linking it would leave the modules under it resolving their dependencies from the checkout's own directory, which does not hold them.
…urces Each application kept its own isopack cache, so applications built from the same sources built and stored every package once apiece. Take METEOR_ISOPACK_CACHE_DIR to name a cache for them to share, per architecture, defaulting to the project's own local directory as before. An entry is reused only by a build that resolves the same sources for it, and builds take a lock per package, so they can share a cache while running at once; a build that waits then gets what the other built instead of building it again. Wiping the cache takes the same locks and leaves the cache directory itself in place, since locks are named after the containing directory. It also skips another build's dot-prefixed temporary directories, which are locked under the name of the package they will become rather than their own, and so would otherwise be taken out from under the build writing them.
OleksandrChekhovskyi
force-pushed
the
fav-164880
branch
from
September 2, 2026 07:58
2460486 to
d26713e
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.