Convert build parallelism to a more synchronous model - #3569
Open
peterebden wants to merge 13 commits into
Open
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This will (hopefully) make the parallelism easier to understand and avoids some of the weird edge cases we had before. The fan-out logic is now all in
src/plzand has been cut out ofsrc/core. The model there is quite a lot more synchronous - essentially it calls a function like Build or Test which builds all a target's dependencies, then that target, blocking until it's done.We can drop a few of the concepts we had before (like "parse mode", "resolving dependencies" and things like that) and we can lose the maps of "pending things" on the build state. It's simpler around things like subincludes because when we get there we just ask to build the thing, we don't have to ask it to parse then figure out at the end of that whether we should trigger things to build or not.
I ran the in-repo perf tests and they are actually faster than before. That's nice to see although it was not expected - it appears to be that we have fewer concurrently live goroutines because all the dependency resolution stuff is gone. I think it's a bit of a parse-only effect though.