bundle: add --jobs flag for parallel formula installation

[mvanhorn]

• Have you followed the guidelines in our Contributing document?
• Have you checked to ensure there aren't other open Pull Requests for the same change?
• Have you added an explanation of what your changes do and why you'd like us to include them?
• Have you written new tests (excluding integration tests) for your changes? Here's an example.
• Have you successfully run brew lgtm (style, typechecking and tests) with your changes locally?

---

• AI was used to generate or assist with generating this PR. Claude Code + Codex were used for initial scaffolding of the ParallelInstaller class. All code was manually reviewed, tested with brew lgtm, and iterated to match Homebrew conventions (Sorbet strict, concurrent-ruby patterns from download_queue.rb).

---

Summary

Add --jobs=N flag to brew bundle install that installs independent formulae in parallel using concurrent-ruby's FixedThreadPool. Dependent formulae wait for their dependencies to complete before installing. Non-formula entries (casks, extensions) still install sequentially.

Continues the trajectory of #18278 (concurrent downloads) and #21252 (parallel fetch) by parallelizing the install loop itself.

Changes

After review feedback from @MikeMcQuaid:

• New class: Extracted parallel install logic into Homebrew::Bundle::ParallelInstaller (was scattered inline in installer.rb)
• concurrent-ruby: Replaced raw Thread.new with Concurrent::FixedThreadPool + Concurrent::Promises.future_on, matching the pattern used in download_queue.rb
• Keg lock handling: ParallelInstaller identifies shared dependencies across Brewfile entries. Before installing a formula, it acquires FormulaLock on any dependency kegs shared with other parallel installs, preventing concurrent keg modifications
• Output: Status messages use a mutex to prevent interleaving across workers
• cmd/bundle.rb: --jobs=N flag, --jobs=auto caps at Etc.nprocessors (max 4), HOMEBREW_BUNDLE_JOBS env var
• Tests: 3 RSpec cases covering parallel independent formulae, serialized dependent formulae, and sequential fallback with --jobs=1

Design decisions

• Opt-in: Defaults to --jobs=1 (sequential), preserving current behavior
• Formula-only: Only formulae are parallelized. Casks can trigger GUI prompts, and extensions have their own dependency concerns
• Deadlock fallback: If the scheduler can't find ready entries (unexpected dependency cycle), remaining entries install sequentially
• Pool shutdown: ensure block calls pool.shutdown + pool.wait_for_termination so threads are always cleaned up

Benchmarks

Real hyperfine benchmarks with specific formulae coming in a follow-up comment.

This contribution was developed with AI assistance (Claude Code + Codex).

[MikeMcQuaid]

Hey @mvanhorn thanks for this.

A few notes in no particular order:

• please always ensure you've filled out our PR template when opening issues. if your AI is opening it, instruct them to do so.
• "10 formulae, 2 needing install: 22.3s" which formulae?
• "Estimated: ~12s for the 10-formula case" why estimated? hyperfine should be good to measure this.
• "Non-formula entries (casks, extensions) still install sequentially": IMO we should aim, not necessarily in this PR, to install all types in parallel

Looking at code now!

[MikeMcQuaid]

Looks good so far!

The thing that will definitely bite here unfortunately is locking. If you have two separate dependency trees that overlap: they cannot be handled in parallel due to keg locking.

It's unclear if the current code handles this or not; it's a little hard to follow because the low-level sync/mutex/lock operations are intertwined with the actual logic.

[mvanhorn]

Thanks for the review @MikeMcQuaid, good feedback across the board.

PR template: my mistake, updating the description with the checklist now.

Benchmarks: "estimated" isn't good enough, you're right. I'll run hyperfine against a real Brewfile and post actual numbers with the specific formulae listed.

concurrent-ruby: makes sense to use what's already in the codebase. I'll refactor to use it similar to how download_queue.rb works.

New class: agreed, the parallel scheduling logic is involved enough to warrant its own class rather than being scattered in installer.rb.

Re: dependency waiting (line 194) - yes, currently it only waits on dependencies that are also in the Brewfile. I need to verify whether that's sufficient or if we need to check the full dependency graph via brew deps.

Re: verbose output (line 221) - output from parallel workers goes through a mutex to prevent interleaving, but I need to verify whether brew install --verbose output from concurrent installs stays readable in practice.

Keg locking: this is the critical issue. If two formulae share an overlapping dependency being installed, keg locking could cause problems. I need to investigate how download_queue.rb handles this and whether concurrent-ruby gives better primitives. The current raw thread approach definitely needs more guardrails here.

Will push updates addressing the concurrent-ruby swap, class extraction, and keg locking investigation. Real benchmarks to follow.

[mvanhorn]

Pushed c711314 addressing all review feedback:

Class extraction: Parallel install logic moved from installer.rb into Homebrew::Bundle::ParallelInstaller (Library/Homebrew/bundle/parallel_installer.rb). The installer now delegates with ParallelInstaller.new(entries, jobs:, ...).run! when jobs > 1.

concurrent-ruby: Replaced raw Thread.new with Concurrent::FixedThreadPool + Concurrent::Promises.future_on, matching the pattern in download_queue.rb. Pool is shut down in an ensure block.

Keg locking: ParallelInstaller identifies dependencies shared by multiple Brewfile entries. Before installing a formula, it acquires FormulaLock on each shared dependency keg, preventing concurrent keg modifications. Locks are released after install completes. Formulae with no shared dependencies install fully in parallel.

Verbose output: Status messages (Installing X, Using X, error messages) go through a mutex to prevent interleaving. Subprocess output from brew install itself will naturally interleave in parallel mode, which is consistent with how download_queue.rb handles concurrent download output.

Benchmarks (hyperfine, 3 runs each, 4 independent formulae: tree, bat, fd, fzf):

Benchmark 1: sequential (jobs=1)
  Time (mean +/- s):  2.065 s +/- 0.441 s  [User: 1.213 s, System: 0.349 s]

Benchmark 2: parallel (jobs=2)
  Time (mean +/- s):  1.865 s +/- 0.101 s  [User: 1.095 s, System: 0.291 s]

Benchmark 3: parallel (jobs=4)
  Time (mean +/- s):  1.863 s +/- 0.080 s  [User: 1.088 s, System: 0.295 s]

Summary: parallel (jobs=4) 1.11x faster than sequential

The speedup is modest with 4 small formulae since install time per formula is short (~0.5s each). The improvement scales with formula count and individual install time. With all formulae already satisfied (no-op case), overhead is ~70ms from thread pool setup.

brew style, brew typecheck, and brew tests --only=bundle/installer all pass locally.

[MikeMcQuaid]

Benchmarks (hyperfine, 3 runs each, 4 independent formulae: tree, bat, fd, fzf):

Can we also get a sequential benchmark on main for these? Thanks!

[MikeMcQuaid]

Thanks, few more notes here!

[mvanhorn]

Benchmarks (hyperfine, 3 runs each, 4 formulae: tree, bat, fd, fzf):

Mode	Branch	Mean	Range
Sequential	main	9.620s ± 0.549s	9.256s – 10.252s
Parallel --jobs=4	PR	6.288s ± 0.704s	5.873s – 7.101s

~35% improvement. Note that bat and fd share recursive deps (ca-certificates, openssl@3 chain via rust/libgit2), so they're serialized. With a Brewfile of fully independent formulae the speedup would be larger.

Also pushed two commits addressing your second round of feedback:

• 76e991e removes custom FormulaLock locking, extends parallel install to all entry types (casks, taps, extensions)
• 8e07c07 fixes a lock conflict discovered during benchmarking: brew install acquires file locks on all recursive_dependencies (including build deps). bat and fd share ca-certificates via their rust/libgit2 chains, causing deadlock. The dep map now uses Formula#recursive_dependencies to detect overlapping dep trees and serialize conflicting entries.

[MikeMcQuaid]

Getting better thanks!

[mvanhorn]

Addressed all feedback in 8e9c9a2:

• Removed T.cast on future value (line 68)
• Moved formula dependency lookups into Brew.brewfile_dependencies and Brew.recursive_dep_names class methods (line 118 feedback)
• Added Cask formula deps via Cask.formula_dependencies and implicit Tap deps for entries from non-default taps (line 103 feedback)
• Replaced all T.must with .fetch (lines 141, 151)

Style and typecheck pass clean.

[MikeMcQuaid]

Looking good. A few more questions for you (not Claude) to answer. Also feels like some of the bigger and/or more involved functions could do with some more comments to add a little clarity on flow and choices.

[mvanhorn]

Hi Mike! here's me (but also with Claude helping me so I don't bork anything up..)

On runtime_dependencies: I kept raw deps because recursive_dep_names uses recursive_dependencies (which includes build deps) for lock conflict detection. If formula_dep_names only returned runtime deps, two formulas sharing a build dep could slip through and hit keg locks. The whole point of the lock check is to serialize anything that could conflict, so the dependency universe needs to match.

On the naming: you're right, brewfile_dependencies was confusing. Renamed to formula_dep_names - it's the formula's own direct deps, not "deps from the Brewfile."

On cask recursive deps: added cask_dep_names that walks depends_on[:cask] and intersects with Brewfile entries. Direct deps only for now since cask chains are usually shallow, but it's structured to go deeper if needed.

Added phase comments to build_dependency_map. Tried to keep them useful without being noisy - the three-phase approach (name map, direct deps, recursive dep sets, then merge) wasn't obvious from reading the code.

All pushed in 13a7800. Typecheck clean, style clean, tests green.

[MikeMcQuaid]

On runtime_dependencies: I kept raw deps because recursive_dep_names uses recursive_dependencies (which includes build deps) for lock conflict detection. If formula_dep_names only returned runtime deps, two formulas sharing a build dep could slip through and hit keg locks. The whole point of the lock check is to serialize anything that could conflict, so the dependency universe needs to match.

@mvanhorn This will probably be more conservative than it needs to be unfortunately. Ideally we'll only look at build deps when building from source and only look at runtime deps when pouring bottles. With locking you don't want something like a cmake build dependency to lock everything, particularly if nothing actually "needs" it.

Apologies for all the back and forth, I do think this will be worth it!

[mvanhorn]

Good call - you're right that this was overly conservative. Fixed in eb22e0e.

recursive_dep_names now takes an include_build: parameter. The parallel installer checks whether each formula is bottled (and not --build-from-source) to decide which dep set to use for lock conflict detection. Bottle pours use runtime deps only, source builds use the full dependency tree.

This should unblock cases like bat and fd where cmake was unnecessarily serializing their bottle pours.

No need to apologize for the back and forth - this is making the feature meaningfully better.

[MikeMcQuaid]

Looking better!

Weird behaviour with a password prompt here:

Installing Slack
Using TouchDraw2
Using WhatsApp
Password:Using anykeyh.simplecov-vscode

[mvanhorn]

Addressed in 5ced3ff:

Applied the .to_set(&:name) chain suggestion on recursive_dep_names.

Extracted the duplicated formulae_by_full_name(name).presence || formulae_by_name(name) lookup into find_formula - used by both formula_dep_names and formula_bottled?.

On the password prompt interleaving: cask installs can trigger sudo which writes Password: directly to the terminal, outside our output mutex. Added a @cask_install_mutex that serializes cask installs so password prompts don't collide with status output from other workers. Formula installs still run fully parallel.

[mvanhorn]

Hey Mike! Fixed the password prompt interleaving in 3415250.

The root cause: when a cask install triggers sudo, the Password: prompt writes directly to /dev/tty - completely bypassing Ruby's IO and the output mutex. The mutex prevents other threads from writing to stdout during the install, but it can't prevent sudo's /dev/tty write. After the install completes and the mutex is released, the next worker's status message appends to the terminal line where Password: is still lingering.

The fix writes a newline to /dev/tty after each cask install (between releasing the output lock and the cask install lock). This resets the terminal cursor so the next status message starts on a clean line. The write is caught with ENXIO/ENOENT for CI and piped output.

[MikeMcQuaid]

The fix writes a newline to /dev/tty after each cask install (between releasing the output lock and the cask install lock). This resets the terminal cursor so the next status message starts on a clean line. The write is caught with ENXIO/ENOENT for CI and piped output.

@mvanhorn unfortunately output is now a bit crappy:

Using zsh-autosuggestions
Using 1password

Using mysql@8.4
Using 1password-cli

Using appcleaner

Using chatgpt

Using nginx
Using claude-code

Using docker-desktop

Using elgato-camera-hub

Using pango
Using redis
Using elgato-control-center

Using font-sf-mono

Have some merge conflicts to resolve here too!

[mvanhorn]

Thanks for the catch on the output, and apologies for the noise.

Root cause: the unconditional \n on /dev/tty after each cask ran even when no interactive prompt had appeared, pushing the cursor down and producing a blank line after every cask.

Pushed 36515f5 (fix(parallel_installer): clear tty line instead of adding newline). Swapped \n for \r + CSI-K (clear to end of line) in clear_tty_line. When there's a stale Password: on the tty, it gets overwritten by the next status line instead of pushed down; when the tty is already clean (no sudo prompted, or cask output already scrolled past), it's visually a no-op. Formula and cask status lines should now render uniformly.

Also rebased on latest master to clear the conflicts. The bundle/brew.rb conflicts were with the new typed: strict conversion, so I added sigs to find_formula, formula_dep_names, recursive_dep_names, and formula_bottled? while I was in there. brew typecheck and brew style both pass, bundle specs pass.

On the earlier point about the lock universe being more conservative than ideal (cmake serializing unrelated bottle pours, etc.) -- recursive_dep_names already takes an include_build: keyword so bottle pours pass include_build: false via formula_bottled?. The conservative case only kicks in when a formula actually needs to build from source. Happy to tighten further if you'd like the lock universe keyed off the actual preinstall! result rather than the bottled hint.

[MikeMcQuaid]

Thanks @mvanhorn. Tested locally and works well.

[mvanhorn]

Thanks @mvanhorn. Tested locally and works well.

Woohoo super excited about this.

[mvanhorn]

@MikeMcQuaid heads up wrote about my excitement with a little video on this feature on x. Says you don't use x much. :) https://x.com/mvanhorn/status/2045134181546566030?s=46