Skip to content

Parallelize repository cloning during dev environment setup#72

Merged
swissspidy merged 4 commits intomainfrom
copilot/parallelize-repository-cloning
Apr 10, 2026
Merged

Parallelize repository cloning during dev environment setup#72
swissspidy merged 4 commits intomainfrom
copilot/parallelize-repository-cloning

Conversation

Copy link
Copy Markdown
Contributor

Copilot AI commented Apr 9, 2026

Sequential git clone calls during composer install/composer update made initial dev environment setup unnecessarily slow. The refresh step already used xargs for parallelism; cloning did not.

Changes

  • New .maintenance/clone-all-repositories.sh: Bash script that orchestrates the full clone + refresh flow. Checks that jq is available at startup with a clear error message. Detects CPU cores via nproc (Linux) or sysctl -n hw.logicalcpu (macOS), defaulting to 4 if neither is available, and caps parallelism at 8 to avoid saturating GitHub connections or IO. The cap can be overridden via the CLONE_JOBS or WPCLI_DEV_JOBS environment variables. Fetches the repository list from the GitHub API using curl -fsS (failing fast on HTTP errors), validates the response is a JSON array (with distinct messages for rate-limit errors vs. unexpected shapes), parses with jq, filters the skip list, applies the destination map, then clones missing repos and refreshes all repos in parallel using xargs -P$CORES.
  • New .maintenance/clone-repository.sh: Bash helper that clones a single repository, accepting exactly <destination> and <clone_url> as arguments (rejects anything other than exactly 2 args).
  • Removed .maintenance/clone-all-repositories.php and .maintenance/clone-repository.php.
  • Updated composer.json: pre-install-cmd and pre-update-cmd now call bash .maintenance/clone-all-repositories.sh.
# Check prerequisites
if ! command -v jq &>/dev/null; then
    echo "Required command 'jq' is not installed or not available in PATH." >&2
    exit 1
fi

# Detect CPU cores, cap at 8, allow env override
CORES="${CLONE_JOBS:-${WPCLI_DEV_JOBS:-${DETECTED_CORES}}}"
[[ "${CORES}" -gt 8 ]] && CORES=8

# Fetch and validate API response
if ! RESPONSE=$(curl -fsS ... ); then exit 1; fi
if ! jq -e 'type == "array"' >/dev/null <<< "${RESPONSE}"; then exit 1; fi

# Clone and refresh in parallel, bounded by core count
printf '%s\n' "${CLONE_LIST[@]}" | xargs -n2 -P"${CORES}" bash clone-repository.sh
printf '%s\n' "${UPDATE_FOLDERS[@]}" | xargs -P"${CORES}" -I% php refresh-repository.php %

Copilot AI changed the title [WIP] Implement parallel repository cloning during development setup Parallelize repository cloning during dev environment setup Apr 9, 2026
Copilot AI requested a review from swissspidy April 9, 2026 08:31
@swissspidy swissspidy requested a review from thelovekesh April 9, 2026 08:47
@swissspidy swissspidy marked this pull request as ready for review April 9, 2026 21:52
@swissspidy swissspidy requested a review from a team as a code owner April 9, 2026 21:52
Copilot AI review requested due to automatic review settings April 9, 2026 21:52

This comment was marked as resolved.

@swissspidy

This comment was marked as resolved.

…n, cap cores to 8 with env override, drop redundant -n1, exact arg count in clone helper

Agent-Logs-Url: https://github.com/wp-cli/wp-cli-dev/sessions/aab74529-eccb-4e94-949e-b1062afb30b2

Co-authored-by: swissspidy <841956+swissspidy@users.noreply.github.com>

This comment was marked as resolved.

@swissspidy swissspidy merged commit 49fc7d1 into main Apr 10, 2026
6 checks passed
@swissspidy swissspidy deleted the copilot/parallelize-repository-cloning branch April 10, 2026 12:40
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Parallelize repository cloning during development environment setup.

4 participants