Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
83 changes: 83 additions & 0 deletions .github/actions/install-linux-packages/action.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,83 @@
# Copyright 2026 Han Li and contributors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

name: Install the Linux packages
description: >-
Installs Ubuntu packages from a cache of their archives, and downloads what the
cache lacks eight files at a time, continuing what was cut and trying the next
mirror when one is slow, in 15 minutes at most (see install.sh), because the
Ubuntu mirror of the runners is slow for some of them at times.

inputs:
packages:
description: The packages, separated by spaces or new lines.
required: true

runs:
using: composite
steps:
- name: Name the cache
id: names
shell: bash
env:
PACKAGES: ${{ inputs.packages }}
run: |
# A cache for each list of packages and each week: a week is long enough
# for many runs to use it, and short enough to stay near the versions
# of the mirror. The newest cache of any earlier week (or what an
# interrupted run saved, see below) starts the cache of a new week,
# which install.sh trims.
hash="$(echo "$PACKAGES" | tr -s '[:space:]' '\n' | sort -u | sha256sum | cut -c1-16)"
echo "prefix=apt-${ImageOS:-ubuntu}-$hash-" >> "$GITHUB_OUTPUT"
echo "key=apt-${ImageOS:-ubuntu}-$hash-$(date -u +%G-W%V)" >> "$GITHUB_OUTPUT"
- name: Let the cache write the archives
shell: bash
run: sudo chmod a+rwx /var/cache/apt/archives
- name: Restore the archives
id: cache
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: /var/cache/apt/archives/*.deb
key: ${{ steps.names.outputs.key }}
restore-keys: ${{ steps.names.outputs.prefix }}
- name: Install the packages
id: install
shell: bash
env:
PACKAGES: ${{ inputs.packages }}
run: |
# shellcheck disable=SC2086 # The list is meant to be split.
"$GITHUB_ACTION_PATH/install.sh" $PACKAGES
# The cache is saved by the user of the runner, and one archive that this user
# cannot read fails the whole save (tar did on one archive of Wails, on
# 2026-10-23, and nothing was saved). It also holds when the installation failed.
- name: Let the cache read the archives
if: always()
shell: bash
run: sudo chmod a+r /var/cache/apt/archives/*.deb || true
- name: Save the archives
if: steps.install.outcome == 'success' && steps.cache.outputs.cache-hit != 'true'
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: /var/cache/apt/archives/*.deb
key: ${{ steps.names.outputs.key }}
# A download that did not finish within its time is not lost: the next run
# starts from the archives it got (by the prefix above) instead of from
# nothing, so that a run on a slow mirror gets further each time.
- name: Save what was downloaded
if: always() && steps.install.outcome == 'failure'
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6.1.0
with:
path: /var/cache/apt/archives/*.deb
key: ${{ steps.names.outputs.prefix }}partial-${{ github.run_id }}-${{ github.run_attempt }}
149 changes: 149 additions & 0 deletions .github/actions/install-linux-packages/fetch.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,149 @@
#!/usr/bin/env bash
# Copyright 2026 Han Li and contributors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Makes one archive of a package available for install.sh, which starts several of
# these at the same time:
#
# fetch.sh <archive directory> <size> <file name> <hash> <address>...
#
# The hash is the one that apt prints, TYPE:digest (SHA256:... with
# -o Acquire::ForceHash=SHA256). apt checks the hash of what it downloads, but takes
# an archive that is in the directory with its size for a good one, so that the
# archive has to be checked here, before it is moved there. An archive that is
# there already is checked too, and removed when it is wrong, which is what a damaged
# cache would otherwise break the installation with.
#
# The addresses are those of the file on the mirrors, in the order to try them. The
# download goes to partial/ first, where apt keeps what it is downloading, and
# continues from what is there, so that a connection that is cut, or dropped for being
# slow, loses nothing.
#
# A connection that gets less than APT_MIN_SPEED bytes a second (262144 by default)
# for APT_SPEED_TIME seconds (10) is dropped, and the download continues from the next
# mirror: a mirror is slow for some runners and not for others, and the speed of the
# next one is not known before it is tried. When every mirror has been slow in turn, a
# slow download is better than none: the best of them continues, and only a stalled
# connection is dropped. A file that no mirror sends in three rounds, or that has the
# wrong hash three times, is given up.
#
# APT_DEADLINE, in seconds since the epoch, is the time when the download ends.
set -uo pipefail

[[ $# -ge 5 ]] || { echo "Usage: fetch.sh <archive directory> <size> <file name> <hash> <address>..." >&2; exit 2; }

directory="$1"
size="$2"
name="$3"
hash="$4"
shift 4
addresses=("$@")
deadline="${APT_DEADLINE:?APT_DEADLINE is the time when the download ends, in seconds since the epoch}"
minimum="${APT_MIN_SPEED:-262144}"
window="${APT_SPEED_TIME:-10}"
final="$directory/$name"
partial="$directory/partial/$name"
error="$(mktemp)"
trap 'rm -f "$error"' EXIT

# The size of what has been downloaded.
have() {
stat --format %s "$partial" 2> /dev/null || echo 0
}

# right <file>: whether the file has the hash that apt printed.
right() {
local tool
case "${hash%%:*}" in
MD5Sum) tool=md5sum ;;
SHA1) tool=sha1sum ;;
SHA256) tool=sha256sum ;;
SHA512) tool=sha512sum ;;
*) return 1 ;;
esac
[[ "$("$tool" "$1" 2> /dev/null | cut -d ' ' -f 1)" == "${hash#*:}" ]]
}

if [[ -e "$final" ]]; then
if [[ "$(stat --format %s "$final")" -eq "$size" ]] && right "$final"; then
exit 0
fi
echo "$name: the archive in the directory is not the one that apt expects; removed"
rm -f "$final"
fi

fruitless=0
wrong=0
last=""
while true; do
if [[ "$(have)" -eq "$size" ]]; then
right "$partial" && break
echo "$name: the hash is not $hash; starting again"
rm -f "$partial"
wrong=$((wrong + 1))
((wrong < 3)) || { echo "$name: no mirror sends the right file" >&2; exit 1; }
# The mirror that completed it is the last one to try again.
addresses=()
for address in "$@"; do
[[ "$address" == "$last" ]] || addresses+=("$address")
done
[[ -z "$last" ]] || addresses+=("$last")
fi
# More than the file has is not a part of it.
[[ "$(have)" -le "$size" ]] || rm -f "$partial"
start="$(have)"
best=""
best_gain=0
for address in "${addresses[@]}"; do
before="$(have)"
[[ "$before" -lt "$size" ]] || break
left=$((deadline - $(date +%s)))
((left > 0)) || exit 1
last="$address"
speed="$(curl --fail --silent --show-error --location --connect-timeout 10 --max-time "$left" \
--speed-limit "$minimum" --speed-time "$window" --continue-at - \
--write-out '%{speed_download}' --output "$partial" "$address" 2> "$error")"
status=$?
# A server that does not continue a download sends all of the file, which curl
# does not take (33): start again.
[[ $status -ne 33 ]] || rm -f "$partial"
gain=$(($(have) - before))
if ((gain > best_gain)); then
best="$address"
best_gain=$gain
fi
if [[ "$(have)" -ne "$size" ]]; then
host="${address#*://}"
echo "$name: $(have) of $size bytes, ${speed%.*} bytes a second from ${host%%/*}: $(tr '\n' ' ' < "$error")"
fi
done
# All of it came, to be checked at the top.
[[ "$(have)" -ne "$size" ]] || continue
if [[ "$(have)" -gt "$start" ]]; then
fruitless=0
else
fruitless=$((fruitless + 1))
((fruitless < 3)) || { echo "$name: no mirror sends it" >&2; exit 1; }
fi
# Every mirror was slow in turn, or failed: go on with the best of them.
minimum=1000
if [[ -n "$best" ]]; then
addresses=("$best")
for address in "$@"; do
[[ "$address" == "$best" ]] || addresses+=("$address")
done
fi
sleep 1
done
mv "$partial" "$final"
160 changes: 160 additions & 0 deletions .github/actions/install-linux-packages/install.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,160 @@
#!/usr/bin/env bash
# Copyright 2026 Han Li and contributors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Installs Ubuntu packages on a runner of GitHub Actions in two steps, which the
# action next to this file (action.yml) wraps in a cache:
#
# 1. Downloads the packages that the cache lacks into /var/cache/apt/archives, in
# 15 minutes at most. fetch.sh does it, eight files at a time and the largest
# first, from the mirrors of the runner: it continues what was cut, tries the
# next mirror when one is slow, and checks the hash of every archive, those of
# the cache too, because apt takes an archive of the right size for a good one.
# Then apt fetches what is left.
# 2. Installs them from the archives, which needs no network.
#
# Why not apt alone: the Ubuntu mirror of the runners is slow for some of them. On
# 2026-10-23 a job got 40 KB/s for a connection while other jobs of the same minute
# downloaded 90 MB in seconds, and its 94 MB did not come in 30 minutes (a minute on a
# good day), which a job of 25 or 45 minutes cannot wait for; a file of 25 MB alone
# takes 10 minutes at that speed. apt downloads from one mirror, over one connection,
# one file after the other, however slow it is.
#
# Finally it removes the archives of versions that the package lists no longer
# have, so that the cache does not grow from week to week.
#
# Usage: install.sh <package>...
#
# Set APT_DOWNLOAD_SECONDS to change the time of the download (900): it has to end
# before the timeout of the job, for the action to save what was downloaded.
# APT_PARALLEL is the number of files that are downloaded at the same time (8; 0
# leaves all of it to apt). fetch.sh has two more, APT_MIN_SPEED and APT_SPEED_TIME.
set -euo pipefail

[[ $# -gt 0 ]] || { echo "Usage: install.sh <package>..." >&2; exit 2; }

packages=("$@")
seconds="${APT_DOWNLOAD_SECONDS:-900}"
parallel="${APT_PARALLEL:-8}"
here="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
archives=/var/cache/apt/archives
deadline=$(($(date +%s) + seconds))

# A request that gets no data for 30 seconds fails and is tried again, up to 3
# times; apt waits for 2 minutes by default. The archives stay where they are
# after the installation: a runner has no hook that removes them, but a
# container of Docker has.
apt_options=(-o Acquire::Retries=3 -o Acquire::http::Timeout=30 -o Acquire::https::Timeout=30
-o APT::Keep-Downloaded-Packages=true)

# apt_get <time limit> <arguments...>: apt-get as root, for at most the time (0 is
# no limit), without questions.
apt_get() {
local time="$1"
shift
sudo timeout "$time" env DEBIAN_FRONTEND=noninteractive apt-get "${apt_options[@]}" "$@"
}

# archive_size <file name>: the size of the archive, 0 when it is not there.
archive_size() {
stat --format %s "$archives/$1" 2> /dev/null || echo 0
}

# missing <list>: the number of archives of the list that are not in the directory
# with their size.
missing() {
local size name count=0
while read -r size name _; do
[[ "$(archive_size "$name")" -eq "$size" ]] || count=$((count + 1))
done < "$1"
echo "$count"
}

# prefetch: makes the archives that the installation needs available in the
# directory, and checks the ones that are there already. It only saves time: what it
# does not fetch, apt does.
prefetch() {
local mirrors list empty line path total before after size name hash address mirror
if ! command -v curl > /dev/null; then
echo "curl is missing, apt downloads everything."
return
fi
# The runners list their mirrors in a file, one for each line, the best first.
mapfile -t mirrors < <(awk 'NF { print $1 }' /etc/apt/apt-mirrors.txt 2> /dev/null)
list="$(mktemp)"
# apt prints 'address' name size hash for each archive (a SHA256 hash is what
# fetch.sh checks), but not for the archives that are in the directory, which it
# takes for good ones: it is shown an empty directory, and fetch.sh checks them. The
# address of a runner starts with that of the file of the mirrors, which stands for
# one of them: the line of an archive has its address on each mirror, for fetch.sh.
# The largest come first, so that none of them is the last and alone.
empty="$(mktemp -d)"
apt_get 2m -o Dir::Cache::Archives="$empty" -o Acquire::ForceHash=SHA256 install --print-uris -y --no-install-recommends "${packages[@]}" 2> /dev/null \
| sed -n "s/^'\([^']*\)' \([^ ]*\) \([0-9]*\) \([^ ]*\).*/\3 \2 \4 \1/p" \
| while read -r size name hash address; do
line="$size $name $hash"
path="${address#mirror+file:/etc/apt/apt-mirrors.txt/}"
if [[ "$path" != "$address" && ${#mirrors[@]} -gt 0 ]]; then
for mirror in "${mirrors[@]}"; do
line+=" ${mirror%/}/$path"
done
else
line+=" $address"
fi
echo "$line"
done \
| sort -k 1,1nr > "$list" || true
rmdir "$empty" 2> /dev/null || true
total="$(wc -l < "$list")"
before="$(missing "$list")"
echo "$before of $total archives are not in $archives."
sudo mkdir -p "$archives/partial"
# One line, a size, a name, a hash and addresses, for each run of fetch.sh, and no
# run when there is no line. fetch.sh ends at the deadline by itself; the limit here
# is for what goes wrong in it.
sudo timeout "$((deadline - $(date +%s) + 60))" \
env APT_DEADLINE="$deadline" APT_MIN_SPEED="${APT_MIN_SPEED:-}" APT_SPEED_TIME="${APT_SPEED_TIME:-}" \
xargs -r -a "$list" -P "$parallel" -L 1 "$here/fetch.sh" "$archives" || true
after="$(missing "$list")"
echo "fetch.sh downloaded $((before - after)) archives."
if ((after > 0)); then
echo "::warning::$after archives are left for apt."
fi
rm -f "$list"
}

for ((attempt = 1; attempt <= 3; attempt++)); do
apt_get 2m update && break
if ((attempt == 3)); then
echo "::error::apt-get update failed in 3 attempts."
exit 1
fi
sleep 5
done

[[ "$parallel" -eq 0 ]] || prefetch

# apt downloads what is still missing, and checks what it downloads, with what is
# left of the time (a minute at least, which is enough to see that nothing is).
left=$((deadline - $(date +%s)))
((left > 60)) || left=60
if ! apt_get "${left}s" install --download-only -y --no-install-recommends "${packages[@]}"; then
echo "::error::Could not download the packages (the limit is $seconds seconds); the mirrors are probably slow."
exit 1
fi

# Killing dpkg half way would leave packages unpacked but not configured, so
# the installation has no time limit; it needs no network.
apt_get 0 install -y --no-install-recommends "${packages[@]}"
apt_get 0 autoclean
Loading
Loading