Skip to content

Mirror every non-powerpc Arch repository; parallel client; Worker; status site - #14

Closed
Azathothas wants to merge 29 commits into
mainfrom
next
Closed

Azathothas wants to merge 29 commits into
mainfrom
next

Conversation

@Azathothas

Copy link
Copy Markdown
Member

Work in progress. The squash onto main follows when CI and a branch Sync pass.

  • 15 targets: x86_64 core, extra and multilib; aarch64 and armv7h core, extra, alarm and aur; loong64 and riscv64 core and extra.
  • Per-target sync chains (sync-target.yml). Verify runs natively on amd64 for every arch, with Architecture = <arch> and trust roots from keyring_image and keyring_from.
  • Upstream defects pass with no per-package config: NUL-only desc members, and upstream-refused packages (the mirror serves upstream's exact bytes and signature).
  • Client v2: parallel prefetch from pacman's own transaction, blobs by %SHA256SUM%, signatures from %PGPSIG%, resume, host blacklist, and no 2 s pause for an absent database signature under the sandbox.
  • Worker v2 at https://ghcr.mirror.archlinux.pkgforge.dev: redirect-only, so pacman's own downloader works.
  • Status site on Pages (hourly), badges and charts, daily bench.
  • CI: hermetic unit legs, a matrix over integration files, Renovate automerge, policy checks.
  • Redundancy: each scheduled workflow restarts the others when theirs did not fire.

Fixes #3 #4 #5 #6 #7 #8 #9 #10 #11 #12 #13.

…_from, upstream-refused verify, signature-aware plan
# Conflicts:
#	worker/src/index.js
#	worker/wrangler.toml
… hourly in status; per-target artifact patterns
…before its dashes; guard against overlapping target names
A signature request never marks its host down, so a failed optional .sig no longer makes the next file from that host miss.
The size of a held .part comes from its metadata (ls -ln), so a .part that the download user cannot read no longer stops the call with an arithmetic error.
After a refused range (curl 33), resume and the per-attempt time limit are off, and the 30 s stop starts again, so the fetch from zero runs.
The remover touches ,open once a second, and the sweep keeps the entries of a run whose ,open is less than 2 minutes old, so a run in another pid namespace keeps its store.
Pages artifact named per run attempt; report starts status.yml only when active; ci.yml group per run outside pull requests; sync.yml queue: max; status sums, badges and page say over how many targets they count, a registry that did not answer is unknown, the history carries a target's last measured files; worker-down needs 50 minutes as well as 2 runs; check-staleness grace from the enabled date; a stale bench shows its age.
- fetch_upstream: MAX_SECONDS bounds the whole call, retries included
  (--retry-max-time, and timeout 5 s after it). plan ends its upstream
  reads within PLAN_BUDGET (1200 s), shared out over the targets; a
  db_server read gets 1/(n + 1) of the target's share.
- mirror-file: with no %PGPSIG%, the .sig comes from the target's servers
  in their own order, the db_servers first, as verify reads it. A server
  that fails is asked last by every file of the shard after it. A .sig
  read ends within 30 s, a download within 120 s plus the time at 64 KiB/s.
- A file pushed unsigned is named in the staged manifest
  (dev.pkgforge.mirror.unsigned). plan asks for its .sig on each run and
  plans it again once upstream signs it, also with the database unchanged.
- registry_token_file: an anonymous read never uses the token cache.
- Every chain artifact carries the plan id. stage reads only the results
  of its plan; the report reads only the summaries of its plan, for
  pending targets; collect reads only the verdicts and the promotion of
  the staged digest. So "Re-run all jobs" reads nothing of the earlier
  attempt.
- Integration cases for the files-database rule: a .files that leaves out
  a package is planned and verified; one that names an extra package is
  refused.
@Azathothas

Copy link
Copy Markdown
Member Author

Landed on main as one commit: bb8b4a8

@Azathothas Azathothas closed this Oct 14, 2026
@Azathothas
Azathothas deleted the next branch October 14, 2026 17:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Prefetch in parallel: 40 core packages take 60.9 s under XferCommand where an official mirror takes 2.6 s

1 participant