From 172d97ad070a68ec9893659d1e56262faa7cbcac Mon Sep 17 00:00:00 2001 From: Jarek Potiuk Date: Thu, 10 Sep 2026 15:13:33 +0200 Subject: [PATCH 1/2] Add a GitHub Actions chapter to the Services and Tools page Document why the shared GitHub Actions runner queues build up and what a project can do about it: the 8-9 September 2026 queue pile-up as a worked example, plus mitigations (PR workflow optimisation, checking for retired runner labels, grouping dependency updates, ubuntu-slim runners, and self-hosted runners). Move the GitHub Actions Policy pointer and the Dependabot / dependency management section into the new chapter, and cross-link the chapter from the policy, secrets, build services and documentation index pages. Also add a "Group security updates" section to the Dependabot page. A `groups` block defaults to applies-to: version-updates, so security updates keep opening one pull request per alert -- and one CI run with it -- unless a group sets applies-to: security-updates. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_0126n4c75RhhntsoyCN28BJP --- content/pages/build-supported-services.md | 6 ++- content/pages/dependabot.md | 59 +++++++++++++++++++++++ content/pages/doc.md | 2 + content/pages/github-actions-policy.md | 4 ++ content/pages/github-actions-secrets.md | 2 + content/pages/services.md | 38 +++++++++++++-- 6 files changed, 107 insertions(+), 4 deletions(-) diff --git a/content/pages/build-supported-services.md b/content/pages/build-supported-services.md index a0bfa68c8..87edef191 100644 --- a/content/pages/build-supported-services.md +++ b/content/pages/build-supported-services.md @@ -88,7 +88,11 @@ GHA integrates with nightlies.apache.org, Dockerhub, Artifactory, Jenkins and mo All Committers have access to GHA and their workflows via `commit` using their LDAP credentials. #### More information -In addition to the official GitHub documentation, Infra has placed some notes on a Confluence wiki page. +In addition to the official GitHub documentation, Infra maintains: + + - [GitHub Actions and Secrets](github-actions-secrets.html), for handling credentials in workflows. + - The [GitHub Actions Policy](github-actions-policy.html), which sets the limits and requirements every ASF workflow must meet. + - [GitHub Actions](services.html#github-actions) on the Services and Tools page, which explains why the shared runner queues build up and how your project can help.

Artifactory

diff --git a/content/pages/dependabot.md b/content/pages/dependabot.md index 6904434b9..f4ef6a1b0 100644 --- a/content/pages/dependabot.md +++ b/content/pages/dependabot.md @@ -80,6 +80,40 @@ You can also split groups by update type or dependency kind: - "major" ``` +Unless you say otherwise, a group covers version updates only. See [Group security updates](#group-security-updates) below to batch security fixes the same way. + +

Group security updates

+ +Every project is scanned now — by GitHub's own Dependabot alerts, and by the scanners that downstream users and their employers run against our releases — so we all see **way** more security updates than we used to, and everyone rushes to fix what the scanners report as quickly as they can. That is the right instinct, but it has a cost: by default a `groups` block applies only to *version* updates, so security updates keep opening **one pull request per alert**, each with a full CI run behind it. A single busy week of advisories in a widely-used library can put hundreds of extra builds into the [shared GitHub Actions queue](services.html#github-actions). + +Set `applies-to` on a group to batch security fixes as well. A group with `applies-to: security-updates` collects every outstanding security fix in that ecosystem into one pull request: + +```yaml +version: 2 +updates: + - package-ecosystem: "maven" + directory: "/" + schedule: + interval: "weekly" + groups: + # One PR for the routine version bumps + maven-dependencies: + applies-to: version-updates + patterns: + - "*" + # One PR for all outstanding security fixes + maven-security: + applies-to: security-updates + patterns: + - "*" +``` + +When `applies-to` is omitted, the group defaults to `version-updates`, which is why a configuration that groups version updates can still produce a flood of individual security PRs. + +Grouping is a trade-off worth understanding: a grouped pull request is all-or-nothing, so one update in the batch that breaks your build holds up the rest of the fixes in it. If your project needs to land a critical fix in isolation, narrow the group's `patterns` to leave that dependency out, or keep a separate group for the dependencies you want to review one at a time. + +For the full syntax, see the Dependabot `groups` configuration reference. +

Use a cooldown period

A **cooldown** delays Dependabot from proposing a new dependency version until it has been published for a minimum number of days. This gives the community time to discover issues with a release — including compromised packages — before your project adopts it. See Why you should use dependency cooldowns for background on the security rationale. @@ -153,6 +187,11 @@ updates: default-days: 4 groups: actions-dependencies: + applies-to: version-updates + patterns: + - "*" + actions-security: + applies-to: security-updates patterns: - "*" @@ -165,6 +204,11 @@ updates: default-days: 4 groups: maven-dependencies: + applies-to: version-updates + patterns: + - "*" + maven-security: + applies-to: security-updates patterns: - "*" @@ -177,6 +221,11 @@ updates: default-days: 4 groups: npm-dependencies: + applies-to: version-updates + patterns: + - "*" + npm-security: + applies-to: security-updates patterns: - "*" @@ -189,6 +238,11 @@ updates: default-days: 4 groups: pip-dependencies: + applies-to: version-updates + patterns: + - "*" + pip-security: + applies-to: security-updates patterns: - "*" @@ -201,6 +255,11 @@ updates: default-days: 4 groups: uv-dependencies: + applies-to: version-updates + patterns: + - "*" + uv-security: + applies-to: security-updates patterns: - "*" ``` diff --git a/content/pages/doc.md b/content/pages/doc.md index 501e1cada..17de6657a 100644 --- a/content/pages/doc.md +++ b/content/pages/doc.md @@ -24,6 +24,8 @@ Infra provides a library of resources that is constantly evolving to reflect the - .asf.yaml features for Git repositories - [GitHub Actions and Secrets](github-actions-secrets.html) - [GitHub Actions policy](github-actions-policy.html) + - [Shared runner queues and how to reduce your load](services.html#github-actions) +- [Dependabot for dependency management](dependabot.html) #### Build and release #### - [Release creation process](release-publishing.html) diff --git a/content/pages/github-actions-policy.md b/content/pages/github-actions-policy.md index 75701eeb3..7c19d9cbe 100644 --- a/content/pages/github-actions-policy.md +++ b/content/pages/github-actions-policy.md @@ -11,6 +11,8 @@ For additional advice on how to use this feature safely, see GitHub Dependabot for the `github-actions` ecosystem * Forking Renovate using the GitHub actions manager + +See the [Dependabot](dependabot.html) page for how to configure it, including [grouping security updates](dependabot.html#group-security-updates) so that a burst of advisories does not turn into a burst of CI runs. ### Resource use Due to misconfigurations in their builds, some projects have been using unsupportable numbers of [GitHub Actions](github-actions-secrets.html). As part of fixing this situation, Infra has established a policy for GitHub Actions use: @@ -24,6 +26,8 @@ Projects whose builds consistently cross the maximum use limits will lose their **Note**: Projects should review these recommended practices from Git: cwiki.apache.org/confluence/spaces/INFRA/pages/430408443/GitHub+Actions+Recommended+Practices. +Runners are a shared resource, so a project can slow every other project down while staying inside these limits. For why queues build up and what your project can do to reduce its share of the load, see [GitHub Actions](services.html#github-actions) on the Services and Tools page. + ### Avoid using the 'pull_request_target' trigger You **MUST NOT** use `pull_request_target` as a trigger on **ANY** action that exports **ANY** confidential credentials or tokens such as `GITHUB_TOKEN` or `NPM_TOKEN`. An explantion of the risks related to this trigger, and the very limited circumstances in which it may be used, is at /cwiki.apache.org/confluence/display/INFRA/GHA-dangers+of+pull_request_target. diff --git a/content/pages/github-actions-secrets.md b/content/pages/github-actions-secrets.md index bd1b7e6d9..bec4f1ca4 100644 --- a/content/pages/github-actions-secrets.md +++ b/content/pages/github-actions-secrets.md @@ -62,3 +62,5 @@ Further information is at these pages: - The [Apache GitHub Actions Policy](github-actions-policy.html) - Advice for GitHub Actions Security + - [GitHub Actions](services.html#github-actions) on the Services and Tools page, on shared runner queues and how to reduce your project's load + - [Dependabot](dependabot.html) for dependency management, which is **required** for the `github-actions` ecosystem diff --git a/content/pages/services.md b/content/pages/services.md index 15d3c81a9..5265177ff 100644 --- a/content/pages/services.md +++ b/content/pages/services.md @@ -19,7 +19,7 @@ Infra maintains a wide range of tools for PMCs, project committers, and the Apac - Integrating your repository with Jira tickets - Source repository publisher/subscriber services - Build services - - Dependency management + - GitHub Actions - Product naming - Code signing - Code quality @@ -150,14 +150,46 @@ Apache supports and models continuous integration and continuous deployment, or Other tools to consider: -* [Dependabot](dependabot.html) for automated dependency updates * Travis CI * AppVeyor -

Dependency management

+

GitHub Actions

+ +GitHub Actions (GHA) runners are a **shared** resource: every ASF project draws on the same pool of GitHub-hosted runners. Queue time is therefore a community concern rather than a per-project one. See [GitHub Actions secrets](github-actions-secrets.html) for credential handling. + +

GitHub Actions Policy

+ +The [GitHub Actions Policy](github-actions-policy.html) sets out the rules every ASF workflow **must** follow, including job concurrency limits, weekly and five-day runner-minute budgets, pinning external actions to a specific git hash, avoiding the `pull_request_target` trigger where credentials are exposed, and the dependency management described below. Projects whose builds consistently cross the maximum use limits can lose access to GitHub Actions until they fix their build configurations. + +

Dependabot and dependency management

[Dependabot](dependabot.html) automatically keeps your project's dependencies up to date and free of known vulnerabilities. It is **required** for the `github-actions` ecosystem (see the [GitHub Actions Policy](github-actions-policy.html)), and recommended for all other ecosystems your project uses. +

Why queues build up

+ +Sometimes GitHub CI queues many workflows, leading to long wait times (with AI-generated contributions likely contributing significantly). In other cases, build misconfigurations or local spikes from cherry-picked or re-run jobs create noisy-neighbour issues for other projects. + +Dependency updates are a growing share of that load. Every project is now scanned — by GitHub's own alerts, by Dependabot, and by the scanners that downstream users run against our releases — so we all see **way** more security updates than we used to, and everyone — Infra included — rushes to fix what the scanners find as quickly as possible. Left ungrouped, each of those fixes arrives as its own pull request with a full CI run behind it, and a single busy week of advisories can put hundreds of extra builds into the shared queue. + +

Example: the queues of 8-9 September 2026

+ +The queues on 8-9 September 2026 were particularly abnormal. Checks conducted on 9 September 2026 — which resulted in this tool: apache/infrastructure-actions#1259 — suggest a combination of factors: + + - **Doris**: Heavy cherry-picking, a large volume of PRs, and old builds referencing retired `macos-13` tags (misconfigured old branches) may have blocked their queue and impacted others. + - **Airflow**: Significant cherry-picking and merging while preparing for the provider release and Airflow 3.3.2 release caused a temporary spike in queued builds. + - **Post-Holiday Activity**: Increased activity following the Labor Day weekend likely accumulated more new PRs, unapproved workflows, and merges across projects. + - **Dependency Updates**: Dependabot upgrades are plentiful now. Scanners flag findings in every project and everyone applies the fixes as fast as they can, so an ungrouped configuration turns each advisory into its own pull request and its own full CI run. + +

How your project can help

+ +While the Infra team is working on a multi-tenant experiment to help isolate neighbours, here are several ways you and your PMC can help mitigate these issues: + + - **Optimize PR workflows**: Run full test suites periodically on main merges or via scheduled "canary" builds rather than on every PR. Use `paths:` filters in GitHub Actions YAML, write custom scripts to determine dynamic build matrix outputs, or utilize tools like Apache Yetus or [Develocity](gradle.html) (develocity.apache.org) for predictive test selection. Feel free to reach out to Infra for advice on the best approach for your project. + - **Review workflows for anomalies**: Check for deprecated labels like `macos-13`, which cause workers to hang in "waiting for runner" status and artificially inflate queue sizes. Also, review custom parallelism settings that might block valid jobs — the [GitHub Actions Policy](github-actions-policy.html) requires a job concurrency level of 20 or less per workflow, and recommends staying at or below 15. + - **Group your dependency updates**: An ungrouped Dependabot configuration opens one pull request, and one full CI run, per dependency. Combine them with `groups`, and add `applies-to: security-updates` so that security fixes are batched as well instead of arriving as a stream of single-dependency PRs. See [grouping security updates](dependabot.html#group-security-updates) for the configuration. + - **Use lightweight runners**: For small, short jobs, consider using `runs-on: ubuntu-slim`. These provide 1 vCPU, 5GB RAM, no Docker engine, and a 15-minute hard limit. They churn much faster during peak times, allowing you to bypass the standard `ubuntu-latest` queue while freeing up capacity for others. + - **Explore self-hosted runners**: Speak with the Infra team about setting up project-specific [self-hosted runners](self-hosted-runners.html), particularly if you have targeted donations or cloud credits available. +

Product naming

See [guidance for choosing a product name](project-names.html) From ff068b7ee2d1cd93c5a64dbf750ba49530532003 Mon Sep 17 00:00:00 2001 From: Jarek Potiuk Date: Thu, 10 Sep 2026 19:19:47 +0200 Subject: [PATCH 2/2] Address review feedback on the GitHub Actions chapter Reviewer edits from @cottage14: - Rework the dependency-update paragraphs on both `services.md` and `dependabot.md` to the reviewer's wording: assert the problem rather than argue it, and drop the emphatic register. - Remove the "Example: the queues of 8-9 September 2026" section. The material belongs in a blog post, not on a reference page. - Drop the "while Infra is working on a multi-tenant experiment" clause from the mitigations intro; time-dependent text ages the page. - Point at a Jira ticket or `users@infra` explicitly instead of "feel free to reach out to Infra". - Assorted line edits: em-dash to a full stop, comma removal, "faster" to "more quickly", "trade-off worth understanding" to "trade-off". From @ppkarwasz's review: - Add a "Pin first-party actions to a major version tag" mitigation. The policy requires a git hash only for external actions; the `apache/*`, `github/*` and `actions/*` namespaces are exempt, so tag-pinning those removes a large share of Dependabot traffic. - Add a "Reuse workflows across repositories" mitigation: shared CI in reusable workflows means an action bump is tested once rather than once per repository. - Link the INFRA "GitHub Actions Recommended Practices" wiki page from `build-supported-services.md`, noting it needs a committer or Member login, using the same URL form as 4cb161fb. Also make the reference to the shared queue from within `services.md` an in-page anchor rather than a link back to the same document. Generated-by: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01RtBfpombmwwbMDPD1JcUkR --- content/pages/build-supported-services.md | 1 + content/pages/dependabot.md | 8 +++++--- content/pages/services.md | 23 +++++++++-------------- 3 files changed, 15 insertions(+), 17 deletions(-) diff --git a/content/pages/build-supported-services.md b/content/pages/build-supported-services.md index 87edef191..15c605097 100644 --- a/content/pages/build-supported-services.md +++ b/content/pages/build-supported-services.md @@ -93,6 +93,7 @@ In addition to the official GitHub documentation, Infra maintains: - [GitHub Actions and Secrets](github-actions-secrets.html), for handling credentials in workflows. - The [GitHub Actions Policy](github-actions-policy.html), which sets the limits and requirements every ASF workflow must meet. - [GitHub Actions](services.html#github-actions) on the Services and Tools page, which explains why the shared runner queues build up and how your project can help. + - The GitHub Actions Recommended Practices wiki page (requires a committer or Member login).

Artifactory

diff --git a/content/pages/dependabot.md b/content/pages/dependabot.md index f4ef6a1b0..4b5e03d0c 100644 --- a/content/pages/dependabot.md +++ b/content/pages/dependabot.md @@ -84,9 +84,11 @@ Unless you say otherwise, a group covers version updates only. See [Group securi

Group security updates

-Every project is scanned now — by GitHub's own Dependabot alerts, and by the scanners that downstream users and their employers run against our releases — so we all see **way** more security updates than we used to, and everyone rushes to fix what the scanners report as quickly as they can. That is the right instinct, but it has a cost: by default a `groups` block applies only to *version* updates, so security updates keep opening **one pull request per alert**, each with a full CI run behind it. A single busy week of advisories in a widely-used library can put hundreds of extra builds into the [shared GitHub Actions queue](services.html#github-actions). +Every project repository is now scanned by GitHub's Dependabot alerts and by the scanners that downstream users and their employers run against ASF project releases, so we now see many more security updates than we used to. Projects rush to fix what the scanners report as quickly as they can. -Set `applies-to` on a group to batch security fixes as well. A group with `applies-to: security-updates` collects every outstanding security fix in that ecosystem into one pull request: +That response has a cost: by default, a `groups` block applies only to **version** updates, so security updates open **one pull request per alert**, each with a full CI run behind it. This can add hundreds of builds into the [shared GitHub Actions queue](services.html#github-actions). + +A group with `applies-to: security-updates` set collects all outstanding security fixes for that group into one pull request: ```yaml version: 2 @@ -110,7 +112,7 @@ updates: When `applies-to` is omitted, the group defaults to `version-updates`, which is why a configuration that groups version updates can still produce a flood of individual security PRs. -Grouping is a trade-off worth understanding: a grouped pull request is all-or-nothing, so one update in the batch that breaks your build holds up the rest of the fixes in it. If your project needs to land a critical fix in isolation, narrow the group's `patterns` to leave that dependency out, or keep a separate group for the dependencies you want to review one at a time. +Grouping is a trade-off: a grouped pull request is all-or-nothing, so one update in the batch that breaks your build holds up the rest of the fixes in it. If your project needs to land a critical fix in isolation, narrow the group's `patterns` to leave that dependency out, or keep a separate group for the dependencies you want to review one at a time. For the full syntax, see the Dependabot `groups` configuration reference. diff --git a/content/pages/services.md b/content/pages/services.md index 5265177ff..99b00069a 100644 --- a/content/pages/services.md +++ b/content/pages/services.md @@ -169,25 +169,20 @@ The [GitHub Actions Policy](github-actions-policy.html) sets out the rules every Sometimes GitHub CI queues many workflows, leading to long wait times (with AI-generated contributions likely contributing significantly). In other cases, build misconfigurations or local spikes from cherry-picked or re-run jobs create noisy-neighbour issues for other projects. -Dependency updates are a growing share of that load. Every project is now scanned — by GitHub's own alerts, by Dependabot, and by the scanners that downstream users run against our releases — so we all see **way** more security updates than we used to, and everyone — Infra included — rushes to fix what the scanners find as quickly as possible. Left ungrouped, each of those fixes arrives as its own pull request with a full CI run behind it, and a single busy week of advisories can put hundreds of extra builds into the shared queue. +Dependency updates are a growing share of that load. Every project repository is now scanned by GitHub's Dependabot alerts and by the scanners that downstream users and their employers run against ASF project releases, so we now see many more security updates than we used to. Projects rush to fix what the scanners report as quickly as they can. -

Example: the queues of 8-9 September 2026

- -The queues on 8-9 September 2026 were particularly abnormal. Checks conducted on 9 September 2026 — which resulted in this tool: apache/infrastructure-actions#1259 — suggest a combination of factors: - - - **Doris**: Heavy cherry-picking, a large volume of PRs, and old builds referencing retired `macos-13` tags (misconfigured old branches) may have blocked their queue and impacted others. - - **Airflow**: Significant cherry-picking and merging while preparing for the provider release and Airflow 3.3.2 release caused a temporary spike in queued builds. - - **Post-Holiday Activity**: Increased activity following the Labor Day weekend likely accumulated more new PRs, unapproved workflows, and merges across projects. - - **Dependency Updates**: Dependabot upgrades are plentiful now. Scanners flag findings in every project and everyone applies the fixes as fast as they can, so an ungrouped configuration turns each advisory into its own pull request and its own full CI run. +That response has a cost: by default, a `groups` block applies only to **version** updates, so security updates open **one pull request per alert**, each with a full CI run behind it. This can add hundreds of builds into the [shared GitHub Actions queue](#github-actions).

How your project can help

-While the Infra team is working on a multi-tenant experiment to help isolate neighbours, here are several ways you and your PMC can help mitigate these issues: +Here are ways you and your PMC can help mitigate this issue: - - **Optimize PR workflows**: Run full test suites periodically on main merges or via scheduled "canary" builds rather than on every PR. Use `paths:` filters in GitHub Actions YAML, write custom scripts to determine dynamic build matrix outputs, or utilize tools like Apache Yetus or [Develocity](gradle.html) (develocity.apache.org) for predictive test selection. Feel free to reach out to Infra for advice on the best approach for your project. - - **Review workflows for anomalies**: Check for deprecated labels like `macos-13`, which cause workers to hang in "waiting for runner" status and artificially inflate queue sizes. Also, review custom parallelism settings that might block valid jobs — the [GitHub Actions Policy](github-actions-policy.html) requires a job concurrency level of 20 or less per workflow, and recommends staying at or below 15. - - **Group your dependency updates**: An ungrouped Dependabot configuration opens one pull request, and one full CI run, per dependency. Combine them with `groups`, and add `applies-to: security-updates` so that security fixes are batched as well instead of arriving as a stream of single-dependency PRs. See [grouping security updates](dependabot.html#group-security-updates) for the configuration. - - **Use lightweight runners**: For small, short jobs, consider using `runs-on: ubuntu-slim`. These provide 1 vCPU, 5GB RAM, no Docker engine, and a 15-minute hard limit. They churn much faster during peak times, allowing you to bypass the standard `ubuntu-latest` queue while freeing up capacity for others. + - **Optimize PR workflows**: Run full test suites periodically on main merges or via scheduled "canary" builds rather than on every PR. Use `paths:` filters in GitHub Actions YAML, write custom scripts to determine dynamic build matrix outputs, or utilize tools like Apache Yetus or [Develocity](gradle.html) (develocity.apache.org) for predictive test selection. Contact Infra via a Jira ticket or by email to `users@infra` for advice on the best approach for your project. + - **Review workflows for anomalies**: Check for deprecated labels like `macos-13`, which cause workers to hang in "waiting for runner" status and artificially inflate queue sizes. Also, review custom parallelism settings that might block valid jobs. The [GitHub Actions Policy](github-actions-policy.html) requires a job concurrency level of 20 or less per workflow, and recommends staying at or below 15. + - **Group your dependency updates**: An ungrouped Dependabot configuration opens one pull request and one full CI run per dependency. Combine them with `groups`, and add `applies-to: security-updates` so that security fixes are batched as well instead of arriving as a stream of single-dependency PRs. See [grouping security updates](dependabot.html#group-security-updates) for the configuration. + - **Pin first-party actions to a major version tag**: The [GitHub Actions Policy](github-actions-policy.html) requires a specific git hash only for **external** actions; actions in the `apache/*`, `github/*` and `actions/*` namespaces may be used without that restriction. Pinning those to a major version tag rather than to a hash lets a patch or minor release arrive without a pull request, which removes a large share of the Dependabot traffic a project sees. External actions still **MUST** be pinned to a specific git hash. + - **Reuse workflows across repositories**: A project with several repositories can keep its shared CI in reusable workflows in one repository and call them from the rest. An action or dependency update is then reviewed and tested once rather than once per repository, which cuts both the number of Dependabot pull requests and the CI runs behind them. + - **Use lightweight runners**: For small, short jobs, consider using `runs-on: ubuntu-slim`. These provide 1 vCPU, 5GB RAM, no Docker engine, and a 15-minute hard limit. They churn much more quickly during peak times, allowing you to bypass the standard `ubuntu-latest` queue while freeing up capacity for others. - **Explore self-hosted runners**: Speak with the Infra team about setting up project-specific [self-hosted runners](self-hosted-runners.html), particularly if you have targeted donations or cloud credits available.

Product naming