Skip to content

CI takes hours to start #25088

Open
Open

Description

@blaginin

Describe the bug

Created more for visibility to others. CI now takes a long time to start across many PRs, both on push and in the merge queue. This seems to have started sometime between 2026-09-08 07:30 and 08:44 UTC.

Examples:
https://github.com/apache/datafusion/actions/runs/34238609607/job/102102700226?pr=25081
#25081
https://github.com/apache/datafusion/actions/runs/34238609420

Additional context

I believe that's ASF issue, https://issues.apache.org/jira/browse/INFRA-28374

Activity

  1. added theissue type on Sep 8, 2026
  2. added
    bugSomething isn't working
    development-processRelated to development process of DataFusion
    on Sep 8, 2026
  3. alamb commented on Sep 8, 2026

    @alamb
    Contributor

    I hit a bunch of timeouts when trying to merge PRs using the merge queue in arrow-rs-object-store:

  4. alamb commented on Sep 10, 2026

    @alamb
    Contributor

    The root cause appears to be our DataFusion github runner usage is consuming all the github runners:

  5. alamb commented on Oct 20, 2026

    @alamb
    Contributor

    This has been happening again a lot recently (taking hours to start running)

    I have been exploring what apache projects are consuming jobs and Claude came up with this one

    gh api --paginate "repos/apache/airflow/actions/runs?status=in_progress&per_page=100" \
      --jq '.workflow_runs[] | select(.head_repository.full_name == "apache/airflow") | "\(.id) \(.head_sha) \(.html_url)"' \
    | while read -r id sha runurl; do
        link=$(gh api "repos/apache/airflow/commits/$sha/pulls" --jq '.[0].html_url // empty' 2>/dev/null)
        gh api --paginate "repos/apache/airflow/actions/runs/$id/jobs?per_page=100" \
          --jq '.jobs[] | select(.status=="in_progress") | select((.labels|join(","))|test("self-hosted")|not) | "\(.labels|join(","))\t\(.started_at)\t\(.workflow_name) / \(.name)"' \
        | sed "s|^|${link:-$runurl}\t|"
      done | sort -t$'\t' -k1,1 -k3,3

    The current output is

      done | sort -t$'\t' -k1,1 -k3,3
    https://github.com/apache/airflow/pull/74789	ubuntu-22.04	2026-10-20T17:32:54Z	Tests (AMD) / MySQL tests: core / DB-core:MySQL:8.0:3.11:API...CLI
    https://github.com/apache/airflow/pull/74789	ubuntu-22.04	2026-10-20T17:32:54Z	Tests (AMD) / MySQL tests: core / DB-core:MySQL:8.0:3.11:Core...Serialization
    https://github.com/apache/airflow/pull/74789	ubuntu-22.04	2026-10-20T17:33:09Z	Tests (AMD) / Low dep tests:core / core:LowestDeps:14:3.11:Core...Serialization
    https://github.com/apache/airflow/pull/74789	ubuntu-22.04	2026-10-20T17:33:09Z	Tests (AMD) / Postgres tests: core / DB-core:Postgres:14:3.11:API...CLI
    https://github.com/apache/airflow/pull/74789	ubuntu-22.04	2026-10-20T17:33:10Z	Tests (AMD) / Postgres tests: core / DB-core:Postgres:14:3.11:Core...Serialization
    https://github.com/apache/airflow/pull/74789	ubuntu-22.04	2026-10-20T17:33:42Z	Tests (AMD) / Sqlite tests: core / DB-core:Sqlite:3.11:Core...Serialization
    https://github.com/apache/airflow/pull/74790	ubuntu-22.04	2026-10-20T17:32:51Z	Tests (AMD) / Sqlite tests: core / DB-core:Sqlite:3.11:API...CLI
    https://github.com/apache/airflow/pull/74790	ubuntu-22.04	2026-10-20T17:32:57Z	Tests (AMD) / Postgres tests: core / DB-core:Postgres:14:3.11:API...CLI
    https://github.com/apache/airflow/pull/74790	ubuntu-22.04	2026-10-20T17:32:58Z	Tests (AMD) / Postgres tests: core / DB-core:Postgres:14:3.11:Core...Serialization
    https://github.com/apache/airflow/pull/74790	ubuntu-22.04	2026-10-20T17:32:58Z	Tests (AMD) / Sqlite tests: core / DB-core:Sqlite:3.11:Core...Serialization
    https://github.com/apache/airflow/pull/74790	ubuntu-22.04	2026-10-20T17:33:06Z	Tests (AMD) / Low dep tests:core / core:LowestDeps:14:3.11:API...CLI
    https://github.com/apache/airflow/pull/74790	ubuntu-22.04	2026-10-20T17:33:06Z	Tests (AMD) / Low dep tests:core / core:LowestDeps:14:3.11:Core...Serialization
    https://github.com/apache/airflow/pull/74790	ubuntu-22.04	2026-10-20T17:33:06Z	Tests (AMD) / MySQL tests: core / DB-core:MySQL:8.0:3.11:API...CLI
    https://github.com/apache/airflow/pull/74790	ubuntu-22.04	2026-10-20T17:33:06Z	Tests (AMD) / Non-DB tests: providers / Non-DB-prov::3.11:-amazon,celer...standard
    https://github.com/apache/airflow/pull/74790	ubuntu-22.04	2026-10-20T17:33:07Z	Tests (AMD) / MySQL tests: core / DB-core:MySQL:8.0:3.11:Core...Serialization
    https://github.com/apache/airflow/pull/74790	ubuntu-22.04	2026-10-20T17:33:27Z	Tests (AMD) / Provider distributions tests / Compat 2.11.1:P3.11:-amazon,celer...standard
    https://github.com/apache/airflow/pull/74790	ubuntu-22.04	2026-10-20T17:33:27Z	Tests (AMD) / Provider distributions tests / Compat 3.0.6:P3.11:-amazon,celer...standard
    https://github.com/apache/airflow/pull/74790	ubuntu-22.04	2026-10-20T17:33:27Z	Tests (AMD) / Provider distributions tests / Compat 3.2.2:P3.11:-amazon,celer...standard
    https://github.com/apache/airflow/pull/74790	ubuntu-22.04	2026-10-20T17:33:27Z	Tests (AMD) / Provider distributions tests / Compat 3.3.2:P3.11:-amazon,celer...standard
    https://github.com/apache/airflow/pull/74790	ubuntu-22.04	2026-10-20T17:33:42Z	Tests (AMD) / Provider distributions tests / Compat 3.1.8:P3.11:-amazon,celer...standard
    https://github.com/apache/airflow/pull/74790	ubuntu-22.04	2026-10-20T17:47:29Z	Tests (AMD) / Build PROD images / Build PROD Regular image 3.11
    https://github.com/apache/airflow/pull/74791	ubuntu-22.04	2026-10-20T17:33:12Z	Tests (AMD) / Postgres tests: core / DB-core:Postgres:14:3.11:API...CLI
    https://github.com/apache/airflow/pull/74791	ubuntu-22.04	2026-10-20T17:33:12Z	Tests (AMD) / Postgres tests: core / DB-core:Postgres:14:3.11:Core...Serialization
    https://github.com/apache/airflow/pull/74791	ubuntu-22.04	2026-10-20T17:33:18Z	Tests (AMD) / CI image checks / Static checks
    https://github.com/apache/airflow/pull/74791	ubuntu-22.04	2026-10-20T17:33:21Z	Tests (AMD) / Provider distributions tests / Compat 3.2.2:P3.11:-amazon,celer...standard
    https://github.com/apache/airflow/pull/74791	ubuntu-22.04	2026-10-20T17:33:39Z	Tests (AMD) / Provider distributions tests / Compat 3.0.6:P3.11:-amazon,celer...standard
    https://github.com/apache/airflow/pull/74791	ubuntu-22.04	2026-10-20T17:33:39Z	Tests (AMD) / Provider distributions tests / Compat 3.3.2:P3.11:-amazon,celer...standard
    https://github.com/apache/airflow/pull/74791	ubuntu-22.04	2026-10-20T17:33:47Z	Tests (AMD) / MySQL tests: core / DB-core:MySQL:8.0:3.11:Core...Serialization
    https://github.com/apache/airflow/pull/74791	ubuntu-22.04	2026-10-20T17:33:48Z	Tests (AMD) / Provider distributions tests / Compat 2.11.1:P3.11:-amazon,celer...standard
    https://github.com/apache/airflow/pull/74791	ubuntu-22.04	2026-10-20T17:33:48Z	Tests (AMD) / Provider distributions tests / Compat 3.1.8:P3.11:-amazon,celer...standard
    https://github.com/apache/airflow/pull/74791	ubuntu-22.04	2026-10-20T17:33:50Z	Tests (AMD) / Sqlite tests: core / DB-core:Sqlite:3.11:Core...Serialization
    https://github.com/apache/airflow/pull/74791	ubuntu-22.04	2026-10-20T17:33:53Z	Tests (AMD) / Sqlite tests: core / DB-core:Sqlite:3.11:API...CLI
    https://github.com/apache/airflow/pull/74791	ubuntu-22.04	2026-10-20T17:34:00Z	Tests (AMD) / Low dep tests:core / core:LowestDeps:14:3.11:API...CLI
    https://github.com/apache/airflow/pull/74791	ubuntu-22.04	2026-10-20T17:34:00Z	Tests (AMD) / Low dep tests:core / core:LowestDeps:14:3.11:Core...Serialization
    https://github.com/apache/airflow/pull/74791	ubuntu-22.04	2026-10-20T17:34:03Z	Tests (AMD) / MySQL tests: core / DB-core:MySQL:8.0:3.11:API...CLI
    https://github.com/apache/airflow/pull/74791	ubuntu-22.04	2026-10-20T17:48:54Z	Tests (AMD) / Build PROD images / Build PROD Regular image 3.11
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:42:16Z	Tests (AMD) / Sqlite tests: core / DB-core:Sqlite:3.11:Core...Serialization
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:42:26Z	Tests (AMD) / Low dep tests:core / core:LowestDeps:14:3.11:API...CLI
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:42:26Z	Tests (AMD) / MySQL tests: core / DB-core:MySQL:8.0:3.11:API...CLI
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:42:26Z	Tests (AMD) / MySQL tests: core / DB-core:MySQL:8.0:3.11:Core...Serialization
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:42:32Z	Tests (AMD) / Provider distributions tests / Compat 3.2.2:P3.11:-amazon,celer...standard
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:42:39Z	Tests (AMD) / Provider distributions tests / Compat 3.0.6:P3.11:-amazon,celer...standard
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:42:39Z	Tests (AMD) / Provider distributions tests / Compat 3.1.8:P3.11:-amazon,celer...standard
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:42:39Z	Tests (AMD) / Provider distributions tests / Providers wheel tests
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:42:40Z	Tests (AMD) / Provider distributions tests / Compat 3.3.2:P3.11:-amazon,celer...standard
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:42:59Z	Tests (AMD) / CI image checks / Static checks
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:43:13Z	Tests (AMD) / Provider distributions tests / Compat 2.11.1:P3.11:-amazon,celer...standard
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:43:16Z	Tests (AMD) / Postgres tests: providers / DB-prov:Postgres:14:3.11:-amazon,celer...standard
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:43:17Z	Tests (AMD) / CI image checks / Build documentation
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:43:19Z	Tests (AMD) / MySQL tests: providers / DB-prov:MySQL:8.0:3.11:-amazon,celer...standard
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:43:19Z	Tests (AMD) / Sqlite tests: providers / DB-prov:Sqlite:3.11:-amazon,celer...standard
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:43:26Z	Tests (AMD) / Sqlite tests: core / DB-core:Sqlite:3.11:API...CLI
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:43:32Z	Tests (AMD) / Low dep tests: providers / prov:LowestDeps:14:3.11:samba...zendesk
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:43:32Z	Tests (AMD) / Postgres tests: core / DB-core:Postgres:14:3.11:Core...Serialization
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:43:35Z	Tests (AMD) / Low dep tests: providers / prov:LowestDeps:14:3.11:apprise...duckdb
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:43:39Z	Tests (AMD) / Generate constraints / Generate constraints for 3.11 on linux/amd64
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:43:52Z	Tests (AMD) / Low dep tests: providers / prov:LowestDeps:14:3.11:airbyte...apache.tinker
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:43:52Z	Tests (AMD) / Postgres tests: core / DB-core:Postgres:14:3.11:API...CLI
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:44:14Z	Tests (AMD) / Low dep tests:core / core:LowestDeps:14:3.11:Core...Serialization
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:44:31Z	Tests (AMD) / Low dep tests: providers / prov:LowestDeps:14:3.11:microsoft.psr...salesforce
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:44:43Z	Tests (AMD) / Low dep tests: providers / prov:LowestDeps:14:3.11:edge3...microsoft.mss
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:44:43Z	Tests (AMD) / Special tests / Latest SQLAlchemy test: core / DB-core:LatestSQLAlchemy-Postgres:14:3.11:API...CLI
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:44:43Z	Tests (AMD) / Special tests / Min SQLAlchemy test: providers / DB-prov:MinSQLAlchemy-Postgres:14:3.11:-amazon,celer...standard
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:44:44Z	Tests (AMD) / Special tests / Latest SQLAlchemy test: core / DB-core:LatestSQLAlchemy-Postgres:14:3.11:Core...Serialization
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:44:50Z	Tests (AMD) / Special tests / Min SQLAlchemy test: core / DB-core:MinSQLAlchemy-Postgres:14:3.11:Core...Serialization
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:44:53Z	Tests (AMD) / Special tests / Latest Boto test: providers / prov:LatestBoto-Postgres:14:3.11:-amazon,celer...standard
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:44:53Z	Tests (AMD) / Special tests / Pendulum2 test: core / core:Pendulum2-Postgres:14:3.11:API...CLI
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:45:02Z	Tests (AMD) / Special tests / Latest Boto test: core / core:LatestBoto-Postgres:14:3.11:API...CLI
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:45:02Z	Tests (AMD) / Special tests / Latest Boto test: core / core:LatestBoto-Postgres:14:3.11:Core...Serialization
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:45:02Z	Tests (AMD) / Special tests / Pendulum2 test: core / core:Pendulum2-Postgres:14:3.11:Core...Serialization
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:45:18Z	Tests (AMD) / Special tests / Latest SQLAlchemy test: providers / DB-prov:LatestSQLAlchemy-Postgres:14:3.11:-amazon,celer...standard
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:45:18Z	Tests (AMD) / Special tests / Min SQLAlchemy test: core / DB-core:MinSQLAlchemy-Postgres:14:3.11:API...CLI
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:45:18Z	Tests (AMD) / Special tests / Pendulum2 test: providers / prov:Pendulum2-Postgres:14:3.11:-amazon,celer...standard
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:45:18Z	Tests (AMD) / Special tests / Quarantined test: providers / Qrnt-prov:Postgres:14:3.11:-amazon,celer...standard
    https://github.com/apache/airflow/pull/74793	ubuntu-22.04	2026-10-20T17:45:23Z	Tests (AMD) / Non-DB tests: providers / Non-DB-prov::3.11:-amazon,celer...standard
    https://github.com/apache/airflow/pull/74794	ubuntu-22.04	2026-10-20T17:41:10Z	Tests (AMD) / MySQL tests: core / DB-core:MySQL:8.0:3.11:API...CLI
    https://github.com/apache/airflow/pull/74794	ubuntu-22.04	2026-10-20T17:41:10Z	Tests (AMD) / MySQL tests: core / DB-core:MySQL:8.0:3.11:Core...Serialization
    https://github.com/apache/airflow/pull/74794	ubuntu-22.04	2026-10-20T17:41:12Z	Tests (AMD) / Low dep tests:core / core:LowestDeps:14:3.11:API...CLI
    https://github.com/apache/airflow/pull/74794	ubuntu-22.04	2026-10-20T17:41:27Z	Tests (AMD) / Low dep tests:core / core:LowestDeps:14:3.11:Core...Serialization
    https://github.com/apache/airflow/pull/74794	ubuntu-22.04	2026-10-20T17:41:37Z	Tests (AMD) / Provider distributions tests / Providers wheel tests
    https://github.com/apache/airflow/pull/74794	ubuntu-22.04	2026-10-20T17:42:22Z	Tests (AMD) / Postgres tests: core / DB-core:Postgres:14:3.11:Core...Serialization
    https://github.com/apache/airflow/pull/74794	ubuntu-22.04	2026-10-20T17:42:23Z	Tests (AMD) / Postgres tests: core / DB-core:Postgres:14:3.11:API...CLI
    https://github.com/apache/airflow/pull/74794	ubuntu-22.04	2026-10-20T17:42:23Z	Tests (AMD) / Sqlite tests: core / DB-core:Sqlite:3.11:API...CLI
    https://github.com/apache/airflow/pull/74794	ubuntu-22.04	2026-10-20T17:42:23Z	Tests (AMD) / Sqlite tests: core / DB-core:Sqlite:3.11:Core...Serialization
    https://github.com/apache/airflow/pull/75231	ubuntu-22.04	2026-10-20T16:59:23Z	Tests (AMD) / Sqlite tests: core / DB-core:Sqlite:3.11:Core...Serialization
    https://github.com/apache/airflow/pull/75231	ubuntu-22.04	2026-10-20T17:41:30Z	Tests (AMD) / Kubernetes tests / K8S System:LocalExecutor-3.12-v1.32.11-true
    https://github.com/apache/airflow/pull/75231	ubuntu-22.04	2026-10-20T17:41:40Z	Tests (AMD) / Kubernetes tests / K8S System:LocalExecutor-3.14-v1.34.11-true
    https://github.com/apache/airflow/pull/75231	ubuntu-22.04	2026-10-20T17:41:43Z	Tests (AMD) / Kubernetes tests / K8S System:LocalExecutor-3.12-v1.32.11-false
    https://github.com/apache/airflow/pull/75231	ubuntu-22.04	2026-10-20T17:41:48Z	Tests (AMD) / Kubernetes tests / K8S System:LocalExecutor-3.11-v1.31.12-false
    https://github.com/apache/airflow/pull/75231	ubuntu-22.04-arm	2026-10-20T17:42:52Z	Tests (ARM) / Kubernetes tests / K8S System:LocalExecutor-3.13-v1.37.0-false
    https://github.com/apache/airflow/pull/75231	ubuntu-22.04	2026-10-20T17:43:54Z	Tests (AMD) / Kubernetes tests / K8S System:LocalExecutor-3.14-v1.34.11-false
    https://github.com/apache/airflow/pull/75231	ubuntu-22.04	2026-10-20T17:44:14Z	Tests (AMD) / Kubernetes tests / K8S System:LocalExecutor-3.11-v1.35.8-false
    https://github.com/apache/airflow/pull/75231	ubuntu-22.04	2026-10-20T17:50:59Z	Tests (AMD) / Kubernetes tests / K8S System:LocalExecutor-3.12-v1.36.4-false
    https://github.com/apache/airflow/pull/75231	ubuntu-22.04	2026-10-20T17:50:59Z	Tests (AMD) / Kubernetes tests / K8S System:LocalExecutor-3.13-v1.37.0-false

    It looks like a few PRs consumed a lot of the runner time (each launches 14 jobs via some matrix)

  6. alamb commented on Oct 20, 2026

    @alamb
    Contributor

    The TLDR of my poking around is that some projects have many queued jobs (often from matrix'd runs) that are consuming significant numbers of runners and the DataFusion (and arrow-rs backlogs) are queued.

    I'll ask the projects politely if there is some way to reduce the runner usage

  7. mbutrovich commented on Oct 20, 2026

    @mbutrovich
    Contributor

    I'll ask the projects politely if there is some way to reduce the runner usage

    Thank you!

  8. alamb commented on Oct 20, 2026

    @alamb
    Contributor

    Current queue across projects

    Queued jobs, repos with 10+ queued runs (jobs are what the runner pool actually serves):
    
    ┌──────────────────────────────┬─────────────┬─────────────┬──────────────────┐
    │             Repo             │ Queued jobs │ Queued runs │ In-progress runs │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ datafusion                   │ 537         │ 108         │ 0                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ nuttx                        │ 349         │ 75          │ 0                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ gravitino                    │ 280         │ 220         │ 2                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ spark                        │ 223         │ 30          │ 3                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ shenyu                       │ 200         │ 62          │ 3                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ arrow-rs                     │ 192         │ 30          │ 0                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ datafusion-sqlparser-rs      │ 187         │ 61          │ 1                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ iceberg-go                   │ 154         │ 91          │ 0                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ druid                        │ 130         │ 31          │ 14               │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ shardingsphere               │ 120         │ 89          │ 0                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ mahout                       │ 118         │ 103         │ 0                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ skywalking-banyandb          │ 100         │ 15          │ 0                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ doris                        │ 99          │ 100         │ 17               │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ grails-core                  │ 97          │ 44          │ 13               │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ ossie                        │ 92          │ 33          │ 0                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ paimon                       │ 88          │ 18          │ 0                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ rocketmq-dashboard           │ 85          │ 17          │ 0                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ maven                        │ 80          │ 18          │ 0                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ tomcat                       │ 76          │ 51          │ 2                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ gluten                       │ 70          │ 55          │ 25               │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ nuttx-apps                   │ 68          │ 22          │ 0                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ pinot                        │ 68          │ 40          │ 1                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ superset-kubernetes-operator │ 64          │ 29          │ 0                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ seatunnel                    │ 60          │ 57          │ 0                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ texera                       │ 59          │ 62          │ 1                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ hudi                         │ 54          │ 30          │ 0                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ mynewt-core                  │ 49          │ 12          │ 0                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ maven-resolver               │ 45          │ 10          │ 0                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ iceberg-python               │ 44          │ 16          │ 0                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ camel                        │ 32          │ 25          │ 0                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ ignite                       │ 30          │ 10          │ 0                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ datafusion-ballista          │ 27          │ 12          │ 0                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ accumulo                     │ 25          │ 10          │ 1                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ hbase                        │ 23          │ 10          │ 0                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ aries                        │ 21          │ 17          │ 0                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ fluss                        │ 20          │ 12          │ 0                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ iceberg                      │ 16          │ 16          │ 3                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ hop                          │ 13          │ 10          │ 0                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ iggy                         │ 10          │ 10          │ 0                │
    ├──────────────────────────────┼─────────────┼─────────────┼──────────────────┤
    │ superset                     │ 9           │ 16          │ 85               │
    └──────────────────────────────┴─────────────┴─────────────┴──────────────────┘
    
  9. tballison commented on Oct 21, 2026

    @tballison

    Over on Tika, we're also seeing > hour long queues. It felt like there was a change a bit more than two weeks ago. The lengthy queues cannot be attributed to post Community Over Code exuberance.

    This is not good.

    Thank you for opening this ticket.

  10. alamb commented on Oct 21, 2026

    @alamb
    Contributor

    Over on Tika, we're also seeing > hour long queues. It felt like there was a change a bit more than two weeks ago. The lengthy queues cannot be attributed to post Community Over Code exuberance.

    I suspect it may be the fact that it is far easier than ever to make PRs via ai agents, and each PR often starts large amounts of CI

    Also the new github stacks feature was released around Oct 6 (approximately the right timeline): https://github.blog/changelog/2026-10-06-stacked-pull-requests-generally-available/

    And I found at least one example where a stack of PRs ended up running the full CI suite for each stacked PR each time the stack got rebased:

    I don't have more data to back up that yet, but will keep searching

  11. alamb commented on Oct 21, 2026

    @alamb
    Contributor

    Here is another snapshot

    Runner usage and queue across apache repos, snapshot 2026-10-21 12:45 UTC
    (jobs are what the runner pool serves; a run fans out into many jobs)
    
    ┌─────────────────────────┬──────────────┬─────────────┬──────────────┬─────────────┐
    │ Repo                    │ Running jobs │ Queued jobs │ Running runs │ Queued runs │
    ├─────────────────────────┼──────────────┼─────────────┼──────────────┼─────────────┤
    │ airflow                 │ 188          │ ?           │ 15           │ 4           │
    │ kvrocks                 │ 67           │ 0           │ 3            │ 0           │
    │ fineract                │ 60           │ ?           │ 2            │ 1           │
    │ iotdb                   │ 50           │ ?           │ 20           │ 7           │
    │ dubbo                   │ 46           │ 0           │ 1            │ 0           │
    │ gravitino               │ 40           │ 200         │ 29           │ 173         │
    │ beam                    │ 39           │ ?           │ 22           │ 8           │
    │ cloudstack              │ 30           │ ?           │ 9            │ 6           │
    │ superset                │ 22           │ 8           │ 21           │ 13          │
    │ kafka                   │ 21           │ ?           │ 9            │ 3           │
    │ activemq                │ 20           │ ?           │ 13           │ 6           │
    │ ozone                   │ 19           │ ?           │ 4            │ 1           │
    │ datafusion-comet        │ 19           │ 0           │ 2            │ 0           │
    │ paimon                  │ 15           │ 80          │ 3            │ 17          │
    │ druid                   │ 13           │ 210         │ 2            │ 43          │
    │ flink                   │ 11           │ 0           │ 1            │ 0           │
    │ nifi                    │ 11           │ 0           │ 5            │ 0           │
    │ groovy                  │ 10           │ 20          │ 3            │ 10          │
    │ arrow                   │ 9            │ 0           │ 5            │ 0           │
    │ spark                   │ 6            │ 147         │ 1            │ 25          │
    │ gluten                  │ 6            │ 29          │ 6            │ 21          │
    │ flink-agents            │ 6            │ ?           │ 1            │ 5           │
    │ tika                    │ 5            │ 52          │ 6            │ 35          │
    │ yunikorn-k8shim         │ 5            │ 0           │ 1            │ 0           │
    │ doris                   │ 4            │ 439         │ 2            │ 428         │
    │ grails-core             │ 4            │ 47          │ 6            │ 20          │
    │ incubator-kie           │ 4            │ 19          │ 2            │ 14          │
    │ doris-flink-connector   │ 4            │ ?           │ 3            │ 9           │
    │ hop                     │ 3            │ 18          │ 2            │ 14          │
    │ brpc                    │ 3            │ ?           │ 1            │ 9           │
    │ camel                   │ 2            │ 107         │ 1            │ 70          │
    │ fluss                   │ 2            │ 105         │ 2            │ 43          │
    │ datafusion              │ 1            │ 123         │ 2            │ 61          │
    │ tomcat                  │ 1            │ 64          │ 1            │ 43          │
    │ hudi                    │ 1            │ 46          │ 1            │ 27          │
    │ incubator-kie-tools     │ 1            │ 13          │ 1            │ 11          │
    │ nuttx                   │ 0            │ 577         │ 1            │ 103         │
    │ arrow-rs                │ 0            │ 446         │ 0            │ 160         │
    │ opendal                 │ 0            │ 288         │ 0            │ 57          │
    │ texera                  │ 0            │ 243         │ 1            │ 201         │
    │ shenyu                  │ 0            │ 235         │ 0            │ 144         │
    │ shardingsphere          │ 0            │ 144         │ 2            │ 63          │
    │ iceberg-go              │ 0            │ 81          │ 0            │ 52          │
    │ hugegraph-ai            │ 0            │ 78          │ 0            │ 32          │
    │ ossie                   │ 0            │ 76          │ 0            │ 36          │
    │ nuttx-apps              │ 0            │ 75          │ 0            │ 19          │
    │ devlake                 │ 0            │ 69          │ 0            │ 55          │
    │ datafusion-sqlparser-rs │ 0            │ 51          │ 0            │ 17          │
    │ calcite                 │ 0            │ 50          │ 0            │ 10          │
    │ ossie-converters        │ 0            │ 41          │ 1            │ 26          │
    │ pinot                   │ 0            │ 40          │ 0            │ 26          │
    │ syncope                 │ 0            │ 34          │ 0            │ 33          │
    │ seatunnel               │ 0            │ 32          │ 0            │ 31          │
    │ cassandra               │ 0            │ 31          │ 0            │ 19          │
    │ hertzbeat               │ 0            │ 23          │ 0            │ 18          │
    │ ofbiz-framework         │ 0            │ 21          │ 0            │ 16          │
    │ rocketmq-clients        │ 0            │ 20          │ 0            │ 17          │
    │ poi                     │ 0            │ ?           │ 0            │ 5           │
    │ logging-log4j2          │ 0            │ ?           │ 0            │ 5           │
    │ ignite                  │ 0            │ ?           │ 0            │ 9           │
    │ tinkerpop               │ 0            │ ?           │ 0            │ 5           │
    │ netbeans                │ 0            │ ?           │ 0            │ 7           │
    │ streampipes             │ 0            │ ?           │ 0            │ 6           │
    │ skywalking-banyandb     │ 0            │ ?           │ 0            │ 9           │
    │ reqsign                 │ 0            │ ?           │ 0            │ 9           │
    │ amoro                   │ 0            │ ?           │ 0            │ 6           │
    │ fesod                   │ 0            │ ?           │ 0            │ 6           │
    └─────────────────────────┴──────────────┴─────────────┴──────────────┴─────────────┘
    Not shown: 96 repos with <5 running jobs, <10 queued jobs and <5 queued runs,
    holding 13 running jobs and 186 queued runs in total.
    Totals: running jobs 761 | queued jobs in scanned repos 4382 | queued runs 2515 | running runs 230
    
  12. alamb commented on Oct 21, 2026

    @alamb
    Contributor

    I spent some time on some more analytics this morning on how many runner minutes are used by different projects

    Here is the data + script I used: Archive.zip

    Note this only includes the github hosted runners (not the RunsOn runners):

    My numbers show that airflow has used used 77.2 days of runner time in the last 17h

    1000 runs of apache/airflow created between 2026-10-20T22:20:11Z and 2026-10-21T15:10:08Z (16.8 h window)
    jobs loaded: 900 runs from cache, 98 from the API, 2 failed  (cache: gh-runner-usage-cache)
    
    conclusion     jobs  share runner-min runner-days
    success       12235  57.1%     111218        77.2
    skipped        7424  34.7%          0         0.0
    cancelled      1234   5.8%       5745         4.0
    in_progress     274   1.3%          0         0.0
    failure         186   0.9%       2304         1.6
    queued           65   0.3%          0         0.0
    total         21418            119267        82.8
    
    average concurrent jobs over the window: 118
    

    By comparison DataFusion only used 11.7 days of runner time in the last 2 days

    1000 runs of apache/arrow-rs created between 2026-10-19T10:20:44Z and 2026-10-21T15:06:27Z (52.8 h window)
    jobs loaded: 0 runs from cache, 1000 from the API, 0 failed  (cache: gh-runner-usage-cache)
    
    conclusion     jobs  share runner-min runner-days
    success        3492  69.7%      16846        11.7
    skipped         913  18.2%          0         0.0
    queued          327   6.5%          0         0.0
    failure         141   2.8%        853         0.6
    cancelled       125   2.5%       1946         1.4
    in_progress      12   0.2%          0         0.0
    total          5010             19645        13.6
    
    average concurrent jobs over the window: 6
    

    arrow-rs used 2.7 days over 17 hours

    1000 runs of apache/datafusion created between 2026-10-20T22:07:51Z and 2026-10-21T15:13:03Z (17.1 h window)
    warning: jobs for run 39541706327 failed: gh: Server Error (HTTP 502)
    jobs loaded: 69 runs from cache, 930 from the API, 1 failed  (cache: gh-runner-usage-cache)
    
    conclusion     jobs  share runner-min runner-days
    skipped        1757  48.8%          0         0.0
    success        1410  39.2%       3943         2.7
    queued          212   5.9%          0         0.0
    cancelled       184   5.1%       2762         1.9
    failure          24   0.7%        140         0.1
    neutral           7   0.2%          0         0.0
    in_progress       5   0.1%          0         0.0
    total          3599              6845         4.8
    
    average concurrent jobs over the window: 7
    
  13. alamb commented on Oct 21, 2026

    @alamb
    Contributor

    There are also some great discussions in the ASF slack
    https://the-asf.slack.com/archives/CBX4TSBQ8/p1792591324182189?thread_ts=1792520326.623569&cid=CBX4TSBQ8

    That have come to the same conclusion

  14. 11 remaining items

  15. alamb commented on Oct 23, 2026

    @alamb
    Contributor

    There is an interesting script that @potiuk pointed me me at in apache/airflow#75395 (reply in thread)

    https://github.com/apache/infrastructure-actions/blob/main/utils/actions-queue-status.py

    It took 30 minutes to run and here is the output.

    Summary line from the script: 677 jobs running, 2073 queued across 151 of 1253 apache repos with Actions.
    
    Top 30 by running
    
    ┌──────────────────┬────────┬─────────┬──────────────────────────────────────────────────────────────┐
    │       repo       │ queued │ running │                          workflows                           │
    ├──────────────────┼────────┼─────────┼──────────────────────────────────────────────────────────────┤
    │ airflow          │      0 │     120 │ Tests (AMD)=112; Tests (ARM)=8                               │
    ├──────────────────┼────────┼─────────┼──────────────────────────────────────────────────────────────┤
    │ grails-core      │     19 │      77 │ CI=81; Build Health=2; CodeQL=2; ...                         │
    ├──────────────────┼────────┼─────────┼──────────────────────────────────────────────────────────────┤
    │ beam             │      0 │      53 │ Build python source distribution and wheels=4; ...           │
    ├──────────────────┼────────┼─────────┼──────────────────────────────────────────────────────────────┤
    │ cloudstack       │      9 │      45 │ CodeQL Analysis=9; Build=1; Coverage Check=1; ...            │
    ├──────────────────┼────────┼─────────┼──────────────────────────────────────────────────────────────┤
    │ superset         │     12 │      40 │ Build & publish docker images=3; CodeQL=1; ...               │
    ├──────────────────┼────────┼─────────┼──────────────────────────────────────────────────────────────┤
    │ groovy           │     94 │      36 │ Build and test=61; Build and test for coverage=5; ...        │
    ├──────────────────┼────────┼─────────┼──────────────────────────────────────────────────────────────┤
    │ nifi             │      0 │      29 │ build=12; code-compliance=3; docker-tests=1; ...             │
    ├──────────────────┼────────┼─────────┼──────────────────────────────────────────────────────────────┤
    │ ozone            │      2 │      28 │ build-branch=28; Running Copilot Code Review=1; ...          │
    ├──────────────────┼────────┼─────────┼──────────────────────────────────────────────────────────────┤
    │ iotdb            │      0 │      23 │ Daily IT=21; Daily UT=2                                      │
    ├──────────────────┼────────┼─────────┼──────────────────────────────────────────────────────────────┤
    │ nuttx-apps       │     32 │      16 │ Build=43; Check=1; Lint=1; ...                               │
    ├──────────────────┼────────┼─────────┼──────────────────────────────────────────────────────────────┤
    │ iggy             │      2 │      16 │ Pre-merge=17; PR Title=1                                     │
    ├──────────────────┼────────┼─────────┼──────────────────────────────────────────────────────────────┤
    │ pulsar           │      6 │      15 │ Pulsar CI=16; Semantic Pull Request=4; CodeQL=1              │
    ├──────────────────┼────────┼─────────┼──────────────────────────────────────────────────────────────┤
    │ tinkerpop        │     14 │      14 │ build-test=28                                                │
    ├──────────────────┼────────┼─────────┼──────────────────────────────────────────────────────────────┤
    │ commons-io       │     37 │      13 │ Java CI=42; CodeQL=4; Dependency Review=2; ...               │
    ├──────────────────┼────────┼─────────┼──────────────────────────────────────────────────────────────┤
    │ camel-quarkus    │     30 │      12 │ Camel Quarkus CI=42                                          │
    ├──────────────────┼────────┼─────────┼──────────────────────────────────────────────────────────────┤
    │ paimon           │     67 │       9 │ CI=76                                                        │
    ├──────────────────┼────────┼─────────┼──────────────────────────────────────────────────────────────┤
    │ netbeans         │     11 │       9 │ NetBeans=20                                                  │
    ├──────────────────┼────────┼─────────┼──────────────────────────────────────────────────────────────┤
    │ hudi             │      0 │       9 │ Java CI=8; Scheduled Workflow=1                              │
    ├──────────────────┼────────┼─────────┼──────────────────────────────────────────────────────────────┤
    │ fluss            │     30 │       7 │ CI=24; Check License=3; Check Documentation=1; ...           │
    ├──────────────────┼────────┼─────────┼──────────────────────────────────────────────────────────────┤
    │ cloudberry       │      3 │       7 │ Apache Cloudberry Build (Rocky Linux)=6; ...                 │
    ├──────────────────┼────────┼─────────┼──────────────────────────────────────────────────────────────┤
    │ iceberg-rust     │      2 │       6 │ CI=6; Bindings Python CI=1; pages build=1                    │
    ├──────────────────┼────────┼─────────┼──────────────────────────────────────────────────────────────┤
    │ kafka            │      0 │       6 │ CI=6                                                         │
    ├──────────────────┼────────┼─────────┼──────────────────────────────────────────────────────────────┤
    │ camel            │     26 │       5 │ Build and test=12; Camel launcher native exe build=1; ...    │
    ├──────────────────┼────────┼─────────┼──────────────────────────────────────────────────────────────┤
    │ datafusion       │      8 │       5 │ Rust=6; Detect breaking changes=6; Dependencies=1            │
    ├──────────────────┼────────┼─────────┼──────────────────────────────────────────────────────────────┤
    │ iceberg          │      3 │       5 │ Spark CI=5; Docs Build CI=2; PR Title Check=1                │
    ├──────────────────┼────────┼─────────┼──────────────────────────────────────────────────────────────┤
    │ cxf              │      0 │       5 │ Build and Test=3; CodeQL=2                                   │
    ├──────────────────┼────────┼─────────┼──────────────────────────────────────────────────────────────┤
    │ hbase            │      0 │       5 │ Yetus General Check=1; Yetus JDK17 Hadoop3 Compile Check=... │
    ├──────────────────┼────────┼─────────┼──────────────────────────────────────────────────────────────┤
    │ incubator-kie    │      0 │       5 │ CI :: Build=2; CI :: Examples=3                              │
    ├──────────────────┼────────┼─────────┼──────────────────────────────────────────────────────────────┤
    │ datafusion-comet │      0 │       4 │ Comet CI=2; Comet CI (label run)=2                           │
    ├──────────────────┼────────┼─────────┼──────────────────────────────────────────────────────────────┤
    │ drill            │      0 │       4 │ Github CI=3; CodeQL=1                                        │
    └──────────────────┴────────┴─────────┴──────────────────────────────────────────────────────────────┘
    
    Top 30 by queued
    
    ┌────────────────────────┬────────┬─────────┬────────────────────────────────────────────────────────────────┐
    │          repo          │ queued │ running │                           workflows                            │
    ├────────────────────────┼────────┼─────────┼────────────────────────────────────────────────────────────────┤
    │ cassandra-gocql-driver │    162 │       0 │ Build=162                                                      │
    ├────────────────────────┼────────┼─────────┼────────────────────────────────────────────────────────────────┤
    │ nuttx                  │    141 │       0 │ Build=23; Check=3; Build Documentation=1; ...                  │
    ├────────────────────────┼────────┼─────────┼────────────────────────────────────────────────────────────────┤
    │ sedona-db              │    135 │       0 │ Graph Update: pip in /python/sedonadb=1; cargo ...             │
    ├────────────────────────┼────────┼─────────┼────────────────────────────────────────────────────────────────┤
    │ shenyu                 │    125 │       1 │ it=56; ci=22; it-k8s=21; e2e=7; Issue Manager=2; ...           │
    ├────────────────────────┼────────┼─────────┼────────────────────────────────────────────────────────────────┤
    │ aries                  │    120 │       0 │ Async - CI Build=2; Blueprint - CI Build=2; ...                │
    ├────────────────────────┼────────┼─────────┼────────────────────────────────────────────────────────────────┤
    │ groovy                 │     94 │      36 │ Build and test=61; Build and test for coverage=5; ...          │
    ├────────────────────────┼────────┼─────────┼────────────────────────────────────────────────────────────────┤
    │ texera                 │     77 │       1 │ Backport Approval Check=15; Backport Auto Label=3; ...         │
    ├────────────────────────┼────────┼─────────┼────────────────────────────────────────────────────────────────┤
    │ shardingsphere         │     76 │       0 │ CI=3; E2E - Agent=3; CI - MCP=1; E2E - MCP=1; ...              │
    ├────────────────────────┼────────┼─────────┼────────────────────────────────────────────────────────────────┤
    │ spark                  │     75 │       0 │ Branch-4.2 CI Scheduler=2; Branch-4.x CI Scheduler=2; ...      │
    ├────────────────────────┼────────┼─────────┼────────────────────────────────────────────────────────────────┤
    │ iceberg-go             │     68 │       3 │ Audit and Verify=8; API compatibility=7; CodeQL=1; ...         │
    ├────────────────────────┼────────┼─────────┼────────────────────────────────────────────────────────────────┤
    │ paimon                 │     67 │       9 │ CI=76                                                          │
    ├────────────────────────┼────────┼─────────┼────────────────────────────────────────────────────────────────┤
    │ gravitino              │     59 │       3 │ Access Control Integration Test=1; ...                         │
    ├────────────────────────┼────────┼─────────┼────────────────────────────────────────────────────────────────┤
    │ pinot                  │     51 │       0 │ PR flow=9; PR flow signal=4; Pinot Compatibility Checks=1; ... │
    ├────────────────────────┼────────┼─────────┼────────────────────────────────────────────────────────────────┤
    │ commons-io             │     37 │      13 │ Java CI=42; CodeQL=4; Dependency Review=2; ...                 │
    ├────────────────────────┼────────┼─────────┼────────────────────────────────────────────────────────────────┤
    │ calcite                │     37 │       1 │ CI=35; Validate Gradle Wrapper=3                               │
    ├────────────────────────┼────────┼─────────┼────────────────────────────────────────────────────────────────┤
    │ tika                   │     36 │       2 │ Docker snapshot - tika-server and tika-grpc=7; ...             │
    ├────────────────────────┼────────┼─────────┼────────────────────────────────────────────────────────────────┤
    │ nuttx-apps             │     32 │      16 │ Build=43; Check=1; Lint=1; ...                                 │
    ├────────────────────────┼────────┼─────────┼────────────────────────────────────────────────────────────────┤
    │ httpd                  │     32 │       1 │ Linux=32; Windows=1                                            │
    ├────────────────────────┼────────┼─────────┼────────────────────────────────────────────────────────────────┤
    │ lucenenet              │     31 │       0 │ Index Compatibility=2; EditorConfig Rules Check=1; ...         │
    ├────────────────────────┼────────┼─────────┼────────────────────────────────────────────────────────────────┤
    │ camel-quarkus          │     30 │      12 │ Camel Quarkus CI=42                                            │
    ├────────────────────────┼────────┼─────────┼────────────────────────────────────────────────────────────────┤
    │ fluss                  │     30 │       7 │ CI=24; Check License=3; Check Documentation=1; ...             │
    ├────────────────────────┼────────┼─────────┼────────────────────────────────────────────────────────────────┤
    │ camel                  │     26 │       5 │ Build and test=12; ...                                         │
    ├────────────────────────┼────────┼─────────┼────────────────────────────────────────────────────────────────┤
    │ accumulo               │     24 │       0 │ QA=24                                                          │
    ├────────────────────────┼────────┼─────────┼────────────────────────────────────────────────────────────────┤
    │ tomcat                 │     23 │       2 │ CodeQL Advanced=6; Manager2 Test=3; Diff Coverage=2; ...       │
    ├────────────────────────┼────────┼─────────┼────────────────────────────────────────────────────────────────┤
    │ grails-core            │     19 │      77 │ CI=81; ...                                                     │
    ├────────────────────────┼────────┼─────────┼────────────────────────────────────────────────────────────────┤
    │ arrow-rs               │     19 │       0 │ CI=19                                                          │
    ├────────────────────────┼────────┼─────────┼────────────────────────────────────────────────────────────────┤
    │ cassandra              │     16 │       0 │ Ant Check=16                                                   │
    ├────────────────────────┼────────┼─────────┼────────────────────────────────────────────────────────────────┤
    │ ossie                  │     16 │       0 │ Auto Label PRs=2; Community PR Helper=1; ...                   │
    ├────────────────────────┼────────┼─────────┼────────────────────────────────────────────────────────────────┤
    │ poi                    │     16 │       0 │ Java CI with Ant=8; Java CI with Gradle=8                      │
    ├────────────────────────┼────────┼─────────┼────────────────────────────────────────────────────────────────┤
    │ commons-math           │     15 │       0 │ Java CI=12; Java CI (Legacy only)=3                            │
    └────────────────────────┴────────┴─────────┴────────────────────────────────────────────────────────────────┘
    

    One thing I think is different than my scripts above is that it counts the number of jobs but I don't think it reflects the runner time.

    So you can see that airflow has 120 running jobs and I suspect the reason it ends up with so many jobs is that each takes many minutes. Other projects have many jobs but I think each job takes much less time and thus doesn't consume so many runner hours

  16. potiuk commented on Oct 23, 2026

    @potiuk
    Member
  17. alamb commented on Oct 23, 2026

    @alamb
    Contributor

    No. It means that we have 10x more PRs opened from our contributors. Note -
    that it is 10x than we usually had, We are just a very popular project and
    we have more than 400 contributors. Look from how many PRs those jobs come.

    Yes, I see. We are feeling something similar in Arrow and DataFusion . DataFusion is only at 1200 contributors, but recently we have seen a deluge of lots of PRs and each new PR (including AI dumps) was consumes a lot of PR time on the apache shared runners

    I think the most effective thing we have done this past week (thank you @kumarUjjawal and @comphead 🙏 ) was to update our CI so Pull Requests ran on the submitters fork / account, not on the apache runners:

    Unfortunately this added quite a bit of complexity to our CI system, but I don't see any way around it.

    @sdf-jkl is working on the same treatment for arrow-rs

  18. alamb commented on Oct 23, 2026

    @alamb
    Contributor

    BTW grails-core (47 runner-days) and beam (12 runner-days) also consume significant amounts of CPU in the last 24 hours

    andrewlamb@Andrews-MacBook-Pro-3:~/Downloads$ python3 usage.py --time 1d apache/grails-core
    665 runs of apache/grails-core created since 2026-10-22T20:33:15Z (24.0 h window)
    warning: jobs for run 39971712658 failed: gh: Server Error (HTTP 502)
    warning: jobs for run 39967410613 failed: gh: Server Error (HTTP 502)
    warning: jobs for run 39876372833 failed: gh: Server Error (HTTP 502)
    jobs loaded: 280 runs from cache, 382 from the API, 3 failed  (cache: gh-runner-usage-cache)
    
                                 github               self-hosted
    conclusion     jobs   share     mins (days)   share     mins (days)
    success        2038   78.0%    53319 (37.0)       -         0 (0.0)
    failure          40    2.9%      1964 (1.4)       -         0 (0.0)
    in_progress      76    0.0%         0 (0.0)       -         0 (0.0)
    queued           19    0.0%         0 (0.0)       -         0 (0.0)
    cancelled       966   19.1%     13089 (9.1)       -         0 (0.0)
    skipped         797    0.0%         0 (0.0)       -         0 (0.0)
    total          3936            68372 (47.5)                 0 (0.0)
    
    andrewlamb@Andrews-MacBook-Pro-3:~/Downloads$ python3 usage.py --time 1d apache/beam
    2648 runs of apache/beam created since 2026-10-22T20:33:21Z (24.0 h window)
    jobs loaded: 1360 runs from cache, 1288 from the API, 0 failed  (cache: gh-runner-usage-cache)
    
                                 github               self-hosted
    conclusion     jobs   share     mins (days)   share     mins (days)
    success        4741   88.1%    15140 (10.5)   92.6%    68974 (47.9)
    failure         122    0.6%       110 (0.1)    4.9%      3659 (2.5)
    in_progress     129    0.0%         0 (0.0)    0.0%         0 (0.0)
    queued           11    0.0%         0 (0.0)    0.0%         0 (0.0)
    cancelled       525   11.3%      1944 (1.3)    2.5%      1828 (1.3)
    neutral           9    0.0%         0 (0.0)    0.0%         0 (0.0)
    skipped         300    0.0%         0 (0.0)    0.0%         0 (0.0)
    total          5837            17194 (11.9)            74461 (51.7)
    
    
  19. rok commented on Oct 23, 2026

    @rok
    Member

    On arrow we limit non-collaborators to 3 PR at a time. Will merge apache/arrow#55164 once apache/infrastructure-asfyaml#135 is complete.

  20. sdf-jkl commented on Oct 23, 2026

    @sdf-jkl
    Member

    @sdf-jkl is working on the same treatment for arrow-rs

    • https://github.com/apache/arrow-rs/pull/11788
      

    @rok I want to try same approach on arrow as well

  21. rok commented on Oct 23, 2026

    @rok
    Member

    @sdf-jkl let's check with @raulcd if he's comfortable with the extra logic. I think we used to do it at sompoint, but that's better over chat.

  22. rok commented on Oct 23, 2026

    @rok
    Member

    @sdf-jkl also please note - merging apache/arrow#55164 would solve this issue too.

  23. comphead commented on Oct 23, 2026

    @comphead
    Contributor

    Thing that concerns me is https://docs.github.com/en/actions/reference/security/securely-using-pull_request_target#default-policy-for-pull_request_target

    that disables pull_request_target even on public repos, which I believe also applicable for contributor forks

  24. comphead commented on Oct 23, 2026

    @comphead
    Contributor

    This is a migration example apache/spark-connect-swift#615

  25. rok commented on Oct 23, 2026

    @rok
    Member

    I wonder if GitHub will start blocking this pattern. Surely we're not only ones with so many PRs.

  26. potiuk commented on Oct 23, 2026

    @potiuk
    Member
  27. comphead commented on Oct 23, 2026

    @comphead
    Contributor

    draft to migrate to workflow_run similar to what Apache Spark currently has #26683

  28. potiuk commented on Oct 23, 2026

    @potiuk
    Member
  29. potiuk commented on Oct 23, 2026

    @potiuk
    Member
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingdevelopment-processRelated to development process of DataFusion

    Type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions