You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Created more for visibility to others. CI now takes a long time to start across many PRs, both on push and in the merge queue. This seems to have started sometime between 2026-09-08 07:30 and 08:44 UTC.
The TLDR of my poking around is that some projects have many queued jobs (often from matrix'd runs) that are consuming significant numbers of runners and the DataFusion (and arrow-rs backlogs) are queued.
I'll ask the projects politely if there is some way to reduce the runner usage
Over on Tika, we're also seeing > hour long queues. It felt like there was a change a bit more than two weeks ago. The lengthy queues cannot be attributed to post Community Over Code exuberance.
Over on Tika, we're also seeing > hour long queues. It felt like there was a change a bit more than two weeks ago. The lengthy queues cannot be attributed to post Community Over Code exuberance.
I suspect it may be the fact that it is far easier than ever to make PRs via ai agents, and each PR often starts large amounts of CI
Note this only includes the github hosted runners (not the RunsOn runners):
My numbers show that airflow has used used 77.2 days of runner time in the last 17h
1000 runs of apache/airflow created between 2026-10-20T22:20:11Z and 2026-10-21T15:10:08Z (16.8 h window)
jobs loaded: 900 runs from cache, 98 from the API, 2 failed (cache: gh-runner-usage-cache)
conclusion jobs share runner-min runner-days
success 12235 57.1% 111218 77.2
skipped 7424 34.7% 0 0.0
cancelled 1234 5.8% 5745 4.0
in_progress 274 1.3% 0 0.0
failure 186 0.9% 2304 1.6
queued 65 0.3% 0 0.0
total 21418 119267 82.8
average concurrent jobs over the window: 118
By comparison DataFusion only used 11.7 days of runner time in the last 2 days
1000 runs of apache/arrow-rs created between 2026-10-19T10:20:44Z and 2026-10-21T15:06:27Z (52.8 h window)
jobs loaded: 0 runs from cache, 1000 from the API, 0 failed (cache: gh-runner-usage-cache)
conclusion jobs share runner-min runner-days
success 3492 69.7% 16846 11.7
skipped 913 18.2% 0 0.0
queued 327 6.5% 0 0.0
failure 141 2.8% 853 0.6
cancelled 125 2.5% 1946 1.4
in_progress 12 0.2% 0 0.0
total 5010 19645 13.6
average concurrent jobs over the window: 6
arrow-rs used 2.7 days over 17 hours
1000 runs of apache/datafusion created between 2026-10-20T22:07:51Z and 2026-10-21T15:13:03Z (17.1 h window)
warning: jobs for run 39541706327 failed: gh: Server Error (HTTP 502)
jobs loaded: 69 runs from cache, 930 from the API, 1 failed (cache: gh-runner-usage-cache)
conclusion jobs share runner-min runner-days
skipped 1757 48.8% 0 0.0
success 1410 39.2% 3943 2.7
queued 212 5.9% 0 0.0
cancelled 184 5.1% 2762 1.9
failure 24 0.7% 140 0.1
neutral 7 0.2% 0 0.0
in_progress 5 0.1% 0 0.0
total 3599 6845 4.8
average concurrent jobs over the window: 7
One thing I think is different than my scripts above is that it counts the number of jobs but I don't think it reflects the runner time.
So you can see that airflow has 120 running jobs and I suspect the reason it ends up with so many jobs is that each takes many minutes. Other projects have many jobs but I think each job takes much less time and thus doesn't consume so many runner hours
No. It means that we have 10x more PRs opened from our contributors. Note -
that it is 10x than we usually had, We are just a very popular project and
we have more than 400 contributors. Look from how many PRs those jobs come.
No. It means that we have 10x more PRs opened from our contributors. Note -
that it is 10x than we usually had, We are just a very popular project and
we have more than 400 contributors. Look from how many PRs those jobs come.
Yes, I see. We are feeling something similar in Arrow and DataFusion . DataFusion is only at 1200 contributors, but recently we have seen a deluge of lots of PRs and each new PR (including AI dumps) was consumes a lot of PR time on the apache shared runners
I think the most effective thing we have done this past week (thank you @kumarUjjawal and @comphead 🙏 ) was to update our CI so Pull Requests ran on the submitters fork / account, not on the apache runners:
Yes, I see. We are feeling something similar in Arrow and DataFusion.
DataFusion only has 1200 contributors, but recently we have seen a deluge
of PRs. Each new PR (including AI dumps) consumes a lot of PR time on the
Apache shared runners
Oh absolutely. This is what I proposed, including the design and 5-stacked
PR implementing weeks ago only to get some PMC members respond, "if it's
about saving time to others we do not care". And since then I spent a lot
of time convincing people and pushing my way through so that it gets
accepted - to the point that I **just** sent lazy consensus result about
it.
You can track the whole discussion there:
https://lists.apache.org/thread.html/krl1bzszlfgfz70m494rm8jq1ph6dr01
Unfortunately, during the last few days while I was waiting for the lazy
consensus to pass, we were somewhat frequently nudged, added to the mailing
list threads, pointed in slakc and some issues were created. That was
quite distracting to our work because we constantly had to constantly
respond to such nudges, despite my explicit explanation to the people
nudging us that we were working on it. This led me to worry that objectors
might break the lazy consensus, further delaying things. It's sometimes
very unproductive to try to motivate motivated people by nudging them. Each
of us has an example of the manager coming every hour and asking "Are we
there yet"? That distracts people and de-focuses them as we all know. Bad
idea in general, especially if you tell "Working on it". And yet we have
not seen any apology for those distractions.
Luckily the consensus was not broken and we can proceed now.
Oh absolutely again - I have proposed (or rather my Magpie skill that I
created for that very purpose) in
https://lists.apache.org/thread.html/6bgpcflsqow9b9b3r022yyf135m63352 -
that you could replace your custom pull_request_target workfow with
.asfyaml feature - that I not only contributed so that others (not only us)
could use it but also developed and contributed the whole testing framework
to infra that infra or anyone else can use to test .ASF YAML changes:
apache/infrastructure-asfyaml#111 and
apache/infrastructure-asfyaml#133 - because my PR
for that waited for weeks.
And yes - we limited PRs to 5 in Airflow. Some of our contributors had 30
PRs, but I convinced others in Airflow and drove the change (including
closing 200 PRs that were exceessive and taking the pain of it). If not
that pre-emptive move, we would likely have 15x more traffic today. Luckily
- we acted on that without any nudging, because we care. And yes it also
required getting to consensus:
You can see lazy consensus here - and earlier discussions I had to
navigate:
https://lists.apache.org/thread.html/901dt9qdqjtqrh7ow5cdnd8sp0dl3llp
BTW. My (or rather my skill's proposal that I sent yesterday) might help
Arrow to use 30% - 40% less of ASF resources - and rather quickly. It also
proposes you to get rid of pull_request targets.
You can also implement more optimizations by following the infra page I
just got merged today when I described all kinds of optimizations with
examples (and links to the Magpie SKILL that will help your agent to do
those optimizations very quickly) :
https://infra.apache.org/github-actions-optimization.html. Some of those
optimizations are basic and simple - and the 30% - 40% optimisation is
mostly driven by switching push "complete builds" to "scheduled" canary
builds we use in Airflow for 2 years now - this is one single optimization
that helps busy repos like ours a lot.
And yes you **MUST** get rid of all pull_request_target workflow by the end
of November as they will just **stop working** ASF infra will **not**
enable it for the organisation. It's far too dangerous of a footgun..
You can also refer to another document I wrote - where I explained one of
the ways you can replace pull_request_target with much safer workflow:
https://infra.apache.org/github-actions-policy.html - see "Safer
replacement for pull_request_target" and examples - again, taken from
Magpie where I tested and perfected it.
I am glad we are all working in the same direction. And I think it would
benefit all of us if we share finding easler - and not in the form "You are
using too much resources - fix it immediately" repeated 7 times in a day
when we are pretty advanced in solving that very problem, documenting it
for others to follow - and hack - even developing tools that others can use
to make their life easier,
BTW. I REALLY heartily invite everyone to use Magpie - the "repo-health"
family of skills - which includes "workflow optimisation skill" - does all
the analysis and prepares improvement proposals that you can ask the agent
to do for you. HEARTILY RECOMMEND.
J,
—
On Fri, Oct 23, 2026 at 11:28 PM Jarek Potiuk ***@***.***> wrote:
> draft to migrate to workflow_run similar to what Apache Spark currently
has #26683 <#26683>
Yep. That is implementation of the design pattern from
https://infra.apache.org/github-actions-policy.html I proposed :). I am
happy it spreads organically :D
J.
On Fri, Oct 23, 2026 at 11:25 PM Oleks V ***@***.***> wrote:
> *comphead* left a comment (apache/datafusion#25088)
> <#25088 (comment)>
>
> draft to migrate to workflow_run similar to what Apache Spark currently
> has #26683 <#26683>
>
> —
> Reply to this email directly, view it on GitHub
> <#25088?email_source=notifications&email_token=AAERMI2OLAPZMZXXY65ZGST5VPEGPA5CNFSNUABFM5UWIORPF5TWS5BNNB2WEL2JONZXKZKDN5WW2ZLOOQXTMMZXGUYDINRSGA32M4TFMFZW63VHNVSW45DJN5XKKZLWMVXHJLDGN5XXIZLSL5RWY2LDNM#issuecomment-6375046207>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AAERMI2SZDPH3LTWNWQOORD5VPEGPAVCNFSNUABFKJSXA33TNF2G64TZHMZTKOBZGE3TGMJYHNEXG43VMU5TKMZYHE2DEMBQGY4KC5QC>
> .
> You are receiving this because you were mentioned.Message ID:
> ***@***.***>
>
Activity
blaginin commentedon Sep 8, 2026
I think that is related to other ASF projects too:
alamb commentedon Sep 8, 2026
I hit a bunch of timeouts when trying to merge PRs using the merge queue in arrow-rs-object-store:
alamb commentedon Sep 10, 2026
The root cause appears to be our DataFusion github runner usage is consuming all the github runners:
alamb commentedon Oct 20, 2026
This has been happening again a lot recently (taking hours to start running)
I have been exploring what apache projects are consuming jobs and Claude came up with this one
The current output is
It looks like a few PRs consumed a lot of the runner time (each launches 14 jobs via some matrix)
alamb commentedon Oct 20, 2026
The TLDR of my poking around is that some projects have many queued jobs (often from matrix'd runs) that are consuming significant numbers of runners and the DataFusion (and arrow-rs backlogs) are queued.
I'll ask the projects politely if there is some way to reduce the runner usage
mbutrovich commentedon Oct 20, 2026
Thank you!
alamb commentedon Oct 20, 2026
Current queue across projects
tballison commentedon Oct 21, 2026
Over on Tika, we're also seeing > hour long queues. It felt like there was a change a bit more than two weeks ago. The lengthy queues cannot be attributed to post Community Over Code exuberance.
This is not good.
Thank you for opening this ticket.
alamb commentedon Oct 21, 2026
I suspect it may be the fact that it is far easier than ever to make PRs via ai agents, and each PR often starts large amounts of CI
Also the new github stacks feature was released around Oct 6 (approximately the right timeline): https://github.blog/changelog/2026-10-06-stacked-pull-requests-generally-available/
And I found at least one example where a stack of PRs ended up running the full CI suite for each stacked PR each time the stack got rebased:
I don't have more data to back up that yet, but will keep searching
alamb commentedon Oct 21, 2026
Here is another snapshot
alamb commentedon Oct 21, 2026
I spent some time on some more analytics this morning on how many runner minutes are used by different projects
Here is the data + script I used: Archive.zip
Note this only includes the github hosted runners (not the RunsOn runners):
My numbers show that airflow has used used 77.2 days of runner time in the last 17h
By comparison DataFusion only used 11.7 days of runner time in the last 2 days
arrow-rs used 2.7 days over 17 hours
alamb commentedon Oct 21, 2026
There are also some great discussions in the ASF slack
https://the-asf.slack.com/archives/CBX4TSBQ8/p1792591324182189?thread_ts=1792520326.623569&cid=CBX4TSBQ8
That have come to the same conclusion
11 remaining items
alamb commentedon Oct 23, 2026
There is an interesting script that @potiuk pointed me me at in apache/airflow#75395 (reply in thread)
https://github.com/apache/infrastructure-actions/blob/main/utils/actions-queue-status.py
It took 30 minutes to run and here is the output.
One thing I think is different than my scripts above is that it counts the number of jobs but I don't think it reflects the runner time.
So you can see that airflow has 120 running jobs and I suspect the reason it ends up with so many jobs is that each takes many minutes. Other projects have many jobs but I think each job takes much less time and thus doesn't consume so many runner hours
potiuk commentedon Oct 23, 2026
alamb commentedon Oct 23, 2026
Yes, I see. We are feeling something similar in Arrow and DataFusion . DataFusion is only at 1200 contributors, but recently we have seen a deluge of lots of PRs and each new PR (including AI dumps) was consumes a lot of PR time on the apache shared runners
I think the most effective thing we have done this past week (thank you @kumarUjjawal and @comphead 🙏 ) was to update our CI so Pull Requests ran on the submitters fork / account, not on the apache runners:
Unfortunately this added quite a bit of complexity to our CI system, but I don't see any way around it.
@sdf-jkl is working on the same treatment for arrow-rs
alamb commentedon Oct 23, 2026
BTW grails-core (47 runner-days) and beam (12 runner-days) also consume significant amounts of CPU in the last 24 hours
rok commentedon Oct 23, 2026
On arrow we limit non-collaborators to 3 PR at a time. Will merge apache/arrow#55164 once apache/infrastructure-asfyaml#135 is complete.
sdf-jkl commentedon Oct 23, 2026
@rok I want to try same approach on arrow as well
rok commentedon Oct 23, 2026
@sdf-jkl let's check with @raulcd if he's comfortable with the extra logic. I think we used to do it at sompoint, but that's better over chat.
rok commentedon Oct 23, 2026
@sdf-jkl also please note - merging apache/arrow#55164 would solve this issue too.
comphead commentedon Oct 23, 2026
Thing that concerns me is https://docs.github.com/en/actions/reference/security/securely-using-pull_request_target#default-policy-for-pull_request_target
that disables
pull_request_targeteven on public repos, which I believe also applicable for contributor forkscomphead commentedon Oct 23, 2026
This is a migration example apache/spark-connect-swift#615
rok commentedon Oct 23, 2026
I wonder if GitHub will start blocking this pattern. Surely we're not only ones with so many PRs.
potiuk commentedon Oct 23, 2026
comphead commentedon Oct 23, 2026
draft to migrate to
workflow_runsimilar to what Apache Spark currently has #26683potiuk commentedon Oct 23, 2026
potiuk commentedon Oct 23, 2026