Repository navigation
[DISCUSSION] Is DataFusion's implementation growing too complicated? Are AI assisted PRs to blame? #26257
Description
TLDR; I worry that the ease of adding features with AI tools is leading DataFusion to become overly complicated and that the additional complexity will eventually slow down / ruin the project
I think @2010YOUY01's way of phrasing my concern in this comment: #24768 (comment)
My imaginary failure mode is, after exceeding some complexity threshold, AI always fix one bug by bringing in two new bugs
AI tools are amazing in terms of how much code you can write and how much it lowers the barriers to contribution. However, I worry about the cumulative effect on performance and understandability over the long term
Recent Examples:
- fix(substrait): Preserve pushed-down table scan offsets and limits #26035 from @viirya where there was quite a bit of test and code to handle LIMIT + OFFSET that overflows an i64 (which I think would never happen in practice);
- fix: do not rewrite
SUM(x + literal)for floating point arguments #26241 which looks very plausible, but I think the veneer of plausibility masks the fact it isn't really a bug.
I am not sure if anyone really has the experience yet to know if this is/will happen but I worry about it
Activity
haochunchang commentedon Oct 13, 2026
Just want to chime-in with my two-cents. :)
imo it is the drift between mental model and real codebase that will cause doubt/fear.
By addressing this, I still read through the AI generated code changes (and/or asking questions or use something like explain-diff skill to quiz myself) to understand as thoroughly as time allowed.
As one of the action items, my idea is to add safety rails that could help us understand the code changes, for example:
Maybe starting by refining the contributor guides section to be the context for AI review agents?
For performance, maybe encouraging PR authors to run benchmark to see if the changes would have performance degradation (since my understanding is that performance is really essential for datafusion)?
I also like the PR limits for each user, so that we can focus on what we have started.
alamb commentedon Oct 13, 2026
The intention was that this page is the context I think: https://datafusion.apache.org/contributor-guide/pr_review.html
Maybe we can add an explicit link 🤔
mbutrovich commentedon Oct 13, 2026
It's a fairly complicated project to begin with, and IME agent-generated PRs often lack the context for proper code reuse (either internally or from stuff like arrow-rs) and like to reinvent the wheel. I find myself looking at a number of PRs (here, Iceberg, Comet, etc.) thinking "don't we have code for this already?"
It's making reviewing even harder because new contributors show up (great!) with limited context and a ton of tokens to burn and dump massive PRs on us that pass the initial smell test, but if you dig into the details you find they were caused by the agent reward hacking or poorly scoped requirements to begin with.
I don't really have a solution here other than "actually we should move more slowly." Review cycles should be longer, particularly as average PR line count seems to be growing. I'm almost wondering, since reviews are so heavily AI-assisted, if we should consider bumping the threshold to two approving reviews.
haochunchang commentedon Oct 13, 2026
Do you mean maybe adding an explicit link to the page inside somewhere that AI agent can automatically pick up when contributors are using agents, for example, AGENTS.md (which already includes the link)?
haochunchang commentedon Oct 13, 2026
I think both longer review cycle, and assisting new-comers' agents to collect context all aim to build understandability among the maintainers. From a new comer's POV, maybe start by AI-assisted PR review can help understanding the codebase more quickly.
( I confess I also have opened massive PR, will address them later.)
pepijnve commentedon Oct 13, 2026
I got the feeling expressed in the description when @bert-beyondloops and I were dealing with incorrect/sub-optimal optimisation passes introduced by leaf expression pushdown. He expressed this concern in #25329 (comment)
I really don't want to throw any of the involved developers under the bus here. Just trying to express that both of us are having a hard time getting our heads around the concepts being introduced into the API (the notion of expression placement in this case). This makes it quite difficult to collaborate on fixing things. At the same time the LLM train keeps on thundering down the tracks while requests for clarification for the human trying to review things (me in this case) go unanswered. The end result is that we've basically given up and just turned the optimisation in question off for now.
phillipleblanc commentedon Oct 13, 2026
I found this post to be aligned with my current thinking on how AI is transforming software engineering: https://thorstenball.com/blog/2026/09/19/what-i-believe-about-the-future-of-software-development/
I don't think this will happen, and if it does then it can be solved. The frontier models are getting so good that most problems can be solved with more tokens.
However... I do think DataFusion falls into this edge case called out by the blog post above:
Telling Claude "make datafusion faster, no mistakes" won't work - that will still require human judgement and creativity. But all of the grunt work and stuff that would have distracted from focusing on that core can be handled by AI.
I do think DataFusion has the benefit of focusing on being a small extensible core rather than trying to solve the world. If anything we should focus more in that direction to enable the "building block economy": https://mitchellh.com/writing/building-block-economy And be ruthless about saying no and telling people to build on top of the building blocks that exist.
manuzhang commentedon Oct 13, 2026
Some personal experience to share. As a reviewer, I try to understand AI's review comments first, push hard on each point, and post comments by hand. As an author, I try to split big AI-assisted PRs into smaller ones to make it easier for review.
We learn from contributing and reviewing, and can steer the wheel when AI is going in the wrong direction. That makes me less worried.
comphead commentedon Oct 13, 2026
Yeah, this question is super fair.
IMO AI or not AI if the project follows to healthy project metrics below, it would mitigate lots of regressions.
The AI development is new normal, and we cannot get rid of it, but we need to make sure the AI generated content follows standards that we expect
Code quality
Testing
mainDependencies and API
Performance
Docs
Process
Please find LOC analytics for last 7 releases, the codebase is growing faster than ever
Some open issues to track code quality metrics:
Performance tracking and benchmarks
Binary size and build time
datafusion-clirequires more than 8GB to build #18471datafusion-clirequires more than 8GB to buildviirya commentedon Oct 13, 2026
To clarify, the PR was not intended to handle LIMIT + OFFSET overflowing an i64. 😄 The part @alamb mentioned was added to address a review comment. I was focused on addressing the comment and didn't think too much about it.
On this topic, I think using AI for coding, reviewing, and contributing to OSS projects like DataFusion is an unstoppable trend. Since we are still in the early wave of this trend, some chaos and concerns are unavoidable. AI is a tool, so ultimately the question is how we use it to achieve higher productivity and better results.
For example, I should have looked more carefully at the comment asking to handle i64 overflow. But even without AI, I might have misjudged the request and implemented the i64 handling anyway. AI just makes that easier, because I can simply ask it to address the comment. So I think everyone is still learning how to use AI at this early stage, and we can do better by reflecting like this each time.
I think encouraging PR authors to run benchmarks is helpful in many cases. With the help of AI, it is actually easier and less tedious to do.
The review process definitely helps, whether it comes from humans or from AI (with human direction).
asolimando commentedon Oct 13, 2026
Sharing a few ideas:
requiring an in-depth design discussion for all non-trivial new features
requiring a real SQL/SLT reproducer for bug fixes
perf- improvements: other than beating benchmarks, improvements should make sense on their own and they should make sense beyond pure numbers, like an intuition of why the perf improvment is good, why it would generalize beyond the current benchmarks, what are the blind spots, etc
Why I think they would help:
would probably reintroduce a bit of the friction AI tools lowered, but of a good kind, people must think things through and be able to condense complex concepts in quality examples and summarized explanations. It would also take a bit longer to get changes in, but for a good reason
is what Andrew has been frequently asking lately, and it helps removing all bugs that can't be reached by the end user, and it would make it easier to spot plausible but marginal bugs
limits the overfitting to existing benchmarks which are never really representative and might makes things worse over time if we overdo
AI amplifies good and bad habits, people have always been showing up in OSS projects trying to improve their downstream use cases without doing the effort to improve the abstractions and the general health of the project, it just became simpler to do that, but some healthy gatekeeping will help keeping quality standards and maintenability
alamb commentedon Oct 13, 2026
yes, sorry @viirya for calling you out with more context. I just thought that was a good example where well meaning interactions resulted in bunch more complexity than maybe we really needed (it was fixed before merging)
manuzhang commentedon Oct 14, 2026
Artificial Analysis just added a new evaluation to test whether AI models can write code maintainers would merge. Some key findings 🤯
andygrove commentedon Oct 14, 2026
Here are my thoughts on this. Dealing with growing complexity and technical debt over time in fast moving projects is not new, but AI has accelerated this exponentially. I haven't done an analysis, but I assume that most contributors and maintainers are putting most of their time into PR creation and PR reviews. When reviewing one PR in isolation we don't tend to question how it relates to all of the existing code in the repo / ecosystem.
In addition to PR reviews, I think we need to do regular audits of the whole codebase - perhaps once per release - and look at things like code complexity and duplication, and auditing for regressions compated to the previous release. We started doing some of this with Comet since the 1.0.0 release.
There are existing tools for analyzing code complexity and quality. I was too lazy to resarch them so I had Claude build me a custom tool overnight that specializes in the DataFusion ecosystem. Here is an epic with some of its findings (it found a few bugs too).
#26309
I'd be interested in hearing opinions on this kind of approach.
pepijnve commentedon Oct 14, 2026
I think that's going to be the most realistic way to deal with the increase in review load (in general in software development; not datafusion specifically). If there are ways to capture the heuristics human reviewers would use in automated, preferably deterministic, tools you have a mechanism to push back against low quality submissions. I believe Greg Kroah-Hartman referred to similar things in the Linux kernel ecosystem in https://www.youtube.com/watch?v=NnV_cWeoo5Q where they have tools to scan for bug patterns. I didn't research that further. I think it was Cocinelle he was referring to in that talk.
comphead commentedon Oct 15, 2026
In addition to metrics provided here
we can try to leverage https://github.com/mozilla/rust-code-analysis/ to define and enforce code complexities on the PR level
bugs, regression, performance should be separate process(tests, fuzzers, benches, etc) running weekly to keep the project healthy.
Scalability - stress tests, weekly
Design, reusability thats an interesting part, not sure I found a good answer, appreciate if anyone shares their thoughts
2010YOUY01 commentedon Oct 16, 2026
TIL -- this tool is cool.
Below is its major metrics
I see these metrics as useful additional references, though they may not fully capture the aspects of code complexity that matter most for maintainability.
To me, the gold standard is how easily readers can understand and reason about the code or PR directly. It would be great if reviews place more emphasis on this.
One key criterion is ease of local reasoning, which I believe is best achieved through good problem decomposition/modeling and deep modules, as described in Heuristic 1 in:
This doesn't always align with static analysis metrics such as Cyclomatic Complexity, Cognitive Complexity, or LoC. For example, a function or module can have a complex internal implementation, but if it exposes a simple interface and keeps that complexity encapsulated, it can still make the overall system easier to understand and maintain.