Skip to content

Backport RustFS for DataFusion CLI S3 tests - #166

Merged
osipovartem merged 6 commits into
df55-physical-udf-instancefrom
df55-rustfs-cli-test
Oct 25, 2026
Merged

osipovartem merged 6 commits into
df55-physical-udf-instancefrom
df55-rustfs-cli-test

Conversation

@osipovartem

@osipovartem osipovartem commented Oct 25, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • Backport the upstream DataFusion 55 RustFS replacement from apache/datafusion#25759 (original #25706) into our df55 fork.
  • Replace the withdrawn MinIO image used by CLI S3 integration tests with pinned RustFS 1.0.0, pre-pull it in CI, and update contributor docs.
  • Repair inherited CI-only issues on the df55 base so the workflow reaches the Docker-backed CLI job: an InformationSchemaConfig test initializer and SLT expectations, Taplo manifest order, generated config/function docs, a clippy test, and a CSV diagnostic assertion that compared nondeterministic backtraces.

Verification

  • The resolved RustFS CLI test file matches the tested upstream backport except for an unrelated upstream-only Spark opt-in test absent from this fork.
  • cargo +1.99.0 fmt --all -- --check, Taplo 0.9.3 formatting (51 manifests), and git diff --check: passed.
  • Independent read-only review of the cumulative diff: approved, no blocking findings.
  • On the preceding commit, fork CI passed 36 checks, including the full amd64 test suite, clippy, and Docker-backed cargo test datafusion-cli (amd64); cli_integration passed 34/34 with no skips, including test_cli, test_aws_options, test_s3_url_fallback, test_object_store_profiling, and rustfs_image_matches_ci_prepull.
  • The only new change on the current head is the exact generated array_reduce documentation diff. Its generated-docs check now passes; remaining code/test jobs are still running. A full local build was deferred because host free space was below the workspace threshold.
  • security_audit reports existing October 2026 advisories in the unchanged Cargo.lock; dependency remediation is outside this test-infrastructure backport.

Closes apache#25705

This morning it seems like Minio pulled its images from quay.io, which
has caused recent CI runs to start failing, e.g.

https://github.com/apache/datafusion/actions/runs/36006680489/job/107657362588

RustFS is a pretty comparable (and more permissively licensed)
alternative that provides a suitable S3 interface for integration tests.

This was substantially authored by Codex, but reviewed by me and tested
locally.

Existing integration tests

No

Signed-off-by: Andrew Duffy <andrew@a10y.dev>
(cherry picked from commit e2ca7f3)
(cherry picked from commit 5ddc41bb19e776f2490acae6b95c12ebd3881eac)
(cherry picked from commit 52580648fa40ce24a28b466044872f59615d6a8c)
(cherry picked from commit 69bb1e18cdb5358b3dd776de89c34ec8de8619c5)
@github-actions github-actions Bot added documentation Improvements or additions to documentation core catalog sqllogictest datasource labels Oct 25, 2026
@osipovartem
osipovartem merged commit 105819d into df55-physical-udf-instance Oct 25, 2026
29 of 38 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants