Repository navigation
GCSFileSystem requires gcp extra at lookup time while S3FileSystem does not #37445
Description
There's an inconsistency in how FileSystems.get_filesystem() handles missing optional dependencies between GCS and S3.
Current Behavior
S3 (without aws extra):
>>> from apache_beam.io import filesystems
>>> filesystems.FileSystems.get_filesystem("s3://blah")
<apache_beam.io.aws.s3filesystem.S3FileSystem at 0x11a0af750>Returns the filesystem object; validation happens later when the filesystem is actually used.
GCS (without gcp extra):
>>> from apache_beam.io import filesystems
>>> filesystems.FileSystems.get_filesystem("gcs://blah")
ValueError: Unable to get filesystem from specified path, please use the correct path or ensure the required dependency is installed, e.g., pip install apache-beam[gcp]. Path specified: gcs://blahRaises immediately because GCSFileSystem isn't registered as a subclass.
Proposed Behavior
Both should behave consistently. GCSFileSystem should be returned from get_filesystem() like S3FileSystem, allowing callers to validate dependencies when the filesystem is actually used rather than at lookup time.
Why This Matters
- Inconsistent API behavior is confusing
- Code that handles multiple filesystem types can't catch/handle GCS gracefully
- Dependency validation at usage time (not lookup time) allows for better error handling and lazy loading patterns
Environment
- Apache Beam version: 2.70.0
- Python version: 3.11
Generated by Claude Code, confirmed by @hjtran
Activity
hjtran commentedon Jan 29, 2026
@damccorm do you have an opinion on this? I'd like a second opinion on the proposed behavior before someone works on it. The fix will likely just be lazy loading
gcsioingcsfilesystem.pydamccorm commentedon Jan 29, 2026
SGTM - I agree with the proposed behavior change and with the fix you're suggesting
RushabhRatnaparkhi commentedon Jan 30, 2026
Hi @hjtran , I want to work on this issue if it is ok.
hjtran commentedon Jan 30, 2026
hjtran commentedon Jan 30, 2026
@RushabhRatnaparkhi , you can comment
.take_issueto assign the issue to yourselfHridayyy1 commentedon Jan 30, 2026
Thanks for confirming! I went through the issue and the filesystem lookup logic to understand the GCS vs S3 inconsistency. Since someone else is already working on this, I’ll step back, but I’m happy to help review or test the fix if needed. I’ll also look out for related issues. Thanks!
RushabhRatnaparkhi commentedon Jan 30, 2026
.take_issue
Simarjeet112 commentedon Feb 5, 2026
Hi, I’d like to work on this issue as a new contributor.
.take-issue
hjtran commentedon Feb 5, 2026
Hi @Simarjeet112 , it looks like a PR has already been put up for this, see #37488
MansiSingh17 commentedon Feb 19, 2026
Hi, I’m Mansi Singh, prospective GSoC 2026 contributor interested in Kafka Streams Runner.
I’d like to work on this issue if no one else is assigned. Thanks!
shunping commentedon Feb 19, 2026
Hi @hjtran. I just came across this issue and saw a lot of interests here (which is great!).
I would like to understand the rationale behind the proposed behavior a bit more. To me, I'm more inclined toward the 'fail-fast' approach of GCSFileSystem, which provides immediate feedback if a required dependency is missing.
Could you elaborate on your specific use case? Particularly, why deferring validation to usage time is preferable for your scenario?
hjtran commentedon Feb 19, 2026
Hey @shunping
For my use case, it's optional whether or not the GCS dependencies are installed. I want to be able to do something like
if Filesystems.get_filesystem(user_provided_path) is GCSFileSystem: raise RuntimeError("You haven't installed GCS dependencies!").I can do this today with
S3Filesystem, but not withGCSFilesystem. I can do a manualif path.startswith("gcs"), but this is a bit awkward especially since S3 doesn't require itshunping commentedon Feb 19, 2026
Thanks for clarifying this. It sounds like you are using
get_filesystemto identify the filesystem type of a file without necessarily interacting with it.If we make this change,
get_filesystem(some_gcs_path)would no longer provide the immediate 'fail-fast' signal for missing dependencies. In that case, I assume you would have other logic to detect whether the required dependencies are actually installed before attempting to use the file, right?4 remaining items
hjtran commentedon Feb 20, 2026
Make GCS filesystem lookup lazy to match S3 behavior (fixes apache#37445
MansiSingh17 commentedon Feb 21, 2026
@hjtran The PreCommit Python Integration failures are Dataflow integration tests that require GCP credentials. These are not related to my changes — the failures are in :sdks:python:test-suites:dataflow:py310:preCommitIT_batch which runs against real GCP infrastructure. Could a maintainer please re-run or confirm these are pre-existing?
Make GCS filesystem lookup lazy to match S3 behavior
Make GCS filesystem lookup lazy to match S3 behavior
Make GCS filesystem lookup lazy to match S3 behavior
wilmerdooley commentedon May 30, 2026
I have opened #38751 for this. It makes the gcsio import in gcsfilesystem.py lazy so GCSFileSystem is returned from FileSystems.get_filesystem() without the gcp extra installed, deferring the dependency error to usage time (matching S3FileSystem), which is the approach @hjtran suggested above.
I built on the earlier attempts (#37488, #37674) and tried to fold in the review feedback from those:
@MansiSingh17, this issue is assigned to you but I did not see an open PR, so I went ahead. Please feel free to use this as a reference or take it forward, and I am happy to coordinate.
Abacn commentedon May 30, 2026
The real difference for gcs and s3 here is due to that in s3io boto3 has been optional at import time. There is a module variable BOTO3_INSTALLED evaluated:
beam/sdks/python/apache_beam/io/aws/s3io.py
Line 43 in c31f232
Then it only prevents s3io object creation at runtime:
beam/sdks/python/apache_beam/io/aws/s3io.py
Line 66 in c31f232
I think a proper fix for gcsfilesystem should be in gcsio to make it follow same pattern
[Python] Defer the GCS dependency check to gcsio, matching s3io
nikhilsh2003 commentedon Sep 4, 2026
Hi @hjtran, I’d like to work on this issue if no one is currently working on it.
From the discussion, it looks like the main problem is with "gs://" paths when the "gcp" extra isn’t installed. "GCSFileSystem" fails during import/registration, so "FileSystems.get_filesystem()" can’t return it, while S3 handles the missing dependency differently.
I’m planning to trace how "GCSFileSystem" is imported and registered, figure out why the import fails without the GCP dependency, and see how we can make the filesystem discoverable without changing the behavior when it’s actually used.
I’ll also add a regression test for the missing "gcp" dependency case and make sure the existing behavior with the "gcp" extra installed still works.
Please let me know if someone is already working on this. Otherwise, I’d be happy to pick it up.
hjtran commentedon Sep 18, 2026
Thanks @shunping !