[Python] ParquetWriter doesn't support all valid s3 paths #46924

Activity

1. graydenshand

   added

   Type: bug

   on Jun 26, 2025
2. github-actions

   added

   Component: Python

   on Jun 26, 2025
3. amoeba

   changed the title [-](python) ParquetWriter doesn't support all valid s3 paths[/-] [+][Python] ParquetWriter doesn't support all valid s3 paths[/+] on Jun 28, 2025
4. thisisnic

   added

   Status: needs championHigh impact issues which aren't being worked on but require a volunteer to move the task forward.High impact issues which aren't being worked on but require a volunteer to move the task forward.

   on Oct 26, 2026
5. thisisnic commented on Oct 26, 2026

   

   thisisnic

   on Oct 26, 2026

   Last edited by thisisnic

   Member

   More actions

   Can reproduce on 26.0.0. Did a brief Claude analysis; looks like a mix of things which aren't percent encoded and error messages not surfacing:

   🤖 :

   The path is parsed as an RFC 3986 URI (uriparser, in cpp/src/arrow/util/uri.cc). In a URI # starts the fragment, and a second # inside a fragment is a syntax error, so s3://my-bucket/a#b#c.parquet fails with "Cannot parse URI ... syntax error at character '#'". The same happens for spaces and [/]. The characters that work ($*&@!) are all legal in a URI path.

   _resolve_filesystem_and_path() in python/pyarrow/fs.py catches that "Cannot parse URI" error and silently falls back to LocalFileSystem, which then rejects the string with the misleading "Expected a local filesystem path, got a URI" seen above. This is the same mechanism as #41365 (whitespace), which #49372 is fixing by raising a proper error instead of falling back.

   There is a worse case with a single #: s3://my-bucket/a#b.parquet parses fine, the fragment is dropped, and FileSystem.from_uri() returns key my-bucket/a, so a write would silently go to the wrong object. FileSystemFromUri() in cpp/src/arrow/filesystem/filesystem.cc should reject a non-empty fragment, since no filesystem uses one.

   Results on 26.0.0 via FileSystem.from_uri(): s3://my-bucket/a#b#c.parquet cannot parse; s3://my-bucket/a#b.parquet resolves to my-bucket/a; s3://my-bucket/a%23b%23c.parquet resolves to my-bucket/a#b#c.parquet; s3://my-bucket/a b.parquet and s3://my-bucket/a[b].parquet cannot parse; s3://my-bucket/a?b.parquet is rejected as an unexpected query parameter; $, *, &, @, ! all resolve correctly.

   Workaround: percent-encode the key (s3://my-bucket/a%23b%23c.parquet), or pass filesystem=S3FileSystem() with the plain key my-bucket/a#b#c.parquet. Neither the from_uri() docstring nor the filesystems docs currently say that URIs must be percent-encoded.