Repository navigation
Conversation
Add an `opendal-hdfs` feature that serves `hdfs://` paths through OpenDAL's `services-hdfs`, libhdfs over JNI, next to `hdfs-native` as apache#1130 agreed. Only the transport differs. Hadoop's Java client reads its configuration from CLASSPATH and resolves nameservices, HA, Kerberos and `fs.defaultFS` itself, so the backend maps a path to the NameNode URI `FileSystem.get` takes and caches one operator per NameNode. The `hdfs-native` backend and the `iceberg` crate are unchanged.
comphead
force-pushed
the
feat-storage-hdfs-libhdfs
branch
from
October 23, 2026 00:36
6d6c7bb to
2b4f03d
Compare
`hdrs` 0.3.4, the libhdfs binding behind `opendal-hdfs`, does not compile on Windows: it uses the Unix-only `libc::O_CLOEXEC` and assumes a 64-bit `tTime`. OpenDAL's own CI leaves `services-hdfs` out of its Windows build as well.
Collaborator
Author
|
I'm planning to test this on the real cluster with kerberos protected |
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
What changes are included in this PR?
#1130 agreed to ship both HDFS clients (consensus). #3111 added
hdfs-native. This PR adds libhdfs as a second transport. Thehdfs-nativebackend and theicebergcrate are unchanged.opendal-hdfsfeature (OpenDAL'sservices-hdfs, experimental, not inopendal-all) addsOpenDalStorageFactory::HdfsandOpenDalStorage::Hdfs.CLASSPATH, so nameservices, HA and Kerberos work as in Hadoop. Nohdfs.*orhadoop.*property applies.hdfs://<authority>/to libhdfs as written and caches one operator per NameNode. Authority-less paths usehdfs:///, which fails like Hadoop whenfs.defaultFSis not HDFS. libhdfs'sdefaultwould fall back to the local disk.OpenDalResolvingStorageFactorystill routeshdfs://tohdfs-native.Why both: through OpenDAL,
hdfs-nativelooks up delegation tokens for a synthetic nameservice, so it misses the tokens Spark executors and YARN containers carry. libhdfs uses the JVM's credentials.Costs: building needs a JDK, and binaries built with the feature need
$JAVA_HOME/lib/serveron the library path. The tests job sets it. The feature does not build on Windows, wherehdrsdoes not compile, so the Windows build job leaves it out, as OpenDAL's CI does.Are these changes tested?
delete_streamkeys andrelativize_path. A mutation check of the new code caught all 7 mutations.make check-clippy,cargo fmt,taplo,cargo macheteandpublic-api.txtpass locally.AI Disclosure
Developed with AI assistance (Claude Code), which drafted the code, tests and this description. I reviewed the changes.