Motivation
Yardl's HDF5 backend has become a net negative for the project. It increases maintenance cost, constrains the evolution of the format, and works against several of Yardl's core design goals. Multiple discussions with downstream users and maintainers have identified HDF5 support as an obstacle to higher-priority work, including indexing, improved tooling, and language support.
Reasons
- HDF5 is fundamentally at odds with Yardl's streaming-first design
Yardl was created as tooling for streaming instrument and signal-processing data. Streaming protocols and file formats are a first-class concept throughout the ecosystem. HDF5, on the other hand, is fundamentally a random-access container format rather than a streaming format, making it an awkward fit for many of the scenarios Yardl is intended to support. The broader ecosystem around Yardl, MRD, and Tyger continues to move toward stream-oriented processing and transport.
- HDF5 support is incomplete across Yardl's language targets
C++, Python, and MATLAB are all first-class Yardl language targets. In practice, HDF5 support has historically been focused on the C++ ecosystem while providing an inconsistent experience across languages. This creates confusion for users, increases testing burden, and makes it difficult to claim feature parity across supported language targets. Evidence from downstream projects and user discussions repeatedly highlights friction around HDF5 interoperability.
- HDF5 increases maintenance burden
Every schema change, feature enhancement, bug fix, and generated-language update must also be implemented, tested, documented, and maintained for the HDF5 backend. The PETSIRD community specifically identified HDF5 support as something that "nobody uses", while noting that it consumes maintenance resources that could instead be invested in features users actively request.
- Plan for binary indexing
A long-standing Yardl goal is to support indexing of binary Yardl files. Indexing enables efficient random access, better developer tooling, faster exploration of large datasets, and more sophisticated applications built on top of Yardl. Once implemented, Yardl's binary file format will support random access - one of the few desirable features of HDF5.
- Removing HDF5 enables better tooling
Several discussions have identified viewer and inspection tools as high-value additions to the Yardl ecosystem. Simplifying around a single binary representation makes these tools significantly easier to build and maintain.
- Reducing external dependencies simplifies adoption
HDF5 is a substantial dependency that complicates builds, packaging, portability, CI, and cross-platform development. Eliminating the dependency reduces complexity for downstream projects and improves the portability of generated code and tools.
Proposed Direction
- Deprecate HDF5 generation in Yardl.
- Focus future investment on the binary streaming format and associated tooling.
- Implement indexing directly in Yardl's native format.
- Build ecosystem tooling (viewers, inspectors, upgrade tools, converters, etc.) on top of the native format rather than HDF5.
- If required, provide standalone import/export or conversion tools rather than maintaining HDF5 as a core serialization backend. This possibility has already been discussed by maintainers.
Expected Benefits
- Lower maintenance burden.
- Clearer cross-language story.
- Better Python and MATLAB experience.
- Alignment with Yardl's streaming architecture.
- Ability to implement indexing.
- Simpler tooling ecosystem.
- Faster progress on features users actually need.
Motivation
Yardl's HDF5 backend has become a net negative for the project. It increases maintenance cost, constrains the evolution of the format, and works against several of Yardl's core design goals. Multiple discussions with downstream users and maintainers have identified HDF5 support as an obstacle to higher-priority work, including indexing, improved tooling, and language support.
Reasons
Yardl was created as tooling for streaming instrument and signal-processing data. Streaming protocols and file formats are a first-class concept throughout the ecosystem. HDF5, on the other hand, is fundamentally a random-access container format rather than a streaming format, making it an awkward fit for many of the scenarios Yardl is intended to support. The broader ecosystem around Yardl, MRD, and Tyger continues to move toward stream-oriented processing and transport.
C++, Python, and MATLAB are all first-class Yardl language targets. In practice, HDF5 support has historically been focused on the C++ ecosystem while providing an inconsistent experience across languages. This creates confusion for users, increases testing burden, and makes it difficult to claim feature parity across supported language targets. Evidence from downstream projects and user discussions repeatedly highlights friction around HDF5 interoperability.
Every schema change, feature enhancement, bug fix, and generated-language update must also be implemented, tested, documented, and maintained for the HDF5 backend. The PETSIRD community specifically identified HDF5 support as something that "nobody uses", while noting that it consumes maintenance resources that could instead be invested in features users actively request.
A long-standing Yardl goal is to support indexing of binary Yardl files. Indexing enables efficient random access, better developer tooling, faster exploration of large datasets, and more sophisticated applications built on top of Yardl. Once implemented, Yardl's binary file format will support random access - one of the few desirable features of HDF5.
Several discussions have identified viewer and inspection tools as high-value additions to the Yardl ecosystem. Simplifying around a single binary representation makes these tools significantly easier to build and maintain.
HDF5 is a substantial dependency that complicates builds, packaging, portability, CI, and cross-platform development. Eliminating the dependency reduces complexity for downstream projects and improves the portability of generated code and tools.
Proposed Direction
Expected Benefits