[Roadmap] TransferQueue Q1 roadmap #20

Interface

• Provide high-level key-value based interface [feat] Introduce high-level key-value (KV) interface #28 [fix,feat] Support MooncakeStore easy init #45 @0oshowero0
• Provide a one-click initialization experience, with clear config structure [refactor] Simplify initialization and improve API usability #26 @0oshowero0

Feature

• User-defined Metadata: Support put/get operations for user-defined metadata in the TQ controller. @0oshowero0
• StreamingDataLoader: Optimize the user experience. [StreamingDataLoader, 4/N] feat: Introduce sample pre-allocation for dynamic streaming #16 @0oshowero0
• Easy-to-use Backend: Provide backend-specific metadata storage capability, simplifying RDMA transport (eliminate the need to use zmq to fetch some dtype & shape info by users). @tianyi-ge [feat] Support store custom_meta in controller for backend-specific info #5, https://gitcode.com/Ascend/TransferQueue/pull/14

Performance

• Improve metadata efficiency [Proposal] Refactor BatchMeta to Columnar Storage to Eliminate O(B×F) Overhead #24 https://gitcode.com/Ascend/TransferQueue/pull/29 https://gitcode.com/Ascend/TransferQueue/pull/28 @mpb159753
• Improve SimpleStorageUnit efficiency [perf] Use zmq.asyncio.Context to accelerate notify_data_update process #38 [perf] Add zmq.proxy to accelerate request processing for SimpleStorageUnit #37 @0oshowero0

Integrations

verl

tracking issue: verl-project/verl#5401 verl-project/verl#4880

• Upstream Merge: Merge TQ integration into the verl main codebase [WIP][data] feat: TransferQueue - integrate TransferQueue into main codebase verl-project/verl#4987
• Multi-modal Support: Optimize TQ performance for multi-modal scenarios.
• Router-Replay: Implement support for router-replay transmission with TQ. @ji-huazhong
• Support next-gen single-controller development

Optimization

• Zero-Copy Serialization: Implement faster, robust zero-copy serialization https://gitcode.com/Ascend/TransferQueue/pull/7, https://gitcode.com/Ascend/TransferQueue/pull/19, https://gitcode.com/Ascend/TransferQueue/pull/22 @mpb159753
• User experience: Optimize initialization of storage manager: [Enhancement] A suggestion regarding StorageManager initialization #6 @0oshowero0

Backend Supports

• Yuanrong Backend: Optimize backend and provide E2E performance benchmarks on both GPU and NPU. @tianyi-ge
• MooncakeStore Backend: Provide easy init capabilities for mooncake [fix,feat] Support MooncakeStore easy init #45 @0oshowero0

Documentation & Tutorial

• Website: Build a dedicated documentation website.
• API Reference: Generate a comprehensive API list.
• Onboarding: Provide guiding docs to simplify the user onboarding process.
• Tutorial: Add pluggable backend tutorial, demonstrate how to switch between supported backends and illustrate the custom_backend_meta usage. @tianyi-ge

CI/CD

• Unified Workflow: Unify CI experience across GitCode and GitHub (addressing missing tests on GitCode).
• E2E Testing: Enable End-to-End tests. @mpb159753 [CI] Add E2E consistency tests with comprehensive data type coverage #35
• Performance Checkpoint: Add periodic performance testing to detect regressions in new PRs.
• Refactoring: Refactor unit test logic for better maintainability.