Rust SDK

It would be great to have Rust SDK in order to create very high-performant yet safe pipelines.

Imported from Jira BEAM-12658. Original Jira may contain additional context.
Reported by: .

@bvolpato

Thanks @bvolpato for pointing me to this #21089
This would be extremely useful for us at Discord. We use Rust heavily and we are using Beam (using python) for data processing. To reduce training serving skew this would be very beneficial.

@gauravchak

@gauravchak , any... Contributions welcome! Do advise if need some resources to get started, for example a related talk: https://www.youtube.com/watch?v=VsGQ2LFeTHY

Hi, I would like to express interest in working on the Rust SDK. I'll create an incubator fork soon.

.take-issue

Work is underway here. Progress may be slow, and early code will look quite rough. I'll be really happy to receive any feedback or collaboration opportunities.

@nivaldoh

@nivaldoh iv been looking for a reason to learn rust. Happy to take on any house keeping work that will slow you down

@dofinn

@dofinn I really appreciate the offer. Currently we have a few TODOs with improvement ideas which I'd be happy to describe in more detail, and we could also try to coordinate effort on the larger tasks as well (which I'm also planning to organize in the main README file) if you're interested. Feel free to open a PR directly in the fork or reach out to me by email (nivaldo.humbertoo@gmail.com) if you'd like.

@nivaldoh

@nivaldoh thanks for doing this!

I will be an early adopter when you are ready for that.

@esadler-hbo

@esadler-hbo thanks for the support!

There's still a lot of work to be done, and I try to keep an updated roadmap here if you or anyone else is interested in a quick overview of the current state of the implementation.

In particular, the user API is starting to take shape, and the snippet below (inspired by the new patterns set by the Typescript SDK) is now functional end-to-end:

let runner = DirectRunner::new();

// Impulse won't be exposed to the user, but it serves as a mock transform for now
let transform = Impulse::new();

runner.run(|root| root.apply(transform)).await;

To anyone interested, any early input on this format would be highly appreciated, as this is the current foundation that I'll be using for everything else. Any other contributions (including early code reviews, even if partial) would be awesome as well.

I also like to help! I have some experience with Rust but am pretty new to Beam or large-scale data processing framework

@nivaldoh

@nivaldoh I think reading the doc will only get me so far and I should probably start working on some implementation and see how it goes. Is there any coder/transform or other stuff you want my help with?

@laysakura

Thank you @laysakura for rebooting this effort!

@robertwb

@robertwb Thank you for creating the basic mechanisms of Beam, such as ParDo and GBK.

@dahlbaek and I are now working on creating a more Rust-like, statically-typed pipeline based on your work. We would also appreciate your contribution to laysakura/beam based on your extensive experience with the TypeScript SDK.

@nivaldoh

Oh, I seem to have missed some traffic on this issue. I received collaborator access to @nivaldoh's fork yesterday, but I'll move development over to @laysakura's fork. :)

I've got some work in progress for data channels on DataSource and DataSink and I'm currently drafting a change to serialization as mentioned on nivaldoh#22.

@sjvanrossum

@sjvanrossum Thank you so much! I sent an invitation to add you as a collaborator of laysakura/beam.

@sjvanrossum

I think it would be helpful to create design documents in order to align our goals and understanding. To start, I have written an initial version of a document titled "Custom Coders for the Beam Rust SDK".

A portion of the proposal outlined in the document has already been implemented and tested, as can be seen here: laysakura#30.

I would especially appreciate it if @sjvanrossum and @dahlbaek, who have recently collaborated with me on the laysakura/beam repository, could take a look and provide any comments or suggestions.

Of course, I welcome feedback from anyone.

@laysakura

@laysakura and all: a little note to see whether interest has dwindled, or just other priorities, etc. It seems this had been taking shape and would be great to eventually get it in a state where we can get this merged into Beam.

@brucearctor

@brucearctor For me, I still has interests but cannot prioritize Beam-related work in my company for a while 😞
It would be great if others lead the Rust SDK's development.

@brucearctor

@brucearctor For me, I still has interests but cannot prioritize Beam-related work in my company for a while 😞 It would be great if others lead the Rust SDK's development.

Makes sense -- our abilities to devote work time to various efforts does change over time.

Sounds like a call for Any/All that are interested to consider stepping in and helping/contribute!

From my side I'm still interested in contributing, but I do it on my own time, and haven't had much to spare lately.

@dahlbaek

From my side I'm still interested in contributing, but I do it on my own time, and haven't had much to spare lately.

@dahlbaek : Totally understandable, and one of the nice things about Open Source!

@brucearctor

@brucearctor I have Coder/DoFn serialization and authoring functionality in the works that I've only been able to progress on again and off again, but and I'm happy to support contributors if they wish to contribute PRs (not sure if my remote branch is fully up to date, but I'll take a peek). My situation matches that of the other contributors, I've unfortunately had to prioritize my core role over this for the past few months.

Just found this --> https://github.com/swiftdiaries/beam-rust hadn't dug deep, unclear the extent of what inside it might be usable here.

Seems like it didn't progress beyond "Hello, world!" unfortunately.

Restructure current modules into a single crate

What happened to this issue? Did everyone gave up on Rust in Beam?

Also check #40508 someone is working on Rust SDK prototype.

Navigation Menu

Repository navigation

Rust SDK #21089

21 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Rust SDK #21089

Description

Activity

gauravchak commented on Oct 11, 2022

brucearctor commented on Oct 31, 2022

nivaldoh commented on Nov 1, 2022

nivaldoh commented on Nov 1, 2022

nivaldoh commented on Nov 12, 2022

dofinn commented on Nov 12, 2022

nivaldoh commented on Nov 12, 2022

esadler-hbo commented on Dec 23, 2022

nivaldoh commented on Dec 24, 2022

TommyCpp commented on Dec 29, 2022

TommyCpp commented on Jan 1, 2023

21 remaining items

robertwb commented on Apr 13, 2023

laysakura commented on Apr 13, 2023

sjvanrossum commented on Apr 21, 2023

laysakura commented on Apr 21, 2023

laysakura commented on May 14, 2023

brucearctor commented on Sep 12, 2023

laysakura commented on Sep 12, 2023

brucearctor commented on Sep 12, 2023

dahlbaek commented on Sep 14, 2023

brucearctor commented on Sep 14, 2023

sjvanrossum commented on Nov 28, 2023

brucearctor commented on Dec 30, 2023

sjvanrossum commented on Feb 1, 2024

CH-Abhinav commented on Oct 24, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions