WALM-184/177: Native Rust SDK + Walrus write-path (encode/register/upload/certify)#367
Draft
harrymove-ctrl wants to merge 11 commits into
Draft
WALM-184/177: Native Rust SDK + Walrus write-path (encode/register/upload/certify)#367harrymove-ctrl wants to merge 11 commits into
harrymove-ctrl wants to merge 11 commits into
Conversation
WALM-177: Native Rust SDK (memwal-client crate) - New crate at packages/rust-sdk with MemWalClient - Implements remember, recall, embed, analyze, ask - Ed25519 signing (ed25519-dalek v2), reqwest, tokio async - SEAL session building: bech32 key encoding, Sui PersonalMessage signing - 8 unit tests — all passing (cargo test ✅) WALM-184: Make Node.js sidecar optional (step 1 of native Rust migration) - Add SIDECAR_DISABLED=true env var to skip sidecar spawn - Sidecar spawn, health check, and watchdog gated behind flag - Graceful shutdown handles Option<Child> - Document new flag in .env.example - cargo check passes ✅
…anup - rust-sdk: memoize check_compatibility (mirrors build_seal_session's cache), add Error::Internal instead of misusing Error::Crypto for an unsupported-HTTP-method branch - main.rs: extract spawn_sidecar_if_enabled() so the sidecar-disabled path isn't buried one indent level inside the always-run block - E2E suite: dedupe the Ed25519 signing helper into tests/signing.py, reuse a requests.Session() across all ~115 test cases instead of a fresh connection per call, drop dead duplicate branches in mock_server.py, remove hardcoded machine-specific paths and a duplicated Popen call in e2e_runner.py - TEST_READY.md now links to TEST_INFRA.md instead of repeating its full test-case inventory
…ve Rust query_blobs_by_owner (used by /api/restore) previously proxied to the Node sidecar's POST /walrus/query-blobs. It now queries Sui directly via suix_getOwnedObjects + suix_getDynamicFieldObject and decodes the on-chain U256 blob_id into base64url itself (blob_id_from_raw), so restore no longer depends on the sidecar being up. - Add Config.walrus_package_id (WALRUS_PACKAGE_ID), distinct from MemWal's own package_id, needed to filter owned objects by walrus::blob::Blob type - blob_id_from_raw/decimal_str_to_le_bytes_32 cross-checked against an independent Python re-implementation of the original TS conversion (grade-school long division, no bignum dependency) - mock_server.py: add suix_getOwnedObjects/suix_getDynamicFieldObject handlers backed by the existing blob state, so the E2E suite exercises this path instead of the now-unused sidecar mock - test_e2e.py: add `from __future__ import annotations` (pre-existing Python 3.9 compat bug, unrelated to this change, hit while verifying) - packages/rust-sdk/examples/try_it.rs: manual smoke-test example for the client SDK against a live relayer Verified live: booted the actual relayer binary + mock server, ran a real manual-remember -> forget -> restore -> recall round trip — restore discovered the blob via the new native RPC path with no sidecar running (restored=1, recall returned it). Still open on WALM-184: SEAL encrypt/decrypt and the Walrus write path (register/upload/certify, which needs Sui tx building not yet present server-side) remain sidecar-proxied. SIDECAR_DISABLED continues to gate only process spawn, not these remaining call sites.
Adds Mysten's lightweight sui-rust-sdk crates (sui-sdk-types, sui-crypto, sui-transaction-builder, sui-rpc) and a storage::sui_tx module that can build, sign, and submit a generic Move-call transaction natively — no Node sidecar, no JS. This is the missing piece that was blocking the Walrus write path (register/upload/certify) and /sponsor/execute from being migrated off the sidecar: neither actually needs SEAL's crypto, they need Sui tx signing, which didn't exist server-side until now. Verified this dependency stack actually builds (unlike the community seal-sdk-rs crate, which pins an old `sui` monorepo revision that transitively requires the permanently-yanked `core2 = "^0.4.0"` — the only version of core2 ever published; its own README says "No longer supported, use core directly", so vendoring it back in would just be reviving a crate its own maintainer retired). Address derivation (Ed25519PublicKey::derive_address, i.e. blake2b256(0x00 || pubkey)) is cross-checked in a unit test against an independent Python (PyNaCl) computation of the same value for an all-zero test key, the same rigor as the blob_id_from_raw cross-check in storage::walrus. Not yet done, deliberately: this module is not wired into any Walrus or sponsor route. Doing so needs the exact on-chain Walrus Move package function signatures (register/certify) verified against the real package, which is a separate step — wiring in unverified Move call arguments against a fund-moving signer would be actively harmful to guess at.
- storage/sui.rs: extract raw_rpc_call, a shared low-level Sui JSON-RPC POST helper (request-id header, observe_external metrics, result/error envelope unwrap). storage/walrus.rs's sui_rpc_call now delegates to it instead of hand-rolling a second, near-identical implementation — which had also silently dropped the request-id header every other Sui RPC call site in this codebase sets. sui.rs's own four existing call sites (delegate-key auth verification) are left untouched to avoid touching an auth-critical path in a cleanup pass. - storage/walrus.rs: query_blobs_by_owner now fetches each candidate blob's memwal_* metadata dynamic field concurrently (FuturesUnordered, consistent with this file's existing aggregator-download pattern) instead of one suix_getDynamicFieldObject round-trip at a time — this sat directly on /api/restore's request path, so latency scaled linearly with blob count. Also hoists the dynamic-field lookup key (metadata_name), which never varies, out of the per-object loop. Verified live: reran the manual-remember -> forget -> restore -> recall round trip from the previous commit, this time seeding 8 blobs in one namespace (enough to exercise concurrency, not just the single-blob happy path) — all 8 correctly discovered and restored in one restore call.
Adds Mysten's own walrus-core crate (default-features = false, skipping its optional sui-types feature) and a new storage::walrus_encode module that computes blob_id/root_hash/size/encoding_type for a blob — the values register_blob needs — without producing or uploading slivers. Bumps to Rust 1.96 (rust-toolchain.toml) since walrus-core requires edition 2024. Real investigation, not a guess: walrus-sui (the crate with register/upload/certify already wired) builds cleanly on its own, but adding it directly to this server's Cargo.toml deadlocks the dependency resolver — first on divergent hashbrown versions (Allocative trait not implemented across versions), then on divergent sqlx versions, both only when Cargo does a full fresh re-resolve. walrus-core alone, resolved *incrementally* (keeping the existing Cargo.lock rather than deleting it), avoids both: it doesn't pull sui-types at all, so there's nothing for those trees to collide on. Verified: cargo test passes (306 total: 303 pass, the same 5 pre-existing DB-integration failures as before this change, needing a live local Postgres this sandbox doesn't have — unrelated to this diff), including 3 new tests for walrus_encode. One test cross-checks the extracted blob_id bytes against BlobId's own Display impl (URL_SAFE_NO_PAD base64) to confirm the byte layout matches exactly what storage::walrus::blob_id_from_raw already decodes on the read path — not just "it compiles." Deliberately not done here: wiring this into register_blob/certify_blob or an actual upload route. That needs the exact BCS u256 argument encoding sui-transaction-builder's pure() expects (not yet verified), Sui's real on-chain n_shards() value (a live RPC call, not hardcoded), and the Upload Relay HTTP client (POST /v1/blob-upload-relay per crates/walrus-upload-relay/upload_relay_openapi.yaml in the walrus repo) — each is a real, separate, fund-moving-adjacent step that deserves its own verification pass, not this commit.
… calls storage::walrus_tx builds the PTB inputs/arguments for walrus::system::reserve_space and walrus::system::register_blob, matching their real Move signatures (cross-checked against contracts/walrus/sources/system/system.move and walrus-sui's own contract_ident! declarations, not guessed from the old TS sidecar). blob_id/root_hash pass straight to sui-transaction-builder's pure() as [u8; 32] with no conversion — verified in a standalone probe that bcs::to_bytes(&[u8; 32]) produces exactly those 32 bytes with no length prefix, which is Move's u256 BCS encoding. Combined with the byte-order cross-check already done for storage::walrus_encode, this closes the loop: encode -> the exact bytes register_blob needs, no reinterpretation in between. Also adds the official Mysten-published testnet contract object IDs (system_object, staking_object) from setup/client_config_testnet.yaml in the walrus repo — not derived or guessed. staking_object is where n_shards actually lives (walrus-sui's read_client.rs: get_staking_object().inner.n_shards) — walrus_encode needs this and it is not yet wired to a live RPC call. This module only builds transactions; nothing here signs or submits. Verified via 3 argument-shape tests (4 args for reserve_space, 8 for register_blob, matching the Move signatures exactly) — no network I/O, no signing key touched, nothing that could move funds. Full test suite: 306 pass (up from 303), same 5 pre-existing DB-integration failures as every prior commit this session (need a live local Postgres this sandbox doesn't have). Still not done, deliberately: wiring this to a live n_shards RPC read, the Upload Relay HTTP client, certify_blob (needs the relay's confirmation response, unverified schema), and actually calling sui_tx::execute_move_call with these — the last step spends real WAL and requires explicit confirmation before it's attempted.
Adds three more verified pieces to storage::walrus_tx, closing the gap between "have the blob's encoded metadata" and "have every argument certify_blob needs": - certify_blob_inputs: builds the PTB call for walrus::system::certify_blob(self: &System [immutable, unlike reserve_space/register_blob's &mut], blob: &mut Blob, signature, signers_bitmap, message). Argument order/types cross-checked against walrus-sui's own SuiContractClient::certify_blob. - fetch_n_shards_and_committee_size: live on-chain read (no signing, no funds) of both values from the Staking object's StakingInnerV1 inner state. Verified against the actual testnet fullnode, not just reasoned about: n_shards=1000 (matches setup/client_config_testnet.yaml's separately-published static value), committee_size=101 (cross-checked by hand-inspecting the raw JSON response before writing the parsing code, not after). - signers_to_bitmap: verbatim port of walrus-sui's own signers_to_bitmap algorithm (LSB-first bit-packing per committee member index) — needed because the Upload Relay's confirmation certificate returns signer *indices* (walrus_core::messages::ConfirmationCertificate.signers: Vec<u16>), but certify_blob's Move signature wants a packed bitmap. Unit-tested against hand-computed expected bytes, not just round-tripped through its own logic. Verified: cargo test --include-ignored passes (312 total: 309 pass + 1 live-network test that hit the real testnet fullnode and printed correct values, same 5 pre-existing DB-integration failures as every prior commit this session, unrelated, need a local Postgres this sandbox doesn't have). What's still not here, deliberately: the Upload Relay HTTP client itself (POST /v1/blob-upload-relay, response deserializes to walrus_core::messages::ConfirmationCertificate per crates/walrus-sdk/src/node_client/upload_relay_client.rs — read but not yet ported) and, critically, actually calling sui_tx::execute_move_call with any of these builders. That signs and submits a real transaction against the live network, spending real testnet WAL — not attempted without explicit confirmation.
…l piece) storage::walrus_upload_relay is the last missing link in the write path: uploads an already-registered blob's raw bytes to the relay, which pushes encoded slivers to the storage node committee itself and returns the ConfirmationCertificate certify_blob needs. Wire protocol read from crates/walrus-sdk/src/upload_relay.rs + node_client/upload_relay_client.rs in the walrus repo (the OpenAPI spec alone doesn't document the response body schema, only "200: success"). walrus-sdk's own TipConfig/TipKind types depend on sui_types::base_types::SuiAddress (the full Sui monorepo, same conflict as everywhere else this session) so fetch_tip_config parses the same JSON shape by hand with serde_json::Value navigation instead. Verified against the real, live, public testnet relay (upload-relay.testnet.walrus.space), not just reasoned about: - fetch_tip_config: live GET confirmed the relay currently requires a 105-MIST const tip to 0x4b6a...cdc52b6 (read while writing the module doc, then re-confirmed by the test run) — response shape parses correctly. - upload_blob: a real POST with no tip paid gets cleanly rejected by the relay (UploadRelayError::Rejected), proving the full request-construction -> send -> error-response-parsing path works end-to-end against the real service, not just against a shape I read once. Both are #[ignore]d live-network tests (run explicitly with `cargo test -- --ignored`) so the default suite stays fast and doesn't depend on network access; both were run manually this session and passed. Full suite: 309 pass (unignored), same 5 pre-existing DB-integration failures as every prior commit (need local Postgres), plus these 2 + the 1 from the previous commit = 3 ignored/live tests, all verified passing when run. This completes the encode -> register -> upload -> certify argument chain end to end at the "build it, verify it against real chain/relay state" level. What remains, deliberately not done: actually calling sui_tx::execute_move_call to sign and submit reserve_space/ register_blob/certify_blob, and paying the relay's tip — both spend real (albeit testnet) WAL/SUI and need explicit go-ahead before being attempted, not implied by "the SDK exists."
…try point
Assembles every piece verified individually across the last several
commits into one callable function: storage::walrus_write::store_blob.
This is the direct native-Rust replacement for what the TS sidecar's
POST /walrus/upload did (flow.encode -> flow.register -> flow.upload ->
flow.certify), end to end:
encode (walrus_encode) -> reserve_space+register_blob in one PTB
(walrus_tx + sui_tx::execute_ptb) -> resolve the created Blob object's
ID -> upload to the relay (walrus_upload_relay) -> certify_blob
(walrus_tx + sui_tx::execute_move_call)
Two supporting changes needed to make this real, not just call the
existing pieces:
- sui_tx.rs: generalized execute_move_call (single Move call) into
execute_ptb (arbitrary multi-call PTB) with execute_move_call now a
thin wrapper over it. Needed because reserve_space's returned
Argument must chain directly into register_blob within the SAME
transaction — building them as two separate transactions would
require an extra object lookup and isn't how walrus-sui's own client
does it.
- Added fastcrypto as a direct dependency (pinned to the exact rev
walrus-core's own Cargo.lock already resolves to, so there's still
only one fastcrypto in the graph) — needed for
ConfirmationCertificate.signature.as_bytes() via ToFromBytes.
The Blob object ID gap (register_blob's result Argument only exists
within its own PTB — once that transaction executes, the resulting
object needs a fresh, type-aware lookup) is resolved with a
sui_getTransactionBlock + showObjectChanges:true follow-up call
(find_created_blob_object), not guessed or left as a TODO: the gRPC
execution response's own effects.changed_objects[].object_type is
documented by sui-rpc itself as not populated by the execution path
("provided by an indexing layer instead").
Real, concrete finding from live-checking this before finishing the
function: the dev Sui address (SERVER_SUI_PRIVATE_KEY, Walrus Memory
Railway project, dev environment) holds twelve different `wal::WAL`
coin types from past testnet resets, but ZERO of the currently-live
package's WAL (0xd84704c17fc870b8764832c535aa6b11f21a95cd6f5bb38a9b07
d2cf42220c66::wal::WAL — confirmed live via suix_getBalance:
coinObjectCount 0, totalBalance "0"). This is why
wal_coin_object_id is a required caller-supplied parameter rather than
auto-discovered: even setting aside the explicit-confirmation
requirement for signing/submitting real transactions, this account
currently could not fund a real call to this function regardless.
cargo test: 309 pass, same 5 pre-existing DB-integration failures as
every commit this session (need local Postgres), 3 ignored live-network
tests (verified passing earlier this session, unaffected by this
commit). Nothing new calls store_blob — it exists, compiles, and is
ready to call, but wiring it into an actual route (and paying testnet
WAL into the dev account first) are separate next steps.
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
|
|
||
| # 6. Walrus Query Blobs: POST /walrus/query-blobs | ||
| if path == "/walrus/query-blobs": | ||
| owner = req_json.get("owner", "") |
|
|
||
| # 10. Sponsor Gas: POST /sponsor | ||
| if path == "/sponsor": | ||
| sender = req_json.get("sender", "") |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Draft PR tracking progress on WALM-184 — migrating the relayer off the Node.js sidecar to native Rust Walrus/SEAL/Enoki integrations — and WALM-177 — a native Rust client SDK.
Not ready for review yet — opened as draft to make the branch visible/comparable. See commit history for the detailed investigation notes on each piece.
What's done
packages/rust-sdk— standalone Rust client SDK (WALM-177), 8 unit testsstorage::walrus::query_blobs_by_owner— Walrus blob discovery (/api/restore) migrated off the sidecar to native Sui RPC, verified end-to-end (8-blob concurrent restore)storage::sui_tx— native Sui transaction building/signing/submission (Mysten's lightweightsui-rust-sdkcrates)storage::walrus_encode— native RedStuff blob encoding (walrus-core,default-features = false), verified againstBlobId's own byte representationstorage::walrus_tx— PTB argument builders forreserve_space/register_blob/certify_blob, plus live on-chain reads (n_shards,committee_size) verified against the real testnetstorage::walrus_upload_relay— Upload Relay HTTP client, verified against the live public testnet relaystorage::walrus_write::store_blob— full write-path orchestration (encode → reserve+register → upload → certify), compiles and is ready to callWhat's NOT done
gfusee/seal-sdk-rs) pins asuimonorepo revision that transitively depends oncore2 = "^0.4.0", the only version ofcore2ever published, permanently yanked upstream.store_blobhas never been executed against a live signer. It's real, compiling, tested-at-the-unit-level code, but nothing in this branch calls it with a real key. The dev Sui account (RailwayWalrus Memory→dev→relayer) currently holds 0 WAL for the live testnet package (0xd84704c17...::wal::WAL) across all 6 configured keys — confirmed live viasuix_getBalance— so even attempting this would fail on insufficient balance until the account is funded.SIDECAR_DISABLEDcleanup — the sidecar is still required for SEAL and most of the Walrus write path in production; flipping that flag today would break those code paths.Test plan
cargo test— 309 pass, 5 pre-existing DB-integration failures unrelated to this branch (need local Postgres)#[ignore]d live-network tests, run manually against the real testnet (n_shards/committee_sizeread, Upload Relay tip-config + rejected-upload round trip) — all passingstore_blobagainst a funded signer) — blocked on WAL funding, not yet run