feat: forest-dev export-state-tree

[hanabi1224]

Summary of changes

This PR adds a dev tool for exporting state trees together with messages, message receipts and events for a tipset range

forest-dev export-state-tree --chain mainnet --from 5915000 --to 5910000
-rw-------. 1 me me  56G Apr 10 14:46 statetree_mainnet_5910000_5915000.forest.car.zst

forest-dev export-state-tree --chain calibnet--from 3550000 --to 3540000
-rw-------. 1 me me 3.2G Apr 10 10:41 statetree_calibnet_3540000_3550000.forest.car.zst

Changes introduced in this pull request:

Reference issue to close (if applicable)

Closes

Other information and links

Change checklist

• I have performed a self-review of my own code,
• I have made corresponding changes to the documentation. All new code adheres to the team's documentation standards,
• I have added tests that prove my fix is effective or that my feature works (if possible),
• I have made sure the CHANGELOG is up-to-date. All user-facing changes should be reflected in this document.

Outside contributions

• I have read and agree to the CONTRIBUTING document.
• I have read and agree to the AI Policy document. I understand that failure to comply with the guidelines will lead to rejection of the pull request.

Summary by CodeRabbit

• New Features

  • Added an export-state-tree CLI command to export parent state trees over a specified tipset range, with configurable DB and output paths; integrated into the dev subcommands.
  • Added an IPLD streaming utility to collect and serialize blocks for state exports efficiently.

• Documentation

  • Added CLI docs for new dev commands: update-checkpoints, archive-missing, export-tipset-lookup, and export-state-tree.

[coderabbitai]

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 81e7fab9-c1eb-495a-a298-bcedd9c357cf

📥 Commits

Reviewing files that changed from the base of the PR and between 344abad and f33f28d.

📒 Files selected for processing (1)

• src/ipld/util.rs

🚧 Files skipped from review as they are similar to previous changes (1)

• src/ipld/util.rs

---

Walkthrough

Adds an export-state-tree dev CLI subcommand and integrates it into the subcommand dispatcher; adds CLI docs entries; implements an IpldStream that streams IPLD blocks from a blockstore and a new command that collects tipset parent state-related roots and writes a forest CAR file for a tipset range.

Changes

Cohort / File(s)	Summary
CLI Documentation docs/docs/users/reference/cli.sh	Added generate_markdown_section calls for new CLI entries: export-state-tree, update-checkpoints, archive-missing, and export-tipset-lookup.
Export State Tree Subcommand src/dev/subcommands/export_state_tree_cmd.rs	New ExportStateTreeCommand (clap Args) that resolves tipsets in a (to..from] range, collects parent state roots, receipts, header message CIDs and event roots, builds an IpldStream, encodes frames into a forest CAR, and writes the output file.
Subcommand Integration src/dev/subcommands/mod.rs	Added export_state_tree_cmd module, Subcommand::ExportStateTree variant, and dispatch path to call cmd.run().await.
IPLD Stream Utility src/ipld/util.rs	Added IpldStream<DB> implementing Stream<Item = anyhow::Result<CarBlock>>: LIFO CID worklist, deduplication, blockstore fetch, DAG-CBOR child extraction, yields blocks or missing-key errors.

Sequence Diagram(s)

sequenceDiagram
    participant CLI as CLI Handler
    participant Chain as ChainStore
    participant DB as ManyCar DB
    participant IPLD as IpldStream
    participant Encoder as CAR Encoder
    participant FS as File System

    CLI->>Chain: Resolve tipsets in range (to..from]
    Chain->>DB: Query tipset blocks/headers
    DB-->>Chain: Return tipset data (headers, receipts)
    Chain-->>CLI: Provide collected roots (state, receipts, event roots, msg CIDs)

    CLI->>IPLD: Create IpldStream(roots)
    loop for each CarBlock
        IPLD->>DB: Get block by CID
        DB-->>IPLD: Return block bytes
        IPLD->>IPLD: Extract child CIDs (DAG-CBOR) -> push to worklist
        IPLD-->>Encoder: Yield CarBlock
        Encoder->>Encoder: Encode/compress frame
    end

    Encoder->>FS: Write temp CAR file
    FS-->>Encoder: Flush/sync
    Encoder->>FS: Persist/rename to output path
    FS-->>CLI: Success

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~50 minutes

Possibly related PRs

• Update known blocks subcommand #6604: Modifies the same dev subcommand wiring in src/dev/subcommands/mod.rs (related enum/dispatch changes).
• tool: forest-dev export-tipset-lookup #6827: Adds other dev export subcommands and similar CAR/export utilities; overlaps in export tooling and subcommand additions.

Suggested reviewers

• LesnyRumcajs
• sudo-shashank

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title 'feat: forest-dev export-state-tree' directly and concisely summarizes the main change: adding the export-state-tree command to forest-dev, matching the PR objectives and file changes.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch hm/export-state-tree-cmd

✨ Simplify code

• Create PR with simplified code
• Commit simplified code in branch hm/export-state-tree-cmd

---

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

Actionable comments posted: 1

🧹 Nitpick comments (2)

src/dev/subcommands/export_state_tree_cmd.rs (1)

48-49: Add rustdoc for run.

ExportStateTreeCommand::run is public and newly introduced, so it should have a brief doc comment like the struct does.

As per coding guidelines, "Document public functions and structs with doc comments"

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@src/dev/subcommands/export_state_tree_cmd.rs` around lines 48 - 49, Add a
brief rustdoc comment for the public async method ExportStateTreeCommand::run
describing its purpose and behavior (e.g., what running the command does and any
important side effects or return behavior). Place the doc comment immediately
above the fn signature using ///, mirroring the style used for the
ExportStateTreeCommand struct and keeping it concise and informative for public
API consumers.

src/ipld/util.rs (1)

425-440: Document IpldStream and IpldStream::new.

Both are new public APIs, but neither has rustdoc yet. Please add short docs covering traversal order and missing-block behavior.

As per coding guidelines, "Document public functions and structs with doc comments"

🤖 Prompt for AI Agents

Verify each finding against the current code and only fix it if needed.

In `@src/ipld/util.rs` around lines 425 - 440, Add rustdoc comments for the public
struct IpldStream and its constructor IpldStream::new: document that IpldStream
traverses IPLD nodes in the order provided by the cid_vec/roots (FIFO or
DFS/BFS—state actual traversal used by the implementation), explain how seen:
CidHashSet prevents revisiting nodes, and describe missing-block behavior (e.g.,
whether missing CIDs cause the stream to yield an error, skip, or terminate).
Place the docs directly above the pub struct IpldStream<DB> and above pub fn
new(db: DB, roots: Vec<Cid>) so users know traversal order, dedup semantics, and
how the stream reacts to unavailable blocks.

🤖 Prompt for all review comments with AI agents

Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@src/dev/subcommands/export_state_tree_cmd.rs`:
- Around line 30-45: Add two boolean flags to ExportStateTreeCommand named
message_receipts and events (both #[arg(long)] with default false) so receipts
and event roots are opt-in rather than always included; update the export
invocation code that reads ExportStateTreeCommand to pass these flags into the
exporter/export_state_tree routine so it only includes message_receipts and
events when those flags are true; keep GC snapshot code that currently requires
receipts/events unchanged but explicitly set message_receipts = true and events
= true where snapshots are created for gc (the GC snapshot creator symbol), and
ensure user-facing callers (e.g., the archive/export command symbol) continue to
use the default false values unless the flags are passed.

---

Nitpick comments:
In `@src/dev/subcommands/export_state_tree_cmd.rs`:
- Around line 48-49: Add a brief rustdoc comment for the public async method
ExportStateTreeCommand::run describing its purpose and behavior (e.g., what
running the command does and any important side effects or return behavior).
Place the doc comment immediately above the fn signature using ///, mirroring
the style used for the ExportStateTreeCommand struct and keeping it concise and
informative for public API consumers.

In `@src/ipld/util.rs`:
- Around line 425-440: Add rustdoc comments for the public struct IpldStream and
its constructor IpldStream::new: document that IpldStream traverses IPLD nodes
in the order provided by the cid_vec/roots (FIFO or DFS/BFS—state actual
traversal used by the implementation), explain how seen: CidHashSet prevents
revisiting nodes, and describe missing-block behavior (e.g., whether missing
CIDs cause the stream to yield an error, skip, or terminate). Place the docs
directly above the pub struct IpldStream<DB> and above pub fn new(db: DB, roots:
Vec<Cid>) so users know traversal order, dedup semantics, and how the stream
reacts to unavailable blocks.

🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

• Push a commit to this branch (recommended)
• Create a new PR with the fixes

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Repository UI

Review profile: CHILL

Plan: Pro

Run ID: 5b8d4c5b-9ef9-43b6-b9c4-c7ede1b9cac7

📥 Commits

Reviewing files that changed from the base of the PR and between be0d0db and dd259a0.

📒 Files selected for processing (4)

• docs/docs/users/reference/cli.sh
• src/dev/subcommands/export_state_tree_cmd.rs
• src/dev/subcommands/mod.rs
• src/ipld/util.rs

[codecov]

Codecov Report

❌ Patch coverage is 0% with 86 lines in your changes missing coverage. Please review.
✅ Project coverage is 63.88%. Comparing base (aae05a5) to head (f33f28d).
⚠️ Report is 5 commits behind head on main.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/dev/subcommands/export_state_tree_cmd.rs	0.00%	64 Missing ⚠️
src/ipld/util.rs	0.00%	21 Missing ⚠️
src/dev/subcommands/mod.rs	0.00%	1 Missing ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
src/dev/subcommands/mod.rs	73.52% <0.00%> (-1.10%)	⬇️
src/ipld/util.rs	55.64% <0.00%> (-5.15%)	⬇️
src/dev/subcommands/export_state_tree_cmd.rs	0.00% <0.00%> (ø)

... and 8 files with indirect coverage changes

---

Continue to review full report in Codecov by Sentry.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update aae05a5...f33f28d. Read the comment docs.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.