Skip to content

Migrate transaction read path to use views - #962

Merged
cjonas9 merged 35 commits into
feature/full-historyfrom
getTxByHash-views
Sep 11, 2026
Merged

cjonas9 merged 35 commits into
feature/full-historyfrom
getTxByHash-views

Conversation

@cjonas9

@cjonas9 cjonas9 commented Aug 27, 2026 •

Copy link
Copy Markdown
Contributor

What

This PR migrates the transaction read paths to ledger bytes/views, following #945. getTransaction (by hash) and getTransactions no longer decode an entire multi-MB LCM to read a handful of per-transaction fields back out of it. Here is what's modified:

  • getTransactionByHash: the row's meta blob is scanned into a view and exactly one transaction is extracted via ingest.LedgerTransactionViewRange
    • removes NewLedgerTransactionReaderFromLedgerCloseMeta + Seek over a fully decoded LCM.
  • getTransactions: each ledger in the page walk is borrowed as raw bytes (readLedgerPage) and walked through views (processTransactionsInLedger)
    • only the page's worth of transactions is materialized
    • page-entry rendering is extracted into transactionInfo, shared with the differential suite below.
  • store.ParseTransactionView takes an ingest.LedgerTransactionView and just reshapes it
    • old re-marshal loop is deleted 🎉
  • store.LedgerReaderTx gains loan-shaped WithLedgerRaw: fn borrows the marshaled LCM, valid only inside the call.
    • for v1: it lends the scanned blob (database/sql clones BLOB scans)
    • for v2: it advances the same walk cursor as GetLedger and lends the step's scratch bytes w/o copy.

Two related correctness fixes were made:

Note that an exhaustive test suite relating to this work exists as a fast-follow PR, #976. Both should be merged around the same time.

Minor behavior changes to note

  • Legacy TransactionMeta V0 is now tolerated as event-free by the view path. This is pinned by the new test TestTransactionsWithLegacyV0Meta
  • getTransactions now returns jrpc2.InternalError instead of jrpc2.InvalidParams when it is unable to read a transaction out of a ledger

Why

See #732. This is a component of adopting ledger views in RPC for query-path performance purposes.

Known Limitations

N/A

Attribution

Substantial parts of this PR are @tamirms's work, cherry-picked from
tamirms/gettransactions-view-walk:

  • the loan-shaped WithLedgerRaw accessor on LedgerReaderTx and its contract
  • the raw-bytes page walk in getTransactions and the shared transactionInfo renderer
  • the Tx-level WithLedgerRaw tests
  • the discovery of the V3/absent-SorobanMeta event-arity wire bug, + its upstream fix (stellar/go-stellar-sdk#5997)

Comment thread cmd/stellar-rpc/internal/rpcv2/adapters/ledger_reader.go
@tamirms

tamirms commented Aug 31, 2026 •

Copy link
Copy Markdown
Contributor

I was looking into rpcv2 perf for getTransactions and ended up building essentially this same migration before spotting your PR — so I'm dropping my branch in favor of yours. A few things I discovered that you should fold in:

  1. The raw-view approach has a wire-visible bug. For TransactionMeta V3 + a Soroban envelope + absent SorobanMeta (a real shape on pubnet protocols 20–22 — a Soroban tx charged but never executed), today's decode path returns contractEventsXdr: [[]] but the view path returns []. The fixture edit in get_transactions_test.go happens to mask exactly this case — please revert it. The root fix is now upstream: ingest: align the transaction view's event arity with GetTransactionEvents go-stellar-sdk#5997 aligns the view path's event arity with GetTransactionEvents and adds a 150-cell differential pinning the two APIs against each other, so once it merges you just bump the SDK pin and both lookups this PR touches (the pager and v1's getTransaction-by-hash) are correct with no RPC-side workaround. If you want to merge before the SDK fix lands, an interim repair exists on tamirms/gettransactions-view-walk (repairV3OperationArity, commit 2042a38) — delete it at the pin bump. I swept the full shape matrix — this is the only divergence.
  2. Use a borrow-shaped accessor instead of returning the bytes. Returning an owned view forces v2 to copy the full ledger per page (~15MB on stress profiles — a few ms against a ~6ms total). Commit d8e0797 on the same branch adds WithLedgerRaw on LedgerReaderTx (v2 delegates to rpcv2: O(tx) allocation for by-hash reads — pooled decode scratch, copy-out, pinned read #961's loan machinery; v1 reads the sqlite blob raw) and composes with the rest of your rewrite unchanged.
  3. Add the differential suite from the same branch (get_transactions_differential_test.go): ~410 cases pinning byte-identical responses old-vs-new across meta versions, fee-bumps, event shapes, cursors, and page boundaries — that's the merge gate for this migration. Also keep the checked uint32ToInt32 rather than a bare cast + nolint.

FWIW, measured on my branch (same migration, borrow accessor): getTransactions limit=50 over the wire went p50 100ms → 5.8ms (~17×), responses byte-identical. The handler is shared, so v1 should see a similar improvement.

@tamirms tamirms added this to the platform sprint 75 milestone Aug 31, 2026
@tamirms tamirms moved this from To Do to Needs Review in Platform Scrum Aug 31, 2026
@cjonas9
cjonas9 force-pushed the getTxByHash-views branch from 49fa209 to f21a36c Compare August 31, 2026 18:46
@cjonas9 cjonas9 linked an issue Aug 31, 2026 that may be closed by this pull request
@cjonas9
cjonas9 marked this pull request as ready for review August 31, 2026 20:11
Copilot AI balanced review requested due to automatic review settings August 31, 2026 20:11

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Migrates transaction reads to zero-copy XDR views, reducing full-ledger decoding and re-marshalling.

Changes:

  • Adds view-based ledger reads for SQLite and RPC v2.
  • Extracts individual transactions or paginated ranges from ledger views.
  • Updates mocks and fixtures for view-backed transaction metadata.

Reviewed changes

Copilot reviewed 11 out of 11 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
internal/store/transaction.go Reshapes transaction views into store records.
internal/store/ledger.go Adds GetLedgerView to the transaction reader.
internal/rpcv2/adapters/ledger_reader.go Implements shared view/decoded ledger walking.
internal/rpcv1/sqlitedb/transaction.go Reads transactions directly from ledger views.
internal/rpcv1/sqlitedb/mocks.go Updates transaction mock extraction.
internal/rpcv1/sqlitedb/ledger.go Implements SQLite ledger-view reads.
internal/methods/mocks.go Extends ledger-reader mocks.
internal/methods/get_transactions.go Migrates pagination to transaction view ranges.
internal/methods/get_transactions_test.go Updates fixtures and view-backed test reader.
internal/methods/get_transaction_test.go Corrects Soroban transaction metadata fixtures.
internal/methods/get_ledgers_test.go Updates encoded metadata expectations.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread cmd/stellar-rpc/internal/rpcv1/sqlitedb/transaction.go
Comment thread cmd/stellar-rpc/internal/rpcv2/adapters/ledger_reader.go Outdated
Copilot AI review requested due to automatic review settings September 1, 2026 17:57

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 17 out of 18 changed files in this pull request and generated 1 comment.

Comment thread cmd/stellar-rpc/internal/methods/get_events.go
Comment thread cmd/stellar-rpc/internal/methods/get_transactions_differential_test.go Outdated
ledger, found, err := readTx.GetLedger(ctx, ledgerSeq)
if err != nil {
return ledger, &jrpc2.Error{
// readLedgerPage borrows ledgerSeq's raw LedgerCloseMeta and runs the page

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

attribution: @tamirms loan-shaped pager design from tamirms/gettransactions-view-walk

@cjonas9 cjonas9 self-assigned this Sep 1, 2026
Copilot AI review requested due to automatic review settings September 1, 2026 22:40

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 17 out of 18 changed files in this pull request and generated 1 comment.

Suppressed comments (1)

cmd/stellar-rpc/internal/methods/get_events.go:278

  • This removes inSuccessfulContractCall from the public v1 getEvents wire response, but the Unreleased changelog does not announce the breaking removal; it only retains the old v23 deprecation notice. Since the dependency change deliberately performs the consumer-side removal, please add an Unreleased breaking-change entry so API consumers are warned.
		EventType:       eventType,
		Ledger:          int32(ledger),
		LedgerClosedAt:  ledgerClosedAt,
		ID:              cursor.String(),
		TransactionHash: txHash,

Comment thread cmd/stellar-rpc/internal/rpcv1/sqlitedb/ledger_test.go Outdated
Base automatically changed from optimize-getHealth to feature/full-history September 2, 2026 17:07
Copilot AI review requested due to automatic review settings September 11, 2026 17:01

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 13 out of 14 changed files in this pull request and generated no new comments.

Copilot AI review requested due to automatic review settings September 11, 2026 17:21

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated no new comments.

Copilot AI review requested due to automatic review settings September 11, 2026 17:24

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated no new comments.

Copilot AI review requested due to automatic review settings September 11, 2026 17:28

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated no new comments.

Comment thread cmd/stellar-rpc/internal/store/ledger.go Outdated
Comment thread cmd/stellar-rpc/internal/rpcv1/sqlitedb/ledger.go Outdated
Copilot AI review requested due to automatic review settings September 11, 2026 19:37

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 14 out of 15 changed files in this pull request and generated no new comments.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Adopt XDR views: migrate transaction read path to utilize XDR views

5 participants