Track in-flight splices for failure reporting and crash recovery - #1080
Draft
jkczyz wants to merge 15 commits into
Draft
Track in-flight splices for failure reporting and crash recovery#1080jkczyz wants to merge 15 commits into
jkczyz wants to merge 15 commits into
Conversation
Wallet sync resolves a funding payment's id for any transaction linked to the record through its conflicting txids, and then adopted that transaction's txid and confirmation outright. A cooperative close conflicts with a pending splice in exactly that way: the splice record would report the close's txid and confirmation under its InteractiveFunding type and contribution figures and graduate as if the splice had confirmed, while the close's own record never received its confirmation. Adopt a transaction only when it is part of the payment's funding history — the record's current txid or a classified candidate. Anything else is recorded under its own txid-keyed id, which also delivers the close's confirmation to the close's own record. Generated with assistance from Claude Code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A queued broadcast whose payment-record classification failed was dropped outright, on the theory that broadcasting a transaction we failed to record would leave it on-chain without a payment. For interactive funding that theory doesn't hold: the counterparty broadcasts the same transaction once the signature exchange completes, so dropping the package keeps nothing off-chain — it only guarantees the round is never recorded as a candidate on our side. The funding-status ownership gate then treats the round's confirmation as foreign to the funding record and re-keys it to a stray duplicate record, which shadows the funding record's txid lookups permanently: the splice payment stays Pending forever while an untyped duplicate holds the confirmation. Keep the package alive instead: requeue it after a short delay and retry classification until it succeeds, holding the broadcast back the whole time. Classification failures are persistence failures, so the retry is unbounded — a store that never recovers keeps the node from functioning anyway — and every failed round is logged. Generated with assistance from Claude Code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The retry test slept a fixed three seconds and assumed classification had failed by then; if writes were re-enabled before the first attempt, the test would pass without any retry happening. Count failed writes in FailSwitchStore and wait for one before re-enabling writes. Also fix the test's store reads to use list_page: the payment store's cache is bounded, so list_filter is unavailable, and this commit did not compile its tests standalone (the conversion had landed in the following commit). Implemented with Claude Code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The retry for a failed classification was a detached tokio::spawn that outlived the node. Its comment claimed a re-send after shutdown would fail because the queue had closed, but the queue receiver lives in the broadcaster and is only dropped with the node, so the re-send succeeded and a stale package would be classified and broadcast after a stop()/start() cycle. Queue failed packages inside the broadcast loop instead and retry them from a timer branch of the same select. New packages keep flowing while a retry waits, and pending retries are dropped when the loop stops. Implemented with Claude Code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A queued classification can retry after a newer candidate of the same funding already classified. The retry carries the candidate history as of its own broadcast, so applying it rotated the record's txid back to the older candidate and shrank the stored candidate history — after which wallet sync could no longer map the newer transaction to the record and would file it as a foreign duplicate. A fresh interactive-funding classification always carries the record's current txid in its history, so one that doesn't is stale: ignore it, and never let a candidate-history update drop stored candidates. Implemented with Claude Code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
LDK re-broadcasts pending claims every 30 seconds (and sweeps once per block) until they confirm, so while the payment store is unavailable, the list of pending retries accumulated a copy per rebroadcast — memory, retry load on the struggling store, and a duplicate broadcast burst on recovery all growing with the outage's duration. A package whose transactions already await a retry is not queued again, and the rest are bounded: at the bound, the oldest waiting non-funding package is dropped to make room — its transactions return with LDK's next periodic rebroadcast — but never a funding package, whose transaction would be left confirming without a recorded candidate. Fee-bumped rebroadcast variants carry new txids, so the bound, not the dedup, is what limits their accumulation. Implemented with Claude Code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Since declining to adopt a conflicting close's confirmation, a funding payment whose transaction was double-spent stayed Pending forever -- nothing wrote a terminal status for an on-chain record -- and the sync loop kept re-queueing the dead transaction for rebroadcast on every tip change. Mark such a record Failed once a conflict from outside its candidate history has confirmed through ANTI_REORG_DELAY while neither its own transaction nor any RBF candidate can still confirm, mirroring the anti-reorg finality the Succeeded transition already assumes. Removing the payment's pending entry then stops the re-queueing. Settling also removes the entry that maps candidate txids to the record, so a later wallet event for a dead candidate falls back to keying by that candidate's txid -- which, for the first candidate, is the record's own id. Skip such events rather than let the generic handling resurrect the settled record, and let a replayed replacement event finish an entry removal a crash interrupted instead of stamping the terminal status into the leftover entry. Implemented with Claude Code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Funding records were keyed by a PaymentId derived from a funding txid: the broadcast txid in the generic classification path, the first negotiated candidate's txid in the interactive path. A txid is no identity for a replaceable transaction — the record deliberately outlives RBF rounds of its funding, so its key carried the txid of whichever round happened to come first, and code could be tempted to re-derive the id from a txid instead of resolving it. Generate the id from the OS entropy source when the record is created, and resolve existing records through their transaction history (find_payment_by_txid) everywhere. RBF stability now comes from resolution instead of derivation. Resolution must share one lock acquisition with the record writes: resolved outside it, the id could go stale against a record wallet sync creates for the same transaction, producing a divergent record — so classification acquires the cross-store lock itself and the write helper now takes the guard. The funding-record surface (classification, candidates, stable ids) debuts in the upcoming release — v0.7.0 shipped splice_in with no record machinery — so changing the scheme now costs nothing, while one release later it would break payment(&PaymentId(funding_txid)) lookups for new records. Generated with assistance from Claude Code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A user-initiated splice dropped before LDK persists it leaves no trace in LDK. Recovering whatever the splice reserved and describing later events about it in terms of the original request both require persisting the splice intent before handing it to LDK, which happens before negotiation and therefore before any funding transaction exists. The pending-payment record was built around an on-chain PaymentDetails carrying a txid, which cannot represent a splice that has not been broadcast yet. Reshape PendingPaymentDetails into an enum: a PendingSplice variant that holds only the generated PaymentId and the splice intent, and a Tracked variant that is the previous record plus an optional intent retained until the splice locks. Add the SpliceIntent and SpliceKind types that record what was handed to LDK and the API call that produced it. Wallet writes to the pending store go through DataStore::mutate, replacing racy read-then-write pairs. They share one helper whose closure re-reads the payment's status inside the critical section — only Pending payments belong in the pending store, and a status read taken outside it can go stale against graduation — and promotes a bare PendingSplice to a Tracked record once a payment exists under its id: a plain payment-tracking merge would silently no-op against the variant, leaving the splice invisible to txid lookups. This is groundwork; nothing constructs a PendingSplice yet. A later commit adds the classification that reads the variant; the entry points that persist splice intents land with the splice tracking built on this. Generated with assistance from Claude Code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A user-initiated splice will be keyed by a PaymentId generated at splice time rather than derived from a candidate's txid, so its splice intent, funding payment, and candidate history all share one record. Teach the classifier to find a pre-broadcast splice intent by its channel and reuse that id, promoting the intent record to a tracked funding payment while preserving the intent until the splice locks. Splices we did not originate (counterparty-initiated or V2 dual-funded opens) fall back to a record any candidate's txid already resolves to, otherwise a freshly generated id. A splice under a generated id is no longer found by the txid-derived lookup, so it leans on find_payment_by_txid's candidate probe to map its txids back to the record. If the intent is already gone when classification runs, the classifier probes those same lookups for a record any candidate already created before generating a fresh id, so a wallet sync that recorded the transaction first and a late classification converge on one record. The generic funding classification already resolves an existing record the same way before generating a fresh id: LDK re-broadcasts a promoted-but-unconfirmed 0conf funding transaction through that path, and a test added here covers the rebroadcast merging into the record classification already created rather than creating a duplicate. Promotion of a pre-broadcast intent in persist_funding_payment_locked is gated on the payment still being Pending, read inside the pending store's critical section like the rest of the write's decision: a payment that confirmed through ANTI_REORG_DELAY before classification must not re-enter the pending store, which graduation and rebroadcast assume holds only Pending payments. No splice intents are created yet; the splice entry points that persist them land in a follow-up — on this branch the intent probe stays dormant. Generated with assistance from Claude Code. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Wallet sync can observe a funding round before classification records it as a candidate: the counterparty broadcasts an interactively funded transaction on its own, so a classification failure being retried here — or a sync poll racing the broadcast queue — leaves the round unrecorded while its events arrive. The funding-status gate rightly reports such a round foreign, and sync re-keys the event to the round's txid-derived id, creating an untyped duplicate record whose pending entry from then on shadows the funding record in txid resolution: even after the round's classification lands, every later event routes to the duplicate, the confirmation strands there, and the funding record never confirms or graduates. Fold the duplicate back in when its round becomes a recorded candidate: adopt its confirmation onto the funding record — through the same status-update path wallet sync uses, so the confirmed candidate's figures land — and remove the duplicate along with its pending entry. A duplicate for a round that never confirmed is dropped without adopting anything; the actively-broadcast candidate stays the record's current txid. The merge runs under the classification's cross-store lock acquisition, so sync cannot interleave, and is idempotent, so the broadcast queue's classification retry can re-run it after a partial failure. The pending entry is removed before the payment record: a retry rediscovers the duplicate through the record, so a failure between the two removals can still be cleaned up, instead of orphaning a pending entry that would shadow txid resolution all over again. Generated with assistance from Claude Code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
LDK only persists a splice once its negotiation reaches AwaitingSignatures, so a splice in flight when the node stops can leave no trace in LDK. Persist each user-initiated splice as an intent record before its contribution is handed to LDK, so such a splice can be recognized at the next startup — releasing whatever the wallet still holds for it, which a later commit adds — and so events about the splice can be described in terms of the original request. A fee bump reuses the channel's existing intent record, so at most one record ever exists per channel. The record is undone when LDK rejects the hand-off synchronously and settled once the splice locks, its failure is surfaced, or its channel closes. A failure event settles the intent only after the event is durably queued — a crash in between leaves the intent for the replayed event to settle, erring toward a duplicate report over a lost one — and only when the event's contribution identifies the recorded splice: a mismatch means the failure concerns an older, superseded attempt with no record of its own. A splice queued behind another pending splice survives the pending splice's lock, so its intent is re-anchored to the new funding rather than settled. Wallet state staged on a splice's behalf is flushed only after the intent record persists, so nothing the wallet reserves for a splice can outlive the record through which a later startup would release it. A splice that fails before the hand-off releases what the wallet holds for it immediately; one LDK rejects has it returned through the DiscardFunding event instead. When a lock settles an intent without spending its inputs — a replacement or counterparty-initiated splice locked instead — the inputs are released for other spends. Once a splice funding payment is classified, the intent is carried on the payment's record until the splice locks; a payment that already graduated instead removes the leftover intent record. Developed with assistance from Claude Code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
An application handling SpliceNegotiationFailed had nothing to act on: the event did not say why the splice failed, nor what the failed call had attempted. Both matter for deciding what to do next — a fee bump lost to a disconnect can simply be re-issued, while the splice it meant to bump may still confirm at the prior feerate. Attach a reason, mapped from LDK's NegotiationFailureReason onto an ldk-node-owned enum so the event's serialization and bindings do not change with LDK's, and the parameters of the originating API call, taken from the persisted splice intent when the failure identifies it. Both fields are optional and serialized as odd TLVs: events written by LDK Node v0.7 read back as None, and v0.7 readers ignore the new fields. Developed with assistance from Claude Code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
LDK only persists a splice once its negotiation reaches AwaitingSignatures, so a splice in flight when the node stops can leave no trace in LDK — and, once splice contributions lock wallet inputs, whatever the wallet reserved for such a splice would stay reserved forever. At startup, reconcile each persisted splice intent against live channel state: release the reservations of any splice LDK no longer holds and drop its record, re-anchor a queued splice whose predecessor locked while the node was down, and keep — minus any inputs no surviving round still claims — those LDK resumes on its own. Recovery is silent: the initiating call already returned, and the channel simply no longer shows a pending splice, so no failure event is fabricated for a splice lost this way. Reconciliation runs before background syncing and broadcasting start, so nothing can act on the stale reservations first. Events LDK replays from its last persisted state (e.g. a DiscardFunding for a splice that died before the node stopped) are likewise consumed before the node is running, so they cannot act on state a new user operation set up since. Developed with assistance from Claude Code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A disconnect during the interactive negotiation fails the splice with PeerDisconnected. The first test asserts that exactly one SpliceNegotiationFailed reaches the user — carrying the reason and the originating request's parameters — and that a new splice initiated afterwards completes with a single funding payment. The window only exists mid-negotiation: a contribution still queued at disconnect is resumed by LDK itself on reconnect, and one awaiting signatures survives re-establishment. The test therefore synchronizes on the counterparty's splice_ack — logged by LDK's peer handler — and stretches the negotiation by funding the splice from many small UTXOs, each of which adds an interactive-tx round trip. A splice dropped by a restart is recovered silently: startup reconciliation releases what the wallet reserved and drops the record without fabricating a failure event. What does reach the user is the failure LDK persisted at shutdown and replays at startup — once, with parameters only when it still matches a kept record. The restart tests cover both cases: a dropped splice-out surfaces without parameters and a further restart stays silent, while a dropped fee bump — whose record reconciliation keeps, since LDK still holds the negotiated splice — surfaces with the bump's parameters. In both, the application re-initiates and the splice completes. A splice confirmed while its node was offline keeps exactly one payment record under its splice-time id regardless of whether wallet sync or classification sees the confirmation first. Developed with assistance from Claude Code. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
👋 Hi! I see this is a draft PR. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Persist every application-initiated splice until LDK is guaranteed to remember it, so that a startup pass can release wallet inputs held for splices lost to a crash, and failure events can say which operation failed and why.
LDK persists a splice only once negotiation reaches
AwaitingSignatures; rounds short of that are failed on reload through theSpliceNegotiationFailed/DiscardFundingevents aChannelManagerwrite records alongside itself. But a splice initiated after the last manager write leaves no trace at all — no event ever comes. Once #1037 lands and splice contributions lock wallet inputs, a splice lost in that window would leave its inputs reserved forever, with nothing left running that knows to release them.What changes
Intent record. The splice is written into its pending payment record (from #1079) before the contribution is handed to LDK, and settled once the splice locks, its failure surfaces, or its channel closes. A fee bump replaces the intent of the splice it bumps — at most one record per channel.
Failure events.
Event::SpliceNegotiationFailedgains the failure'sreason(mirroring LDK's negotiation-failure reasons) and the initiating request'sparameters(In/Out/FeeBump). Both areNonefor splices this node didn't initiate; they're new odd TLVs, so events written by v0.7.0 still read and v0.7.0 readers skip them.Startup reconciliation. Runs before syncing and event processing, checking each record against the reloaded channel:
AwaitingSignatures, resumed on reconnect, orNegotiatedwith our contribution — keeps its record. Only inputs no surviving candidate spends are released; a fee bump lost with the restart may have reserved extras.Recovery is silent — the initiating call already returned and the channel shows no pending splice, so no failure event is fabricated. The failures LDK replays on reload are consumed before the node is marked running.
Notes for reviewers
DiscardFundinghandler that releases a discarded contribution's inputs, and a companion commit proposed there makes coin selection stage its locks so they reach disk only with the intent record.funding_contributedalso queues aSpliceNegotiationFailed, so a rejected call both returns an error and surfaces a failure event. Benign — the event no longer matches a recorded intent by then.Last in the PR stack replacing #930 for this release, per the discussion there; stacked on #1057 → #1079 (funding payment model).
Developed with assistance from Claude Code.