[Bug]: Node.listBalances ANR in receive flow

[piotr-iohk]

What happened?

Google Play Console reports a new production ANR cluster on Android mainnet 2.3.0 / build 182.

Play cluster label: split_config.arm64_v8a.apk | uniffi_ldk_node_fn_method_node_list_balances

The ANR occurs while the receive flow refreshes wallet state and calls into LDK Node through UniFFI/JNA:

ReceiveSheet -> WalletViewModel.refreshReceiveState -> LightningRepo.syncState -> LightningRepo.getBalances -> LightningService.getBalances -> Node.listBalances

Play Vitals release dashboard snapshot:

• Release: 2.3.0 / build 182
• Type: ANR
• Affected users: 1
• Events: 1
• Share: 5.0%
• Last occurred: ~4 days ago

Related epic: #986

Expected behavior

Opening the receive flow and refreshing wallet/Lightning state should not block input dispatching or trigger an ANR. Balance retrieval through Node.listBalances should complete quickly or be coordinated off the UI-critical path with timeout/error handling.

Steps to Reproduce

No deterministic repro confirmed. Suggested scenario from the Play stack:

1. Use production mainnet Bitkit Android 2.3.0 / build 182.
2. Open the receive flow.
3. Allow ReceiveSheet to refresh receive state.
4. App calls WalletViewModel.refreshReceiveState, then LightningRepo.syncState, then Node.listBalances.
5. Observe Play ANR: input dispatching timed out.

Logs / Screenshots / Recordings

Play Console stack trace:

#00  pc 0x000000000009f7dc  /apex/com.android.runtime/lib64/bionic/libc.so (syscall+28)
#01  pc 0x0000000000cf9ef4  /data/app/.../to.bitkit.../split_config.arm64_v8a.apk
#02  pc 0x0000000000661bd0  /data/app/.../to.bitkit.../split_config.arm64_v8a.apk
#03  pc 0x00000000007b56c8  /data/app/.../to.bitkit.../split_config.arm64_v8a.apk
#04  pc 0x000000000079c024  /data/app/.../to.bitkit.../split_config.arm64_v8a.apk
#05  pc 0x00000000006183bc  /data/app/.../to.bitkit.../split_config.arm64_v8a.apk
#06  pc 0x000000000061983c  /data/app/.../to.bitkit.../split_config.arm64_v8a.apk
#07  pc 0x0000000000963e7c  /data/app/.../to.bitkit.../split_config.arm64_v8a.apk (uniffi_ldk_node_fn_method_node_list_balances+68)
#08  pc 0x000000000001404c  /data/app/.../to.bitkit.../split_config.arm64_v8a.apk (ffi_tramp_arch+30212096) (BuildId: 8549526bbd95df90b51b87eaf518d627a449eb24)
#09  pc 0x0000000000010a68  /data/app/.../to.bitkit.../split_config.arm64_v8a.apk (ffi_tramp_free+30212096) (BuildId: 8549526bbd95df90b51b87eaf518d627a449eb24)
#10  pc 0x00000000000063c8  /data/app/.../to.bitkit.../split_config.arm64_v8a.apk (getCallbackAddress+30212096) (BuildId: 8549526bbd95df90b51b87eaf518d627a449eb24)
at com.sun.jna.Native.invokeStructure (Native method)
at com.sun.jna.Native.invokeStructure (Native.java:2196)
at com.sun.jna.Function.invoke (Function.java:448)
at com.sun.jna.Function.invoke (Function.java:364)
at com.sun.jna.Library$Handler.invoke (Library.java:270)
at java.lang.reflect.Proxy.invoke (Proxy.java:1009)
at org.lightningdevkit.ldknode.UniffiLib.uniffi_ldk_node_fn_method_node_list_balances (ldk_node.android.kt)
at org.lightningdevkit.ldknode.Node$listBalances$lambda$35$$inlined$uniffiRustCall$1.invoke (ldk_node.android.kt:14656)
at org.lightningdevkit.ldknode.Node$listBalances$lambda$35$$inlined$uniffiRustCall$1.invoke (ldk_node.android.kt:313)
at org.lightningdevkit.ldknode.UniffiRustCallStatusHelper.withReference (ldk_node.android.kt:413)
at org.lightningdevkit.ldknode.Node.listBalances (ldk_node.android.kt:14900)
at to.bitkit.services.LightningService.getBalances (LightningService.kt:1106)
at to.bitkit.repositories.LightningRepo.getBalances (LightningRepo.kt:1397)
at to.bitkit.repositories.LightningRepo.syncState (LightningRepo.kt:1326)
at to.bitkit.viewmodels.WalletViewModel$refreshReceiveState$1.invokeSuspend (WalletViewModel.kt:403)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith (ContinuationImpl.kt:34)
at kotlinx.coroutines.internal.DispatchedContinuationKt.resumeCancellableWith (DispatchedContinuation.kt:375)
at kotlinx.coroutines.intrinsics.CancellableKt.startCoroutineCancellable (Cancellable.kt:26)
at kotlinx.coroutines.CoroutineStart.invoke (CoroutineStart.kt:358)
at kotlinx.coroutines.AbstractCoroutine.start (AbstractCoroutine.kt:134)
at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch (Builders.common.kt:53)
at kotlinx.coroutines.BuildersKt.launch (unavailable:1)
at kotlinx.coroutines.BuildersKt__Builders_commonKt.launch$default (Builders.common.kt:44)
at kotlinx.coroutines.BuildersKt.launch$default (unavailable:1)
at to.bitkit.viewmodels.WalletViewModel.refreshReceiveState (WalletViewModel.kt:401)
at to.bitkit.ui.screens.wallets.receive.ReceiveSheetKt$ReceiveSheet$2$1.invokeSuspend (ReceiveSheet.kt:59)
at kotlin.coroutines.jvm.internal.BaseContinuationImpl.resumeWith (ContinuationImpl.kt:34)
at kotlinx.coroutines.DispatchedTask.run (DispatchedTask.kt:100)
at androidx.compose.ui.platform.AndroidUiDispatcher.performTrampolineDispatch (AndroidUiDispatcher.android.kt:79)
at androidx.compose.ui.platform.AndroidUiDispatcher.access$performTrampolineDispatch (AndroidUiDispatcher.android.kt:41)
at androidx.compose.ui.platform.AndroidUiDispatcher$dispatchCallback$1.run (AndroidUiDispatcher.android.kt:57)
at android.os.Handler.handleCallback (Handler.java:1095)
at android.os.Handler.dispatchMessageImpl (Handler.java:135)
at android.os.Handler.dispatchMessage (Handler.java:125)
at android.os.Looper.loopOnce (Looper.java:269)
at android.os.Looper.loop (Looper.java:367)
at android.app.ActivityThread.main (ActivityThread.java:9333)
at java.lang.reflect.Method.invoke (Native method)
at com.android.internal.os.RuntimeInit$MethodAndArgsCaller.run (RuntimeInit.java:566)
at com.android.internal.os.ZygoteInit.main (ZygoteInit.java:929)

Bitkit Version

2.3.0 / build 182, production mainnet package to.bitkit.

Device / OS

TBD from Play Console details.

Reproducibility

Rarely (once or twice)

Additional context

This is part of the Play production crash/vitals follow-up tracked in #986.

This appears distinct from the existing native crash/timeout tickets:

• crash: Production SIGABRT in arm64 native code #982 tracks raw native SIGABRT crashes.
• [Bug]: $Proxy4.uniffi_ldk_node_fn_free_node — TimeoutException on Node destroy (Play 2.2.0) #984 tracks free_node timeout during node teardown/finalizer cleanup.

This new cluster points to a blocking Node.listBalances call during receive-state refresh. It may still benefit from the ongoing Android native-symbol/release-chain hardening work, but the observed symptom is an ANR rather than a crash.

[piotr-iohk]

New related Play ANR seen on 2.3.0 / build 182.
https://play.google.com/console/u/1/developers/5690237613698939049/app/4973032799696771282/vitals/crashes/250bd59288786da8e3f3ec1362570cde/details?versionCode=182&isUserPerceived=true

This appears to be the same receive-flow / LightningRepo.syncState ANR class, but blocking in Node.listPeers instead of Node.listBalances.

Play cluster label:
split_config.arm64_v8a.apk | uniffi_ldk_node_fn_method_node_list_peers

Release dashboard snapshot:

• Type: ANR
• Affected version: 182 (2.3.0)
• Affected users: 1
• Events: 1
• Last occurred: ~6 days ago

Call path:
ReceiveSheet -> WalletViewModel.refreshReceiveState -> LightningRepo.syncState -> LightningRepo.getPeers -> LightningService.getPeers -> Node.listPeers

Stack trace:

#00  pc 0x00000000000bf45c  /apex/com.android.runtime/lib64/bionic/libc.so (syscall+28)
#01  pc 0x0000000000cfa090  /data/app/.../to.bitkit.../split_config.arm64_v8a.apk
#02  pc 0x0000000000661c5c  /data/app/.../to.bitkit.../split_config.arm64_v8a.apk
#03  pc 0x0000000000791a90  /data/app/.../to.bitkit.../split_config.arm64_v8a.apk
#04  pc 0x000000000079ba58  /data/app/.../to.bitkit.../split_config.arm64_v8a.apk
#05  pc 0x000000000060cfe8  /data/app/.../to.bitkit.../split_config.arm64_v8a.apk
#06  pc 0x0000000000619894  /data/app/.../to.bitkit.../split_config.arm64_v8a.apk
#07  pc 0x0000000000963fc4  /data/app/.../to.bitkit.../split_config.arm64_v8a.apk (uniffi_ldk_node_fn_method_node_list_peers+60)
#08  pc 0x000000000001404c  /data/app/.../to.bitkit.../split_config.arm64_v8a.apk (ffi_tramp_arch+30212096) (BuildId: 8549526bbd95df90b51b87eaf518d627a449eb24)
#09  pc 0x0000000000010a68  /data/app/.../to.bitkit.../split_config.arm64_v8a.apk (ffi_tramp_free+30212096) (BuildId: 8549526bbd95df90b51b87eaf518d627a449eb24)
#10  pc 0x00000000000063c8  /data/app/.../to.bitkit.../split_config.arm64_v8a.apk (getCallbackAddress+30212096) (BuildId: 8549526bbd95df90b51b87eaf518d627a449eb24)
...
at org.lightningdevkit.ldknode.UniffiLib.uniffi_ldk_node_fn_method_node_list_peers (ldk_node.android.kt)
at org.lightningdevkit.ldknode.Node.listPeers (ldk_node.android.kt:14960)
at to.bitkit.services.LightningService.getPeers (LightningService.kt:1109)
at to.bitkit.repositories.LightningRepo.getPeers (LightningRepo.kt:1413)
at to.bitkit.repositories.LightningRepo.syncState (LightningRepo.kt:1324)
at to.bitkit.viewmodels.WalletViewModel$refreshReceiveState$1.invokeSuspend (WalletViewModel.kt:403)
at to.bitkit.ui.screens.wallets.receive.ReceiveSheetKt$ReceiveSheet$2$1.invokeSuspend (ReceiveSheet.kt:59)

[ovitrif]

The build 182 native stacks are now mapped against the exact production arm64 ELF.

Artifact match

Bitkit v2.3.0 uses ldk-node-android:0.7.0-rc.46 from 3ace915250296cbbc3069b2c23030d2cdf7ba834. The released APK, Maven AAR, and committed rc.46 Android library are byte-identical:

• libldk_node.so: 14,213,792 bytes
• SHA-256: fbb450b145054f0682d966c5326cf88a477c9efd1c82b3c74e70d0bb7acabc48
• .text: 0x561280..0xd338d0

The artifact is stripped and has no GNU build ID or published rc.46 debug-symbol classifier. The mappings below use the exact ELF's exported symbols and control flow together with the exact rc.46 source/dependency graph.

Node.listBalances

#01 0xcf9ef4 std futex wait / RwLock write contention
#02 0x661bd0 std::sys::sync::rwlock::futex::RwLock::write_contended
#03 0x7b56c8 lightning::sync::fairrwlock::FairRwLock<T>::read
#04 0x79c024 ldk_node::Node::list_balances
#05 0x6183bc list_balances path through total_anchor_channels_reserve_sats
#06 0x61983c UniFFI callback wrapper
#07 0x963e7c uniffi_ldk_node_fn_method_node_list_balances + 68

The inlined source path is Node::list_balances → total_anchor_channels_reserve_sats → ChannelManager::list_channels → short_to_chan_info.read(). FairRwLock::read queues new readers behind an already waiting writer. This trace proves lock contention and fairness queuing; it does not prove a deadlock or identify the writer. A separate rust-lightning issue needs an all-thread trace that identifies the holder.

Node.listPeers

#01 0xcfa090 std futex wait / RwLock read contention
#02 0x661c5c std::sys::sync::rwlock::futex::RwLock::read_contended
#03 0x791a90 ldk_node::peer_store::PeerStore::list_peers
#04 0x79ba58 ldk_node::Node::list_peers
#05 0x60cfe8 UniFFI scaffolding
#06 0x619894 UniFFI callback wrapper
#07 0x963fc4 uniffi_ldk_node_fn_method_node_list_peers + 60

PeerStore::add_peer and remove_peer retain peers.write() across synchronous persistence. VssStore::write waits for remote persistence through block_in_place and internal_runtime.block_on, so network-duration storage waits block list_peers readers. This remains on the fork's current main and is now tracked in synonymdev/ldk-node#99. Upstream lightningdevkit/ldk-node#919 contains the established lock-scope fix.

The Android-owned action remains moving the synchronous native snapshot reads in WalletViewModel.refreshReceiveState / LightningRepo.syncState from Main to the injected background dispatcher or serialized LDK service queue, then publishing the resulting UI state on Main.

[ovitrif]

I rechecked Play Console on 2026-08-01; the data was last updated on 2026-07-31 at 23:00.

• Neither the Node.listBalances cluster nor the Node.listPeers cluster appears in the current user-perceived 28-day issue list.
• The direct Node.listPeers cluster view reports no affected users, events, or last occurrence for Jul 4 – Aug 1.
• fix: release peer lock before persistence ldk-node#101 fixes the listPeers peer-store persistence lock and is integrated through merged fix: integrate ldk crash fixes #1122 using ldk-node-android:0.7.0-rc.62.
• The original listBalances trace identified separate LDK/rust-lightning lock contention and has not recurred in this window.
• The off-Main Android change described above remains defensive hardening owned by this issue.

@piotr-iohk, I propose closing this production incident and reopening it if either cluster recurs on a current release. Does that match your Play monitoring?

[piotr-iohk]

Play catch-up (Aug 17): 2.4.0 / 187 still shows 1 listBalances ANR (1 user) on the version dashboard. (https://play.google.com/console/u/1/developers/5690237613698939049/app/4973032799696771282/tracks/production)

Keeping this open — rc.62 on master may help listPeers but not the listBalances/syncState-on-Main path here, and none of that is in production yet.

Let's revisit after patch release.