fix(mqtt): bound QoS0 outbox and publish synchronously to stop heap exhaustion + packet drops#26
Merged
agessaman merged 3 commits intoJul 11, 2026
Conversation
QoS0 packet/raw publishes are forced into the esp-mqtt outbox (store=true, async) so packet topics keep flowing, but the outbox has no size bound of its own — esp-mqtt frees entries only on send-ack or ~30s expiry. On a stalled or slow uplink (socket still "connected") QoS0 frames accumulate on internal heap without limit, driving the heap exhaustion/fragmentation seen in the field. Cap the outbox at the application level: PsychicMqttClient::setOutboxLimit() records a per-client byte cap, and publish() drops a QoS0 message (returns -2) when esp_mqtt_client_get_outbox_size() is already at/over the cap, before enqueuing. The bridge's existing processPacketQueue retry/drop path handles the -2 as backpressure. Caps: 16 KiB PSRAM / 8 KiB non-PSRAM (outbox lives on internal heap, so non-PSRAM is the fragmentation-sensitive case). Portable across IDF 4.4 and 5 via esp_mqtt_client_get_outbox_size(); esp-mqtt's own outbox.limit config is not used (its enqueue path does not reliably enforce it for QoS0, and the app-level guard fires before enqueue regardless). Adds getOutboxSize()/getOutboxLimit()/getOutboxDrops() and surfaces per-slot outbox size/cap/drops via a throttled logMemoryStatus() in the MQTT task loop (MQTT_DEBUG-gated) to confirm the bound on-target.
The esp-mqtt task drains only one QUEUED outbox item per loop iteration, and each iteration blocks up to MQTT_POLL_READ_TIMEOUT_MS (1s) on esp_transport_poll_read. With little inbound traffic that caps throughput at ~1 message/second per connection, so even a light packet rate (~1.2/s) outruns the drain: the outbox pins at its cap and ~20-30% of QoS0 packets are dropped as backpressure. The poll timeout is a compile-time constant baked into the precompiled esp-mqtt lib, so the async drain rate cannot be raised on the Arduino/IDF 4.4 toolchain. Route QoS0 packet publishes through esp_mqtt_client_publish() (async=false) so they write straight to the socket, bypassing the outbox drain entirely — QoS0 no longer touches the outbox. QoS1 status keeps the async/outbox + retransmit path. The esp-mqtt task releases its API lock before the poll, so a synchronous publish from the (Core-0, prio-1) MQTT task acquires the lock and writes immediately; a stalled socket blocks only that task (mesh RX on Core 1 and the WiFi/TCP stack are unaffected), bounded by a new setNetworkTimeout() lowered to 2500ms so a first stall fails fast and flips the slot to disconnected. The outbox cap from the previous commit stays as a dormant safety net. Retools the MQTT_DEBUG diagnostic from outbox size/drops (now always ~0) to per-slot publish ok/err counts, the live signal for delivery health, with 1-based slot numbering to match the status line.
…stats CLI The 30s "MQTT: Memory" line was useful during the outbox/sync-publish investigation but is spam for production. Gate the periodic logMemoryStatus() call in the MQTT task loop behind MQTT_MEMORY_DEBUG (a dedicated diagnostics flag, not enabled by plain MQTT_DEBUG or production builds) and stop the heltec_v3 variant from force-enabling MQTT_MEMORY_DEBUG on its observer_mqtt envs (now commented out to match heltec_v4). logMemoryStatus() itself is kept intact for opt-in debugging. Expose the same data on demand via a new `get mqtt.stats` CLI command backed by MQTTBridge::formatMqttStatsReply(): free/max heap, queue depth, outbox total, and per-slot publish ok/err counts (1-based, matching the msgs: line). Fits the 160-byte reply buffer at 6 slots; returns "(bridge not running)" when down.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Problem
Observer nodes running the MQTT bridge exhausted/fragmented internal heap under a
stalled-but-still-"connected" TLS uplink, and separately dropped ~20–30% of QoS0
packets even under light load. Two independent root causes, both in the QoS0 publish
path.
Root causes
(
store=true) which has no size bound of its own — esp-mqtt frees entries only onsend-ack or ~30s expiry. On a slow/half-open uplink, QoS0 frames accumulated on
internal heap without limit.
loop iteration, and each iteration blocks up to
MQTT_POLL_READ_TIMEOUT_MS(1s) onesp_transport_poll_read. With little inbound traffic that caps throughput at~1 message/second per connection — below the ~1.2 pkt/s inflow — so the outbox pinned
at its cap and the overflow was dropped. The poll timeout is a compile-time constant
baked into the precompiled esp-mqtt lib, so the async drain can't be sped up on the
Arduino/IDF 4.4 toolchain.
Changes
PsychicMqttClient::setOutboxLimit()+ a per-publish guard usingesp_mqtt_client_get_outbox_size()): over the cap,publish()returns-2and thebridge's existing retry/drop path applies backpressure. Caps: 16 KiB PSRAM / 8 KiB
non-PSRAM. Portable across IDF 4.4 and 5; esp-mqtt's own
outbox.limitconfig is notused (it doesn't reliably cover the QoS0 enqueue path).
esp_mqtt_client_publish(),async=false): writesstraight to the socket, bypassing the drain ceiling entirely — QoS0 no longer touches
the outbox. QoS1 status keeps the durable async/outbox + retransmit path. A stalled
socket blocks only the Core-0 prio-1 MQTT task (mesh RX on Core 1 and the WiFi/TCP
stack are unaffected), bounded by a new
setNetworkTimeout()lowered to 2500 ms so afirst stall fails fast and flips the slot to disconnected. The outbox cap remains as a
dormant safety net.
ok/errcounters. The periodic 30s heap/stats serialline is gated behind
MQTT_MEMORY_DEBUG(off for production and plainMQTT_DEBUGbuilds; the heltec_v3 variant's explicit enable is now commented out). The same data is
available on demand via a new
get mqtt.statsCLI command.On-target validation (~2h each)
Outbox=0throughout,Max(largest free block)rock-stable 106–114 KB, ~9 publish errors over ~14k publishes (~0.06%, all at
disconnect moments), 15 reconnects, zero crashes.
Outbox=0, 2 errors over ~8.5k publishes, zerocrashes.
Freestable.Both boards: the original unbounded-outbox growth is gone and QoS0 drops fell to ~0.
Testing
Station_G2_repeater_observer_mqttandHeltec_v3_repeater_observer_mqtt(IDF 4.4 toolchain).
get mqtt.statsconfirmed working on-target.