Skip to content

fix(mqtt): bound QoS0 outbox and publish synchronously to stop heap exhaustion + packet drops#26

Merged
agessaman merged 3 commits into
mqtt-bridge-implementation-flexfrom
fix/mqtt-outbox-bound
Jul 11, 2026
Merged

fix(mqtt): bound QoS0 outbox and publish synchronously to stop heap exhaustion + packet drops#26
agessaman merged 3 commits into
mqtt-bridge-implementation-flexfrom
fix/mqtt-outbox-bound

Conversation

@agessaman

@agessaman agessaman commented Jul 11, 2026

Copy link
Copy Markdown
Owner

Problem

Observer nodes running the MQTT bridge exhausted/fragmented internal heap under a
stalled-but-still-"connected" TLS uplink, and separately dropped ~20–30% of QoS0
packets even under light load. Two independent root causes, both in the QoS0 publish
path.

Root causes

  1. Unbounded outbox. QoS0 packet publishes were forced into the esp-mqtt outbox
    (store=true) which has no size bound of its own — esp-mqtt frees entries only on
    send-ack or ~30s expiry. On a slow/half-open uplink, QoS0 frames accumulated on
    internal heap without limit.
  2. ~1 msg/s drain ceiling. The esp-mqtt task sends only one QUEUED outbox item per
    loop iteration, and each iteration blocks up to MQTT_POLL_READ_TIMEOUT_MS (1s) on
    esp_transport_poll_read. With little inbound traffic that caps throughput at
    ~1 message/second per connection — below the ~1.2 pkt/s inflow — so the outbox pinned
    at its cap and the overflow was dropped. The poll timeout is a compile-time constant
    baked into the precompiled esp-mqtt lib, so the async drain can't be sped up on the
    Arduino/IDF 4.4 toolchain.

Changes

  • Bound the outbox (PsychicMqttClient::setOutboxLimit() + a per-publish guard using
    esp_mqtt_client_get_outbox_size()): over the cap, publish() returns -2 and the
    bridge's existing retry/drop path applies backpressure. Caps: 16 KiB PSRAM / 8 KiB
    non-PSRAM. Portable across IDF 4.4 and 5; esp-mqtt's own outbox.limit config is not
    used (it doesn't reliably cover the QoS0 enqueue path).
  • Publish QoS0 synchronously (esp_mqtt_client_publish(), async=false): writes
    straight to the socket, bypassing the drain ceiling entirely — QoS0 no longer touches
    the outbox. QoS1 status keeps the durable async/outbox + retransmit path. A stalled
    socket blocks only the Core-0 prio-1 MQTT task (mesh RX on Core 1 and the WiFi/TCP
    stack are unaffected), bounded by a new setNetworkTimeout() lowered to 2500 ms so a
    first stall fails fast and flips the slot to disconnected. The outbox cap remains as a
    dormant safety net.
  • Diagnostics: per-slot publish ok/err counters. The periodic 30s heap/stats serial
    line is gated behind MQTT_MEMORY_DEBUG (off for production and plain MQTT_DEBUG
    builds; the heltec_v3 variant's explicit enable is now commented out). The same data is
    available on demand via a new get mqtt.stats CLI command.

On-target validation (~2h each)

  • Station G2 (PSRAM, 4 brokers): Outbox=0 throughout, Max (largest free block)
    rock-stable 106–114 KB, ~9 publish errors over ~14k publishes (~0.06%, all at
    disconnect moments), 15 reconnects, zero crashes.
  • Heltec V3 (non-PSRAM, 2 brokers): Outbox=0, 2 errors over ~8.5k publishes, zero
    crashes. Free stable.

Both boards: the original unbounded-outbox growth is gone and QoS0 drops fell to ~0.

Testing

  • Builds clean: Station_G2_repeater_observer_mqtt and Heltec_v3_repeater_observer_mqtt
    (IDF 4.4 toolchain).
  • get mqtt.stats confirmed working on-target.

QoS0 packet/raw publishes are forced into the esp-mqtt outbox (store=true,
async) so packet topics keep flowing, but the outbox has no size bound of its
own — esp-mqtt frees entries only on send-ack or ~30s expiry. On a stalled or
slow uplink (socket still "connected") QoS0 frames accumulate on internal heap
without limit, driving the heap exhaustion/fragmentation seen in the field.

Cap the outbox at the application level: PsychicMqttClient::setOutboxLimit()
records a per-client byte cap, and publish() drops a QoS0 message (returns -2)
when esp_mqtt_client_get_outbox_size() is already at/over the cap, before
enqueuing. The bridge's existing processPacketQueue retry/drop path handles the
-2 as backpressure. Caps: 16 KiB PSRAM / 8 KiB non-PSRAM (outbox lives on
internal heap, so non-PSRAM is the fragmentation-sensitive case).

Portable across IDF 4.4 and 5 via esp_mqtt_client_get_outbox_size(); esp-mqtt's
own outbox.limit config is not used (its enqueue path does not reliably enforce
it for QoS0, and the app-level guard fires before enqueue regardless).

Adds getOutboxSize()/getOutboxLimit()/getOutboxDrops() and surfaces per-slot
outbox size/cap/drops via a throttled logMemoryStatus() in the MQTT task loop
(MQTT_DEBUG-gated) to confirm the bound on-target.
The esp-mqtt task drains only one QUEUED outbox item per loop iteration, and
each iteration blocks up to MQTT_POLL_READ_TIMEOUT_MS (1s) on esp_transport_poll_read.
With little inbound traffic that caps throughput at ~1 message/second per
connection, so even a light packet rate (~1.2/s) outruns the drain: the outbox
pins at its cap and ~20-30% of QoS0 packets are dropped as backpressure. The
poll timeout is a compile-time constant baked into the precompiled esp-mqtt lib,
so the async drain rate cannot be raised on the Arduino/IDF 4.4 toolchain.

Route QoS0 packet publishes through esp_mqtt_client_publish() (async=false) so
they write straight to the socket, bypassing the outbox drain entirely — QoS0 no
longer touches the outbox. QoS1 status keeps the async/outbox + retransmit path.
The esp-mqtt task releases its API lock before the poll, so a synchronous publish
from the (Core-0, prio-1) MQTT task acquires the lock and writes immediately; a
stalled socket blocks only that task (mesh RX on Core 1 and the WiFi/TCP stack
are unaffected), bounded by a new setNetworkTimeout() lowered to 2500ms so a
first stall fails fast and flips the slot to disconnected.

The outbox cap from the previous commit stays as a dormant safety net. Retools
the MQTT_DEBUG diagnostic from outbox size/drops (now always ~0) to per-slot
publish ok/err counts, the live signal for delivery health, with 1-based slot
numbering to match the status line.
…stats CLI

The 30s "MQTT: Memory" line was useful during the outbox/sync-publish
investigation but is spam for production. Gate the periodic logMemoryStatus()
call in the MQTT task loop behind MQTT_MEMORY_DEBUG (a dedicated diagnostics
flag, not enabled by plain MQTT_DEBUG or production builds) and stop the
heltec_v3 variant from force-enabling MQTT_MEMORY_DEBUG on its observer_mqtt
envs (now commented out to match heltec_v4). logMemoryStatus() itself is kept
intact for opt-in debugging.

Expose the same data on demand via a new `get mqtt.stats` CLI command backed by
MQTTBridge::formatMqttStatsReply(): free/max heap, queue depth, outbox total,
and per-slot publish ok/err counts (1-based, matching the msgs: line). Fits the
160-byte reply buffer at 6 slots; returns "(bridge not running)" when down.
@agessaman
agessaman merged commit e7a8224 into mqtt-bridge-implementation-flex Jul 11, 2026
1 check passed
@agessaman
agessaman deleted the fix/mqtt-outbox-bound branch July 11, 2026 19:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant