Skip to content

Don't queue background callbacks on the heap while the heap is torn down - #11935

Open
dhalbert wants to merge 2 commits into
adafruit:mainfrom
dhalbert:i2s-teardown-callback
Open

dhalbert wants to merge 2 commits into
adafruit:mainfrom
dhalbert:i2s-teardown-callback

Conversation

@dhalbert

@dhalbert dhalbert commented Nov 20, 2026 •

Copy link
Copy Markdown
Collaborator

Diagnosis by @nucino. Fix by Claude, reviewed by @dhalbert.

Problem

On soft reload, stop_mp() calls background_callback_reset(), which unlinks queued callbacks that are on the heap. It then calls gc_deinit(), which runs finalisers, and frees the heap.

But hardware owned by a heap object will keep interrupting until its finaliser stops it. On ESP32-S3, the I2SOut on_sent interrupt can queue a callback that lives inside the I2SOut object between those two calls. The callback is still queued after the heap is freed, and running it causes a hard fault.

Fix

The fix is to prevent heap-related callbacks from running while the heap is being torn down.

  • background_callback_reset() now sets a flag. While it is set, background_callback_add() drops any callback that is on the heap or whose data is on the heap.
  • A new routine background_callback_heap_ready() clears the flag. start_mp() calls it after gc_init().
  • gc_ptr_on_heap() is now put in ITCM. On i.MX RT, the USB interrupt can run during a flash write, and it calls background_callback_add(), which now calls gc_ptr_on_heap() during heap teardown. So that routine shouldn't be in flash.

This fix will work for any driver whose interrupt queues a callback stored in a heap object, not only I2SOut.

Testing

Metro ESP32-S3, running the reproducer from #11822 with 20,000 small bytearrays allocated before I2SOut. The extra heap objects make the I2SOut finaliser run later, which widens the window.

Firmware Hard faults
main, unpatched 5 in 21 reloads
this PR 1 in 1,216 reloads

A debug build showed that the remaining fault is a separate bug: the mdns.Server finaliser asserts in heap_caps_free() inside mdns_free(). It is not addressed here.

stop_mp() unlinks queued callbacks on the heap, then runs finalisers and
frees the heap. Until a finaliser stops its hardware, an interrupt can queue
a callback that lives in a heap object again, and it would still be pending
after the heap is freed. On ESP32-S3, the I2SOut on_sent interrupt did this
and caused hard faults at soft reload (adafruit#11822).

background_callback_reset() now sets a flag that makes
background_callback_add() drop callbacks on the heap until
background_callback_heap_ready() is called after the next gc_init().

gc_ptr_on_heap() moves to ITCM, because on i.MX RT the USB interrupt can
queue a callback during a flash write.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Guard common-hal/audiobusio/PDMIn.c and the EVSYS handler's call into it
with CIRCUITPY_AUDIOBUSIO_PDMIN, as I2SOut.c is guarded with
CIRCUITPY_AUDIOBUSIO_I2SOUT.

metro_m0_express and sparkfun_redboard_turbo de_DE builds overflowed flash
by 28 bytes. Turning off PDMIn leaves about 2.6 KB free. audiobusio.I2SOut
is still available.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@dhalbert
dhalbert requested a review from a team November 21, 2026 03:14

@mikeysklar mikeysklar left a comment •

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed on a Metro ESP32-S3:

main faulted on reload 2

This PR ran 30 clean.

Should the check also cover callbacks queued after gc_deinit() zeroes the heap areas?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ESP32-S3: hard fault at soft reload while I2SOut is playing

2 participants