Skip to content

Espressif: Add ESP-IDF component generator and int8 wake-word profile - #24293

Open
JakeStevens wants to merge 2 commits into
pytorch:mainfrom
JakeStevens:espressif-idf-component
Open

JakeStevens wants to merge 2 commits into
pytorch:mainfrom
JakeStevens:espressif-idf-component

Conversation

@JakeStevens

@JakeStevens JakeStevens commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Summary

Add examples/espressif/build_idf_component.sh, which generates a
source-vendored, registry-shaped ESP-IDF component from this
repository, following the same approach as the Arduino library
packaging but with an ESP-IDF layout.

The generator produces the schema headers, vendors the runtime and
needed kernel sources (using the same build_variables.bzl lists as
the CMake build), copies the component template from #24292 and the
shared PAL from #24291, and writes et_generated.cmake, Kconfig,
idf_component.yml, licenses, and a VERSION file recording the
ExecuTorch commit and per-profile op-set hashes. Kernel
registration is generated per profile in idf_profiles/<name>.ops;
the component's Kconfig selects which profile compiles, so switching
profiles needs no regeneration. This PR ships one example profile,
wakeword_int8, which mixes portable ATen glue ops with Cadence
backend generic (portable) int8 kernels.

Consumers of the generated component need only ESP-IDF — no host
CMake build and no flatc. Application code and models are not
included; they stay with the application.

Stacked on #24292

Test plan

  • Generated the component:
    ./examples/espressif/build_idf_component.sh --profiles wakeword_int8 --out /tmp/et-idf-out
    (component version 1.6.0~3, 80 sources / 292 headers; the
    known exporter/runtime version-skew warning is printed when the
    installed Python package was built from a different commit).
  • Verified the generated src/platform/esp_pal.cpp is
    byte-identical to examples/espressif/platform/esp_pal.cpp
    (cmp).
  • Built a minimal ESP-IDF consumer that vendors the generated
    component and calls et_model_create, for esp32s3 and
    esp32c5 with ESP-IDF v6.0.1 (idf.py set-target <target> build);
    both complete with Project build complete.
  • Ran end to end on Muse Gadget SDK w/ Box 3

@JakeStevens JakeStevens added the release notes: examples Changes to any of our example LLMs integrations, such as Llama3 and Llava label Oct 29, 2026
@pytorch-bot

pytorch-bot Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/pytorch/executorch/24293

Note: Links to docs will display an error until the docs builds have been completed.

❌ 1 New Failure, 19 Pending, 2 Unclassified Failures

As of commit 81bead7 with merge base 822cb60 (image):

NEW FAILURE - The following job has failed:

UNCLASSIFIED FAILURES - DrCI could not classify the following jobs because the workflow did not run on the merge base. The failures may be pre-existing on trunk or introduced by this PR:

This comment was automatically generated by Dr. CI and updates every 15 minutes.

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. label Oct 29, 2026
@JakeStevens
JakeStevens force-pushed the espressif-idf-component branch 4 times, most recently from 7efdb0e to c897b6b Compare October 30, 2026 18:10
@JakeStevens
JakeStevens marked this pull request as ready for review October 30, 2026 20:48
@JakeStevens
JakeStevens requested a review from rascani October 30, 2026 20:48

@rascani rascani left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Should we add this to CI too?


config ET_ENABLE
bool "Build the ExecuTorch runtime into the firmware"
default y

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should default this to n so we don't bloat users unless they need this.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

For the IDF component, WDYT about having it include all ops/delegates we might use, and then use configuration or cmake to select what what actually gets used. I'm concerned we'd start creating N x M profiles for all the model combinations we'd want to support.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Agreed, this is the better long-term shape. Punting to a follow-up PR so this one stays focused.

@JakeStevens
JakeStevens force-pushed the espressif-idf-component branch from c897b6b to 8fa3ce4 Compare November 2, 2026 18:16
Add the hand-written glue the component generator (next PR) stamps out:
a registry-shaped CMakeLists.txt gated by CONFIG_ET_ENABLE with
WHOLE_ARCHIVE kernel registration and size options (-Os, program
verification, debug asserts, kernel registry size), the
idf_component.yml and README templates, and et_model, a small C
inference API over Program/Method whose method, planned and optional
scratch arenas are caller-provided or placed with heap_caps. The
component carries only the core runtime and BufferDataLoader, and the
shared PAL is copied in by the generator rather than forked here.

Authored with AI assistance using Claude Code.
build_idf_component.sh generates a self-contained, source-vendored
ESP-IDF component from this repository, modeled on the Arduino
packaging script:

- Schema headers generated with a host flatc (built from
  third-party/flatbuffers when unavailable); consumers need only
  ESP-IDF.
- Per-profile kernel registration via ExecuTorch selective-build
  codegen. Profiles may mix aten:: ops (portable kernels) with
  cadence:: ops (Cadence backend generic/portable kernels, via
  backends/cadence/aot/functions.yaml with the selection filtered
  to the profile's roots). Each profile gets its own generated
  registration; Kconfig selects one without regeneration.
- Core runtime and kernel sources vendored from the same
  build_variables.bzl lists the CMake build uses.
- VERSION records the ExecuTorch commit and per-profile op-set
  SHA-256 + kernel counts, so model exports can be checked against
  the vendored component.

- The component is off until CONFIG_ET_ENABLE=y, so vendoring it
  leaves firmware unchanged.
- idf_component_smoke/ is a minimal consumer. CI
  (test_espressif_idf_component.sh) generates every shipped profile,
  builds the consumer for ESP32-S3 against each, and runs the smoke
  profile on the emulator.

Profiles: smoke (add/mul, for CI) and wakeword_int8 (Hey Muse
StreamingWakeWord, Cadence-backend int8 export). README documents the
generated-component flow next to the existing executor runner.

Authored with AI assistance using Claude Code.
@JakeStevens
JakeStevens force-pushed the espressif-idf-component branch from 8fa3ce4 to 81bead7 Compare November 2, 2026 18:32
@JakeStevens

Copy link
Copy Markdown
Contributor Author

Should we add this to CI too?

✅ Added the ESP-IDF component (ESP32-S3) job.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Facebook bot. Authors need to sign the CLA before a PR can be reviewed. release notes: examples Changes to any of our example LLMs integrations, such as Llama3 and Llava

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants