Skip to content

[3/3] xtensa/esp32s3: Per-process address environments and POSIX fork() - #19797

Draft
casaroli wants to merge 13 commits into
apache:masterfrom
casaroli:xtensa-split-3-esp32s3
Draft

[3/3] xtensa/esp32s3: Per-process address environments and POSIX fork()#19797
casaroli wants to merge 13 commits into
apache:masterfrom
casaroli:xtensa-split-3-esp32s3

Conversation

@casaroli

@casaroli casaroli commented Aug 11, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Summary

Split from #19772, part 3 of 3, as asked in review.

It needs both earlier parts: part 1 for up_fork(), and part 2 for BUILD_KERNEL on Xtensa. Those two do not depend on each other and can be merged in any order.

This is the ESP32-S3 half: a per-process address environment, and fork() on top of it.

The address environment is built on the SoC's cache MMU, not on anything in the Xtensa core. The core contributes only region protection — coarse 512 MB regions with no paging — so the per-process mapping comes entirely from the MMU that maps external flash and PSRAM into the address space, 64 KB and 32 KB pages respectively. This is the reason the code is chip code and not architecture code: a non-Espressif LX7 would share none of it.

up_addrenv_fork() duplicates an address environment rather than sharing it, which is what separates fork() from vfork().

The page pool is deliberately left unmapped in the kernel address space. It is carved out of the PSRAM the user processes run from, and the external memory permissions are indexed by physical address, so a permanent kernel window onto the pool would be a window onto every process's memory that no permission setting could close. The kernel reaches a pool page through a small scratch mapping instead, established for one operation and invalidated afterwards.

Two review points are addressed here.

ARCH_HAVE_FORK is now selected by the architecture instead of being defaulted from inside its own definition, so the condition sits where a reader of arch/Kconfig will look for it. It repeats the ARCH_ADDRENV dependency, because a select bypasses depends on.

The chip no longer duplicates the page pool settings. ARCH_PGPOOL_PBASE and ARCH_PGPOOL_SIZE were reachable only under ARCH_PGPOOL_MAPPING, which is why the duplication existed. A physical base and a size describe the pool whether or not it is statically mapped — only a virtual base needs the mapping — so those two move out of that block and the chip uses them.

Impact

The ESP32-S3 gains CONFIG_BUILD_KERNEL and fork(). Existing flat and protected configurations are unaffected.

Testing

Board: ESP32-S3-DevKitC. Two modules, an ESP32-S3-WROOM-2 N32R8V with 32 MB octal flash and 8 MB octal PSRAM, and an ESP32-S3-WROOM-1 N8R2 with 8 MB quad flash and 2 MB embedded PSRAM.

Host: macOS 15 on Apple Silicon, xtensa-esp32s3-elf-gcc 12.2.0.

module configuration build mode flash result
WROOM-2 esp32s3-devkit:ostest flat octal vfork() passes, status 0
WROOM-2 esp32s3-devkit:knsh protected octal vfork() passes, status 0
WROOM-2 esp32s3-devkit:kernel_oct kernel octal vfork() and fork() pass, status 0
WROOM-1 N8R2 esp32s3-devkit:ostest flat DIO vfork() passes
WROOM-1 N8R2 esp32s3-devkit:kernel_n8r2 kernel DIO vfork() and fork() pass, status 0 
user_main: vfork() test
vfork_test: Child 6 ran and exited before the parent resumed
user_main: fork() test
fork_test: Child running independently (child)
fork_test: Parent and child had independent memory
ostest_main: Exiting with status 0

tools/checkpatch.sh -c -u -m -g reports no errors.

depends-on: /pull/19795 /pull/19796 apache/nuttx-apps/pull/3685

@github-actions github-actions Bot added Area: Documentation Improvements or additions to documentation Arch: xtensa Issues related to the Xtensa architecture Size: XL The size of the change in this PR is very large. Consider breaking down the PR into smaller pieces. Board: xtensa labels Aug 11, 2026
@casaroli
casaroli marked this pull request as draft August 11, 2026 14:23
@casaroli casaroli changed the title xtensa/esp32s3: Per-process address environments and POSIX fork() [3/3] xtensa/esp32s3: Per-process address environments and POSIX fork() Aug 11, 2026
@github-actions

github-actions Bot commented Aug 11, 2026 •
edited
Loading

Copy link
Copy Markdown

MemBrowse Memory Report

esp32-devkitc

@casaroli
casaroli force-pushed the xtensa-split-3-esp32s3 branch from 43336c4 to c5c1dcc Compare August 11, 2026 15:42
@casaroli

casaroli commented Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

Keeping this as draft until we merge both prerequisites

@casaroli casaroli mentioned this pull request Aug 11, 2026
Open
casaroli added 13 commits August 11, 2026 22:03
@casaroli
arch/xtensa: Provide vfork().
2391a81 
Xtensa selected neither fork primitive, so vfork() was simply absent.  This
wires it onto the two-primitive semantics.

There is no assembly entry point and none is needed.  Every exception entry
already runs SPILL_ALL_WINDOWS, so the whole context of the calling thread is
in its exception frame and copying its stack copies a complete frame chain.
A flat build reaches that frame through SYS_save_context, issued inline so
that the recorded stack pointer belongs to a frame that stays alive for the
whole operation; a build with syscalls reaches it through xcp.sregs, recorded
by xtensa_swint() for the duration of the call.

The stack copy needs more than a relocated stack pointer here.  A windowed
ABI stores each frame's caller stack pointer absolutely, in the base save
area below the frame, so a copy taken at a different address still names the
parent throughout and the child's first retw would underflow onto the
parent's stack.  xtensa_fork_rebase() walks that chain and adds the
relocation offset to each link.  The copy also starts one base save area
below the stack pointer rather than at it, because the frame the child
resumes into keeps its caller's spilled a0-a3 there.

Ported from the per-architecture work, reduced to the two primitives.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
@casaroli
mm/pgalloc: support 32 KB and 64 KB page sizes
b1daf04 
Add CONFIG_MM_PGSIZE == 32768 and 65536 to the page-size switch (and the
Kconfig help text). The 64 KB size matches the ESP32-S3 cache-MMU page
granularity, so an address-environment port there can use one mm_pgalloc()
page per cache-MMU page (naturally 64 KB-aligned by the granule allocator)
instead of coalescing several smaller pages. Inert for existing configs:
MM_PGSIZE is only used when CONFIG_MM_PGALLOC is enabled (BUILD_KERNEL).

Assisted-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
@casaroli
xtensa: Support BUILD_KERNEL.
4f564e4 
Add what a kernel build needs on Xtensa:  a crt0 for a user process, the
kernel stack allocation that a system call switches to, the syscall entry and
return path for an unprivileged caller, and the initial register state that
starts a user task at EL0 with its save area on the kernel stack.

On the ESP32-S3 the arch code that runs while the flash mapping is in flux
moves to IRAM, and the kernel heap is placed above the user .bss so that
up_allocate_kheap() and the user address environment do not overlap.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
@casaroli
xtensa/esp32s3: add recoverable cache-attribute fault dispatcher (Uni…
0c05141 
…t B)

Route the precise cache-attribute permission faults -- Load/Store/InstrFetch
Prohibited (EXCCAUSE 28/29/20) -- from xtensa_user() to a new dispatcher,
esp32s3_pagefault_dispatch().  On a serviced fault the register frame is
returned so the exception vector's RFE re-executes the faulting instruction;
otherwise it declines to the existing panic path.  Gated by
CONFIG_ESP32S3_PAGEFAULT (default n, depends on BUILD_PROTECTED); the build
is unchanged when the option is off.

This is the recoverable-fault primitive the address-environment / demand-paging
work builds on.  Proven on the ESP32-S3-DevKitC WROOM-2:

- A precise LoadProhibited carries a tracking EXCVADDR (the exact faulting
  address), and RFE cleanly re-executes the faulted load on return -- verified
  with CONFIG_ESP32S3_PAGEFAULT_SELFTEST (the identical instruction restarts
  three times, then steps past, and the task resumes with the shell alive).
- ESP32-S3 PMS (World Controller) permission violations are NOT delivered as
  these precise causes; they raise the asynchronous DRAM0/IRAM0 PMS-monitor
  interrupt, so PMS is an isolation (kill) mechanism, not a restartable one.

No regression: esp32s3-devkit:knsh (WROOM-2) boots to nsh and ostest passes
with the option enabled.

Assisted-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
@casaroli
xtensa/esp32s3: keep octal-flash init in IRAM for the protected build
f01eb70 
The protected kernel linker (kernel-space.ld) placed the octal (OPI)
flash bring-up helpers -- esp_rom_spiflash / esp_rom_opiflash_*,
spi_flash_oct_flash_init, mmu_hal, mspi_timing_*, bootloader_flash*,
efuse_hal/efuse_utility, esp_mmu_map and esp32s3_spi_timing -- in mapped
flash.  During configure_cpu_caches() / spi_flash_init_chip_state() in
__start these run while the flash mapping is being reconfigured, which
faults (illegal instruction) on octal-flash modules such as the
ESP32-S3-WROOM-2.  Quad-flash parts never exercise the OPI path, so the
problem was latent.

Place those functions in .iram0.text (mirroring the flat sections
script) so they are safe to execute during flash reconfiguration.

Assisted-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
@casaroli
xtensa/esp32s3: Implement per-process address environments.
512efe9 
Give the ESP32-S3 the arch_addrenv_t machinery that BUILD_KERNEL needs:  a
per-process page directory built from the 64 KiB MMU pages of the chip, with
allocation, teardown, and the vaddr-to-paddr translation that the kernel uses
to reach a user buffer.

The MMU, PMS and WCL primitives are exposed as an arch API first, because the
address environment code and the protected user split both need them and
neither owns them.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
@casaroli
boards/esp32s3-devkit: add a kernel-build configuration
c6456f8 
kernel_oct, with the user-program layout and the boot ROMFS a kernel build
loads its programs from.  The ROMFS placeholder is rebuilt with the image,
the generated copy is ignored, and the programs are given stack sizes and
room for a fork() child.

Folds in:
  esp32s3-devkit: user-program layout and boot ROMFS for kernel builds
  boards/esp32s3-devkit: add a kernel-build configuration
  boards/esp32s3-devkit: give kernel_oct's programs their stacks back
  esp32s3-devkit: ignore the generated boot ROMFS
  boards/esp32s3-devkit: rebuild the ROMFS placeholder with the image
  boards/esp32s3-devkit: leave kernel_oct room for a fork() child

Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
@casaroli
xtensa/esp32s3: Reach a page pool page through a scratch mapping.
a7a86ce 
The page pool is carved out of the PSRAM that user processes run from, and
the external memory permissions are indexed by physical address, so a
permanent kernel window onto the pool is a window onto every process, which
no permission setting can close.

Stop mapping the pool.  The kernel reaches a pool page through a small
scratch region instead, mapped for one operation and invalidated afterwards.
esp32s3_pgmap() takes a slot, esp32s3_pgunmap() releases it, and
ARCH_KMAP_VBASE and ARCH_KMAP_NPAGES describe the region.  Two slots are
enough, because the deepest user is up_addrenv_fork(), which holds a source
and a destination page at once.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
@casaroli
arch/xtensa: Provide POSIX fork() on the ESP32-S3.
8c5e993 
up_addrenv_fork() duplicates an address environment into freshly allocated
pages mapped at the same virtual addresses.  The text, data and heap regions
of the source are walked one page at a time and copied into fresh pages hung
off the child's own directory, using the two kmap slots that
CONFIG_ARCH_KMAP_NPAGES reserves for exactly this.

xtensa_fork.c already took both paths:  a child that keeps the parent's stack
addresses needs no relocation, which is what a duplicated address environment
gives it.  Only the hook and the Kconfig default were missing.

fork() is offered on a kernel build, which is the only mode with per-process
address environments.

Verified on an ESP32-S3-WROOM-2 with esp32s3-devkit:kernel_oct.  ostest
reports "Parent and child had independent memory" and exits with status 0.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
@casaroli
boards/esp32s3-devkit: Add kernel_n8r2, a kernel build for 2 MB PSRAM.
be6dcf3 
kernel_oct targets a WROOM-2 N32R8V:  octal flash, and 8 MB of PSRAM for the
page pool.  The defaults size the pool for that part, with 8 pages of 64 KiB
for each of the text, data and heap regions, so 1.5 MB per process.  fork()
duplicates the address environment, so a parent and a child need 3 MB at once
and a module with 2 MB of PSRAM cannot do it.

kernel_n8r2 sizes the same build for such a module.  Each region is 2 pages,
so a process takes 384 KiB and a fork() peaks at 768 KiB, inside a 1.5 MB pool
placed at 0x80000 to leave the start of the PSRAM alone.

The flash is quad and runs in DIO mode, so this configuration also exercises
the CONFIG_ESP32S3_FLASH_MODE_OCT guard in kernel-space.ld from the quad side,
which kernel_oct cannot.

This is tight by construction.  ostest has 115 KiB of text against a 128 KiB
text region.  A larger program needs a module with more PSRAM, not a larger
pool.

Verified on an ESP32-S3-DevKitC with an N8R2 module, 8 MB flash in DIO mode
and 2 MB of embedded quad PSRAM.  ostest reports "Parent and child had
independent memory" and exits with status 0.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
@casaroli
Documentation/esp32s3-devkit: Describe the kernel build configurations.
f83f531 
Describe kernel_oct and kernel_n8r2 next to the other configurations of this
board.

The entry for kernel_oct carries what a user needs and cannot guess:  a KERNEL
build is the only mode with fork() on this chip, the page pool is reached
through a scratch mapping rather than a permanent window, the ROMFS is linked
into the kernel image so a change to an application needs the whole
export-import-mkromfsimg-relink chain, how to confirm that the ROMFS is really
in the image, and that the shell needs the full path of a program.

The entry for kernel_n8r2 states its limit.  Each region of a process is 2
pages, ostest has 115 KiB of text against a 128 KiB text region, and a larger
program needs a module with more PSRAM.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
@casaroli
arch/xtensa: Address the review of the fork series.
6d648f6 
Two points from the review of apache#19772 that belong with the ESP32-S3 work.

ARCH_HAVE_FORK is now selected by the architecture rather than defaulted from
inside its own definition, so the condition sits where a reader of arch/Kconfig
will look for it.  It repeats the ARCH_ADDRENV dependency, because a select
bypasses depends on and without that an architecture could offer fork() with no
address environment to duplicate.

The page pool no longer carries chip-specific copies of settings the common
address environment already defines.  ARCH_PGPOOL_PBASE and ARCH_PGPOOL_SIZE
were only reachable under ARCH_PGPOOL_MAPPING, which does not apply here:  the
pool is deliberately left unmapped, because it is carved out of the PSRAM the
user processes run from and the external memory permissions are indexed by
physical address.  But a physical base and a size describe the pool whether or
not it is mapped -- only a virtual base needs the mapping -- so those two move
out of that block and the chip uses them.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
@casaroli
xtensa/esp32s3: Wire the chip into the kernel build.
3be3fe2 
The common Xtensa BUILD_KERNEL support needs the chip to say what it can do
and where its memory goes.

The chip selects the address environment options it now implements, keeps the
kernel and user heaps apart, and the linker scripts separate kernel from user
text and data so the two worlds can be given different permissions.

Split out of the same change as the common code, so that arch/xtensa/src/common
can be reviewed without the chip in the way.

Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Signed-off-by: Marco Casaroli <marco.casaroli@gmail.com>
@casaroli
casaroli force-pushed the xtensa-split-3-esp32s3 branch from c5c1dcc to 3be3fe2 Compare August 11, 2026 20:06
@github-actions

github-actions Bot commented Aug 11, 2026

Copy link
Copy Markdown

❌ Cross-repo dependency could not be applied

The Build report says the declared dependency PR(s) could not be applied, so CI did not run against the combined code:

Reason: cherry-pick failed (if your PR has merge commits, rebase instead)

CI run: https://github.com/apache/nuttx/actions/runs/31531214704

@casaroli casaroli mentioned this pull request Aug 11, 2026
Open
@casaroli

casaroli commented Aug 11, 2026

Copy link
Copy Markdown
Contributor Author

This work is now in #19798, together with the ESP32-S3 chip code and the user mode isolation that a kernel build needs.

Please review #19798 instead of this one.

This pull request stays open for reference, and it will be closed when #19798 is merged.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@Donny9 Donny9 Awaiting requested review from Donny9 Donny9 is a code owner
@jerpelea jerpelea Awaiting requested review from jerpelea jerpelea is a code owner
@fdcavalcanti fdcavalcanti Awaiting requested review from fdcavalcanti fdcavalcanti is a code owner
@tmedicci tmedicci Awaiting requested review from tmedicci tmedicci is a code owner
@eren-terzioglu eren-terzioglu Awaiting requested review from eren-terzioglu eren-terzioglu is a code owner
@acassis acassis Awaiting requested review from acassis acassis is a code owner

At least 2 approving reviews are required to merge this pull request.

Assignees

No one assigned

Labels

Arch: xtensa Issues related to the Xtensa architecture Area: Documentation Improvements or additions to documentation Board: xtensa Size: XL The size of the change in this PR is very large. Consider breaking down the PR into smaller pieces.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@casaroli