Skip to content

P105: integrate SwissPPNFTL reclamation into normal writes - #93

Open
shaynelegault wants to merge 11 commits into
OrchardLinux:mainfrom
shaynelegault:p105/08-ftl-reclamation
Open

shaynelegault wants to merge 11 commits into
OrchardLinux:mainfrom
shaynelegault:p105/08-ftl-reclamation

Conversation

@shaynelegault

@shaynelegault shaynelegault commented Nov 6, 2026 •

Copy link
Copy Markdown
Contributor

This draft implements foreground garbage collection in the SwissPPNFTL writer for the 64 GB P105. The writer copies live data, publishes a native-format checkpoint, recovers the map and verifies the retired blocks before reuse. The full transaction runs on synthetic NAND. Physical P105 writes still append into verified blank blocks, and the installer retains its space warning.

Changes

  • Preserve the original user-sequence bits when relocating live pages. Verify ownership across the whole device map, including the 16 internal keys beyond the disk. Qualifying a clean-free destination requires zero saved mappings and a full data/OOB blank check.
  • Reserve a fresh two-block context group. After the verified base page, sample both controller statistics records. Read back each checkpoint page, recover with both generations visible, retire the old occupied context lane by lane, and cold-load again before donor erase. Reconcile wear before making a released block reusable.
  • Recompute block live counts, the global valid-page count and per-block user-sequence summaries. Preserve the native timing of cached allocation counters. Support healthy four-replica VFL bank rotation, including a source table update that can follow the latest complete checkpoint.
  • Keep physical P105 erase and SLC rotation disabled. The device capability remains reclaims = false.

Verification

  • An independent replay of the original 13G36 firmware selected donor 3 from two candidates, relocated four live keys (101–104) to block 32, updated the map and saved a checkpoint using fresh context blocks 33 and 34. The 88-byte VFL and 464-byte controller records matched the Linux accounting code byte for byte. NAND responses and the initial state were modeled in RAM.

  • Across separate comparisons, all 552 statistics bytes matched at 426 sampling points, including counter rollover. Allocation and ordered erase decisions matched across 600 block tables. Donor finalization matched across 341 captured block records.

  • The C reader recovered all 15,625,016 keys in the archived image read-only. With the missing context tails explicitly modeled as erased on a RAM overlay, it recovered 22,455 ranges through all 95 incomplete checkpoint prefixes and 15 partial old-context retirements. Healthy VFL rotation reproduced all 72 captured target pages, including OOB, and preserved all 256 source pages.

  • All 72 host sanitizer tests, 20 ARMv7 object configurations and 50 catalog tests passed. All 45 ARMv7 patches reproduced the kernel source tree from pinned Linux v6.12. Both ARM64 series retained their 651 original patches.

  • The 64 GB iPad booted the kernel built for bd8916c. The CI artifact at 7549fbb contains the identical ARMv7 kernel image and P105 device tree. Its kernel notes also match the digest measured in RAM. All 16 raw MTD devices rejected programs while SFTL owned the disk. Both partition headers and 29 live iOS Data pages matched across Linux and native iOS. A later native recheck reproduced the pre-erase hashes of 254 existing System and Data files. P105 device measurements

  • The paired iOS service returned 69 live provider counters in each of 48 readings. In two comparisons of opposite read orders, the counter changes from either interval reproduced every byte of the other interval’s 8,766-byte ending reply. Host comparison tool

  • On the iPad, the collector can now take native disk statistics on both sides of each flash sample and confirm the same iOS boot. One 64 KiB System read counted as one disk request, 16 VFL pages and four lower-provider counts. For the same 1 MiB, 4 KiB reads produced 256 disk requests and 256 lower-provider counts. Using 64 KiB reads produced 16 disk requests and 64 lower-provider counts. Both repetitions agreed, with zero disk I/O in four control intervals. A separate run of the published collector around a single read also picked up 12 KiB written during the interval. Native capture procedure

  • The Linux recorder starts at the first controller request. A two-bus test with simulated registers reproduced a reversal between preparing commands and starting I/O. The revised recorder matched the independent register observer on all 20 launches and four status replies. The capture checker rejects missing events. A P105 hardware run of this diagnostic kernel and a witness for firmware activity across the handoff are pending. Linux recorder · Trace checker

  • An ARMv7 guest reproduced a defect in the published block driver. Even with a provider that supported erase and checkpoints, a write through the disk reached the append path without invoking the planner. The candidate fix connects the policy when the disk is registered. A buffered write and sync then triggered reclamation on synthetic NAND. A later write reused the erased block, and a fresh driver probe recovered all five relocated pages and the last write. All 74 kernel tests passed in this configuration. The change awaits a physical P105 test before export to Orchard. Candidate Linux fix and regression tests

  • CI passed at 7549fbb, including the ARMv7 kernel build. GitHub Actions run

Kernel source · P105 bring-up evidence · GitHub Actions

Review scope and follow-up qualification

Henrik’s fusion plan places this PR first and keeps P105 erase disabled. The code and existing read-only hardware evidence are ready for review in that scope. The tests below are required before enabling physical erase and advertising reclamation. Fusion order

The next physical test needs a trustworthy counter history from the final iOS sample through Linux’s first flash request. An ordered trace can cover that interval, or the device can demonstrate that flash activity has stopped before the sample and stays stopped until Linux takes over.

Physical erase remains disabled. The P105 still needs a measured erase footprint in both normal and SLC views, recovery after an interrupted erase, and an ordinary write that triggers reclamation and reuses the released block. A subsequent cold boot must recover the new data and preserve the existing iOS volumes and sampled files. Scrub and bad-member replacement also remain unqualified.

@shaynelegault
shaynelegault force-pushed the p105/08-ftl-reclamation branch from e1c4d55 to 9e0fd8e Compare November 6, 2026 23:43
@shaynelegault
shaynelegault marked this pull request as ready for review November 7, 2026 05:08
henrikklee added a commit that referenced this pull request Nov 7, 2026
Add the append writes of #91 to kernel/patches/armv7/mtd: the iOS disk
of the 64 GB iPad mini 1 becomes writable. Writing is part of the driver;
the default-off MTD_APPLE_SFTL_WRITE option and the write_enable module
parameter are gone. The disk stays read-only only when the replayed
journal does not end at the allocation floor. Without reclamation (#93),
writes fail once the blank blocks are used up. The patch keeps Shayne
Legault as author and notes the changes and his original commit IDs.
The bring-up notes and the board README follow the driver as carried.

Assisted-by: LLM
henrikklee added a commit to shaynelegault/hardware that referenced this pull request Nov 7, 2026
Add the append writes of OrchardLinux#91 to kernel/patches/armv7/mtd: the iOS disk
of the 64 GB iPad mini 1 becomes writable. Writing is part of the driver;
the default-off MTD_APPLE_SFTL_WRITE option and the write_enable module
parameter are gone. The disk stays read-only only when the replayed
journal does not end at the allocation floor. Without reclamation (OrchardLinux#93),
writes fail once the blank blocks are used up. The patch keeps Shayne
Legault as author and notes the changes and his original commit IDs.
The bring-up notes and the board README follow the driver as carried.

Assisted-by: LLM
@henrikklee
henrikklee marked this pull request as draft November 17, 2026 16:47
@henrikklee

Copy link
Copy Markdown
Contributor

Thanks, Shayne. Marking this as a draft until reclamation is finished: we'll merge it once the erase and GC side that uses this planner is in place and it runs end to end. When you get there, 32-bit kernel patches now live in kernel/patches/armv7/mtd/ (listed in kernel/series/apple-armv7), without .patch.toml scope files.

Export the offline SwissPPNFTL checkpoint and reclamation work from the merged ARMv7 append writer. Carry the kernel changes in the ARMv7 MTD series and document the current append-only behavior and remaining runtime work.

Assisted-by: LLM
@shaynelegault
shaynelegault force-pushed the p105/08-ftl-reclamation branch from 9e0fd8e to 3aff97d Compare November 19, 2026 03:38
Export the verified H2FMI erase operation, live page copier and SFTL allocator from the ARMv7 kernel at ccd56d3c440c. Record the synthetic end-to-end tests and the native checkpoint and hardware qualification still required for P105.

Assisted-by: LLM
@shaynelegault shaynelegault changed the title P105: offline SwissPPNFTL checkpoint encoder and GC transaction planner P105: integrate SwissPPNFTL reclamation into normal writes Nov 19, 2026
Export the native-format save and recovery path, healthy VFL rotation and
block-accounting guards to the ARMv7 MTD series. Record the captured-media
and interruption results, and the provider accounting still needed before
enabling physical erase and reuse.

Assisted-by: LLM
Export the native provider counters, free-block and wear transitions,
fresh context allocation and checkpoint publication order. Record the
firmware replay, the integrity checks for unlisted context sites and the
remaining physical qualification requirements for erase and reuse.

Assisted-by: LLM
Serialize raw programs against the lifetime of the recovered SFTL map. Stage
a board-specific module to capture both physical views and controller status,
and compare paired observations along with adjacent and same-index controls.
Publish matching PR kernel and measurement module artifacts for device tests.

Keep physical erase disabled pending measured coverage, continuous runtime
accounting and cold-boot preservation.

Assisted-by: LLM
Use the checkout SHA provided by the pull request job when packaging the
P105 kernel and its capture module. The runner and the build container use
different owners for the worktree.

Assisted-by: LLM
Stage the ARMv7 measurement build in the kernel directory layout consumed
by the RAM project. Include the linked kernel notes digest so a running
build can be identified independently of the release string.

Let hardware-check start on kernels without the optional hotplug sysctl.
Fence the P105 SFTL disk and partitions as they register and verify their
mount and read-only state around the bounded storage probe.

Assisted-by: LLM
@henrikklee

Copy link
Copy Markdown
Contributor

@shaynelegault, a heads-up so nothing collides with your work here.

Project Cascadia now has its own P105 NAND writer (rzxvx's apple_ppn/apple_ftl, at Cascadia ad28ba4). It runs on the 16 GB iPad mini with iOS 8.4.1, and one full write, reclaim and iOS reboot cycle has completed on hardware. We compared it with your merged stack and with this PR: p105-nand-merge-proposal.md.

The proposal keeps your structure (H2FMI provider, apple-sftl, LwVM parser, then this PR) as the base. On top of it we'd port, as separate commits credited to rzxvx:

  • 16 GB geometry and iOS 8.4.1 context compatibility
  • giving up only the failed superblock on a program/erase failure instead of stopping the bus
  • torn-superblock handling at mount
  • a few controller details
  • test-only on-device fault injection

That work goes on p105-nand-fusion, branched from your head bd8916c. Your branch stays untouched; we rebase when you push, and this PR lands first as yours.

You're welcome to review it, take any of it over, or tell us to hold off on a part you're already doing.

henrikklee added a commit that referenced this pull request Nov 19, 2026
The P105's translation layer registered only on the 64 GB NAND package.
Port Cascadia's 16 GB layout into it (patch 0018, authored by rzxvx):
four banks of 1064 blocks of 256 pages, 1055 superblocks, the VFL in
blocks 1062/1063, iOS 8.4.1's context and summary rules. A 16 GB P105 now
gets a read-only applesftl disk of 16,000,000,000 bytes. 0017 first moves
#93's fixed 64 GB constants into a geometry descriptor, unchanged in
behaviour for the 64 GB package.

KUnit gains a synthetic 16 GB medium (3 cases); all 84 apple-sftl and
apple-h2fmi cases pass. Untested on hardware.

Assisted-by: LLM
Capture the native iOS counters over paired USB and reject mismatched flash geometry. Document the verified raw-write exclusion and the iOS pages read through both operating systems. Keep reclamation pending its physical qualification.

Assisted-by: LLM
henrikklee added a commit that referenced this pull request Nov 19, 2026
A torn page at the end of an open block made the whole P105 medium
unreadable. Patch 0022 (authored by rzxvx) ports Cascadia's
give_up_torn: the load skips that one page and its four sequences, and
the first write moves the block's live pages elsewhere. Only malformed
metadata on the block's last programmed page counts as torn; read
errors and unknown statuses still stop the load, as #93 decided.

KUnit: 89 cases pass. Untested on hardware.

Assisted-by: LLM
henrikklee added a commit that referenced this pull request Nov 19, 2026
…ted write-failure handling and 16 GB support ported on top

Compares Shayne Legault's merged H2FMI/apple-sftl stack and draft #93
with rzxvx's apple_ppn/apple_ftl in Project Cascadia ad28ba4, and lists
what each side keeps, what is ported and in which order it is qualified.

Assisted-by: LLM
henrikklee added a commit that referenced this pull request Nov 19, 2026
The P105's translation layer registered only on the 64 GB NAND package.
Port Cascadia's 16 GB layout into it (patch 0018, authored by rzxvx):
four banks of 1064 blocks of 256 pages, 1055 superblocks, the VFL in
blocks 1062/1063, iOS 8.4.1's context and summary rules. A 16 GB P105 now
gets a read-only applesftl disk of 16,000,000,000 bytes. 0017 first moves
#93's fixed 64 GB constants into a geometry descriptor, unchanged in
behaviour for the 64 GB package.

KUnit gains a synthetic 16 GB medium (3 cases); all 84 apple-sftl and
apple-h2fmi cases pass. Untested on hardware.

Assisted-by: LLM
henrikklee added a commit that referenced this pull request Nov 19, 2026
A torn page at the end of an open block made the whole P105 medium
unreadable. Patch 0022 (authored by rzxvx) ports Cascadia's
give_up_torn: the load skips that one page and its four sequences, and
the first write moves the block's live pages elsewhere. Only malformed
metadata on the block's last programmed page counts as torn; read
errors and unknown statuses still stop the load, as #93 decided.

KUnit: 89 cases pass. Untested on hardware.

Assisted-by: LLM
henrikklee added a commit that referenced this pull request Nov 19, 2026
…oposal

Patch 0023 (authored by rzxvx) adds CONFIG_MTD_APPLE_H2FMI_FAULT_INJECT,
default n: debugfs fault attributes make a page program fail before or
after it reaches the NAND, or an erase fail, so the give-up paths can be
exercised on a tablet as Cascadia did. KUnit gains Cascadia's fault
script on synthetic media: 90 cases pass.

The merge proposal now lists what each port became, compares #93's
native context encoder with Cascadia's iOS 8.4.1 encoder field by field
(port 2, offline: the w0c dump is private), and marks what needs a
tablet or the dump in its order section. device.toml notes that the
16 GB read path is untested.

Assisted-by: LLM
Replay the native counter increments from one controlled interval against the other starting sample. Byte-check the complete responses to detect any evidence of read order outside the provider counters. Record the pre-erase file checks on both iOS volumes.

Assisted-by: LLM
@henrikklee

Copy link
Copy Markdown
Contributor

@shaynelegault, here are the details for the read-only pass Henrik mentioned.

p105-nand-fusion (bea4fa2) is rebased on your 59bdee5. Your 0001-0016 are unchanged. On top are mtd 0017-0023:

  • 0017: a geometry descriptor in place of the fixed 64 GB constants. Behaviour should not change.
  • 0018: the 16 GB layout, read-only.
  • 0019: SDR bring-up when iBoot leaves the controller in DDR. This also turns on STRICT_DEVMEM/IO_STRICT_DEVMEM.
  • 0020: package reset after a failed transaction, up to 8 times.
  • 0021: a failed program gives up only its block and moves the live pages.
  • 0022: a torn last page of an open block is skipped at load.
  • 0023: test-only fault injection through debugfs, default n.

KUnit passes 90/90 (81 before), and bin/orchard build --device p105 examples/minimal-boot builds. To fit 0021/0022, four of your writer tests now expect the give-up instead of a stop, and one relocation test corrupts a different page. Nothing has run on a P105 yet.

Your 64 GB read-only qualification from today (p105-sftl-device-qualification.md, 59bdee5) is the best check for this branch. Rerunning the same procedure with the fusion kernel shows whether 0017's descriptor and 0020-0022 changed anything on a healthy medium. Keep everything read-only and unmounted, as in your run. It should produce the same results:

  • 12,793 map runs and 32 native context records recovered
  • block 896's logical reads match their physical addresses
  • raw MTD writes return EBUSY on all 16 devices
  • LwVM headers pass their CRC

Any new apple-sftl or h2fmi lines in dmesg are the interesting part.

Leave writes and erase off. The give-up, torn-page and fault-injection paths come after a clean read, against a disposable Linux volume.

The 16 GB path (0018) needs a 16 GB iPad mini on iOS 8.4.1, which we don't have here. If you know someone with one, that would be the next unit. The proposal doc lists everything that still needs hardware: order section.

henrikklee added a commit that referenced this pull request Nov 19, 2026
…ted write-failure handling and 16 GB support ported on top

Compares Shayne Legault's merged H2FMI/apple-sftl stack and draft #93
with rzxvx's apple_ppn/apple_ftl in Project Cascadia ad28ba4, and lists
what each side keeps, what is ported and in which order it is qualified.

Assisted-by: LLM
henrikklee added a commit that referenced this pull request Nov 19, 2026
The P105's translation layer registered only on the 64 GB NAND package.
Port Cascadia's 16 GB layout into it (patch 0018, authored by rzxvx):
four banks of 1064 blocks of 256 pages, 1055 superblocks, the VFL in
blocks 1062/1063, iOS 8.4.1's context and summary rules. A 16 GB P105 now
gets a read-only applesftl disk of 16,000,000,000 bytes. 0017 first moves
#93's fixed 64 GB constants into a geometry descriptor, unchanged in
behaviour for the 64 GB package.

KUnit gains a synthetic 16 GB medium (3 cases); all 84 apple-sftl and
apple-h2fmi cases pass. Untested on hardware.

Assisted-by: LLM
henrikklee added a commit that referenced this pull request Nov 19, 2026
A torn page at the end of an open block made the whole P105 medium
unreadable. Patch 0022 (authored by rzxvx) ports Cascadia's
give_up_torn: the load skips that one page and its four sequences, and
the first write moves the block's live pages elsewhere. Only malformed
metadata on the block's last programmed page counts as torn; read
errors and unknown statuses still stop the load, as #93 decided.

KUnit: 89 cases pass. Untested on hardware.

Assisted-by: LLM
henrikklee added a commit that referenced this pull request Nov 19, 2026
…oposal

Patch 0023 (authored by rzxvx) adds CONFIG_MTD_APPLE_H2FMI_FAULT_INJECT,
default n: debugfs fault attributes make a page program fail before or
after it reaches the NAND, or an erase fail, so the give-up paths can be
exercised on a tablet as Cascadia did. KUnit gains Cascadia's fault
script on synthetic media: 90 cases pass.

The merge proposal now lists what each port became, compares #93's
native context encoder with Cascadia's iOS 8.4.1 encoder field by field
(port 2, offline: the w0c dump is private), and marks what needs a
tablet or the dump in its order section. device.toml notes that the
16 GB read path is untested.

Assisted-by: LLM
Check the kernel identity and complete request sequence before exporting the ordered controller timeline. Record the buffer requirement measured with the archived startup workload and the remaining firmware handoff gap.

Assisted-by: LLM
Bracket each flash sample with the twelve iOS block counters and verify the boot identity when this option is selected. Record the observed grouping for individual and batched reads on the 64 GB P105.

Assisted-by: LLM
henrikklee added a commit that referenced this pull request Nov 19, 2026
…ted write-failure handling and 16 GB support ported on top

Compares Shayne Legault's merged H2FMI/apple-sftl stack and draft #93
with rzxvx's apple_ppn/apple_ftl in Project Cascadia ad28ba4, and lists
what each side keeps, what is ported and in which order it is qualified.

Assisted-by: LLM
henrikklee added a commit that referenced this pull request Nov 19, 2026
The P105's translation layer registered only on the 64 GB NAND package.
Port Cascadia's 16 GB layout into it (patch 0018, authored by rzxvx):
four banks of 1064 blocks of 256 pages, 1055 superblocks, the VFL in
blocks 1062/1063, iOS 8.4.1's context and summary rules. A 16 GB P105 now
gets a read-only applesftl disk of 16,000,000,000 bytes. 0017 first moves
#93's fixed 64 GB constants into a geometry descriptor, unchanged in
behaviour for the 64 GB package.

KUnit gains a synthetic 16 GB medium (3 cases); all 84 apple-sftl and
apple-h2fmi cases pass. Untested on hardware.

Assisted-by: LLM
henrikklee added a commit that referenced this pull request Nov 19, 2026
A torn page at the end of an open block made the whole P105 medium
unreadable. Patch 0022 (authored by rzxvx) ports Cascadia's
give_up_torn: the load skips that one page and its four sequences, and
the first write moves the block's live pages elsewhere. Only malformed
metadata on the block's last programmed page counts as torn; read
errors and unknown statuses still stop the load, as #93 decided.

KUnit: 89 cases pass. Untested on hardware.

Assisted-by: LLM
henrikklee added a commit that referenced this pull request Nov 19, 2026
…oposal

Patch 0023 (authored by rzxvx) adds CONFIG_MTD_APPLE_H2FMI_FAULT_INJECT,
default n: debugfs fault attributes make a page program fail before or
after it reaches the NAND, or an erase fail, so the give-up paths can be
exercised on a tablet as Cascadia did. KUnit gains Cascadia's fault
script on synthetic media: 90 cases pass.

The merge proposal now lists what each port became, compares #93's
native context encoder with Cascadia's iOS 8.4.1 encoder field by field
(port 2, offline: the w0c dump is private), and marks what needs a
tablet or the dump in its order section. device.toml notes that the
16 GB read path is untested.

Assisted-by: LLM
@shaynelegault
shaynelegault marked this pull request as ready for review November 20, 2026 15:41
@shaynelegault

Copy link
Copy Markdown
Contributor Author

The 64 GB whole-disk read passed with the local fusion fix.

@henrikklee

Copy link
Copy Markdown
Contributor

@shaynelegault great, thanks for running it! What was the local fix? Could you push it, either to your PR branch or as a commit on top of p105-nand-fusion (with git commit -s), so we can fold it in? Did the results otherwise match your earlier run (12,793 map runs, 32 context records, block 896)?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants