Repository navigation
P105: integrate SwissPPNFTL reclamation into normal writes - #93
shaynelegault wants to merge 11 commits into
Conversation
e1c4d55 to
9e0fd8e
Compare
Add the append writes of #91 to kernel/patches/armv7/mtd: the iOS disk of the 64 GB iPad mini 1 becomes writable. Writing is part of the driver; the default-off MTD_APPLE_SFTL_WRITE option and the write_enable module parameter are gone. The disk stays read-only only when the replayed journal does not end at the allocation floor. Without reclamation (#93), writes fail once the blank blocks are used up. The patch keeps Shayne Legault as author and notes the changes and his original commit IDs. The bring-up notes and the board README follow the driver as carried. Assisted-by: LLM
Add the append writes of OrchardLinux#91 to kernel/patches/armv7/mtd: the iOS disk of the 64 GB iPad mini 1 becomes writable. Writing is part of the driver; the default-off MTD_APPLE_SFTL_WRITE option and the write_enable module parameter are gone. The disk stays read-only only when the replayed journal does not end at the allocation floor. Without reclamation (OrchardLinux#93), writes fail once the blank blocks are used up. The patch keeps Shayne Legault as author and notes the changes and his original commit IDs. The bring-up notes and the board README follow the driver as carried. Assisted-by: LLM
|
Thanks, Shayne. Marking this as a draft until reclamation is finished: we'll merge it once the erase and GC side that uses this planner is in place and it runs end to end. When you get there, 32-bit kernel patches now live in kernel/patches/armv7/mtd/ (listed in kernel/series/apple-armv7), without .patch.toml scope files. |
Export the offline SwissPPNFTL checkpoint and reclamation work from the merged ARMv7 append writer. Carry the kernel changes in the ARMv7 MTD series and document the current append-only behavior and remaining runtime work. Assisted-by: LLM
9e0fd8e to
3aff97d
Compare
Export the verified H2FMI erase operation, live page copier and SFTL allocator from the ARMv7 kernel at ccd56d3c440c. Record the synthetic end-to-end tests and the native checkpoint and hardware qualification still required for P105. Assisted-by: LLM
Export the native-format save and recovery path, healthy VFL rotation and block-accounting guards to the ARMv7 MTD series. Record the captured-media and interruption results, and the provider accounting still needed before enabling physical erase and reuse. Assisted-by: LLM
Export the native provider counters, free-block and wear transitions, fresh context allocation and checkpoint publication order. Record the firmware replay, the integrity checks for unlisted context sites and the remaining physical qualification requirements for erase and reuse. Assisted-by: LLM
Serialize raw programs against the lifetime of the recovered SFTL map. Stage a board-specific module to capture both physical views and controller status, and compare paired observations along with adjacent and same-index controls. Publish matching PR kernel and measurement module artifacts for device tests. Keep physical erase disabled pending measured coverage, continuous runtime accounting and cold-boot preservation. Assisted-by: LLM
Use the checkout SHA provided by the pull request job when packaging the P105 kernel and its capture module. The runner and the build container use different owners for the worktree. Assisted-by: LLM
Stage the ARMv7 measurement build in the kernel directory layout consumed by the RAM project. Include the linked kernel notes digest so a running build can be identified independently of the release string. Let hardware-check start on kernels without the optional hotplug sysctl. Fence the P105 SFTL disk and partitions as they register and verify their mount and read-only state around the bounded storage probe. Assisted-by: LLM
|
@shaynelegault, a heads-up so nothing collides with your work here. Project Cascadia now has its own P105 NAND writer (rzxvx's The proposal keeps your structure (H2FMI provider, apple-sftl, LwVM parser, then this PR) as the base. On top of it we'd port, as separate commits credited to rzxvx:
That work goes on You're welcome to review it, take any of it over, or tell us to hold off on a part you're already doing. |
The P105's translation layer registered only on the 64 GB NAND package. Port Cascadia's 16 GB layout into it (patch 0018, authored by rzxvx): four banks of 1064 blocks of 256 pages, 1055 superblocks, the VFL in blocks 1062/1063, iOS 8.4.1's context and summary rules. A 16 GB P105 now gets a read-only applesftl disk of 16,000,000,000 bytes. 0017 first moves #93's fixed 64 GB constants into a geometry descriptor, unchanged in behaviour for the 64 GB package. KUnit gains a synthetic 16 GB medium (3 cases); all 84 apple-sftl and apple-h2fmi cases pass. Untested on hardware. Assisted-by: LLM
Capture the native iOS counters over paired USB and reject mismatched flash geometry. Document the verified raw-write exclusion and the iOS pages read through both operating systems. Keep reclamation pending its physical qualification. Assisted-by: LLM
A torn page at the end of an open block made the whole P105 medium unreadable. Patch 0022 (authored by rzxvx) ports Cascadia's give_up_torn: the load skips that one page and its four sequences, and the first write moves the block's live pages elsewhere. Only malformed metadata on the block's last programmed page counts as torn; read errors and unknown statuses still stop the load, as #93 decided. KUnit: 89 cases pass. Untested on hardware. Assisted-by: LLM
…ted write-failure handling and 16 GB support ported on top Compares Shayne Legault's merged H2FMI/apple-sftl stack and draft #93 with rzxvx's apple_ppn/apple_ftl in Project Cascadia ad28ba4, and lists what each side keeps, what is ported and in which order it is qualified. Assisted-by: LLM
The P105's translation layer registered only on the 64 GB NAND package. Port Cascadia's 16 GB layout into it (patch 0018, authored by rzxvx): four banks of 1064 blocks of 256 pages, 1055 superblocks, the VFL in blocks 1062/1063, iOS 8.4.1's context and summary rules. A 16 GB P105 now gets a read-only applesftl disk of 16,000,000,000 bytes. 0017 first moves #93's fixed 64 GB constants into a geometry descriptor, unchanged in behaviour for the 64 GB package. KUnit gains a synthetic 16 GB medium (3 cases); all 84 apple-sftl and apple-h2fmi cases pass. Untested on hardware. Assisted-by: LLM
A torn page at the end of an open block made the whole P105 medium unreadable. Patch 0022 (authored by rzxvx) ports Cascadia's give_up_torn: the load skips that one page and its four sequences, and the first write moves the block's live pages elsewhere. Only malformed metadata on the block's last programmed page counts as torn; read errors and unknown statuses still stop the load, as #93 decided. KUnit: 89 cases pass. Untested on hardware. Assisted-by: LLM
…oposal Patch 0023 (authored by rzxvx) adds CONFIG_MTD_APPLE_H2FMI_FAULT_INJECT, default n: debugfs fault attributes make a page program fail before or after it reaches the NAND, or an erase fail, so the give-up paths can be exercised on a tablet as Cascadia did. KUnit gains Cascadia's fault script on synthetic media: 90 cases pass. The merge proposal now lists what each port became, compares #93's native context encoder with Cascadia's iOS 8.4.1 encoder field by field (port 2, offline: the w0c dump is private), and marks what needs a tablet or the dump in its order section. device.toml notes that the 16 GB read path is untested. Assisted-by: LLM
Replay the native counter increments from one controlled interval against the other starting sample. Byte-check the complete responses to detect any evidence of read order outside the provider counters. Record the pre-erase file checks on both iOS volumes. Assisted-by: LLM
|
@shaynelegault, here are the details for the read-only pass Henrik mentioned.
KUnit passes 90/90 (81 before), and Your 64 GB read-only qualification from today (
Any new Leave writes and erase off. The give-up, torn-page and fault-injection paths come after a clean read, against a disposable Linux volume. The 16 GB path (0018) needs a 16 GB iPad mini on iOS 8.4.1, which we don't have here. If you know someone with one, that would be the next unit. The proposal doc lists everything that still needs hardware: order section. |
…ted write-failure handling and 16 GB support ported on top Compares Shayne Legault's merged H2FMI/apple-sftl stack and draft #93 with rzxvx's apple_ppn/apple_ftl in Project Cascadia ad28ba4, and lists what each side keeps, what is ported and in which order it is qualified. Assisted-by: LLM
The P105's translation layer registered only on the 64 GB NAND package. Port Cascadia's 16 GB layout into it (patch 0018, authored by rzxvx): four banks of 1064 blocks of 256 pages, 1055 superblocks, the VFL in blocks 1062/1063, iOS 8.4.1's context and summary rules. A 16 GB P105 now gets a read-only applesftl disk of 16,000,000,000 bytes. 0017 first moves #93's fixed 64 GB constants into a geometry descriptor, unchanged in behaviour for the 64 GB package. KUnit gains a synthetic 16 GB medium (3 cases); all 84 apple-sftl and apple-h2fmi cases pass. Untested on hardware. Assisted-by: LLM
A torn page at the end of an open block made the whole P105 medium unreadable. Patch 0022 (authored by rzxvx) ports Cascadia's give_up_torn: the load skips that one page and its four sequences, and the first write moves the block's live pages elsewhere. Only malformed metadata on the block's last programmed page counts as torn; read errors and unknown statuses still stop the load, as #93 decided. KUnit: 89 cases pass. Untested on hardware. Assisted-by: LLM
…oposal Patch 0023 (authored by rzxvx) adds CONFIG_MTD_APPLE_H2FMI_FAULT_INJECT, default n: debugfs fault attributes make a page program fail before or after it reaches the NAND, or an erase fail, so the give-up paths can be exercised on a tablet as Cascadia did. KUnit gains Cascadia's fault script on synthetic media: 90 cases pass. The merge proposal now lists what each port became, compares #93's native context encoder with Cascadia's iOS 8.4.1 encoder field by field (port 2, offline: the w0c dump is private), and marks what needs a tablet or the dump in its order section. device.toml notes that the 16 GB read path is untested. Assisted-by: LLM
Check the kernel identity and complete request sequence before exporting the ordered controller timeline. Record the buffer requirement measured with the archived startup workload and the remaining firmware handoff gap. Assisted-by: LLM
Bracket each flash sample with the twelve iOS block counters and verify the boot identity when this option is selected. Record the observed grouping for individual and batched reads on the 64 GB P105. Assisted-by: LLM
…ted write-failure handling and 16 GB support ported on top Compares Shayne Legault's merged H2FMI/apple-sftl stack and draft #93 with rzxvx's apple_ppn/apple_ftl in Project Cascadia ad28ba4, and lists what each side keeps, what is ported and in which order it is qualified. Assisted-by: LLM
The P105's translation layer registered only on the 64 GB NAND package. Port Cascadia's 16 GB layout into it (patch 0018, authored by rzxvx): four banks of 1064 blocks of 256 pages, 1055 superblocks, the VFL in blocks 1062/1063, iOS 8.4.1's context and summary rules. A 16 GB P105 now gets a read-only applesftl disk of 16,000,000,000 bytes. 0017 first moves #93's fixed 64 GB constants into a geometry descriptor, unchanged in behaviour for the 64 GB package. KUnit gains a synthetic 16 GB medium (3 cases); all 84 apple-sftl and apple-h2fmi cases pass. Untested on hardware. Assisted-by: LLM
A torn page at the end of an open block made the whole P105 medium unreadable. Patch 0022 (authored by rzxvx) ports Cascadia's give_up_torn: the load skips that one page and its four sequences, and the first write moves the block's live pages elsewhere. Only malformed metadata on the block's last programmed page counts as torn; read errors and unknown statuses still stop the load, as #93 decided. KUnit: 89 cases pass. Untested on hardware. Assisted-by: LLM
…oposal Patch 0023 (authored by rzxvx) adds CONFIG_MTD_APPLE_H2FMI_FAULT_INJECT, default n: debugfs fault attributes make a page program fail before or after it reaches the NAND, or an erase fail, so the give-up paths can be exercised on a tablet as Cascadia did. KUnit gains Cascadia's fault script on synthetic media: 90 cases pass. The merge proposal now lists what each port became, compares #93's native context encoder with Cascadia's iOS 8.4.1 encoder field by field (port 2, offline: the w0c dump is private), and marks what needs a tablet or the dump in its order section. device.toml notes that the 16 GB read path is untested. Assisted-by: LLM
|
The 64 GB whole-disk read passed with the local fusion fix. |
|
@shaynelegault great, thanks for running it! What was the local fix? Could you push it, either to your PR branch or as a commit on top of |
This draft implements foreground garbage collection in the SwissPPNFTL writer for the 64 GB P105. The writer copies live data, publishes a native-format checkpoint, recovers the map and verifies the retired blocks before reuse. The full transaction runs on synthetic NAND. Physical P105 writes still append into verified blank blocks, and the installer retains its space warning.
Changes
reclaims = false.Verification
An independent replay of the original 13G36 firmware selected donor 3 from two candidates, relocated four live keys (101–104) to block 32, updated the map and saved a checkpoint using fresh context blocks 33 and 34. The 88-byte VFL and 464-byte controller records matched the Linux accounting code byte for byte. NAND responses and the initial state were modeled in RAM.
Across separate comparisons, all 552 statistics bytes matched at 426 sampling points, including counter rollover. Allocation and ordered erase decisions matched across 600 block tables. Donor finalization matched across 341 captured block records.
The C reader recovered all 15,625,016 keys in the archived image read-only. With the missing context tails explicitly modeled as erased on a RAM overlay, it recovered 22,455 ranges through all 95 incomplete checkpoint prefixes and 15 partial old-context retirements. Healthy VFL rotation reproduced all 72 captured target pages, including OOB, and preserved all 256 source pages.
All 72 host sanitizer tests, 20 ARMv7 object configurations and 50 catalog tests passed. All 45 ARMv7 patches reproduced the kernel source tree from pinned Linux v6.12. Both ARM64 series retained their 651 original patches.
The 64 GB iPad booted the kernel built for
bd8916c. The CI artifact at7549fbbcontains the identical ARMv7 kernel image and P105 device tree. Its kernel notes also match the digest measured in RAM. All 16 raw MTD devices rejected programs while SFTL owned the disk. Both partition headers and 29 live iOS Data pages matched across Linux and native iOS. A later native recheck reproduced the pre-erase hashes of 254 existing System and Data files. P105 device measurementsThe paired iOS service returned 69 live provider counters in each of 48 readings. In two comparisons of opposite read orders, the counter changes from either interval reproduced every byte of the other interval’s 8,766-byte ending reply. Host comparison tool
On the iPad, the collector can now take native disk statistics on both sides of each flash sample and confirm the same iOS boot. One 64 KiB System read counted as one disk request, 16 VFL pages and four lower-provider counts. For the same 1 MiB, 4 KiB reads produced 256 disk requests and 256 lower-provider counts. Using 64 KiB reads produced 16 disk requests and 64 lower-provider counts. Both repetitions agreed, with zero disk I/O in four control intervals. A separate run of the published collector around a single read also picked up 12 KiB written during the interval. Native capture procedure
The Linux recorder starts at the first controller request. A two-bus test with simulated registers reproduced a reversal between preparing commands and starting I/O. The revised recorder matched the independent register observer on all 20 launches and four status replies. The capture checker rejects missing events. A P105 hardware run of this diagnostic kernel and a witness for firmware activity across the handoff are pending. Linux recorder · Trace checker
An ARMv7 guest reproduced a defect in the published block driver. Even with a provider that supported erase and checkpoints, a write through the disk reached the append path without invoking the planner. The candidate fix connects the policy when the disk is registered. A buffered write and sync then triggered reclamation on synthetic NAND. A later write reused the erased block, and a fresh driver probe recovered all five relocated pages and the last write. All 74 kernel tests passed in this configuration. The change awaits a physical P105 test before export to Orchard. Candidate Linux fix and regression tests
CI passed at
7549fbb, including the ARMv7 kernel build. GitHub Actions runKernel source · P105 bring-up evidence · GitHub Actions
Review scope and follow-up qualification
Henrik’s fusion plan places this PR first and keeps P105 erase disabled. The code and existing read-only hardware evidence are ready for review in that scope. The tests below are required before enabling physical erase and advertising reclamation. Fusion order
The next physical test needs a trustworthy counter history from the final iOS sample through Linux’s first flash request. An ordered trace can cover that interval, or the device can demonstrate that flash activity has stopped before the sample and stays stopped until Linux takes over.
Physical erase remains disabled. The P105 still needs a measured erase footprint in both normal and SLC views, recovery after an interrupted erase, and an ordinary write that triggers reclamation and reuses the released block. A subsequent cold boot must recover the new data and preserve the existing iOS volumes and sampled files. Scrub and bad-member replacement also remain unqualified.