diff --git a/docs/developers/android-sparse-file.md b/docs/developers/android-sparse-file.md index 94d77ba..629185c 100644 --- a/docs/developers/android-sparse-file.md +++ b/docs/developers/android-sparse-file.md @@ -44,3 +44,50 @@ without always writing at the beginning of the partition. The solution is to use the ``CHUNK_TYPE_DONTCARE``, which will skip some blocks of the output partition in the beginning of the 2+ images. + +Each split fragment is a fully valid, self-contained sparse file: + +- its ``total_blks`` field in the sparse header always reports the full block + count of the *original*, unsplit image, not just the blocks contained in + that particular fragment; +- it is prefixed with a ``CHUNK_TYPE_DONTCARE`` chunk covering every block + already written by previous fragments; +- it is suffixed with a ``CHUNK_TYPE_DONTCARE`` chunk covering every block that + will be written by later fragments; +- the total serialized size of the fragment (file header + all chunk headers + + payloads) never exceeds ``max-download-size``. + +## Splitting raw (non-sparse) files + +Raw binary files can also be split into a sequence of ``max-download-size``-bounded +sparse fragments, using the exact same splitting algorithm described above. + +The raw file is treated as a single logical ``CHUNK_TYPE_RAW`` region spanning +the whole image: + +```text +total_blks = ceil(file_size / block_size) +``` + +where ``block_size`` defaults to 4096 bytes. This synthetic RAW region is fed +into the same fragment-splitting logic used for real sparse files (prefix/suffix +``CHUNK_TYPE_DONTCARE`` padding, header/overhead accounting bounded by +``max-download-size``), so raw files gain automatic splitting support without +needing any format conversion of the file on disk. + +If the raw file size is not a multiple of ``block_size``, the final block is +transparently zero-padded up to the block boundary in the generated sparse +payload only; the source file itself is never modified. + +This is implemented by ``snagrecover.protocols.fastboot.Fastboot.flash_image()``, +which: + +1. Reads ``max-download-size`` from U-Boot (raises an error if it can't be read, + or if it is 0). +2. If the file's size fits within ``max-download-size``, downloads and flashes + it directly with no splitting. +3. Otherwise, detects whether the file is an Android sparse file (by checking + its magic cookie) or a raw binary file, and splits it accordingly: + sparse files are split with the existing chunk-parsing logic, raw files are + split via the synthetic single-RAW-region approach described above. +4. Downloads and flashes each fragment in turn. diff --git a/docs/snagfactory.md b/docs/snagfactory.md index 21b64c7..278131a 100644 --- a/docs/snagfactory.md +++ b/docs/snagfactory.md @@ -90,10 +90,14 @@ configuration variables, that are common to all tasks. It must contain the following values: ```yaml -target-device: The device configured as the Fastboot flashing backend in U-Boot. Either 'mmc' or 'nand'. +target-device: The device configured as the Fastboot flashing backend in U-Boot. Either 'mmc', 'ufs' or 'nand'. fb-buffer-addr: The size in bytes of the Fastboot buffer. eraseblk-size: The size in bytes of an erase block for MTD targets fb-buffer-size: (optional) The size in bytes of the Fastboot buffer. This can only be used to reduce the default U-Boot buffer size. +usb-wait-timeout: (optional) The number of seconds to wait for the board's USB Fastboot + device to (re-)enumerate before giving up, for example after U-Boot boots following + snagrecover. This is useful for boards which take longer than the default (~9 seconds) + to boot up to the U-Boot Fastboot prompt. ``` These variables may be updated during the factory flashing process, e.g.: @@ -311,6 +315,21 @@ Example: skip-pwr-cycle: False ``` +#### UFS backends + +`target-device: ufs` is supported for boards whose storage is UFS rather +than eMMC/MTD. With a `ufs` target-device: + +- `eraseblk-size` is **not** required (same as for `mmc`), since UFS is + LBA-addressable like eMMC. +- The **`gpt`**, **`flash`**, **`mtd-parts`** and **`emmc-hwpart`** tasks are + **not supported currently** for `ufs` targets and will raise a clear + configuration error if used. This is because U-Boot addresses UFS storage + through the `scsi` command family rather than the `mmc`-specific commands + these tasks generate. +- Instead, use the **`run`** task to issue the equivalent raw commands + directly + ## libusb-win32 diff --git a/docs/snagflash.md b/docs/snagflash.md index c306fcb..0fe890c 100644 --- a/docs/snagflash.md +++ b/docs/snagflash.md @@ -41,6 +41,7 @@ download: erase: flash: flash_sparse:: +flash_image:: boot continue reboot @@ -60,9 +61,21 @@ Example: snagflash -P fastboot -p 0483:0afb -f download:boot.img -f flash:0:1 -f boot ``` -The ``flash_sparse`` command will download and flash a android sparse file with -fastboot protocol. For details about the file format, see the [sparse file format -partial documentation](developers/android-sparse-file.md). +The ``flash_image`` command downloads and flashes an image file to a partition, +transparently handling both raw binary files and android sparse files. It reads +the ``max-download-size`` Fastboot variable from U-Boot: if the file fits within +this size, it is downloaded and flashed directly with no splitting. If the file +is bigger, it is automatically split into several android sparse fragments +(each bounded by ``max-download-size``, headers included) which are downloaded +and flashed one after another. This works for both raw binary files (which are +converted to an equivalent single-region sparse image on the fly, with no +change to the source file) and existing android sparse files. For details +about the sparse file format, see the [sparse file format partial +documentation](developers/android-sparse-file.md). + +The ``flash_sparse`` command is a backward-compatible alias for +``flash_image``, kept for users relying on the historical command name. It +behaves identically. For more information on Fastboot commands, see the [fastboot specification](https://android.googlesource.com/platform/system/core/+/refs/heads/master/fastboot/README.md) diff --git a/src/snagfactory/config.py b/src/snagfactory/config.py index 70d0d14..353cb85 100644 --- a/src/snagfactory/config.py +++ b/src/snagfactory/config.py @@ -161,6 +161,7 @@ def str_rule(pattern: str): "fb-buffer-size": int_rule, "fb-buffer-addr": int_rule, "eraseblk-size": int_rule, + "usb-wait-timeout": int_rule, } tasks_rule = { diff --git a/src/snagfactory/fastboot.py b/src/snagfactory/fastboot.py index 8d38e03..5f929b9 100644 --- a/src/snagfactory/fastboot.py +++ b/src/snagfactory/fastboot.py @@ -70,6 +70,7 @@ def __init__(self, config: dict, num: int, globals: dict): fb_addr = self.require_global("fb-buffer-addr") fb_size = self.get_global("fb-buffer-size") self.target_device = self.require_global("target-device") + self.usb_wait_timeout = self.get_global("usb-wait-timeout") self.cmds = [ f"set fb-addr {fb_addr}", @@ -79,7 +80,10 @@ def __init__(self, config: dict, num: int, globals: dict): if fb_size is not None: self.cmds.append(f"set fb-size {fb_size}") - if not self.target_device.startswith("mmc"): + is_mmc_target = self.target_device.startswith("mmc") + is_ufs_target = self.target_device.startswith("ufs") + + if not is_mmc_target and not is_ufs_target: self.eraseblk_size = self.require_global("eraseblk-size") self.cmds.append(f"set eraseblk-size {self.eraseblk_size}") @@ -125,6 +129,7 @@ def attach(self, board): "port": self.port, "fastboot_cmd": [], "interactive_cmds": self.cmds, + "usb_wait_timeout": self.usb_wait_timeout, } self.args = FastbootArgs(args) @@ -157,7 +162,10 @@ class FastbootMTDTask(FastbootTask): def __init__(self, config: dict, num: int, globals: dict): super().__init__(config, num, globals) - if self.target_device.startswith("mmc"): + is_mmc_target = self.target_device.startswith("mmc") + is_ufs_target = self.target_device.startswith("ufs") + + if is_mmc_target or is_ufs_target: raise SnagFactoryConfigError( f"the '{self.name}' task is only supported on mtd backends" ) @@ -202,7 +210,8 @@ def __init__(self, config: dict, num: int, globals: dict): if not self.target_device.startswith("mmc"): raise SnagFactoryConfigError( - f"the '{self.name}' task is only supported on mmc backends" + f"the '{self.name}' task is only supported on mmc backends. " + f"For 'ufs' backends, use the 'run' task with raw commands." ) self.device_num = int(self.target_device[-1]) @@ -239,6 +248,15 @@ def get_cmds(self): class FastbootTaskFlash(FastbootTask): + def __init__(self, config: dict, num: int, globals: dict): + super().__init__(config, num, globals) + + if self.target_device.startswith("ufs"): + raise SnagFactoryConfigError( + f"the '{self.name}' task is not supported on 'ufs' backends. " + f"Use the 'run' task instead." + ) + def get_cmds(self): for entry in self.config: part = entry.get("part", "") diff --git a/src/snagflash/android_sparse_file/utils.py b/src/snagflash/android_sparse_file/utils.py index 2710e24..c3f33d0 100644 --- a/src/snagflash/android_sparse_file/utils.py +++ b/src/snagflash/android_sparse_file/utils.py @@ -23,12 +23,15 @@ # Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA. import logging +import os from snagflash.android_sparse_file.sparse import ( AndroidSparseFile, AndroidChunkHeader, SPARSE_CHUNKHEADER_LEN, SPARSE_FILEHEADER_LEN, + DEFAULT_BLOCK_SIZE, + MAGIC, CHUNK_TYPE_DONTCARE, CHUNK_TYPE_RAW, CHUNK_TYPE_FILL, @@ -56,6 +59,46 @@ def chunk_type_name(chunk_type): return CHUNK_TYPE_NAMES.get(chunk_type, f"UNKNOWN(0x{chunk_type:04X})") +def is_sparse_file(path): + """ + Check whether the file at 'path' is a valid Android sparse image, by + inspecting its magic cookie (first 4 bytes, little-endian). + """ + with open(path, "rb") as f: + magic_bytes = f.read(4) + if len(magic_bytes) < 4: + return False + magic = int.from_bytes(magic_bytes, byteorder="little") + return magic == MAGIC + + +class ZeroPaddedReader: + """ + Wraps a raw binary file object so that read(n) always returns exactly n + bytes, zero-padding the final short read at true EOF. + + This lets a raw (non-sparse) file be fed directly into + process_raw_chunk(), which normally expects to read exact, + block-aligned payloads out of a real Android sparse file. Raw files are + usually not a multiple of block_size, so the final block must be + zero-padded up to the block boundary. Once EOF has been reached, any + further reads return all zero bytes, as a safety net. + """ + + def __init__(self, fd): + self.fd = fd + self.eof = False + + def read(self, n): + if self.eof: + return b"\x00" * n + data = self.fd.read(n) + if len(data) < n: + self.eof = True + data += b"\x00" * (n - len(data)) + return data + + class SplitFragmentState: """ Holds the accumulated state for the sparse fragment currently being built @@ -446,3 +489,84 @@ def split_streaming(path, dest, bufsize): finally: sparse_file.close() + + +def split_streaming_raw(path, dest, bufsize, block_size=DEFAULT_BLOCK_SIZE): + """ + Generator that yields one split sparse file at a time, built from a raw + (non-sparse) binary image, for immediate processing. + + The raw file is treated as a single logical CHUNK_TYPE_RAW region + spanning the whole image. This synthetic chunk is split into + bufsize-limited sparse fragments by reusing the exact same + process_raw_chunk()/flush_fragment() machinery used by split_streaming() + for real sparse files, so the resulting fragments have the same + guarantees: + + - Every fragment's sparse header reports the full logical block count + of the entire raw image (not just the blocks it contains). + - Every fragment starts with a DONT_CARE prefix covering blocks already + flushed in earlier fragments, and ends with a DONT_CARE suffix + covering blocks that will be flushed in later fragments. + - Each fragment (header + prefix + RAW payload + suffix) never exceeds + bufsize bytes. + + If the raw file size is not a multiple of block_size, the final block + is transparently zero-padded up to the block boundary (the padding only + exists in the generated sparse payload, the source file is untouched). + + Args: + path: Path to input raw binary file + dest: Path for temporary output file (will be reused for each split) + bufsize: Maximum size for each output file + block_size: Sparse image block size in bytes (default: DEFAULT_BLOCK_SIZE) + + Yields: + Path to each split file (same path, but content changes each iteration) + """ + file_size = os.path.getsize(path) + total_blks = (file_size + block_size - 1) // block_size # ceil division + + # Pre-flight validation: ensure bufsize can hold at least one block with all headers + min_required = ( + SPARSE_FILEHEADER_LEN # 28 bytes: file header + + 2 * SPARSE_CHUNKHEADER_LEN # 24 bytes: prefix + one data chunk header + + block_size # At least one block of data + + SPARSE_CHUNKHEADER_LEN # 12 bytes: suffix reserve + ) + if bufsize <= min_required: + raise IOError( + f"Buffer size {bufsize} too small. Need at least {min_required} bytes " + f"to fit one {block_size}-byte block with headers" + ) + + blocks_done = 0 + state = SplitFragmentState() + + logger.debug( + f"Starting streaming raw split: total_blocks={total_blks}, block_size={block_size}" + ) + + # Synthesize a single RAW chunk header covering the entire raw image + header = AndroidChunkHeader() + header.type = CHUNK_TYPE_RAW + header.size = total_blks + + with open(path, "rb") as raw_fd: + input_fd = ZeroPaddedReader(raw_fd) + + raw_gen = process_raw_chunk( + input_fd, header, state, blocks_done, bufsize, block_size, dest, total_blks + ) + while True: + try: + flushed_fragment = next(raw_gen) + yield flushed_fragment + except StopIteration as stop: + blocks_done = stop.value + break + + # Final flush: emit any remaining staged chunks as the last fragment + frag = flush_fragment(state, dest, block_size, total_blks) + if frag: + yield frag diff --git a/src/snagflash/fastboot.py b/src/snagflash/fastboot.py index 312fca7..06c99e7 100644 --- a/src/snagflash/fastboot.py +++ b/src/snagflash/fastboot.py @@ -27,9 +27,12 @@ cli_error, access_error, prettify_usb_addr, + USB_RETRIES, + USB_INTERVAL, ) import sys import logging +from math import ceil logger = logging.getLogger("snagflash") @@ -79,7 +82,13 @@ def fastboot(args): if usb_addr is None: access_error("USB Fastboot", args.port) - dev = get_usb(usb_addr, ready_check=fastboot_ready_check) + usb_wait_timeout = getattr(args, "usb_wait_timeout", None) + if usb_wait_timeout is None: + retries = USB_RETRIES + else: + retries = max(0, ceil(usb_wait_timeout / USB_INTERVAL) - 1) + + dev = get_usb(usb_addr, retries=retries, ready_check=fastboot_ready_check) dev.default_timeout = int(args.timeout) fast = fb.Fastboot(dev, timeout=dev.default_timeout) @@ -114,6 +123,7 @@ def fastboot(args): "oem_bootbus", "reset", "flash_sparse", + "flash_image", } for cmd in args.fastboot_cmd: diff --git a/src/snagrecover/protocols/fastboot.py b/src/snagrecover/protocols/fastboot.py index 3e1e537..b67c62d 100644 --- a/src/snagrecover/protocols/fastboot.py +++ b/src/snagrecover/protocols/fastboot.py @@ -24,7 +24,11 @@ from typing import Optional, Union from snagrecover import utils -from snagflash.android_sparse_file.utils import split_streaming +from snagflash.android_sparse_file.utils import ( + split_streaming, + split_streaming_raw, + is_sparse_file, +) import logging @@ -239,11 +243,10 @@ def reset(self): self.dev.write(self.ep_out, packet, timeout=self.timeout) - def flash_sparse(self, args: str): + def get_max_download_size(self) -> int: """ - Download and flash an android sparse file. - If the file is too big, it's splitting into - smaller android sparse files. + Read the "max-download-size" Fastboot variable from U-Boot. + Raises a FastbootError if it cannot be read, or if it is 0. """ try: maxsize = int(self.getvar("max-download-size"), 0) @@ -253,51 +256,33 @@ def flash_sparse(self, args: str): ) from e if maxsize == 0: raise FastbootError("Fastboot variable max-download-size is 0") - arg_list = args.split(":", 1) - cnt = len(arg_list) - if cnt != 2: - raise FastbootError( - f"Wrong arguments count {cnt}, expected 2. Given {args}" - ) - fname = arg_list[0] - if not os.path.exists(fname): - raise FastbootError(f"File {fname} does not exist") - - # Verify the file is a valid Android sparse file by checking magic cookie - try: - with open(fname, "rb") as f: - magic_bytes = f.read(4) - if len(magic_bytes) < 4: - raise FastbootError( - f"File {fname} is too small to be a valid sparse file" - ) - magic = int.from_bytes(magic_bytes, byteorder="little") - if magic != 0xED26FF3A: - raise FastbootError( - f"File {fname} is not a valid Android sparse file. " - f"Expected magic 0xED26FF3A, got 0x{magic:08X}" - ) - logger.info(f"Verified {fname} is a valid Android sparse file") - except IOError as e: - raise FastbootError(f"Failed to read file {fname}: {e}") from e + return maxsize - part = arg_list[1] - - # Use streaming approach to minimize memory usage - # Each split file is created, downloaded, flashed, and then reused for the next split - # This allows processing of arbitrarily large sparse files with constant memory usage + def split_and_flash(self, splitter, fname: str, part: str, maxsize: int): + """ + Split 'fname' into a series of android sparse fragments (each no + bigger than 'maxsize') using the given 'splitter' generator + function, then download and flash each fragment in turn to 'part'. + + 'splitter' is expected to have the same signature/semantics as + split_streaming()/split_streaming_raw(): splitter(path, dest, bufsize) + is a generator yielding the path to each fragment file (dest, reused + and overwritten for every fragment). + + Each split file is created, downloaded, flashed, and then reused for + the next split. This allows processing of arbitrarily large images + with constant memory usage. + """ with tempfile.TemporaryDirectory() as tmp: - temppath = os.path.join(tmp, "sparse.img") + temppath = os.path.join(tmp, "split.img") try: # Count total splits upfront for "split X/N" logging below - total_splits = sum(1 for _ in split_streaming(fname, temppath, maxsize)) + total_splits = sum(1 for _ in splitter(fname, temppath, maxsize)) split_count = 0 - logger.info( - f"Starting streaming sparse file flash ({total_splits} split(s))..." - ) + logger.info(f"Starting streaming flash ({total_splits} split(s))...") - for split_file in split_streaming(fname, temppath, maxsize): + for split_file in splitter(fname, temppath, maxsize): split_count += 1 logger.info( f"Processing split {split_count}/{total_splits}: Downloading {split_file}" @@ -327,4 +312,73 @@ def flash_sparse(self, args: str): f"Successfully flashed {split_count}/{total_splits} split file(s) to {part}" ) except Exception as e: - raise FastbootError(f"Streaming sparse flash failed: {e}") from e + raise FastbootError(f"Streaming flash failed: {e}") from e + + def flash_image(self, args: str): + """ + Download and flash an image file (raw binary or android sparse) to + a partition, handling both formats transparently: + + 1. Reads "max-download-size" from U-Boot (fails if unavailable or 0). + 2. If the file fits within max-download-size, downloads and flashes + it directly, with no splitting required. + 3. If the file is bigger than max-download-size: + - If it's an Android sparse file, it's split into smaller sparse + fragments. + - If it's a raw binary file, it's split by synthesizing a single + logical RAW region covering the whole image, then splitting + that into smaller sparse fragments the same way. This gives + raw files automatic splitting support that they otherwise + don't have with a plain download()+flash(). + + Each emitted fragment (sparse or raw-derived) is downloaded and + flashed in turn, with progress logged as "split X/N". + """ + arg_list = args.split(":", 1) + cnt = len(arg_list) + if cnt != 2: + raise FastbootError( + f"Wrong arguments count {cnt}, expected 2. Given {args}" + ) + fname = arg_list[0] + part = arg_list[1] + + if not os.path.exists(fname): + raise FastbootError(f"File {fname} does not exist") + + maxsize = self.get_max_download_size() + + filesize = os.path.getsize(fname) + + if filesize <= maxsize: + logger.info( + f"File {fname} ({filesize} bytes) fits within " + f"max-download-size (0x{maxsize:x}), flashing directly" + ) + self.download(fname) + self.flash(part) + return + + if is_sparse_file(fname): + logger.info( + f"File {fname} is an Android sparse file, splitting to fit " + f"max-download-size (0x{maxsize:x})" + ) + self.split_and_flash(split_streaming, fname, part, maxsize) + else: + logger.info( + f"File {fname} is a raw binary file, splitting into sparse " + f"fragments to fit max-download-size (0x{maxsize:x})" + ) + self.split_and_flash(split_streaming_raw, fname, part, maxsize) + + def flash_sparse(self, args: str): + """ + Download and flash an android sparse file. + + This is a backward-compatible alias for flash_image(), kept for + users relying on the historical "flash_sparse" command name. + flash_image() already handles both android sparse files and raw + binary files transparently, so this simply forwards to it. + """ + self.flash_image(args)