Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
47 changes: 47 additions & 0 deletions docs/developers/android-sparse-file.md
Original file line number Diff line number Diff line change
Expand Up @@ -44,3 +44,50 @@ without always writing at the beginning of the partition.

The solution is to use the ``CHUNK_TYPE_DONTCARE``, which will skip some blocks of the output
partition in the beginning of the 2+ images.

Each split fragment is a fully valid, self-contained sparse file:

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This paragraph doesn't seem related to the commit?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

You're right that this paragraph isn't strictly about the change introduced in this commit (automatic raw-to-sparse conversion). It documents a pre-existing property of the sparse-splitting logic (the prefix/suffix CHUNK_TYPE_DONTCARE padding and total_blks behavior) that was already implemented before this patch, but wasn't previously documented.

I added it here because the new "Splitting raw (non-sparse) files" section builds directly on top of this fragment-splitting behavior (it reuses the same prefix/suffix DONTCARE padding and size-accounting logic), so having it spelled out just above gives readers the full picture of how a fragment is structured before explaining how raw files reuse that same mechanism.

we can:

  • keep it as is (since it fills a documentation gap directly relevant to understanding the new raw-splitting section), or
  • move it into a separate follow-up commit/PR that documents the existing sparse-splitting behavior on its own, keeping this commit focused strictly on the raw-image conversion feature.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm okay with keeping this in the current commit, however please add a small paragraph in the commit log to describe this change.

@BakyarajM BakyarajM Oct 15, 2026 •

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Addressed. Modified the commit message.


- its ``total_blks`` field in the sparse header always reports the full block
count of the *original*, unsplit image, not just the blocks contained in
that particular fragment;
- it is prefixed with a ``CHUNK_TYPE_DONTCARE`` chunk covering every block
already written by previous fragments;
- it is suffixed with a ``CHUNK_TYPE_DONTCARE`` chunk covering every block that
will be written by later fragments;
- the total serialized size of the fragment (file header + all chunk headers +
payloads) never exceeds ``max-download-size``.

## Splitting raw (non-sparse) files

Raw binary files can also be split into a sequence of ``max-download-size``-bounded
sparse fragments, using the exact same splitting algorithm described above.

The raw file is treated as a single logical ``CHUNK_TYPE_RAW`` region spanning
the whole image:

```text
total_blks = ceil(file_size / block_size)
```

where ``block_size`` defaults to 4096 bytes. This synthetic RAW region is fed
into the same fragment-splitting logic used for real sparse files (prefix/suffix
``CHUNK_TYPE_DONTCARE`` padding, header/overhead accounting bounded by
``max-download-size``), so raw files gain automatic splitting support without
needing any format conversion of the file on disk.

If the raw file size is not a multiple of ``block_size``, the final block is
transparently zero-padded up to the block boundary in the generated sparse
payload only; the source file itself is never modified.

This is implemented by ``snagrecover.protocols.fastboot.Fastboot.flash_image()``,
which:

1. Reads ``max-download-size`` from U-Boot (raises an error if it can't be read,
or if it is 0).
2. If the file's size fits within ``max-download-size``, downloads and flashes
it directly with no splitting.
3. Otherwise, detects whether the file is an Android sparse file (by checking
its magic cookie) or a raw binary file, and splits it accordingly:
sparse files are split with the existing chunk-parsing logic, raw files are
split via the synthetic single-RAW-region approach described above.
4. Downloads and flashes each fragment in turn.
21 changes: 20 additions & 1 deletion docs/snagfactory.md
Original file line number Diff line number Diff line change
Expand Up @@ -90,10 +90,14 @@ configuration variables, that are common to all tasks. It must contain the
following values:

```yaml
target-device: The device configured as the Fastboot flashing backend in U-Boot. Either 'mmc<num>' or 'nand'.
target-device: The device configured as the Fastboot flashing backend in U-Boot. Either 'mmc<num>', 'ufs<num>' or 'nand'.
fb-buffer-addr: The size in bytes of the Fastboot buffer.
eraseblk-size: The size in bytes of an erase block for MTD targets
fb-buffer-size: (optional) The size in bytes of the Fastboot buffer. This can only be used to reduce the default U-Boot buffer size.
usb-wait-timeout: (optional) The number of seconds to wait for the board's USB Fastboot
device to (re-)enumerate before giving up, for example after U-Boot boots following
snagrecover. This is useful for boards which take longer than the default (~9 seconds)
to boot up to the U-Boot Fastboot prompt.
```

These variables may be updated during the factory flashing process, e.g.:
Expand Down Expand Up @@ -311,6 +315,21 @@ Example:
skip-pwr-cycle: False
```

#### UFS backends

`target-device: ufs<num>` is supported for boards whose storage is UFS rather
than eMMC/MTD. With a `ufs<num>` target-device:

- `eraseblk-size` is **not** required (same as for `mmc<num>`), since UFS is
LBA-addressable like eMMC.
- The **`gpt`**, **`flash`**, **`mtd-parts`** and **`emmc-hwpart`** tasks are
**not supported currently** for `ufs<num>` targets and will raise a clear
configuration error if used. This is because U-Boot addresses UFS storage
through the `scsi` command family rather than the `mmc`-specific commands
these tasks generate.
- Instead, use the **`run`** task to issue the equivalent raw commands
directly


## libusb-win32

Expand Down
19 changes: 16 additions & 3 deletions docs/snagflash.md
Original file line number Diff line number Diff line change
Expand Up @@ -41,6 +41,7 @@ download:<filepath>
erase:<part>
flash:<part>
flash_sparse:<sparsefilepath>:<partition>
flash_image:<filepath>:<partition>
Comment thread
rgantois marked this conversation as resolved.
Resolved
boot
continue
reboot
Expand All @@ -60,9 +61,21 @@ Example:
snagflash -P fastboot -p 0483:0afb -f download:boot.img -f flash:0:1 -f boot
```

The ``flash_sparse`` command will download and flash a android sparse file with
fastboot protocol. For details about the file format, see the [sparse file format
partial documentation](developers/android-sparse-file.md).
The ``flash_image`` command downloads and flashes an image file to a partition,
transparently handling both raw binary files and android sparse files. It reads
the ``max-download-size`` Fastboot variable from U-Boot: if the file fits within
this size, it is downloaded and flashed directly with no splitting. If the file
is bigger, it is automatically split into several android sparse fragments
(each bounded by ``max-download-size``, headers included) which are downloaded
and flashed one after another. This works for both raw binary files (which are
converted to an equivalent single-region sparse image on the fly, with no
change to the source file) and existing android sparse files. For details
about the sparse file format, see the [sparse file format partial
documentation](developers/android-sparse-file.md).

The ``flash_sparse`` command is a backward-compatible alias for
``flash_image``, kept for users relying on the historical command name. It
behaves identically.

For more information on Fastboot commands, see the [fastboot
specification](https://android.googlesource.com/platform/system/core/+/refs/heads/master/fastboot/README.md)
Expand Down
1 change: 1 addition & 0 deletions src/snagfactory/config.py
Original file line number Diff line number Diff line change
Expand Up @@ -161,6 +161,7 @@ def str_rule(pattern: str):
"fb-buffer-size": int_rule,
"fb-buffer-addr": int_rule,
"eraseblk-size": int_rule,
"usb-wait-timeout": int_rule,
}

tasks_rule = {
Expand Down
24 changes: 21 additions & 3 deletions src/snagfactory/fastboot.py
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,7 @@ def __init__(self, config: dict, num: int, globals: dict):
fb_addr = self.require_global("fb-buffer-addr")
fb_size = self.get_global("fb-buffer-size")
self.target_device = self.require_global("target-device")
self.usb_wait_timeout = self.get_global("usb-wait-timeout")

self.cmds = [
f"set fb-addr {fb_addr}",
Expand All @@ -79,7 +80,10 @@ def __init__(self, config: dict, num: int, globals: dict):
if fb_size is not None:
self.cmds.append(f"set fb-size {fb_size}")

if not self.target_device.startswith("mmc"):
is_mmc_target = self.target_device.startswith("mmc")
is_ufs_target = self.target_device.startswith("ufs")

if not is_mmc_target and not is_ufs_target:
self.eraseblk_size = self.require_global("eraseblk-size")
self.cmds.append(f"set eraseblk-size {self.eraseblk_size}")

Expand Down Expand Up @@ -125,6 +129,7 @@ def attach(self, board):
"port": self.port,
"fastboot_cmd": [],
"interactive_cmds": self.cmds,
"usb_wait_timeout": self.usb_wait_timeout,
}

self.args = FastbootArgs(args)
Expand Down Expand Up @@ -157,7 +162,10 @@ class FastbootMTDTask(FastbootTask):
def __init__(self, config: dict, num: int, globals: dict):
super().__init__(config, num, globals)

if self.target_device.startswith("mmc"):
is_mmc_target = self.target_device.startswith("mmc")
is_ufs_target = self.target_device.startswith("ufs")

if is_mmc_target or is_ufs_target:
raise SnagFactoryConfigError(
f"the '{self.name}' task is only supported on mtd backends"
)
Expand Down Expand Up @@ -202,7 +210,8 @@ def __init__(self, config: dict, num: int, globals: dict):

if not self.target_device.startswith("mmc"):
raise SnagFactoryConfigError(
f"the '{self.name}' task is only supported on mmc backends"
f"the '{self.name}' task is only supported on mmc backends. "
f"For 'ufs' backends, use the 'run' task with raw commands."
)

self.device_num = int(self.target_device[-1])
Expand Down Expand Up @@ -239,6 +248,15 @@ def get_cmds(self):


class FastbootTaskFlash(FastbootTask):
def __init__(self, config: dict, num: int, globals: dict):
super().__init__(config, num, globals)

if self.target_device.startswith("ufs"):
raise SnagFactoryConfigError(
f"the '{self.name}' task is not supported on 'ufs' backends. "
f"Use the 'run' task instead."
)

def get_cmds(self):
for entry in self.config:
part = entry.get("part", "")
Expand Down
124 changes: 124 additions & 0 deletions src/snagflash/android_sparse_file/utils.py
Original file line number Diff line number Diff line change
Expand Up @@ -23,12 +23,15 @@
# Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301, USA.

import logging
import os

from snagflash.android_sparse_file.sparse import (
AndroidSparseFile,
AndroidChunkHeader,
SPARSE_CHUNKHEADER_LEN,
SPARSE_FILEHEADER_LEN,
DEFAULT_BLOCK_SIZE,
MAGIC,
CHUNK_TYPE_DONTCARE,
CHUNK_TYPE_RAW,
CHUNK_TYPE_FILL,
Expand Down Expand Up @@ -56,6 +59,46 @@ def chunk_type_name(chunk_type):
return CHUNK_TYPE_NAMES.get(chunk_type, f"UNKNOWN(0x{chunk_type:04X})")


def is_sparse_file(path):
"""
Check whether the file at 'path' is a valid Android sparse image, by
inspecting its magic cookie (first 4 bytes, little-endian).
"""
with open(path, "rb") as f:
magic_bytes = f.read(4)
if len(magic_bytes) < 4:
return False
magic = int.from_bytes(magic_bytes, byteorder="little")
return magic == MAGIC


class ZeroPaddedReader:
"""
Wraps a raw binary file object so that read(n) always returns exactly n
bytes, zero-padding the final short read at true EOF.

This lets a raw (non-sparse) file be fed directly into
process_raw_chunk(), which normally expects to read exact,
block-aligned payloads out of a real Android sparse file. Raw files are
usually not a multiple of block_size, so the final block must be
zero-padded up to the block boundary. Once EOF has been reached, any
further reads return all zero bytes, as a safety net.
"""

def __init__(self, fd):
self.fd = fd
self.eof = False

def read(self, n):
if self.eof:
return b"\x00" * n
data = self.fd.read(n)
if len(data) < n:
self.eof = True
data += b"\x00" * (n - len(data))
return data


class SplitFragmentState:
"""
Holds the accumulated state for the sparse fragment currently being built
Expand Down Expand Up @@ -446,3 +489,84 @@ def split_streaming(path, dest, bufsize):

finally:
sparse_file.close()


def split_streaming_raw(path, dest, bufsize, block_size=DEFAULT_BLOCK_SIZE):
"""
Generator that yields one split sparse file at a time, built from a raw
(non-sparse) binary image, for immediate processing.

The raw file is treated as a single logical CHUNK_TYPE_RAW region
spanning the whole image. This synthetic chunk is split into
bufsize-limited sparse fragments by reusing the exact same
process_raw_chunk()/flush_fragment() machinery used by split_streaming()
for real sparse files, so the resulting fragments have the same
guarantees:

- Every fragment's sparse header reports the full logical block count
of the entire raw image (not just the blocks it contains).
- Every fragment starts with a DONT_CARE prefix covering blocks already
flushed in earlier fragments, and ends with a DONT_CARE suffix
covering blocks that will be flushed in later fragments.
- Each fragment (header + prefix + RAW payload + suffix) never exceeds
bufsize bytes.

If the raw file size is not a multiple of block_size, the final block
is transparently zero-padded up to the block boundary (the padding only
exists in the generated sparse payload, the source file is untouched).

Args:
path: Path to input raw binary file
dest: Path for temporary output file (will be reused for each split)
bufsize: Maximum size for each output file
block_size: Sparse image block size in bytes (default: DEFAULT_BLOCK_SIZE)

Yields:
Path to each split file (same path, but content changes each iteration)
"""
file_size = os.path.getsize(path)
total_blks = (file_size + block_size - 1) // block_size # ceil division

# Pre-flight validation: ensure bufsize can hold at least one block with all headers
min_required = (
SPARSE_FILEHEADER_LEN # 28 bytes: file header
+ 2 * SPARSE_CHUNKHEADER_LEN # 24 bytes: prefix + one data chunk header
+ block_size # At least one block of data
+ SPARSE_CHUNKHEADER_LEN # 12 bytes: suffix reserve
)
if bufsize <= min_required:
raise IOError(
f"Buffer size {bufsize} too small. Need at least {min_required} bytes "
f"to fit one {block_size}-byte block with headers"
)

blocks_done = 0
state = SplitFragmentState()

logger.debug(
f"Starting streaming raw split: total_blocks={total_blks}, block_size={block_size}"
)

# Synthesize a single RAW chunk header covering the entire raw image
header = AndroidChunkHeader()
header.type = CHUNK_TYPE_RAW
header.size = total_blks

with open(path, "rb") as raw_fd:
input_fd = ZeroPaddedReader(raw_fd)

raw_gen = process_raw_chunk(
input_fd, header, state, blocks_done, bufsize, block_size, dest, total_blks
)
while True:
try:
flushed_fragment = next(raw_gen)
yield flushed_fragment
except StopIteration as stop:
blocks_done = stop.value
break

# Final flush: emit any remaining staged chunks as the last fragment
frag = flush_fragment(state, dest, block_size, total_blks)
if frag:
yield frag
12 changes: 11 additions & 1 deletion src/snagflash/fastboot.py
Original file line number Diff line number Diff line change
Expand Up @@ -27,9 +27,12 @@
cli_error,
access_error,
prettify_usb_addr,
USB_RETRIES,
USB_INTERVAL,
)
import sys
import logging
from math import ceil

logger = logging.getLogger("snagflash")

Expand Down Expand Up @@ -79,7 +82,13 @@ def fastboot(args):
if usb_addr is None:
access_error("USB Fastboot", args.port)

dev = get_usb(usb_addr, ready_check=fastboot_ready_check)
usb_wait_timeout = getattr(args, "usb_wait_timeout", None)
if usb_wait_timeout is None:
retries = USB_RETRIES
else:
retries = max(0, ceil(usb_wait_timeout / USB_INTERVAL) - 1)

dev = get_usb(usb_addr, retries=retries, ready_check=fastboot_ready_check)
dev.default_timeout = int(args.timeout)

fast = fb.Fastboot(dev, timeout=dev.default_timeout)
Expand Down Expand Up @@ -114,6 +123,7 @@ def fastboot(args):
"oem_bootbus",
"reset",
"flash_sparse",
"flash_image",
}

for cmd in args.fastboot_cmd:
Expand Down
Loading
Loading