Layered/incremental snapshots (#1926)

## Summary

Snapshots share immutable memory layers. A capture copies only the pages
changed since its parent snapshot, and `Sandbox::snapshot` switches the
sandbox to the capture so the next one is incremental too. This makes
`snapshot()` much faster for larger sandboxes. It also improves memory
density: each snapshot holds only its changed pages and shares the rest
with its parent, where `main` holds a full copy of guest memory per
snapshot. See [HIP
0003](proposals/0003-incremental-snapshots/README.md).

Deviations from the merged HIP 0003 draft (the HIP is updated):

* The VM maps each layer whole, one mapping per layer. A layer stays in
a snapshot while it gives at least one page. Its pages that a newer
layer replaced stay mapped, so a guest can read data it overwrote before
a snapshot from the older layers of that snapshot.
* Each blob holds guest data followed by the page tables of the snapshot
that created it, as on `main`. A restore copies the newest page tables
into scratch.
* `Sandbox::snapshot` installs the capture, because otherwise every
later capture copies all pages changed since the last restore again.
Installing removes `map_region` and `map_file_cow` regions, because the
snapshot holds their mapped pages and later blobs may reuse their
addresses.
* Snapshots from earlier versions are rejected instead of loaded as one
layer. This is simpler, and there is little reason to keep backwards
compatibility here.

## Changes

* Snapshot memory is a list of immutable shared blobs, one per layer,
each mapped whole.
* Capture keeps parent layers that still give a page and copies changed
pages into one new blob.
* `Sandbox::snapshot` installs the capture into the running sandbox.
Scratch pages it frees are zeroed.
* The OCI format stores ordered data layers and a transport layer
(config v4, ABI 7). The config names the layer whose page tables a
restore uses.
* Snapshot performance: adjacent page-table walk mappings are merged,
and page-table pages are cached during capture.
* KVM: `KVM_X86_QUIRK_SLOT_ZAP_ALL` is disabled where supported, so
deleting a memory slot flushes only that slot.
* New benchmarks `snapshots/call_and_create` and
`snapshots/restore_alternate` cover the cost moved into the next call
and restores to a non-current snapshot.

## Breaking changes

* `Sandbox::snapshot` removes `map_region` and `map_file_cow` regions
from the sandbox, which only `restore` did before. Region pages the
guest has not mapped become unreachable at the snapshot instead of at
the next restore. Snapshots hold the same pages as before.
* `PtRootFinder` receives a `SnapshotMemoryReader` and returns
`Result<Vec<u64>>`.
* A snapshot holds at most 30 memory layers, and a capture fails when it
would exceed this. This bounds restore cost by limiting the number of
hypercalls to map memory.

## Commits

1. Add `call_and_create` and `restore_alternate` snapshot benchmarks.
Lands first so the benchmarks run before and after.
2. Implement layered and incremental snapshots.
3. Install captured snapshots into the sandbox.
4. vmem: merge adjacent mappings in `walk_va_spaces`.
5. Cache page-table pages in `PageTableReader`.
6. Disable `KVM_X86_QUIRK_SLOT_ZAP_ALL`.
7. PR feedback.

## Benchmarks

Compared to `main`. `restore`, `restore_alternate` and `call_and_create`
compare to the first commit, which adds the last two.

* MSHV: Linux 6.6 mshv.
* KVM: Linux 6.17.

### Highlights

| Benchmark | MSHV | KVM |
|---|---|---|
| `snapshots/create/default` | 440 µs → 249 µs (−43%) | 380 µs → 262 µs
(−31%) |
| `snapshots/create/large` | 171 ms → 1.68 ms (−99%) | 162 ms → 1.7 ms
(−99%) |
| `snapshots/call_and_create/medium` | 38.1 ms → 726 µs (−98%) | 35 ms →
778 µs (−98%) |
| `snapshots/restore_alternate/large` | 4.41 ms → 106 µs (−98%) | 1.66
ms → 131 µs (−92%) |
| `snapshots/restore/large` | 1.2 ms → 91.6 µs (−92%) | 403 µs → 52 µs
(−87%) |

Guest calls show no consistent change.

### Regressions

| Benchmark | MSHV | KVM | Cause |
|---|---|---|---|
| `snapshot_files/load_snapshot_unverified/default` | 88.5 µs → 112 µs
(**+27%**) | 93.3 µs → 123 µs (**+31%**) | Opens and maps 2 data blobs
instead of 1 |
| `snapshot_files/load_snapshot/default` | 617 µs → 701 µs (**+14%**) |
613 µs → 701 µs (**+14%**) | Also hashes 88 KiB more: base layer pages
that the newer layer replaced, and the base layer's page tables |
| `snapshot_files/save_snapshot/default` | 853 µs → 941 µs (**+10%**) |
35.5 ms → 43.4 ms (**+22%**) | Writes, hashes and fsyncs 2 data blobs
and 88 KiB more. fsync is slow on the KVM host |
| `snapshot_files/cold_start_via_snapshot/default` | 1.8 ms → 1.97 ms
(**+9%**) | 2.59 ms → 2.85 ms (**+10%**) | Includes `load_snapshot` |
| `sandboxes/sandbox_from_snapshot/small` | 823 µs → 812 µs (~) | 877 µs
→ 978 µs (**+12%**) | Maps 2 layers instead of 1. Each costs a memory
slot on KVM and a map call on MSHV. MSHV shows it only at `default`
(+11%) |

Kernels before 6.12 cannot disable `KVM_X86_QUIRK_SLOT_ZAP_ALL`, so each
memory slot change flushes all guest mappings. WHP and aarch64 were not
benchmarked.

A saved snapshot stores each layer in full. Base layer pages that a
newer layer replaced stay in the file, which adds 88 KiB for `default`.

Closes #1956
Closes #1551

### Full results

<details>
<summary>All benchmarks on both hosts (click to expand)</summary>

Before → after (change). Negative is faster. **Bold** is a regression. ~
is no consistent change. A change needs agreement between a 0.5 s and a
3 s run, or a 3 s run alone.

#### snapshots

| Benchmark | Size | MSHV | KVM |
|---|---|---|---|
| `call_and_create` | default | 556 µs → 372 µs (−33%) | 471 µs → 409 µs
(−13%) |
| `call_and_create` | small | 2.67 ms → 416 µs (−84%) | 2.65 ms → 466 µs
(−82%) |
| `call_and_create` | medium | 38.1 ms → 726 µs (−98%) | 35 ms → 778 µs
(−98%) |
| `call_and_create` | large | 172 ms → 1.78 ms (−99%) | 161 ms → 1.85 ms
(−99%) |
| `create` | default | 440 µs → 249 µs (−43%) | 380 µs → 262 µs (−31%) |
| `create` | small | 2.57 ms → 291 µs (−89%) | 2.47 ms → 324 µs (−87%) |
| `create` | medium | 41.4 ms → 606 µs (−99%) | 36.9 ms → 606 µs (−98%)
|
| `create` | large | 171 ms → 1.68 ms (−99%) | 162 ms → 1.7 ms (−99%) |
| `restore` | default | 64.9 µs → 65.7 µs (~) | 20.6 µs → 21 µs (~) |
| `restore` | small | 66.3 µs → 67 µs (~) | 21.4 µs → 21.5 µs (~) |
| `restore` | medium | 74.5 µs → 71.1 µs (−5%) | 26.4 µs → 25.7 µs (~) |
| `restore` | large | 1.2 ms → 91.6 µs (−92%) | 403 µs → 52 µs (−87%) |
| `restore_alternate` | default | 164 µs → 78.7 µs (−52%) | 248 µs →
96.5 µs (−61%) |
| `restore_alternate` | small | 270 µs → 79.7 µs (−71%) | 268 µs → 93 µs
(−65%) |
| `restore_alternate` | medium | 901 µs → 83.4 µs (−91%) | 255 µs → 103
µs (−60%) |
| `restore_alternate` | large | 4.41 ms → 106 µs (−98%) | 1.66 ms → 131
µs (−92%) |

#### guest_calls

| Benchmark | Size | MSHV | KVM |
|---|---|---|---|
| `call` | default | 45.8 µs → 45.3 µs (~) | 18.4 µs → 17.3 µs (~) |
| `call` | small | 46.2 µs → 43.8 µs (−5%) | 18.5 µs → 17.7 µs (~) |
| `call` | medium | 47.3 µs → 43.8 µs (~) | 19.8 µs → 17.3 µs (~) |
| `call` | large | 45.4 µs → 44 µs (~) | 18.5 µs → 18 µs (~) |
| `call_with_host_function` | default | 77.5 µs → 75.2 µs (~) | 32.9 µs
→ 34.5 µs (~) |
| `call_with_host_function` | small | 75.4 µs → 74.1 µs (~) | 32.8 µs →
34.7 µs (~) |
| `call_with_host_function` | medium | 74 µs → 75.2 µs (~) | 34.8 µs →
32.8 µs (~) |
| `call_with_host_function` | large | 73.5 µs → 75.2 µs (~) | 33.5 µs →
32.7 µs (~) |
| `call_with_restore` | default | 140 µs → 133 µs (~) | 64 µs → 62.9 µs
(~) |
| `call_with_restore` | small | 141 µs → 140 µs (~) | 64.4 µs → 63.8 µs
(~) |
| `call_with_restore` | medium | 147 µs → 138 µs (~) | 70.4 µs → 67.9 µs
(~) |
| `call_with_restore` | large | 201 µs → 166 µs (~) | 102 µs → 89.1 µs
(~) |
| `different_thread` | - | 75.5 µs → 74 µs (~) | 42.4 µs → 42.9 µs (~) |
| `interrupt_latency` | - | 47.8 µs → 48.9 µs (~) | 32.8 µs → 35.9 µs
(~) |

#### guest_functions_with_large_parameters

| Benchmark | Size | MSHV | KVM |
|---|---|---|---|
| `guest_call_with_large_parameters` | - | 164 ms → 163 ms (~) | 872 ms
→ 916 ms (~) |

#### sample_workloads

| Benchmark | Size | MSHV | KVM |
|---|---|---|---|
| `24K_in_8K_out_c` | - | 56.3 µs → 56.5 µs (~) | 26.9 µs → 27.6 µs (~)
|
| `24K_in_8K_out_rust` | - | 54.6 µs → 56.1 µs (~) | 26.6 µs → 27.9 µs
(~) |

#### sandboxes

| Benchmark | Size | MSHV | KVM |
|---|---|---|---|
| `create_initialized` | default | 1.59 ms → 1.72 ms (~) | 2.62 ms → 2.8
ms (~) |
| `create_initialized` | small | 4.44 ms → 4.35 ms (~) | 4.72 ms → 4.58
ms (~) |
| `create_initialized` | medium | 11.8 ms → 11.8 ms (~) | 9.9 ms → 9.81
ms (~) |
| `create_initialized` | large | 37.8 ms → 38.2 ms (~) | 27.6 ms → 27.5
ms (~) |
| `create_initialized_and_drop` | default | 22.4 ms → 21.4 ms (~) | 10.9
ms → 14.1 ms (~) |
| `create_initialized_and_drop` | small | 29.4 ms → 28.9 ms (~) | 14.8
ms → 13.3 ms (~) |
| `create_initialized_and_drop` | medium | 31.9 ms → 32.2 ms (~) | 19.9
ms → 20.7 ms (~) |
| `create_initialized_and_drop` | large | 58.8 ms → 57.9 ms (~) | 41.7
ms → 39.6 ms (~) |
| `sandbox_from_snapshot` | default | 599 µs → 665 µs (**+11%**) | 865
µs → 971 µs (~) |
| `sandbox_from_snapshot` | small | 823 µs → 812 µs (~) | 877 µs → 978
µs (**+12%**) |
| `sandbox_from_snapshot` | medium | 2.17 ms → 2.02 ms (~) | 975 µs →
1.01 ms (~) |
| `sandbox_from_snapshot` | large | 6.29 ms → 6.06 ms (~) | 1.06 ms →
1.22 ms (**+15%**) |

#### snapshot_files

| Benchmark | Size | MSHV | KVM |
|---|---|---|---|
| `cold_start_via_evolve` | default | 1.67 ms → 1.61 ms (~) | 3.05 ms →
2.96 ms (~) |
| `cold_start_via_evolve` | small | 4.74 ms → 4.55 ms (~) | 5.12 ms →
5.76 ms (~) |
| `cold_start_via_evolve` | medium | 12 ms → 12.3 ms (~) | 10.3 ms → 11
ms (~) |
| `cold_start_via_evolve` | large | 37.6 ms → 38.8 ms (~) | 27.9 ms →
29.1 ms (~) |
| `cold_start_via_snapshot` | default | 1.8 ms → 1.97 ms (**+9%**) |
2.59 ms → 2.85 ms (**+10%**) |
| `cold_start_via_snapshot` | small | 9.14 ms → 8.93 ms (~) | 8.55 ms →
8.95 ms (~) |
| `cold_start_via_snapshot` | medium | 54.7 ms → 53.8 ms (~) | 49.8 ms →
50.9 ms (~) |
| `cold_start_via_snapshot` | large | 204 ms → 205 ms (~) | 193 ms → 194
ms (~) |
| `cold_start_via_snapshot_unverified` | default | 1.31 ms → 1.4 ms (~)
| 2.06 ms → 2.2 ms (~) |
| `cold_start_via_snapshot_unverified` | small | 2.18 ms → 2.08 ms (−4%)
| 2.03 ms → 2.17 ms (~) |
| `cold_start_via_snapshot_unverified` | medium | 3.68 ms → 3.59 ms (~)
| 2.2 ms → 2.25 ms (~) |
| `cold_start_via_snapshot_unverified` | large | 7.7 ms → 7.73 ms (~) |
2.33 ms → 2.59 ms (~) |
| `load_snapshot` | default | 617 µs → 701 µs (**+14%**) | 613 µs → 701
µs (**+14%**) |
| `load_snapshot` | small | 6.65 ms → 6.71 ms (~) | 6.49 ms → 6.68 ms
(~) |
| `load_snapshot` | medium | 50.3 ms → 50.4 ms (~) | 48.1 ms → 48.8 ms
(~) |
| `load_snapshot` | large | 197 ms → 198 ms (~) | 190 ms → 192 ms (~) |
| `load_snapshot_unverified` | default | 88.5 µs → 112 µs (**+27%**) |
93.3 µs → 123 µs (**+31%**) |
| `load_snapshot_unverified` | small | 89.6 µs → 113 µs (**+26%**) |
95.1 µs → 119 µs (**+25%**) |
| `load_snapshot_unverified` | medium | 90.4 µs → 114 µs (**+26%**) |
96.3 µs → 120 µs (**+24%**) |
| `load_snapshot_unverified` | large | 94 µs → 118 µs (**+26%**) | 99 µs
→ 125 µs (**+27%**) |
| `save_snapshot` | default | 853 µs → 941 µs (**+10%**) | 35.5 ms →
43.4 ms (**+22%**) |
| `save_snapshot` | small | 8.97 ms → 9.09 ms (~) | 55.2 ms → 61.5 ms
(~) |
| `save_snapshot` | medium | 68.2 ms → 69.6 ms (~) | 462 ms → 402 ms (~)
|
| `save_snapshot` | large | 270 ms → 271 ms (~) | 2.03 s → 1.75 s (~) |
</details>

---------

Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

Author: ludfjig

CHANGELOG.md

@@ -5,12 +5,29 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 ## [Prerelease] - Unreleased
 
 ### Added
-* `Sandbox::snapshot_with_options` takes a snapshot with `SnapshotOptions`. `SnapshotOptions::scratch_size` gives a restored sandbox a different scratch size. A guest can boot with the scratch its initialisation needs and run guest function calls from a snapshot with only the scratch they need, so each restore costs less. It can also give a guest more scratch than it booted with.
+* `Sandbox::snapshot_with_options` takes a snapshot with `SnapshotOptions`. `SnapshotOptions::scratch_size` gives the sandbox, and later restores of the snapshot, a different scratch size. A guest can boot with the scratch its initialisation needs and run guest function calls from a snapshot with only the scratch they need, so each restore costs less. It can also give a guest more scratch than it booted with.
 
 ### Changed
 * Core dumps no longer record the guest binary path. The process name and command line are always `<unknown>`, as they already were for sandboxes created from snapshots by @ludfjig in https://github.com/hyperlight-dev/hyperlight/pull/1975
 
 * **Breaking:** Generated libc bindings use `usize` for `size_t` arguments and return values, including `strlen` and `memcpy`. Rust callers that use `u64` or `c_ulong` need explicit conversions. The C ABI is unchanged.
+* **Breaking:** Snapshots use ABI 8 and config schema v4. Existing snapshots
+  must be regenerated.
+* **Breaking:** `PtRootFinder` receives a `SnapshotMemoryReader` and returns a
+  `Result<Vec<u64>>`. Read snapshot bytes with `snapshot.read_exact(gpa, buffer)?`
+  and return roots with `Ok(roots)`.
+* Related snapshots share immutable memory layers. This reduces capture time
+  and memory use. A guest can read data it overwrote before a snapshot from the
+  older layers of that snapshot.
+* KVM: deleting a memory slot flushes only that slot's guest mappings on Linux
+  6.12 and later. This speeds up the first guest call after a snapshot,
+  restore, or unmap changes memory slots.
+* **Breaking:** `Sandbox::snapshot` installs the captured snapshot into the
+  sandbox and removes `map_region` and `map_file_cow` regions, which only
+  `restore` removed before. Region pages the guest has not mapped into its page
+  tables become unreachable at the snapshot instead of at the next restore.
+* `vmem::walk_va_spaces` merges adjacent mappings of the same kind into one
+  mapping.
 
 * **Breaking:** The transport arena sits one unmapped guard page below the reserved scratch-top pages, at an address independent of the scratch size, and the page tables at the scratch base. The guest's page allocator stops at the arena GPA the host publishes in the scratch-top metadata, so guests must be rebuilt. `hyperlight_common::layout::scratch_allocator_limit_gpa` is renamed `scratch_reserved_top_gpa`, the start of the reserved pages. An allocator that stops there runs into the arena. The minimum scratch size grows by that guard page. Snapshot ABI 7: existing snapshots must be regenerated.
 

docs/paging-development-notes.md

@@ -117,10 +117,10 @@ the metadata at the top of the scratch region and grows downward.
 
 When the host takes a snapshot of a guest, it will traverse the guest
 page tables, collecting every (non-page-table) physical page that is
-mapped (outside of the scratch map) in the guest. It will write out a
-new compacted snapshot with precisely those pages in order, and a new
-set of page tables which produce precisely the same virtual memory
-layout, except for the scratch map.
+mapped (outside of the scratch map) in the guest. It will reuse pages
+unchanged since earlier snapshots, write the rest in order to a new
+compacted blob, and build a new set of page tables which produce
+precisely the same virtual memory layout, except for the scratch map.
 
 ### Pre-sizing the scratch region
 

docs/snapshot-oci-format.md

@@ -22,24 +22,28 @@ path/
   blobs/sha256/
     <manifest-digest>                 OCI image manifest JSON
     <config-digest>                   Hyperlight config JSON
-    <snapshot-digest>                 raw memory bytes
-                                      (`memory_size` bytes)
+    <data-digest>                     one raw blob per snapshot blob:
+                                      guest data, then page tables
     <transport-digest>                canonical virtqueue rings
 ```
 
 Four blob kinds per tag:
 
 * **manifest** (`application/vnd.oci.image.manifest.v1+json`). Tiny JSON
-  pointer record selected via `index.json`. References one config and
-  two layers by digest.
-* **config** (`application/vnd.hyperlight.snapshot.config.v3+json`). The
+  pointer record selected via `index.json`. References one config,
+  the snapshot blobs, and one transport layer by digest.
+* **config** (`application/vnd.hyperlight.snapshot.config.v4+json`). The
   snapshot descriptor: arch, hypervisor, CPU vendor, ABI version,
   resume address and captured registers, memory and transport layout,
-  registered host functions, snapshot generation counter, and namespaced
-  application metadata. Loaded eagerly and fully parsed.
-* **layer / memory** (`application/vnd.hyperlight.snapshot.memory.v1`).
-  The raw guest memory image, exactly `memory_size` bytes. mmap'd on
-  restore.
+  registered host functions, the start address, data length, and page-table
+  length of each blob, the blob with the page tables for a restore, snapshot
+  generation counter, and namespaced application metadata.
+  Loaded eagerly and fully parsed.
+* **layers / data** (`application/vnd.hyperlight.snapshot.memory.v2`).
+  One raw, host-page-aligned blob per snapshot blob, in config order. Each
+  blob holds guest data followed by the page-table tree of the snapshot that
+  created it. Each blob is mmap'd on restore, and the guest maps only its
+  data.
 * **layer / transport**
   (`application/vnd.hyperlight.snapshot.transport.v1`). A bounded
   binary image of the canonical G2H and H2G rings. Loading checks ring
@@ -48,6 +52,8 @@ Four blob kinds per tag:
 The runtime queue protocol and canonical checkpoint are described in
 [Virtqueue host and guest communication](./virtio-host-guest-communication.md).
 
+Manifest layers are ordered data, then transport.
+
 Blob filenames are the sha256 of the blob bytes, so identical blobs
 across tags are stored once.
 
@@ -95,7 +101,7 @@ A single saved `Snapshot` consists of exactly:
   config blob for tooling visibility,
 * one **manifest** blob (referenced by that index entry),
 * one **config** blob (referenced by the manifest's `config` field),
-* one memory **layer** blob,
+* one or more data **layer** blobs,
 * one transport **layer** blob.
 
 Saving two snapshots under different tags into the same `path`
@@ -138,12 +144,12 @@ podman), `go-containerregistry` (crane), and `regclient`.
 ## Read semantics
 
 `Snapshot::load(path, reference)` reads a snapshot. It does not check
-the manifest, config, memory, or transport blobs against their sha256 digests.
+the manifest, config, data, or transport blobs against their sha256 digests.
 `reference` is an
 [`OciReference`], either a tag that matches the
 `org.opencontainers.image.ref.name` annotation or the manifest
 digest returned by `save`. `Snapshot::checked_load` adds the digest
-check on all four blobs, catching accidental corruption on disk.
+check on every blob, catching accidental corruption on disk.
 Both run every other check (OCI structure, descriptor sizes, schema
 versions, arch / hypervisor / CPU vendor / ABI tags, layout bounds,
 entrypoint bounds). The caller is responsible for trusting the source.

docs/snapshot-versioning.md

@@ -18,25 +18,25 @@ A snapshot carries four independently evolvable version markers:
   size), and the calling convention for guest function entry. A change
   to any of these breaks older snapshots unless the loader adds a
   compat path.
-* **Snapshot blob encoding**, `MT_SNAPSHOT_V1`
-  (`application/vnd.hyperlight.snapshot.memory.v1`), aliased as
+* **Snapshot blob encoding**, `MT_SNAPSHOT_V2`
+  (`application/vnd.hyperlight.snapshot.memory.v2`), aliased as
   `MT_SNAPSHOT_CURRENT`. This is the on-wire format of the snapshot
   blob: framing, section ordering, alignment, dirty/zero-page elision,
   anything about how the bytes are packed inside the OCI layer.
 * **Transport blob encoding**, `MT_TRANSPORT_V1`
   (`application/vnd.hyperlight.snapshot.transport.v1`), aliased as
   `MT_TRANSPORT_CURRENT`. This is the binary encoding of canonical
   virtqueue state stored outside the memory layer.
-* **Config schema**, `MT_CONFIG_V3`
-  (`application/vnd.hyperlight.snapshot.config.v3+json`), aliased as
+* **Config schema**, `MT_CONFIG_V4`
+  (`application/vnd.hyperlight.snapshot.config.v4+json`), aliased as
   `MT_CONFIG_CURRENT`. This is the JSON shape of the config blob:
   field names, types, required vs optional, the descriptors the loader
   needs in order to reconstruct the sandbox (memory sizes, buffer
   sizes, `abi_version`, `hyperlight_version`, etc.). Renaming a field,
   changing its type, or adding a required field is a schema change and
-  bumps this constant. Version 3 describes the virtqueue-only memory layout
-  and requires a transport layer. Config v1 and v2 are incompatible with
-  the current ABI.
+  bumps this constant. Version 4 describes snapshot blobs, the blob with
+  the page tables, and transport geometry, and requires a transport
+  layer. Config v1, v2, and v3 are incompatible with the current ABI.
 
 ABI 6 requires checkpoint readiness after guest initialization and permits
 H2G prefill to omit retained slots. Earlier snapshots must be regenerated.
@@ -45,6 +45,9 @@ ABI 7 places the transport arena below the reserved scratch-top pages, at an
 address independent of the scratch size, and the page tables at the scratch
 base. Earlier snapshots must be regenerated.
 
+ABI 8 stores snapshot memory as layers. Earlier snapshots must be
+regenerated.
+
 The `OCI_LAYOUT_VERSION` constant is pinned by the OCI image-layout
 spec at `1.0.0`.