Repository navigation
fix(restore): replace scratch zero with fresh allocation on Windows - #1765
Merged
danbugs merged 1 commit intoAug 26, 2026
Merged
Conversation
danbugs
requested review from
andreiltd,
jprendes,
ludfjig,
simongdavies,
squillace and
syntactically
as code owners
August 25, 2026 18:03
danbugs
added
A-perf
Contributor
There was a problem hiding this comment.
Pull request overview
This PR optimizes snapshot restore behavior in hyperlight-host by avoiding full zeroing of large scratch regions on Windows, instead relying on fresh demand-zero allocations to reduce RSS spikes and restore latency.
Changes:
- On Windows,
restore_snapshot()allocates a new scratch region even when the scratch size is unchanged, avoiding a full-regionfill(0)/memset. - Adds an in-code rationale explaining the Windows-specific performance motivation and why Linux behavior is unchanged.
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
danbugs
force-pushed
the
danbugs/fix-restore-rss-v0.16
branch
3 times, most recently
from
August 25, 2026 18:29
0a18a36 to
4565fd5
Compare
jsturtevant
previously approved these changes
Aug 25, 2026
syntactically
previously approved these changes
Aug 25, 2026
syntactically
left a comment
syntactically
left a comment
Member
There was a problem hiding this comment.
This looks like the functionally correct change to unblock the Windows unikraft performance issues. I do have a couple of minor nits below, especially around communicating why we are using (and that we should replace) this fairly odd heuristic.
danbugs
dismissed stale reviews from syntactically and jsturtevant
via
August 25, 2026 20:04
ded51f4
danbugs
force-pushed
the
danbugs/fix-restore-rss-v0.16
branch
from
August 25, 2026 20:04
4565fd5 to
ded51f4
Compare
ludfjig
reviewed
Aug 25, 2026
danbugs
force-pushed
the
danbugs/fix-restore-rss-v0.16
branch
from
August 25, 2026 21:33
ded51f4 to
385173c
Compare
syntactically
previously approved these changes
Aug 25, 2026
danbugs
force-pushed
the
danbugs/fix-restore-rss-v0.16
branch
from
August 25, 2026 21:46
385173c to
89c24b2
Compare
ludfjig
approved these changes
Aug 25, 2026
danbugs
force-pushed
the
danbugs/fix-restore-rss-v0.16
branch
from
August 25, 2026 23:21
89c24b2 to
7c879b4
Compare
danbugs
enabled auto-merge (squash)
August 26, 2026 00:26
restore_snapshot() zeroed scratch memory via scratch_mem.zero() when the size was unchanged. On Linux, KVM-only builds use MADV_DONTNEED (lazy zero) and mshv3 builds fall through to fill(0). On Windows there was no lazy path at all — zeroing memsets the entire region. For a Node.js guest with 448 MiB scratch, this caused: - RSS spike from ~9 MiB to ~453 MiB on each restore - Rewind time of ~138 ms (Windows) Gate the fresh-allocation path behind Windows only. On Windows, ExclusiveSharedMemory::new provides demand-zero pages (page-file- backed), so physical memory is consumed only as the guest touches pages. On Linux, keep the existing zero() path (KVM: MADV_DONTNEED, mshv3: fill(0)). Signed-off-by: danbugs <danilochiarlone@gmail.com>
danbugs
force-pushed
the
danbugs/fix-restore-rss-v0.16
branch
from
August 26, 2026 01:04
7c879b4 to
2a76cbc
Compare
danbugs
added a commit
that referenced
this pull request
Oct 8, 2026
A restore zeroed all of scratch on MSHV and replaced the mapping on Windows (#1765). Zeroing costs time in the size of scratch however little the guest wrote, and a fresh mapping costs a fault on every page the next run touches (about 13 us each on WHP). Now, where the hypervisor logs the guest's writes, a restore zeroes only those pages, the pages the host wrote, and the page-table copy (#1766). hyperlight_vm/dirty_log.rs decides when the log is used: - WHP: always. Tracking costs the guest nothing measurable there. - MSHV: tracking makes the guest's first write to each page after a read fault to the hypervisor (about 1 us nested). Each restore compares reading the log, zeroing what was written and those faults with the fastest measured full zero. Three costlier restores in a row switch tracking off for 256 restores, then it is retried. Two restores zero everything before it first starts, so the measure is warm, and a new scratch range starts over. - KVM keeps the in-place reset; the log is not read. Simpleguest restore plus call on MSHV (nested Azure VM): 64 MiB scratch 6.5 ms to 0.63 ms, 256 MiB 23 ms to 1.8 ms; small scratch unchanged. On WHP (bare metal): 352 KiB 0.54 ms to 0.07 ms, 16 MiB 0.97 ms to 0.13 ms, and 16 MiB with 8 MiB written per call 28 ms to 2 ms. Signed-off-by: danbugs <danilochiarlone@gmail.com>
danbugs
added a commit
that referenced
this pull request
Oct 8, 2026
Without a dirty log (Windows on ARM64, or if tracking fails), a restore on Windows replaced the scratch mapping (#1765). That costs about 450 us however small scratch is, and a fault on every page the next run touches, so small sandboxes were slower than zeroing in place. Now scratch up to RESIDENT_SCRATCH_MAX (16 MiB) is zeroed in place, and larger scratch is still replaced. Zeroing in place makes all of scratch resident, which #1765 avoided, so the cap is the one the KVM reset keeps resident, now shared by both. On WHP, restore plus call with scratch zeroed in place against replaced: 352 KiB 75 us against 516 us, 16 MiB 685 us against 953 us. Signed-off-by: danbugs <danilochiarlone@gmail.com>
danbugs
added a commit
that referenced
this pull request
Oct 9, 2026
A restore zeroed scratch by dropping it with MADV_DONTNEED on KVM (fill(0) in builds with mshv3), by fill(0) on MSHV, and by mapping fresh memory on Windows (#1765). Each costs time in the size of scratch, or a fault on every page the next run touches. On KVM, every reset now reads /proc/self/pagemap for all of scratch. A page with memory of its own that holds data is zeroed and kept, so the next run takes no faults on it. One that holds only zeros was not written since the last reset and is dropped, so what stays resident follows what runs write. Swapped pages and shared pages holding data are dropped. The rule is the same from the first restore on. Scratch is backed 4 KiB at a time on KVM, so a reset zeroes only what was touched. On Windows, scratch of up to 16 MiB is now zeroed in place, which is about 2x faster than a fresh mapping (over 5x when the guest writes a MiB or more); larger scratch is still replaced, so it does not all become resident. HostSharedMemory now notes the pages its writes touch, and zero_written zeroes only a given set of pages plus those, for resets that know what the guest wrote (next commits). Signed-off-by: danbugs <danilochiarlone@gmail.com>
danbugs
added a commit
that referenced
this pull request
Oct 9, 2026
A restore zeroed scratch by dropping it with MADV_DONTNEED on KVM (fill(0) in builds with mshv3), by fill(0) on MSHV, and by mapping fresh memory on Windows (#1765). Each costs time in the size of scratch, or a fault on every page the next run touches. On KVM, every reset now reads /proc/self/pagemap for all of scratch. A page with memory of its own that holds data is zeroed and kept, so the next run takes no faults on it. One that holds only zeros was not written since the last reset and is dropped, so what stays resident follows what runs write. Swapped pages and shared pages holding data are dropped. The rule is the same from the first restore on. Scratch is backed 4 KiB at a time on KVM, so a reset zeroes only what was touched. On Windows, scratch of up to 16 MiB is now zeroed in place, which is about 2x faster than a fresh mapping (over 5x when the guest writes a MiB or more); larger scratch is still replaced, so it does not all become resident. HostSharedMemory now notes the pages its writes touch, and zero_written zeroes only a given set of pages plus those, for resets that know what the guest wrote (next commits). Signed-off-by: danbugs <danilochiarlone@gmail.com>
danbugs
added a commit
that referenced
this pull request
Oct 11, 2026
A restore zeroed scratch by dropping it with MADV_DONTNEED on KVM (fill(0) in builds with mshv3), by fill(0) on MSHV, and by mapping fresh memory on Windows (#1765). Each costs time in the size of scratch, or a fault on every page the next run touches. On KVM, every reset now reads /proc/self/pagemap for all of scratch. A page with memory of its own that holds data is zeroed and kept, so the next run takes no faults on it. One that holds only zeros was not written since the last reset and is dropped, so what stays resident follows what runs write. Swapped pages and shared pages holding data are dropped. The rule is the same from the first restore on. Scratch is backed 4 KiB at a time on KVM, so a reset zeroes only what was touched. On Windows, scratch of up to 16 MiB is now zeroed in place, which is about 2x faster than a fresh mapping (over 5x when the guest writes a MiB or more); larger scratch is still replaced, so it does not all become resident. HostSharedMemory now notes the pages its writes touch, and zero_written zeroes only a given set of pages plus those, for resets that know what the guest wrote (next commits). Signed-off-by: danbugs <danilochiarlone@gmail.com>
danbugs
added a commit
that referenced
this pull request
Oct 13, 2026
A restore zeroed scratch by dropping it with MADV_DONTNEED on KVM (fill(0) in builds with mshv3), by fill(0) on MSHV, and by mapping fresh memory on Windows (#1765). Each costs time in the size of scratch, or a fault on every page the next run touches. On KVM, every reset now reads /proc/self/pagemap for all of scratch. A page with memory of its own that holds data is zeroed and kept, so the next run takes no faults on it. One that holds only zeros was not written since the last reset and is dropped, so what stays resident follows what runs write. Swapped pages and shared pages holding data are dropped. The rule is the same from the first restore on. Scratch is backed 4 KiB at a time on KVM, so a reset zeroes only what was touched. On Windows, scratch of up to 16 MiB is now zeroed in place, which is about 2x faster than a fresh mapping (over 5x when the guest writes a MiB or more); larger scratch is still replaced, so it does not all become resident. HostSharedMemory now notes the pages its writes touch, and zero_written zeroes only a given set of pages plus those, for resets that know what the guest wrote (next commits). Signed-off-by: danbugs <danilochiarlone@gmail.com>
danbugs
added a commit
that referenced
this pull request
Oct 14, 2026
perf(mem): reset scratch in place and log host writes to it A restore zeroed scratch by dropping it with MADV_DONTNEED on KVM (fill(0) in builds with mshv3), by fill(0) on MSHV, and by mapping fresh memory on Windows (#1765). Each costs time in the size of scratch, or a fault on every page the next run touches. On KVM, every reset now reads /proc/self/pagemap for scratch, as ranges of pages in use with PAGEMAP_SCAN on Linux 6.7 or newer. A page with memory of its own that holds data is zeroed and kept, so the next run takes no faults on it. One that holds only zeros is kept as it is: a guest may write zeros to it every run, and dropping it would make every run fault on it again. Every 16th reset gives those back instead, so what stays resident follows what recent runs wrote. Swapped pages and shared pages holding data are dropped. Scratch is backed 4 KiB at a time on KVM, so a reset zeroes only what was touched. Pagemap is opened once per process. On Windows, scratch of up to 16 MiB is now zeroed in place, which is about 2x faster than a fresh mapping (over 5x when the guest writes a MiB or more); larger scratch is still replaced, so it does not all become resident. HostSharedMemory now notes the pages its writes touch, and writes with exclusive access count as touching all of it. zero_written zeroes only a given set of pages plus those, for resets that know what the guest wrote (next commits). Signed-off-by: danbugs <danilochiarlone@gmail.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
When restoring a snapshot,
restore_snapshot()can fall into two paths:zero()on the existing regionUnless you exclude default features and use KVM only,
zero()falls to a codepath thatmemsets the entire scratch region with zeroes. With big scratch regions like is often the case with Hyperlight+Unikraft guests (e.g., 448 MiB for Node.js), this causes a massive RSS spike and significantly increases restore time.With this change, the default path on Windows uses the same fresh-allocation strategy as when the scratch sizes differ — the OS provides demand-zero pages (page-file-backed), so physical memory is consumed only as the guest touches pages.
This is gated behind Windows only. The MSHV path on Linux is left unchanged for now to avoid accidentally regressing small MSHV workloads, and may be updated in the future after more thorough benchmarking.