Restore the guest TSC exactly on KVM by ludfjig · Pull Request #2015 · hyperlight-dev/hyperlight

ludfjig · 2026-11-02T23:54:00Z

Restore does not rewind the guest TSC on KVM when it comes less than about a second after the previous restore. Restore now writes a decoy TSC value half the counter range away just before the snapshot value, in the same KVM_SET_MSRS call. KVM then applies the snapshot value exactly, with no extra ioctl.

Fixes #2014

Copilot

🟡 Changes recommended

The regression test can falsely fail due to scheduling delays and Hyper-V TSC_ADJUST behavior, while the documentation incorrectly guarantees rewind behavior across all backends.

3 open issues

What changed in this PR

This PR addresses #2014 by making KVM restore the guest TSC reliably during closely spaced sandbox restores, without an extra ioctl.

Changes:

Write a decoy TSC before the snapshot value in the same KVM_SET_MSRS call, with special handling for zero.
Add guest counter reads and regression tests for repeated restores and zero values.
Document guest-time behavior and record the fix in the changelog.

File	Description
`src/tests/rust_guests/simpleguest/src/main.rs`	Exposes `ReadTsc` for host tests.
`src/hyperlight_host/src/sandbox/initialized.rs`	Tests TSC rewind and zero-value restore.
`src/hyperlight_host/src/hypervisor/virtual_machine/kvm/x86_64.rs`	Adds paired TSC writes to bypass KVM synchronization.
`docs/msr.md`	Describes guest timing and the KVM restore mechanism.
`CHANGELOG.md`	Records the KVM TSC restore fix.

_{Review effort: Balanced}

Restore did not rewind the guest TSC on KVM when restores were less than a second apart. KVM ignores a host TSC write within one second of the value it expects and treats it as a vCPU sync. Write a decoy TSC value half the counter range away immediately before the restored value, in the same KVM_SET_MSRS call. KVM then applies the restored value exactly, with no extra ioctl. KVM also syncs a zero write, so write a restored value of 0 as 1. Add tests that restore repeatedly and that restore a zero TSC. Fixes hyperlight-dev#2014 Signed-off-by: Ludvig Liljenberg <4257730+ludfjig@users.noreply.github.com>

hyperlight-gh-bot · 2026-11-03T02:25:51Z

Benchmark Results

Measured commit: 7591df80d098
Baseline commit: 35dd23664883

kvm / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`373.55 µs` (➖ 1.00x slower)
`byte_chunks`		`751.35 ns` (➖ 1.03x faster)
`vec_bytes`		`577.60 ns` (➖ 1.02x faster)

payload_allocation

	`slot_pool_segmented`
`65536`	`142.18 ns` (➖ 1.00x slower)
`262144`	`526.73 ns` (➖ 1.01x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`14.06 ms` (➖ 1.08x slower)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`7.68 ns` (➖ 1.00x faster)	`7.68 ns` (➖ 1.00x faster)	`7.97 ns` (➖ 1.02x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`100.91 µs` (➖ 1.01x faster)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`1.91 µs` (➖ 1.09x faster)	`2.05 µs` (➖ 1.08x slower)
`262144`	`7.28 µs` (➖ 1.02x slower)	`7.06 µs` (➖ 1.04x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`1.04 µs` (➖ 1.07x faster)	`1.02 µs` (➖ 1.08x faster)
`65536`	`6.18 µs` (➖ 1.08x faster)	`6.24 µs` (➖ 1.01x faster)
`262144`	`26.87 µs` (➖ 1.03x faster)

kvm / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`751.43 µs` (➖ 1.07x slower)
`byte_chunks`		`764.37 ns` (➖ 1.07x slower)
`vec_bytes`		`574.98 ns` (➖ 1.06x slower)

payload_allocation

	`slot_pool_segmented`
`65536`	`172.04 ns` (➖ 1.06x slower)
`262144`	`557.66 ns` (➖ 1.06x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`15.53 ms` (➖ 1.06x slower)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`7.54 ns` (➖ 1.04x slower)	`7.53 ns` (➖ 1.05x slower)	`7.54 ns` (➖ 1.06x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`56.92 µs` (➖ 1.10x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.35 µs` (➖ 1.10x slower)	`2.33 µs` (➖ 1.07x slower)
`262144`	`8.32 µs` (➖ 1.08x slower)	`8.30 µs` (➖ 1.08x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`765.77 ns` (➖ 1.06x slower)	`770.85 ns` (➖ 1.05x slower)
`65536`	`7.46 µs` (➖ 1.03x slower)	`7.49 µs` (➖ 1.05x slower)
`262144`	`30.65 µs` (➖ 1.02x faster)

mshv3 / amd (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`291.01 µs` (➖ 1.04x slower)
`byte_chunks`		`943.70 ns` (➖ 1.04x faster)
`vec_bytes`		`707.70 ns` (➖ 1.02x faster)

payload_allocation

	`slot_pool_segmented`
`65536`	`254.94 ns` (➖ 1.30x slower)
`262144`	`733.38 ns` (➖ 1.00x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`60.44 ms` (➖ 1.05x faster)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`9.53 ns` (➖ 1.02x faster)	`9.51 ns` (➖ 1.05x faster)	`9.87 ns` (➖ 1.01x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`108.05 µs` (➖ 1.02x faster)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.24 µs` (➖ 1.00x faster)	`2.39 µs` (➖ 1.02x faster)
`262144`	`9.54 µs` (➖ 1.01x faster)	`10.46 µs` (➖ 1.00x slower)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`1.27 µs` (➖ 1.00x slower)	`1.30 µs` (➖ 1.05x slower)
`65536`	`8.36 µs` (➖ 1.25x faster)	`8.50 µs` (➖ 1.13x faster)
`262144`	`36.69 µs` (➖ 1.04x faster)

mshv3 / intel (Linux) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`674.74 µs` (➖ 1.00x faster)
`byte_chunks`		`950.44 ns` (➖ 1.03x slower)
`vec_bytes`		`666.44 ns` (➖ 1.04x slower)

payload_allocation

	`slot_pool_segmented`
`65536`	`258.03 ns` (➖ 1.05x slower)
`262144`	`663.06 ns` (➖ 1.01x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`70.56 ms` (➖ 1.00x faster)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`8.79 ns` (➖ 1.01x slower)	`8.75 ns` (➖ 1.00x slower)	`8.70 ns` (➖ 1.01x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`53.67 µs` (➖ 1.01x faster)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.29 µs` (➖ 1.01x faster)	`2.28 µs` (➖ 1.01x faster)
`262144`	`8.23 µs` (➖ 1.03x slower)	`7.94 µs` (➖ 1.02x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`899.96 ns` (➖ 1.01x faster)	`885.06 ns` (➖ 1.03x faster)
`65536`	`7.43 µs` (➖ 1.01x faster)	`7.50 µs` (➖ 1.00x faster)
`262144`	`38.92 µs` (➖ 1.01x faster)

hyperv-ws2025 / amd (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`1.89 ms` (➖ 1.01x slower)
`byte_chunks`		`1.18 µs` (➖ 1.00x faster)
`vec_bytes`		`764.68 ns` (➖ 1.01x slower)

payload_allocation

	`slot_pool_segmented`
`65536`	`231.26 ns` (➖ 1.01x faster)
`262144`	`797.55 ns` (➖ 1.00x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`11.01 ms` (➖ 1.01x faster)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`10.63 ns` (➖ 1.03x faster)	`10.61 ns` (➖ 1.04x faster)	`10.60 ns` (➖ 1.04x faster)

snapshot_files

	`load_snapshot_unverified`
`small`	`918.79 µs` (➖ 1.40x slower)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.87 µs` (➖ 1.06x slower)	`2.51 µs` (➖ 1.00x faster)
`262144`	`8.53 µs` (➖ 1.10x faster)	`8.58 µs` (➖ 1.09x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`1.34 µs` (➖ 1.03x faster)	`1.36 µs` (➖ 1.03x slower)
`65536`	`8.67 µs` (➖ 1.06x faster)	`8.89 µs` (➖ 1.07x faster)
`262144`	`37.68 µs` (➖ 1.01x faster)

hyperv-ws2025 / intel (Windows) (➖ stable)

No benchmark improved or regressed.

Benchmark Results

function_call_codec

	`decode_vec_bytes_copy`	`encode_control`
	`2.83 ms` (➖ 1.04x faster)
`byte_chunks`		`1.19 µs` (➖ 1.00x slower)
`vec_bytes`		`770.84 ns` (➖ 1.02x slower)

payload_allocation

	`slot_pool_segmented`
`65536`	`209.01 ns` (➖ 1.02x faster)
`262144`	`749.48 ns` (➖ 1.02x slower)

sandboxes

	`create_initialized_and_drop`
`medium`	`12.76 ms` (➖ 1.14x faster)

slot_pool

	`alloc_dealloc_4096`	`alloc_dealloc_128`	`alloc_dealloc_1500`
	`10.35 ns` (➖ 1.03x slower)	`9.70 ns` (➖ 1.00x slower)	`9.64 ns` (➖ 1.00x slower)

snapshot_files

	`load_snapshot_unverified`
`small`	`576.79 µs` (➖ 1.33x faster)

virtq_readonly

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`65536`	`2.31 µs` (➖ 1.00x faster)	`2.28 µs` (➖ 1.02x slower)
`262144`	`7.65 µs` (➖ 1.01x slower)	`7.54 µs` (➖ 1.02x faster)

virtq_readwrite

	`slot_pool_segmented`	`slot_pool_segmented_fragmented`
`8192`	`1.10 µs` (➖ 1.07x slower)	`1.06 µs` (➖ 1.02x slower)
`65536`	`8.35 µs` (➖ 1.02x faster)	`8.28 µs` (➖ 1.00x faster)
`262144`	`42.81 µs` (➖ 1.00x faster)

_{Reported by cargo ci bench-report --candidate run:41561462818 --config-file bench_report.toml.}

danbugs

Small question. Otherwise, LGTM

ludfjig added the C-bug For PRs that fix bugs label Nov 2, 2026

Copilot AI balanced review requested due to automatic review settings November 2, 2026 23:54

ludfjig requested review from andreiltd, danbugs, dblnz, devigned, jprendes, jsturtevant, simongdavies, squillace and syntactically as code owners November 2, 2026 23:54

Copilot started reviewing on behalf of ludfjig November 2, 2026 23:54

View session

Copilot AI reviewed Nov 3, 2026

View reviewed changes

Comment thread src/hyperlight_host/src/sandbox/initialized.rs Resolved

Comment thread src/hyperlight_host/src/sandbox/initialized.rs Resolved

Comment thread docs/msr.md Resolved

ludfjig marked this pull request as draft November 3, 2026 00:08

ludfjig force-pushed the kvm-tsc-restore branch from e02c90e to 9d8531c Compare November 3, 2026 00:36

ludfjig mentioned this pull request Nov 3, 2026

Improve handling of TSC on snapshot/restore #1696

Open

ludfjig force-pushed the kvm-tsc-restore branch from c334b69 to 7591df8 Compare November 3, 2026 01:42

ludfjig marked this pull request as ready for review November 3, 2026 22:56

ludfjig mentioned this pull request Nov 3, 2026

Restore does not rewind the guest TSC on WHP #2021

Closed

danbugs reviewed Nov 3, 2026

View reviewed changes

Comment thread src/hyperlight_host/src/hypervisor/virtual_machine/kvm/x86_64.rs Resolved

ludfjig mentioned this pull request Nov 4, 2026

Restore the guest TSC exactly on WHP #2022

Merged

danbugs approved these changes Nov 4, 2026

View reviewed changes

ludfjig merged commit 3294ee3 into hyperlight-dev:main Nov 4, 2026
153 of 185 checks passed

Conversation

ludfjig commented Nov 2, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

🟡 Changes recommended

Uh oh!

Uh oh!

Uh oh!

Uh oh!

hyperlight-gh-bot Bot commented Nov 3, 2026

Benchmark Results

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Benchmark Results

function_call_codec

payload_allocation

sandboxes

slot_pool

snapshot_files

virtq_readonly

virtq_readwrite

Uh oh!

danbugs left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

ludfjig commented Nov 2, 2026 •

edited

Loading