Repository navigation
Release v0.21.0 - #269
Merged
Merged
Release v0.21.0#269
Conversation
Bump the kernel (plat-hyperlight-v2 9f414729 -> 3b7cba88), rebuild the kernels and the native-kernel fixture, and stop sizing the heap: - Hyperlight 0.19 gives the guest no heap region (hyperlight-dev/hyperlight#1968), and the kernel took its 1 MiB boot stack from it: on 0.19 the guest crashed at boot ("Boot stack alloc failed"). The boot stack is in the kernel's .bss now, which a snapshot keeps as it kept the heap, with a canary in its lowest word. The halt that ends boot checks it, so an overflow during boot fails the boot. Each later entry from the host (every guest function call, and the checkpoint before a snapshot) checks it too, catching later use of the stack by the init thread (a native main() without LIBUKBOOT_MAINTHREAD that outlives the boot halt) or a stray write, so such a guest fails a snapshot rather than being captured; a snapshot with no call since the last snapshot or restore enters no guest code, and captures memory the boot halt or a checkpoint has already checked. The end of a call's entry, before its result, checks it for the call's own work, so a call that ran the stack down fails rather than returning. A shutdown checks it before reporting the exit, so a call in which the stack was run down and the guest then shut down fails rather than ending cleanly. The report goes to the output hluk collects, or through DebugPrint for an entry that may be a checkpoint, during which the host refuses a host call; the crash bypasses UK_CRASH, whose reboot action would re-enter the shutdown's check. - The kernel's PEB matches Hyperlight 0.19's, init_data alone. - hluk no longer sizes the heap: HEAP_SIZE and its two set_heap_size calls go, as nothing uses the region now, and the SNAPSHOT_CONTRACT doc in build.rs no longer lists a heap among what a snapshot depends on. The hyperlight main this tree builds against still maps one, at its default size, unused. - plat/hyperlight selects paging, the only way it has to allocate, so the elfloader and native-kernel defconfigs no longer set LIBUKPAGING and LIBUKFALLOC themselves; the resolved .config and the kernels built from it are unchanged by that. It also refuses a build without a boot allocator, which its heap needs, and drops pagetable64.S, whose only symbol nothing reads with paging on. The CHANGELOG notes these for native-kernel builders. The kernel boots on hyperlight 0.19.0 and on the hyperlight main this tree builds against (6c17f7cd). Signed-off-by: danbugs <danilochiarlone@gmail.com>
hyperlight-host and hyperlight-common move to the 0.19.0 release, which has scratch resize (hyperlight-dev/hyperlight#1947) and scratch_allocated_bytes (hyperlight-dev/hyperlight#2019), so the [patch.crates-io] git pin goes from the crate, the clickhouse integration and the pptx-gen demo, and the crate can be published to crates.io again. The three lockfiles take 0.19.0 from crates.io; in re-resolving them cargo also moves some crates' windows-sys dependency to versions already in the locks (mostly 0.61.2). 0.19 drops the guest heap region (hyperlight-dev/hyperlight#1968), and with it SandboxConfiguration::set_heap_size, which the previous commit stopped calling. The CHANGELOG's 0.19 entry says what the re-exported hyperlight_host 0.19 changes, that a native kernel built against an earlier plat/hyperlight must be rebuilt, and, folding in the previous commit's note, what plat/hyperlight now requires and checks; the transport arena entry's own rebuild note goes. SNAPSHOT_CONTRACT stays 8: it was bumped for the transport arena after v0.20.1 and has not been released. Dev builds since then carry it, and their snapshots are refused all the same: the kernel change (the previous commit) rolls the key's other half, k<hash>, the embedded kernel's SHA-256, and Hyperlight's own snapshot ABI 9 refuses them besides. Closes #233 Signed-off-by: danbugs <danilochiarlone@gmail.com>
The restricted-DNS fix (#248) changed what the network policy checks and had no entries: the AI_ADDRCONFIG fix under Fixed, and NetworkFilter being asked at each send on a connected UDP socket, rather than at its connect(), under Changed. The CI changes #260 (a wedged HTTP demo fails once its five-minute wait for an answer runs out, instead of at the job's 90-minute timeout) and #263 (known nested-virt failures on the Linux arm64 runner no longer fail the run) are noted under Changed, as v0.19.1 noted its CI change. Signed-off-by: danbugs <danilochiarlone@gmail.com>
changelog-notes named the newest other tag matching v* as the previous release, for GitHub's PR list: verl-hyperlight-v0.1.1 sorts above v0.20.1, so the notes for v0.21.0 would have compared against the verl release's commit instead of v0.20.1, and once a newer release is tagged, the notes for an older one would compare against it. Take the release just before the given one among the vX.Y.Z tags, sorting them field by field with sort -t. -k1,1n -k2,2n -k3,3n (rather than sort -V, which older macOS lacks) and matching them as fixed strings. Pass GitHub the v-prefixed tag when the version is given without its v. Signed-off-by: danbugs <danilochiarlone@gmail.com>
Signed-off-by: danbugs <danilochiarlone@gmail.com>
There was a problem hiding this comment.
🔵 Needs a closer look
The Hyperlight 0.19 boot-stack ABI change requires rebuilt-kernel correspondence checks and cross-architecture boot, snapshot, and restore validation beyond the supplied textual evidence.
No open issues 🎉
What changed in this PR
Prepares v0.21.0 for crates.io publication by replacing the Hyperlight git pin with released 0.19 dependencies, addressing #233. Aligns host configuration with the updated kernel and documents release compatibility changes.
Changes:
- Bumps the crate to 0.21.0, adopts Hyperlight 0.19, and removes git dependency overrides.
- Removes obsolete heap sizing and redundant paging settings.
- Updates release notes and version examples, and fixes previous-release tag selection for generated notes.
| File | Description |
|---|---|
| tests/fixtures/native-kernel/README.md | Clarifies that the platform selects paging. |
| tests/fixtures/native-kernel/defconfig | Removes redundant paging and allocator selections. |
| src/lib.rs | Removes obsolete guest heap sizing. |
| justfile | Selects the preceding release tag and normalizes its prefix. |
| integrations/clickhouse/Cargo.toml | Removes Hyperlight git overrides. |
| integrations/clickhouse/Cargo.lock | Records released Hyperlight dependencies and crate version. |
| install.sh | Updates the release-version example comment. |
| install.ps1 | Updates the release-version example comment. |
| docs/templates.md | Updates template substitution examples to 0.21.0. |
| docs/manifest.md | Updates image, cache, and version examples. |
| demos/pptx-gen/Cargo.toml | Removes Hyperlight git overrides. |
| demos/pptx-gen/Cargo.lock | Records released Hyperlight dependencies and crate version. |
| defconfig-elfloader | Relies on platform-selected paging and allocator support. |
| CHANGELOG.md | Adds the release heading, compatibility notes, and missing entries. |
| Cargo.toml | Sets version 0.21.0 and adopts Hyperlight 0.19 without git overrides. |
| Cargo.lock | Locks released Hyperlight packages and version 0.21.0. |
| build.rs | Removes heap layout from the snapshot-contract documentation. |
Review effort: Balanced
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Cuts v0.21.0, and builds against the hyperlight 0.19.0 release instead of the hyperlight main git pin, so the crate can go to crates.io again (closes #233). The release carries snapshot resizing (
AppSandbox::snapshot_with_scratch_mb, #162),AppSandbox::dirty_bytes(#259),SandboxBuilder::on_interrupt_handle(#264), the kernel's scratch fixes, name lookups withAI_ADDRCONFIGunder a restricted network policy (#248), thehluk buildcommand checks (#227, #235, #236, #237), the Python CA bundle in theagentandpython-shellimages (#245) and/proc/mountsfor PowerShell (#246).0.21.0 rather than 0.20.2: the library gains
pubitems (snapshot_with_scratch_mb,dirty_bytes,on_interrupt_handle), hyperlight-host moves from 0.18 to 0.19, and snapshots from 0.20.1 are refused. No hluk-definedpubitem is removed or renamed; those that carryhyperlight_hosttypes now carry 0.19's:HostFunctions,Snapshot(and what takes or returns one),Error::Hyperlight, and the interrupt handleAppSandbox::interrupt_handlereturns andSandboxBuilder::on_interrupt_handlehands over. The crate also re-exportshyperlight_hostitself, now 0.19, so code that uses it may need changes: itsSandboxConfiguration::set_heap_size,SandboxConfiguration::DEFAULT_HEAP_SIZE,SandboxBuilder::heap_sizeandSandboxBuilder::get_heap_size(hyperlight's builder, not hluk's) are gone. For Hyperlight 0.19's other breaking changes (among themPtRootFindergets aSnapshotMemoryReaderand returnsResult<Vec<u64>>,Sandbox::snapshotremovesmap_regionandmap_file_cowregions, snapshot ABI 9), see its CHANGELOG. ANetworkFilteris now asked at each send on a connected UDP socket rather than at itsconnect(), as it already was at an unconnectedsendto()(#248).Hyperlight 0.19 gives the guest no heap region (hyperlight-dev/hyperlight#1968), and the kernel took its 1 MiB boot stack from it, so on 0.19 every guest crashed at boot ("Boot stack alloc failed"). The kernel bump here (unikraft
plat-hyperlight-v29f414729 -> 3b7cba88) keeps the boot stack in the kernel's.bss, which a snapshot keeps as it kept the heap region, with a canary in its lowest word, and makes the kernel's PEB match 0.19's. The halt that ends boot checks the canary, so an overflow during boot fails the boot. Each later entry from the host, every guest function call and the checkpoint before a snapshot, checks it too, catching later use of the stack by the init thread (a nativemain()withoutLIBUKBOOT_MAINTHREADthat outlives the boot halt) or a stray write, so such a guest fails a snapshot rather than being captured (a snapshot with no call since the last snapshot or restore enters no guest code, and captures memory the boot halt or a checkpoint has already checked). The end of a call's entry, before its result, checks it for the call's own work, so a call that ran the stack down fails rather than returning. A shutdown checks it before reporting the exit, so a call in which the stack was run down and the guest then shut down fails rather than ending cleanly. At each of the four points the guest crashes, assertions or not, and withoutUK_CRASH, whose reboot crash action would re-enter the shutdown's check. At the boot halt, in a shutdown and at the start and end of an ordinary call's entry the report ("Boot stack canary overwritten (stack overflow?)") goes to the output hluk collects (take_output,drain_output, the echo); for any other entry, which may be a checkpoint, it goes out through DebugPrint, since the host refuses a host call (the console's HostWriteErr and HostWrite included) during a checkpoint and would otherwise end the run withHostCallDuringCheckpointbefore the report reached it.plat/hyperlightnow selects paging, the only way it has to allocate, so the defconfigs no longer set it. The kernel boots on hyperlight 0.19.0 and on the hyperlight main this tree built against before (6c17f7cd).Five commits:
just build-kernel,just build-native-kernel;just verify-kernelandjust verify-native-kernelmatch).defconfig-elfloaderand the native fixture's defconfig dropLIBUKPAGINGandLIBUKFALLOC:plat/hyperlightnow selects paging, which selectsLIBUKFALLOC; the resolved.configand the kernels are unchanged by it.plat/hyperlightalso refuses a build without a boot allocator, and dropspagetable64.S, whose only symbol (bpt_unmap_mrd) nothing reads with paging on. The CHANGELOG notes these for native-kernel builders. hluk stops sizing the heap:HEAP_SIZEand the twoset_heap_sizecalls go, as nothing uses the region now, and theSNAPSHOT_CONTRACTdoc inbuild.rsno longer lists a heap (the hyperlight main the tree builds against at this commit, 6c17f7cd, still maps one, at its default size, unused).hyperlight-hostandhyperlight-commonmove to 0.19, from crates.io: the[patch.crates-io]git pin goes fromCargo.toml,integrations/clickhouse/Cargo.tomlanddemos/pptx-gen/Cargo.toml, and the three lockfiles take 0.19.0 (in re-resolving them cargo also moves some crates'windows-sysdependency to versions already in the locks, mostly 0.61.2). The CHANGELOG notes the move, as earlier bumps were noted, what the re-exportedhyperlight_host0.19 changes, that a native kernel built against an earlierplat/hyperlightmust be rebuilt, and, folding in the kernel commit's note, thatplat/hyperlightnow selects paging, refuses a build without a boot allocator, and crashes the guest when it finds the boot stack's canary overwritten, so the transport arena entry drops its own rebuild sentence.SNAPSHOT_CONTRACTstays 8: it was bumped for the transport arena after v0.20.1 and has not been released. The dev builds since then carry it, and their snapshots are refused all the same: the kernel change rolls the key's other half (k<hash>, the embedded kernel's SHA-256), and Hyperlight's own snapshot ABI 9 refuses them too.AI_ADDRCONFIGfix under Fixed, theNetworkFilterbeing asked at each connected UDP send under Changed), and ci: stop a wedged HTTP demo from hanging the whole job #260 (a wedged HTTP demo fails once its five-minute wait for an answer runs out, instead of holding the arm64 nightly's demos job to its 90-minute timeout) and ci(arm64): soft-fail the known nested-virt failures on linux #263 (the known nested-virt failures on the Linux arm64 runner no longer fail the run) under Changed, as v0.19.1 noted its CI change. Give the nested-virt arm64 runner room on the timeouts it still blows #244 only gives tests and examples longer timeouts on the slow runner, so it gets no entry.just changelog-notes, whichrelease.ymlruns for the release notes, took the newest otherv*tag as the previous release for GitHub's PR list, andverl-hyperlight-v0.1.1sorts abovev0.20.1. It now takes the release just before the given one among thevX.Y.Ztags (v0.21.0compares againstv0.20.1,v0.20.1againstv0.20.0), sorted field by field withsort -t. -k1,1n -k2,2n -k3,3nrather thansort -V, which older macOS lacks, and matched as fixed strings; it also passes GitHub thev-prefixed tag when the version is given without itsv. A release-tooling fix, which the CHANGELOG has never recorded, so it gets no entry.versionto 0.21.0 inCargo.tomland the three lockfiles (hyperlight-unikraftonly); rename the## [Prerelease]CHANGELOG heading to## [v0.21.0], with a fresh empty Prerelease above, and note under Changed that snapshots from v0.20.1 and earlier are refused (the transport arena entry no longer says so as well); bump the illustrativev0.20.1image/version examples indocs/manifest.md,docs/templates.mdand the install scripts tov0.21.0.No Python package needs a new floor:
verl-hyperlightandopenai-agents-hyperlightneed hluk 0.19.0,smolagents-hyperlight(and somcp-server-hyperlight) hluk 0.18.0.After merge: Create release on
main, then Publish to crates.io withdry_run = false, now that the crate has no git dependency.