Repository navigation
tests: HVT e2e tests fail due to seccomp filter stacking #648
Description
The HVT e2e tests fail due to a seccomp filter stacking issue in pkg/unikontainers/hypervisors/hvt.go. This is reproducible on main and unrelated to any recent changes.
Affected Tests
The following HVT test cases fail across the ctr, nerdctl, and crictl suites:
| Suite | Test Name |
|---|---|
ctr |
Hvt-rumprun-hello |
ctr |
Hvt-mirage-hello |
ctr |
Hvt-mirage-block-test |
nerdctl |
Hvt-rumprun-capture-hello |
nerdctl |
Hvt-rumprun-ping-redis |
nerdctl |
Hvt-rumprun-ping-redis-with-block |
nerdctl |
Hvt-mirage-net |
nerdctl |
Hvt-rumprun-with-seccomp |
nerdctl |
Hvt-rumprun-UserGroup |
crictl |
Hvt-rumprun-redis |
crictl |
Hvt-mirage-net |
Unaffected Backends
All other VMM backends — spt, qemu, firecracker, cloud-hypervisor are unaffected since they do not apply seccomp filters in PreExec().
Activity
cmainas commentedon May 11, 2026
Hello @namansh70747 ,
yes, this is a known issue. The problem is that we want to keep the seccomp filter for hvt as tight as possible, but due to Go's execution model, there are cases where some system calls might get executed after the seccomp filter enforcement. This depends on the Go version, distribution, etc..
I am not sure what would be the right approach to resolve this, since we keep adding more and more system calls in the allow list.
pocopepe commentedon May 12, 2026
hey @cmainas, think i found it
FilterFlagTSyncinapplySeccompFilter()syncs the filter to every Go runtime thread the moment it loads. those threads hit syscalls outside the hvt allowlist and SIGSYS.setupUser()already pins the goroutine withLockOSThread()beforePreExecso dropping TSync should be enough, filter still gets inherited by solo5-hvt through execve.found a second one hiding behind it too: Go's runtime calls
prctl(PR_SET_VMA)during heap growth (shows up in the syslog write path), which isn't in the allowlist, so the locked thread itself was crashing betweenapplySeccompFilterandsyscall.Exec. adding prctl sorts it.got both running locally with a regression test for the TSync side. want me to open a pr?
cmainas commentedon May 13, 2026
Hello @pocopepe ,
I am afraid this might be a problem yet again in the future due to some changes in Go's runtime or different distro. And her eis the thing we continue to add more and more system calls in the allow list just to satisfy code, while what we really care about is the hvt binary. Maybe we need to set these seccomp filters from C inside hvt and not in Go. This way we do not need to allow system calls like exec, prctl etc.
pocopepe commentedon May 13, 2026
tried something. basically wrapped prctl + seccomp + execve into one tiny C function with cgo so they all fire together and the Go runtime never sees the filter. think that works? or would a separate C helper be cleaner?
cmainas commentedon May 14, 2026
The we will have problem with dynamic linking against libseccomp. I think we should do that in hvt.
pocopepe commentedon May 14, 2026
looked into the solo5 side, Solo5/solo5#282 has been open since 2020, seccomp for the hvt tender on Linux never landed. Is this what you're looking for? pulling the Go filter without that means hvt runs unrestricted which is worse. are we blocking on that?
cmainas commentedon May 15, 2026
As far as I understand there is a discussion for a seccomp filter in hvt. If this is the case then yes, the hvt knows better which system calls it needs and it can create a good filter without including exec or anything else that is required from any other solution before executing the hvt binary.
pocopepe commentedon May 16, 2026
got a pr up for solo5, something I noticed on running e2e with the compiled binary, Hvt-rumprun-without-seccomp will fail now since the filter always applies for hvt. do we need to look into our tests again if modifying hvt is our way forward?
cmainas commentedon May 18, 2026
If the tests fail, we will update them accordingly.
pocopepe commentedon May 28, 2026
Hey @cmainas, the solo5 PR landed and v0.11.0 is out so gave the approach a go locally. Removed
applySeccompFilter()fromhvt.go, dropped thego-seccomp-bpfdependency, and removedHvt-rumprun-without-seccompas discussed.Tested with solo5 v0.11.0. Mirage and Spt-rumprun tests all seem to pass. The
Hvt-rumprun-*tests crash inside the guest though withSolo5: ABORT: Fatal trap, cr2=0x0during rumprun init. Swapping in solo5 v0.9.3 brings rumprun up fine, so wondering if the existing rumprun test images were compiled against older solo5 bindings and aren't playing well with v0.11.0? Not sure if that warrants a separate issue or if you'd rather fold it into the CI version bump conversation.cmainas commentedon May 29, 2026
Hello @pocopepe , these are good news. We can remove the seccomp filters form urunc then. One question though. Does the new support allow to disable seccomp?
The current images target older versions of Solo5 (specifically v0.9.3) as mentioned in the documentation. We will need to update them, but we try to automate this process.
pocopepe commentedon May 29, 2026
Nope, I've hardcoded in the allowlist with what I found on the syscall trace, so there wouldn't be any way to explicitly disable seccomp. On the automation side, are you thinking of a separate repo for the test images similar to monitors-build? Should I raise a pr referencing this issue with regards to the seccomp clean up?
cmainas commentedon Jun 1, 2026
Hello @pocopepe ,
here is the repository for the automation of the Sample images building (https://github.com/urunc-dev/unikernel-build). It is WIP, but you can check this branch https://github.com/urunc-dev/unikernel-build/tree/ref_build_from_changed_files for the overall structure.
github-actions commentedon Aug 1, 2026
This issue is stale because it has been open 60 days with no activity. Remove stale label or comment or this will be closed in 15 days.
3 remaining items
pocopepe commentedon Aug 7, 2026
Hi @cmainas,
Turns out it's not the bindings, it's a bug in solo5. Since v0.11.0 the hvt tender carves the net ring off guest memory with 4KB alignment instead of 2MB, so guests relying on that invariant break. One line fix, and our HVT tests are fixed. Heads up, I was lost on the stack trace so I used AI to help read it. Reproduced it myself on v0.12.0 with solo5's own test unikernels though.
I'll open an issue on solo5, couldn't find any mentioning it so far. Should I open one on unikernel-build too? Made some hvt changes there I'd like to go through with you.
cmainas commentedon Aug 7, 2026
Hello @pocopepe ,
feel free to open a PR in unikernel-build. There is going to be a heavy re-write of the repository though.
pocopepe commentedon Aug 13, 2026
hey @cmainas,
issue fixed on solo5's side, v0.12.1 is out with it. mind if I open a PR on urunc to drop the go side seccomp filter aand change out the docs to match the solo5 version on setup, alongside one on monitors-build to bump the version, that'd close this out for good. also started on unikernel-build's hvt side: urunc-dev/unikernel-build#2.
cmainas commentedon Aug 13, 2026
Hello @pocopepe ,
yes, feel free to do so. We will also need to update the CI, https://github.com/urunc-dev/monitors-build release etc.
chore: format hvt.go
pocopepe commentedon Aug 14, 2026
Hi @cmainas,
opened prs on both. ci's covered in the urunc one along with the docs. monitors-build is still waiting on workflow approval, once that merges and the release is out I'll push one more commit updating the hardcoded tag in installation.md, but the crux of the logic is up on #974
cmainas commentedon Aug 17, 2026
Hello @pocopepe ,
urunc-dev/monitors-build#5 builds Solo5 correctly and is merged.
docs: point the monitors bundle with solo5 v0.12.1 release
pocopepe commentedon Aug 17, 2026
Hi @cmainas, I've pushed the changes from my side
fix: drop the go side seccomp filter for hvt
docs: point the monitors bundle with solo5 v0.12.1 release
fix: drop the go side seccomp filter for hvt
fix: drop the go side seccomp filter for hvt
fix: drop the go side seccomp filter for hvt
fix: drop the go side seccomp filter for hvt