Repository navigation
fix(linux/vulkan): don't query video capabilities on a device without video encode - #6054
Open
miguelangel-nubla wants to merge 1 commit into
Open
miguelangel-nubla wants to merge 1 commit into
miguelangel-nubla wants to merge 1 commit into
Conversation
|
… video encode During encoder probing, init_codec_options() asks the Vulkan device for its video encode quality levels. The loader resolves vkGetPhysicalDeviceVideoCapabilitiesKHR for any instance, but on a device whose driver has no video encode, such as Mesa's lavapipe (the only Vulkan device on GPUs without a Vulkan driver, e.g. older Intel iGPUs), the call jumps to a null address: Sunshine segfaulted while probing the Vulkan encoder, at startup or on a stream request. The quality query now runs only when FFmpeg enabled VK_KHR_video_encode_queue on the device; otherwise the driver's default is kept and the encoder fails to open as usual. The test fixture enables the extension, and a new test checks that a device without it is never asked for capabilities.
miguelangel-nubla
force-pushed
the
fix/linux-vulkan-no-video-encode
branch
from
November 22, 2026 17:15
d2da842 to
1d0405a
Compare
|
miguelangel-nubla
marked this pull request as ready for review
November 22, 2026 19:37
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.



Description
During encoder probing,
vk_vram_t::init_codec_options()asks the Vulkan device for its video encode quality levels throughquery_and_validate_quality_level(). The loader resolvesvkGetPhysicalDeviceVideoCapabilitiesKHRfor any instance, so the null check on the function pointer passes, but on a device whose driver has no video encode, such as Mesa's lavapipe, the call jumps to address 0 and Sunshine segfaults. lavapipe is the only Vulkan device on hosts whose GPU has no Vulkan driver (here an Intel Sandy Bridge iGPU, HD 2000), so any probe that reaches the Vulkan encoder with a display found crashes Sunshine there: every start withencoder = vulkan, and with another encoder configured whenever it fails and the probe falls through to Vulkan.The crash, from systemd-coredump on that host (Sunshine 2026.1018.5342):
The quality query now runs only when FFmpeg enabled
VK_KHR_video_encode_queueon the device (AVVulkanDeviceContext::enabled_dev_extensions). Otherwise the driver's default quality is kept, and the encoder fails to open as it should, so probing goes on to the next encoder.Screenshot
Issues Fixed or Closed
Roadmap Issues
Testing
VulkanEncodeDeviceTest.SkipsQualityQueryWithoutVideoEncodeSupportintests/unit/platform/linux/test_vulkan_encode.cpp: for each codec, a fake device without the extension keeps the driver default and is never asked for capabilities. The existing fixture now lists the extension as enabled, so the other quality tests run as before.VulkanEncodeDeviceTest.*tests pass.master(c8ea783), together with my two other open PRs (choosing KMS capture later when no display was lit at startup, and retrying the encoder probe before a stream), with the Linux dependencies of the nixpkgs Sunshine package, and ran the wholetest_sunshinesuite in a container without any display: allVulkanEncodeDeviceTest.*tests pass, and compared with plainmasterin the same container it adds no failures (the same 10, which need a display, a network or packaging scripts).Device does not support the VK_KHR_video_encode_queue extension!, and goes on to the next encoder, with no crash.uv.lock) reports no changes for the touched files.master(ab6140e). Built withscripts/linux_build.sh(no CUDA) in an Ubuntu 22.04 arm64 container and rantest_sunshineunder Xvfb: 971 tests, 965 passed, 6 skipped, 0 failed.Type of Change
Checklist
AI Usage
See our AI usage policy.