Skip to content

fix(linux/vulkan): don't query video capabilities on a device without video encode - #6054

Open
miguelangel-nubla wants to merge 1 commit into
SunshineStream:masterfrom
miguelangel-nubla:fix/linux-vulkan-no-video-encode
Open

miguelangel-nubla wants to merge 1 commit into
SunshineStream:masterfrom
miguelangel-nubla:fix/linux-vulkan-no-video-encode

Conversation

@miguelangel-nubla

@miguelangel-nubla miguelangel-nubla commented Nov 5, 2026 •

Copy link
Copy Markdown

Description

During encoder probing, vk_vram_t::init_codec_options() asks the Vulkan device for its video encode quality levels through query_and_validate_quality_level(). The loader resolves vkGetPhysicalDeviceVideoCapabilitiesKHR for any instance, so the null check on the function pointer passes, but on a device whose driver has no video encode, such as Mesa's lavapipe, the call jumps to address 0 and Sunshine segfaults. lavapipe is the only Vulkan device on hosts whose GPU has no Vulkan driver (here an Intel Sandy Bridge iGPU, HD 2000), so any probe that reaches the Vulkan encoder with a display found crashes Sunshine there: every start with encoder = vulkan, and with another encoder configured whenever it fails and the probe falls through to Vulkan.

The crash, from systemd-coredump on that host (Sunshine 2026.1018.5342):

#0  0x0000000000000000 n/a (n/a + 0x0)
#1  vk::vk_vram_t::init_codec_options(AVCodecContext*, AVDictionary**)
#2  video::make_avcodec_encode_session(...)
#3  video::make_encode_session(...)
#4  video::validate_config(...)
#5  video::validate_encoder(...)
#6  video::probe_encoders()

The quality query now runs only when FFmpeg enabled VK_KHR_video_encode_queue on the device (AVVulkanDeviceContext::enabled_dev_extensions). Otherwise the driver's default quality is kept, and the encoder fails to open as it should, so probing goes on to the next encoder.

Screenshot

Issues Fixed or Closed

Roadmap Issues

Testing

  • New unit test VulkanEncodeDeviceTest.SkipsQualityQueryWithoutVideoEncodeSupport in tests/unit/platform/linux/test_vulkan_encode.cpp: for each codec, a fake device without the extension keeps the driver default and is never asked for capabilities. The existing fixture now lists the extension as enabled, so the other quality tests run as before.
  • Regression check: with the new check disabled, the new test fails (the capabilities query is made); with it, all VulkanEncodeDeviceTest.* tests pass.
  • Built on master (c8ea783), together with my two other open PRs (choosing KMS capture later when no display was lit at startup, and retrying the encoder probe before a stream), with the Linux dependencies of the nixpkgs Sunshine package, and ran the whole test_sunshine suite in a container without any display: all VulkanEncodeDeviceTest.* tests pass, and compared with plain master in the same container it adds no failures (the same 10, which need a display, a network or packaging scripts).
  • On Sunshine 2026.1018.5342 with this change: a NixOS VM test (GDM login screen, lavapipe as the only Vulkan device) in which, without this change, a stream request's probe reached the Vulkan encoder and Sunshine segfaulted as above (it depends on when the display wakes during the probe); with it, the same test passes, with the launch answering 200. On the real host (Sandy Bridge iGPU, lavapipe the only Vulkan device): the probe reaches the Vulkan encoder, logs Device does not support the VK_KHR_video_encode_queue extension!, and goes on to the next encoder, with no crash.
  • clang-format 23.1.0 (the version in uv.lock) reports no changes for the touched files.
  • Rebased onto master (ab6140e). Built with scripts/linux_build.sh (no CUDA) in an Ubuntu 22.04 arm64 container and ran test_sunshine under Xvfb: 971 tests, 965 passed, 6 skipped, 0 failed.

Type of Change

  • feat: New feature (non-breaking change which adds functionality)
  • fix: Bug fix (non-breaking change which fixes an issue)
  • docs: Documentation only changes
  • style: Changes that do not affect the meaning of the code (white-space, formatting, missing semicolons, etc.)
  • refactor: Code change that neither fixes a bug nor adds a feature
  • perf: Code change that improves performance
  • test: Adding missing tests or correcting existing tests
  • build: Changes that affect the build system or external dependencies
  • ci: Changes to CI configuration files and scripts
  • chore: Other changes that don't modify src or test files
  • revert: Reverts a previous commit
  • BREAKING CHANGE: Introduces a breaking change (can be combined with any type above)

Checklist

  • Code follows the style guidelines of this project
  • Code has been self-reviewed
  • Code has been commented, particularly in hard-to-understand areas
  • Code docstring/documentation-blocks for new or existing methods/components have been added or updated
  • Tests have been added or updated for new or modified functionality, including regression tests for bug fixes

AI Usage

See our AI usage policy.

  • AI tools were used while preparing this PR

@sonarqubecloud

sonarqubecloud Bot commented Nov 6, 2026

Copy link
Copy Markdown

… video encode

During encoder probing, init_codec_options() asks the Vulkan device for its
video encode quality levels. The loader resolves
vkGetPhysicalDeviceVideoCapabilitiesKHR for any instance, but on a device
whose driver has no video encode, such as Mesa's lavapipe (the only Vulkan
device on GPUs without a Vulkan driver, e.g. older Intel iGPUs), the call
jumps to a null address: Sunshine segfaulted while probing the Vulkan
encoder, at startup or on a stream request.

The quality query now runs only when FFmpeg enabled
VK_KHR_video_encode_queue on the device; otherwise the driver's default is
kept and the encoder fails to open as usual. The test fixture enables the
extension, and a new test checks that a device without it is never asked
for capabilities.
@miguelangel-nubla
miguelangel-nubla force-pushed the fix/linux-vulkan-no-video-encode branch from d2da842 to 1d0405a Compare November 22, 2026 17:15
@sonarqubecloud

Copy link
Copy Markdown

@miguelangel-nubla
miguelangel-nubla marked this pull request as ready for review November 22, 2026 19:37

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant