Update device agnostic encoding to support multiple devices

[eromomon]

Enable hardware-agnostic encoding in MultiStreamEncoder (XPU plugin support)

Signed-off-by: Edgar Romo Montiel edgar.romo.montiel@intel.com
Co-Authored-by: Dmitry Rogozhkin dmitry.v.rogozhkin@intel.com

Summary

This PR makes MultiStreamEncoder hardware-agnostic by moving every device-specific step of the video-encoding pipeline behind the existing DeviceInterface abstraction. The immediate motivation is enabling Intel XPU hardware encoding through the external torchcodec plugin mechanism, without requiring any further changes to the core encoder.

Companion PR (XPU plugin implementation):
intel/torchlib-xpu#58

Motivation

Prior to this change, Encoder.cpp had CUDA-specific branches for codec discovery, pixel-format selection, hardware-frame context setup, and tensor→AVFrame conversion. That prevented any other backend (XPU, or any future device) from providing hardware encoding without patching the core encoder.

By promoting those steps to virtual hooks on DeviceInterface, each backend (CPU, CUDA, XPU plugin, …) implements only the pieces it needs, and MultiStreamEncoder remains device-neutral.

What changed

• DeviceInterface new / generalized virtual methods used by the encoder:
  • find_codec(codec_id, is_decoder=false) allow a device to substitute the container’s default codec with its hardware equivalent.
  • get_encoding_pixel_format(av_codec, user_pixel_format) pixel-format selection is delegated to the device.
  • register_hardware_device_with_codec(codec_context) no-op on CPU, attaches the AVHWDeviceContext on HW backends.
  • setup_hardware_frame_context_for_encoding(codec_context) no-op on CPU, attaches an AVHWFramesContext on HW backends.
  • convert_tensor_to_av_frame_for_encoding(tensor, frame_index, codec_context) device owns tensor→AVFrame conversion (upload, tiling, colour conversion, etc.).
• Encoder.cpp (MultiStreamEncoder::initialize_video_stream and add_frames) CUDA-specific branches removed and replaced with unconditional calls to the hooks above. The CPU path is preserved via default no-op implementations.
• CpuDeviceInterface / CudaDeviceInterface existing behaviour preserved by implementing the new hooks (CUDA keeps NV12 default, hardware frames context, and nvenc codec substitution).
• Plugin surface (test/plugin/, torchcodec/_core/DeviceInterface.h) unchanged in shape, the XPU plugin implements the same virtual interface out-of-tree.

Design notes

• Backwards compatible: CPU and CUDA paths take the exact same code path as before; only indirection level changed.
• No new public Python API. This is a C++ core refactor.
• The XPU plugin lives out-of-tree (see companion PR) and is loaded via the existing plugin registration mechanism (register_device_interface).

Testing

• XPU encoding: manually verified end-to-end on Intel XPU hardware using the plugin from intel/torchlib-xpu

Risk / compatibility

• Internal refactor only. No API break, no behavioural change for existing CPU/CUDA users.
• All new virtual methods have safe default implementations (no-op or STD_TORCH_CHECK(false, ...)) so unrelated backends fail loudly rather than silently misbehave.

Follow-ups (out of scope)

• Upstream XPU-specific tests once the plugin is stabilised.
• Consider similarly generalising the audio-encoding path when a hardware audio backend becomes relevant.

---

Thanks in advance for the review, happy to split further or adjust the hook naming/signatures if that helps land this cleanly.

CC: @dvrogozh, @NicolasHug, @Dan-Flores

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/meta-pytorch/torchcodec/1500

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 7ab694a with merge base fb8e33d ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.

[NicolasHug]

Thanks @eromomon this mostly LGTM - on top of the minor suggestion below, could you also add the new APIs you're using to the third-party tests.

[NicolasHug]

We should remove this one now that we have it in the CudaDeviceInterface class.

[eromomon]

Done

[eromomon]

Thanks! good catch. Now it is done
I've extended [DummyDeviceInterface] in test/third-party-interface/ThirdPartyInterfaceTest.cpp to override the two encoder-side APIs this PR introduces on [DeviceInterface]:
[get_encoding_pixel_format(...)] the new virtual.
[setup_hardware_frame_context_for_encoding(...)], same signature as before, but its default is now a no-op instead of throwing.

The existing [test_third_party_interface.py] cases (pkgconfig, with_prefix, no_ffmpeg) build and dlopen the updated dummy as-is, so no changes to the Python side or [CMakeLists.txt] were needed.

Let me know if I need to address something else.

[NicolasHug]

Thanks @eromomon , LGTM