Remove dependency on pytorch

[NicolasHug]

Sooooo this makes TorchCodec torch-free: torch is optional and VideoDecoder returns numpy or cupy arrays when torch isn't there.

This PR is obviously never going to get merged, and there is no current plan to enable this, this is purely exploratory. Just wanted to explore the feasibility of this. (If you read this and you need TorchCodec to be torch-agnostic, let us know, this is useful signal).

Looks like it's doable, the main implication being that we now need a torch-agnostic data structure to store the frame outputs, and it needs all the operations that we used to rely on torch for (for CPU and CUDA): allocation, division, concatenation, etc. We can probably drastically reduce the surface of those if we're OK with not supporting all features in the non-torch paths, which is probably OK (I mean, decord has exactly the same constraint and they're not re-implementation all torch.tensor operations).

I'm still not entirely confident about the packaging story: what kind of wheels do we build, and where do we push them. Also unclear about the output type for the CUDA non-torch path. On CPU it should clearly be numpy.

Pasting below claude's design - it's a bit noisy. I particularly don't follow the justifications around the bindings, maybe we could simplify everything with nanobind - the first step would be simplify our current bindings (in main) since the situation is already messy.

Making PyTorch optional in TorchCodec — design & implementation

Branch: torch-optional-decoder (branched off main).

This document explains the goal, the architecture, the key decisions (with the
alternatives we rejected and why), the user experience, what's verified, and
what's intentionally left for later.

Update: torch-free decoding now covers GPU as well as CPU. With no
torch installed, CPU decode returns numpy and CUDA decode returns cupy
(both zero-copy via DLPack). See §4.11–§4.13 for the CUDA decisions and
§10 (Implications for future development) for what this means when you add
features going forward.

---

1. Goal

Make PyTorch an optional runtime dependency of TorchCodec's video decoder:

• If torch is installed → VideoDecoder returns torch.Tensor (today's
  behavior, unchanged, including torch.compile support).
• If torch is not installed → VideoDecoder still works and returns
  numpy.ndarray.
• If torch is installed but the user wants numpy, they can opt in
  (torchcodec.set_bridge("numpy")).

Target end-state, in one sentence: one wheel that decodes to torch tensors when
torch is present and to numpy (CPU) / cupy (CUDA) arrays when it isn't.

Scope: the decoder, on CPU and CUDA. Torch-free CUDA decode returns cupy
(uint8). Out of scope for now (still require torch, raise a clear error
otherwise): audio decoding, encoders, samplers, transforms, and float32 output
— see §8 and §10.

---

2. User experience

Install

Neither torch nor numpy is a hard dependency (this mirrors the torch
ecosystem convention of not pinning torch). TorchCodec needs at least one of
them at runtime:

pip install torchcodec            # bring your own torch OR numpy
pip install torch torchcodec      # full features, torch.Tensor output
pip install numpy torchcodec      # torch-free, numpy output

The "bridge" (decord-inspired)

A global, thread-safe (ContextVar) setting selects the output array type. The
VideoDecoder API itself is unchanged.

import torchcodec
from torchcodec.decoders import VideoDecoder

torchcodec.get_bridge()             # "torch" if torch installed else "numpy"

d = VideoDecoder("video.mp4")
d[0]                                # torch.Tensor (torch present) or np.ndarray

torchcodec.set_bridge("numpy")      # even with torch installed -> numpy
d[0]                                # np.ndarray (zero-copy on CPU)

torchcodec.set_bridge("torch")      # raises clearly if torch isn't installed

Behavior matrix:

torch installed	device	bridge	d[0] type
yes	cpu/cuda	"torch" (default)	torch.Tensor
yes	cpu	"numpy"	numpy.ndarray
no	cpu	"numpy" (default)	numpy.ndarray
no	cuda	"numpy" (default)	cupy.ndarray (GPU; see §4.13)
no	any	"torch"	error

Torch-free CUDA decode requires cupy installed (the DLPack consumer); the
returned object is DLPack-capable, so jax/torch can also consume the frame.

---

3. The core problem and the architecture

TorchCodec's C++ core was built on torch::stable::Tensor for four distinct
things: (1) a tensor data structure, (2) allocation (CPU + CUDA caching
allocator), (3) ops (copy_, to, narrow, permute, cat, rot90, div,
contiguous, …), and (4) CUDA stream/context/device-guard. To make torch
optional we had to provide torch-free equivalents of #1–#3 (#4 stays torch for
now — GPU is deferred).

DLPack does not solve this — see §4.1.

The architecture is three C++ libraries + a Python dispatch layer:

                     ┌───────────────────────────── Python ─────────────────────────────┐
                     │  VideoDecoder ── _bridge (output type) ── ops.py (dispatch)        │
                     │                                   │                                │
                     │            torch present?  ┌──────┴───────┐  torch absent?         │
                     │            _torch_ops.py   │              │  _numpy_ops.py         │
                     └────────────────│───────────┘              └──────────│─────────────┘
                                      │ torch.ops                           │ pybind
   libtorchcodec_custom_opsN.so  ◀────┘  (TORCH adapter: registers          │
     (links torch; the adapter)         allocator+CUDA backend hooks,       │
                                        converts tc<->torch::stable)        │
                                                  │                          │
   libtorchcodec_pybind_opsN.so  ◀─────────────────────────────────────────┘
     (torch-FREE pybind frontend; returns DLPack capsules -> numpy)
                                                  │
   libtorchcodec_coreN.so  ◀───────────── tc::Tensor (torch-FREE) ──────────
     (FFmpeg + the decode engine; CPU links NO torch)

• core is now torch-symbol-free (it uses tc::Tensor, not
  torch::stable::Tensor) on both CPU and CUDA. A torch CUDA build still
  links torch (the allocator/compute hooks route to torch), but a torch-free
  CUDA build (ENABLE_CUDA=1 ENABLE_TORCH=0) links only CUDA::cudart and
  compiles TCCudaBackend.cu, which provides the allocator + GPU kernels with no
  torch at all. The CUDA stream/context/device-guard are reached through
  torch-free seams (CUDAStreamHook.h + CudaDeviceGuard); when torch is
  present the adapter registers torch's current-stream getter so behavior is
  unchanged.
• custom_ops is the torch adapter: it keeps the torch.ops.torchcodec_ns.*
  custom-op registrations (so torch.compile works), and it registers the
  allocator/compute hooks that wire torch into the torch-free core.
• pybind is the torch-free frontend: it exposes decode ops that return
  DLPack capsules (consumed as numpy/cupy/jax via from_dlpack).

At runtime the loader (_internally_replaced_utils.py) loads custom_ops via
torch.ops.load_library when torch is importable, otherwise loads core via
ctypes + the pybind module. The same installed files serve both worlds —
custom_ops is simply never loaded when torch is absent.

---

4. Key decisions (and rejected alternatives)

4.1 Our own tc::Tensor, not "just DLPack"

Decision: introduce a small torch-free tensor type, tc::Tensor
(TCTensor.{h,cpp}) — owns {data_ptr, shape, strides, dtype, device, deleter}, implements the op subset the core needs, and exports DLPack.

Rejected: "just use DLPack as the tensor type." DLPack is purely an
exchange format — a C struct + capsule protocol. It has no allocation, no
ops, no memory management. The core needs to compute (copy, cast, permute,
cat, normalize), so we needed a real (if minimal) tensor type. DLPack is used
only at the edges (handing buffers to numpy/torch/cupy).

4.2 torch interop via from_blob+deleter, not DLPack

Phase 0 finding: PyTorch's stable ABI does not expose DLPack
(DLManagedTensor appears nowhere under torch/csrc/; ATen/dlpack.h is
guarded off when TORCH_TARGET_VERSION is defined, which the core sets). But
torch::stable::from_blob with a custom deleter exists (≥ 2.11).

Decision: the torch adapter converts tc::Tensor ↔ torch::stable::Tensor
zero-copy via from_blob + a deleter that holds a reference to the source
(TCStableConvert.h). DLPack is used only on the torch-free pybind frontend.

4.3 Allocator is a pluggable hook → torch's caching allocator when present

Decision: tc::Tensor never calls cudaMalloc itself. Allocation goes
through a process-wide hook (tc::setAllocator). The torch adapter registers an
allocator that uses torch::stable::empty — i.e. torch's CUDA caching
allocator — when torch is present.

Why / rejected alternatives: we deliberately do not reimplement a CUDA
caching allocator (the "slippery slope"). We checked two reference libraries:

• decord: device allocator is plain cudaMalloc/cudaFree; its
  NDArrayPool recycling free-list is constructed with capacity 0, i.e.
  effectively per-frame allocation. No caching allocator.
• PyNvVideoCodec: a grow-once, reuse-forever std::vector of output buffers
  (steady-state zero alloc) + a manual lock/unlock free-list. No cuMemPool.

Conclusion: nobody ships a fancy GPU allocator for a video decoder, and a video
decoder's output frames are uniform-shape so a trivial reuse ring suffices.
When torch is present, torch's caching allocator is strictly better than
either, so we use it. For the future torch-free GPU path, plain per-frame
cudaMalloc (decord-proven) or a ~10-line reuse ring is enough — no allocator
rabbit hole.

4.4 A pluggable compute backend for non-CPU ops

Decision: tc::Tensor implements all ops natively for CPU. For non-CPU
devices each data-touching op dispatches to a registered backend
(tc::registerDeviceBackend). The torch adapter registers a CUDA backend that
runs each op via torch. Metadata-only views (narrow/select/permute) need no
backend (pure stride math, device-agnostic).

This is why the CUDA path "just works" while torch is present without the core
containing GPU kernels: the hooks route GPU allocation + compute to torch.

4.5 The bridge selects the output type, not the execution backend

This was the subtlest decision. Two models were considered:

• (A, chosen) bridge = output type. Execution backend is chosen by torch
  availability (torch present → torch path; absent → pybind path). The bridge
  only decides how the result is wrapped: "torch" → tensor as-is; "numpy" →
  tensor.numpy() (zero-copy on CPU).
• (B, rejected) bridge = execution backend. "torch" → torch path,
  "numpy" → pybind path, even with torch installed.

Why B was rejected: the decoder handle types differ between backends — the
torch path's handle is a torch.Tensor (a pointer laundered into a tensor,
which is exactly what lets it flow through torch.compile graphs), while the
pybind path's handle is a plain int. A decoder is therefore bound to one
backend for its whole life (handle, ops, and metadata calls). Model B would
require both backends live simultaneously, the bridge locked at decoder-creation
time, threading "which backend" through every op and metadata call, and
torch.compile only on the torch bridge anyway. Model A is less code: one
execution path per environment, one handle type, metadata untouched, and the
"go through torch then convert" is a single zero-copy .numpy(). This also
matches what decord actually does (one decode engine, bridge converts at output).

4.6 Keep torch.compile → keep shipping custom_ops

Decision: the CPU wheel still contains custom_ops (the torch.ops
adapter), so torch users keep native tensors and torch.compile.

Rejected: a "pure DLPack" wheel that always decodes via pybind and wraps
from_dlpack → torch in Python. It's the simplest possible binary, but it
drops torch.compile (which needs the custom-op registration) — a
regression for existing torch users.

4.7 One wheel, not two — the CPU core must not link torch

Decision: the CPU core does not link torch even when built with
ENABLE_TORCH=ON (it's torch-symbol-free; only the CUDA core links torch). So a
single CPU wheel, built with torch, ships core (torch-free) + custom_ops
(torch) + pybind, and loads in a torch-free environment.

Rejected: (a) a separate torch-free wheel — more packaging/UX surface;
(b) a "fat" wheel bundling two full core builds. Neither is needed once the core
doesn't link torch.

4.8 numpy is not a hard dependency

Decision: pyproject.toml declares neither torch nor numpy; numpy imports
are guarded (_bridge.py). A torch-only user doesn't pull numpy; a numpy-only
user doesn't pull torch. Requested-but-missing → clear error.

Rejected: declaring numpy in dependencies. It would force numpy on
torch-only users and didn't match the "leave it to the user" convention.

4.9 Module split for the Python dispatch

Decision: ops.py became a thin loader+dispatcher that re-exports either
_torch_ops.py (the original torch body, moved verbatim → byte-identical
torch path) or _numpy_ops.py (pybind → numpy). This keeps the torch path
unchanged and isolates the torch-free implementation.

4.10 TC_CHECK and moving metadata JSON

• STD_TORCH_CHECK (torch) was replaced by a torch-free TC_CHECK/
  TC_CHECK_INDEX (TCError.h) across the CPU core, mapping to
  RuntimeError/IndexError exactly as before.
• The metadata-JSON serialization lived in custom_ops.cpp (torch side); it was
  moved to a torch-free MetadataJson.{h,cpp} so the pybind frontend can read
  metadata (needed by VideoDecoder.__init__). fmt::to_string → std::to_chars
  (shortest round-trip, preserves precision).
• _video_decoder.py/_core/_decoder_utils.py use from __future__ import annotations so torch-typed annotations don't need torch at import, and the
  output_dtype=torch.uint8 default param value became a module constant.

4.11 CUDA decoupled from torch via three small seams (not a rewrite)

The CUDA device interfaces touched torch in only three places, all in
CUDACommon: the current stream (aoti_torch_get_current_cuda_stream), context
init, and a device guard. Decision: replace each with a torch-free seam
rather than forking the CUDA code:

• a stream-provider hook (CUDAStreamHook.h, a void*-typed
  std::function): the torch adapter registers torch's current-stream getter so
  the torch build is byte-for-byte unchanged; torch-free uses the default stream.
• initializeCudaContext already allocated through the allocator hook, so it is
  torch-agnostic for free.
• CudaDeviceGuard — a cudaSetDevice RAII guard replacing torch's DeviceGuard.

This is the CUDA analog of what §4.4 did for compute: keep one set of CUDA code,
route the torch-specific bits through hooks.

4.12 Torch-free GPU compute = cudaMalloc + a few kernels (TCCudaBackend.cu)

Decision: for a torch-free CUDA build, register tc's allocator and compute
backend with a torch-free implementation: a per-frame cudaMalloc/cudaFree
allocator (the decord/PyNvVideoCodec approach from §4.3) and a CUDA compute
backend — copy_ (cudaMemcpy), zero_ (cudaMemset), and contiguous (a
small strided-copy kernel, needed because the decoded frame is a non-contiguous
CHW view). Ops only needed for float32 / transforms / audio
(toDtype/div/rot90/cat) are stubbed to raise a clear error.

Rejected: writing the full GPU op set now. The default uint8 decode path
only needs alloc + copy + zero + contiguous; the rest can be added when float32
/ transforms are made torch-free (§10).

4.13 Torch-free GPU output is cupy (the bridge passes GPU arrays through)

Decision: a torch-free CUDA frame comes back as a cupy.ndarray. The
op layer converts the decoded DLPack frame device-aware: CPU → numpy, CUDA →
cupy. The pybind frontend returns a self-describing _DLPackFrame
(correct __dlpack_device__), so the conversion (and any other DLPack consumer)
sees the real device. to_bridge_array passes GPU arrays through unchanged under
the "numpy" bridge (numpy is the host-only member of its family; on GPU the
natural output is the GPU array).

Rejected: (a) returning a raw DLPack capsule and making the user pick a
framework — worse ergonomics; cupy is the de-facto CUDA-DLPack array and the
object is still DLPack-capable for jax/torch. (b) adding a dedicated "cupy"
bridge value — GPU output is implicitly cupy regardless of bridge, since numpy
can't hold it and torch is absent. A "cupy" (or generic device) bridge can be
added later if needed.

---

5. Phase-by-phase (commit map)

Phase	What	Commits
0	DLPack spike: stable ABI has no DLPack; from_blob+deleter is the interop path	(finding)
A	torch-free core: tc::Tensor+DLPack, allocator hook, Device API, compute-backend hook, torch wiring, full torch::stable→tc swap, TC_CHECK	8335362, e49ad54, 17b837a, f81405e, eab9ff2, 68b390f, e2e1889, 09a8552
B	torch adapter converts at the op boundary	(folded into e2e1889)
C	torch-free Python frontend: pybind decode ops, metadata move, ops.py split, frame/loader/metadata torch-optional, import torchcodec+VideoDecoder torch-free	a90652e, 7ae8afa, e57bf2c, a888cf1, d8b59b3, b2f9505, 97f0a48
D	bridge, single runtime-optional wheel, numpy-optional, CI, formatting	8f7ca31, cce7485, d72bba7, f0af9d5
E	torch-free CUDA: decouple CUDA from torch (stream hook, device guard, TC_CHECK); TCCudaBackend.cu (alloc + kernels); ENABLE_CUDA && !ENABLE_TORCH build; cupy output; smoke + GPU CI; third-party-interface + Windows <array> fixes	bf3621e, 53a31a6, 5e10f45, …, 50aed53

---

6. Verification

Three real conda environments were used:

• codeccuda — Python 3.10, torch (CUDA 12.6), FFmpeg 8. The torch path
  (CPU + CUDA).
• codecfree — Python 3.12, no torch (import torch genuinely fails),
  FFmpeg 8, numpy. The torch-free CPU path.
• codecfreecuda — Python 3.12, no torch, conda cuda-nvcc +
  cuda-cudart-dev + cupy, FFmpeg 7. The torch-free CUDA path.

Results:

• Torch path unchanged: ~983 tests pass (decoders/ops/metadata/frame/bridge),
  plus 787 samplers/transforms/encoders earlier; CPU + CUDA decode work.
• Torch path on the torch-free core (the CPU single-wheel config): torch
  tensors + numpy bridge, 124 cpu/bridge tests pass.
• Genuinely torch-free (codecfree): import torchcodec; VideoDecoder(path)
  returns numpy for []/get_frame_at/get_frames_in_range/get_frames_at/
  get_frames_played_at/metadata/len, with torch never imported; built libs
  link no torch (ldd).
• Genuinely torch-free CUDA (codecfreecuda): VideoDecoder(path, device="cuda") returns cupy uint8 (single + batch, both the NVDEC
  default and ffmpeg backends), torch never imported, core links no torch.
  GPU frame sums match the torch CUDA decode bit-for-bit (e.g. frame 10 sum =
  4213828). Third-party-interface: 3 tests pass.
• C++ gtests (TCTensorTest, TCTorchBackendTest): pass. (TCTorchBackendTest
  includes a real GPU CPU→CUDA→div→CPU round-trip through torch's allocator.)
• Lint: pre-commit run --all-files passes (ufmt + clang-format + flake8).

Tests/CI added:

• test/smoke_test_no_torch.py — loads the pybind module directly, decodes to
  numpy, asserts torch never imported.
• test/smoke_test_no_torch_package.py — full import torchcodec + VideoDecoder
  torch-free.
• test/test_bridge.py — bridge behavior in the torch env.
• .github/workflows/no_torch_smoke.yaml — (job 1) torch-free build + ldd + both
  smokes; (job 2) single-wheel-runtime-optional: build CPU wheel with torch,
  verify torch+numpy-bridge output, uninstall torch, verify same install
  still decodes numpy.

Pre-existing failures (NOT caused by this work, confirmed on base): 2
GetAudioMetadata C++ gtests fail due to the installed FFmpeg's header-reported
audio duration (13.056 vs 13.25); FFmpeg-version dependent.

---

7. Key files

New (torch-free core/frontend):

• src/torchcodec/_core/dlpack.h — vendored, trimmed DLPack header.
• src/torchcodec/_core/TCTensor.{h,cpp} — tc::Tensor, ops, allocator +
  compute-backend hooks, DLPack export/import.
• src/torchcodec/_core/TCError.h — torch-free TC_CHECK/TC_CHECK_INDEX + visibility macros.
• src/torchcodec/_core/TCStableConvert.h — zero-copy tc::Tensor ↔ torch::stable::Tensor.
• src/torchcodec/_core/TCTorchBackend.cpp — registers torch allocator + CUDA backend (in custom_ops).
• src/torchcodec/_core/MetadataJson.{h,cpp} — torch-free metadata JSON.
• src/torchcodec/_core/_torch_ops.py / _numpy_ops.py — the two backends.
• src/torchcodec/_bridge.py — set_bridge/get_bridge + DLPack-based conversion.

Modified (made torch-optional / swapped to tc::Tensor): all CPU-core .cpp/.h
(Frame, DeviceInterface, SingleStreamDecoder, CpuDeviceInterface,
Encoder, AVIOTensorContext, SwScale, WavDecoder, Metadata,
FFMPEGCommon, FilterGraph, Transform, ValidationUtils, StreamOptions,
…); custom_ops.cpp, pybind_ops.cpp; ops.py, _internally_replaced_utils.py,
_metadata.py, _frame.py, _core/_decoder_utils.py, decoders/_video_decoder.py,
decoders/__init__.py, __init__.py; CMakeLists.txt (both), setup.py,
pyproject.toml.

New (torch-free CUDA):

• src/torchcodec/_core/CUDAStreamHook.h — void*-typed stream-provider seam.
• src/torchcodec/_core/TCCudaBackend.cu — torch-free CUDA allocator + compute
  backend (kernels), compiled into core only for ENABLE_CUDA && !ENABLE_TORCH.
• test/smoke_test_no_torch_cuda.py — public-API GPU decode → cupy, no torch.

Build flags: ENABLE_TORCH (default ON) gates building custom_ops + torch
hooks. ENABLE_CUDA gates the CUDA device interfaces; it no longer requires
ENABLE_TORCH. The build matrix:

• ENABLE_TORCH=1 ENABLE_CUDA=0 — CPU wheel (core torch-free, custom_ops
  torch); runs with or without torch at runtime.
• ENABLE_TORCH=1 ENABLE_CUDA=1 — torch CUDA build (core links torch; GPU compute
  via torch hooks).
• ENABLE_TORCH=0 ENABLE_CUDA=0 — torch-free CPU.
• ENABLE_TORCH=0 ENABLE_CUDA=1 — torch-free CUDA (core links CUDA::cudart,
  builds TCCudaBackend.cu).

---

8. What's missing / future work

Done since the original CPU scope:

• GPU decoding torch-free (§4.11–4.13): uint8 video on CUDA returns cupy with
  no torch. ENABLE_CUDA no longer requires ENABLE_TORCH.

Deferred by design (still require torch; raise a clear error torch-free):

• output_dtype="float32" — needs a uint8→float32 cast + divide. On CPU tc
  already implements both natively (it's gated off at the Python layer only); on
  CUDA the toDtype/div backend ops are stubbed (need two small kernels).
• transforms (resize/crop/rotate) — the torchcodec.transforms module
  imports torch (transform-spec building), and GPU rot90 is stubbed.
• audio decoding/WAV, encoders, samplers — not ported to the
  torch-free frontend (audio is CPU-only; nothing torch-fundamental).
• file-like/bytes/tensor sources in the torch-free path (file paths work).
• torch.compile only applies on the torch bridge (numpy/cupy users don't compile).

Packaging caveats to verify in the real wheel build (not locally testable):

• custom_ops links libtorch, so auditwheel/delocate must EXCLUDE libtorch
  from bundling (as the existing torch wheels already do). Otherwise the wheel
  would ship its own libtorch and defeat torch-optionality.
• A torch-free CUDA wheel must source libcudart (and FFmpeg's CUDA support)
  without torch — in the validation env these come from the conda CUDA toolkit.
  This is the "CUDA deps currently ride in via torch" problem; a real torch-free
  GPU wheel needs to depend on / bundle the CUDA runtime itself.

Nice-to-haves:

• A "cupy" bridge target (trivial once torch-free GPU lands).
• A context-manager form of the bridge (with torchcodec.bridge("numpy"):).
• Clearer error if neither torch nor numpy is installed (today: a plain
  numpy ModuleNotFoundError at import).
• mypy not run locally; lazy annotations should keep it green but verify in CI.

Testing limitation encountered: the two validation envs use different Python
versions (3.10 vs 3.12), so a single built .so set can't be shared across them
(pybind ABI). The CI single-wheel-runtime-optional job is the canonical
one-environment proof (build with torch → uninstall torch → still works).

---

9. How to build / use

# Full (torch) build — unchanged from before:
pip install -e ".[dev]" --no-build-isolation

# Torch-free source build (CPU only; needs ffmpeg, pybind11, numpy, no torch):
ENABLE_TORCH=0 ENABLE_CUDA=0 pip install -e . --no-build-isolation

# The published CPU wheel is built WITH torch (ENABLE_TORCH=1, ENABLE_CUDA=0):
# its core links no torch, so it also runs in torch-free environments.

Runtime:

import torchcodec
from torchcodec.decoders import VideoDecoder
frame = VideoDecoder("video.mp4")[0]        # torch.Tensor or np.ndarray
torchcodec.set_bridge("numpy")              # force numpy (torch optional)

---

10. Implications for future development (read this before adding features)

The big mental shift: the decoder core is no longer "a torch program." It is
a torch-free C++ engine (tc::Tensor) with torch wired in as one optional
backend. If you add to the decoder path, you can't just reach for a torch op and
move on. Concretely:

10.1 You cannot use torch:: in the core

The CPU core is torch-symbol-free and the torch-free CUDA core links no torch at
all. #include <torch/...> or torch::stable::... in libtorchcodec_core is a
regression (it would re-break the torch-free build and the single-wheel
property). Torch may only appear in the torch adapter (custom_ops.cpp,
TCTorchBackend.cpp) and _torch_ops.py. Use TC_CHECK/TC_CHECK_INDEX, not
STD_TORCH_CHECK, in the core.

10.2 A new data-touching op: what's genuinely new vs. what was always there

Any operation that reads/writes tensor data (a normalization, a new layout, a
crop, …) is no longer "just call torch." The op lives on tc::Tensor, and
non-CPU work goes through the DeviceBackend hook (TCTensor.h). Adding op foo
touches up to four places — but most of that is not new work:

1. CPU native impl in TCTensor.cpp — NEW. Plain C++.
2. DeviceBackend field for foo — trivial (one struct field).
3. torch CUDA backend in TCTorchBackend.cpp — basically free. It's a
   one-line call to the torch op you would have called before the migration
   anyway. Pre-migration the core already did GPU foo by calling torch; that
   call just moved into the adapter.
4. torch-free CUDA kernel in TCCudaBackend.cu — NEW (only if foo must
   work on GPU without torch; otherwise stub it to raise).

So the honest cost of a new GPU-capable op is a CPU C++ implementation + a
CUDA kernel. Steps 2–3 are nearly free (a field + the torch call we always
made). And if the op is only needed for a torch-only feature, step 4 is a stub.

Worked example — division (div, used by float32 normalization /255):

// 1. CPU native — NEW (TCTensor.cpp): a plain loop.
Tensor div(const Tensor& self, double other) {
  if (self.device().type() != DeviceType::CPU)
    return requireBackend(self.device(), "div").div(self, other);   // -> backend
  Tensor out = empty(self.sizes(), self.scalar_type(), self.device());
  forEachIndex(self.sizes(), [&](const auto& idx) {
    storeFromDouble(elemPtr(out, idx), dt,
                    loadAsDouble(elemPtr(self, idx), dt) / other);
  });
  return out;
}

// 3. torch CUDA — ~FREE (TCTorchBackend.cpp): the torch call we'd have made anyway.
backend.div = [](const tc::Tensor& self, double other) {
  return fromStable(stableDiv(toStable(self), other));   // torch does the work
};

// 4. torch-free CUDA — NEW, and currently STUBBED (div is float32-only, deferred):
backend.div = [unsupported](const tc::Tensor&, double) -> tc::Tensor {
  unsupported("div");   // would be a ~10-line elementwise divide kernel
};

The default uint8 decode path must keep working in all four build modes;
torch-only extras (like float32 div) may stub step 4 and gate in _numpy_ops.py.

10.2a How decord avoids almost all of this (and why we don't)

We checked decord. Its core needs far fewer ops than we do, so it largely
sidesteps the CPU-impl-plus-kernel burden:

• decord's NDArray is a DLPack container, not a compute tensor. Its only
  methods are Empty, CopyFrom/CopyTo (incl. device transfer), and
  CreateView/CreateOffsetView (shape/stride views). There is no div, no
  permute, no cat, no rot90, no astype in decord's core.
• The one piece of real compute it needs — color conversion + resize +
  optional /255 normalization — is fused into a single CUDA kernel
  (src/improc/improc.cu, the normalized ? 1.f : 255.f factor) plus CUDA
  texture hardware for scaling; on CPU it's FFmpeg swscale/filtergraph, not
  hand-written loops. So decord has exactly one CPU path (FFmpeg) and one GPU
  kernel for its whole pipeline.
• Everything else — layout (it returns HWC), dtype, batching, further math —
  is deferred to the consuming framework via the bridge. The user does
  .permute(...), .float()/255, stacking, etc. in torch/numpy/cupy.

Why torchcodec carries more ops: we do more in the decoder — permute to
NCHW, float32 normalization, decode-time transforms (resize/crop/rotate →
rot90), and batch assembly (cat) — to hand back ready-to-use tensors. That
design choice is what creates the tc::Tensor op surface (div, permute,
cat, rot90, contiguous, to) and hence the per-op CPU+kernel cost. If we
ever wanted to shrink that cost, the decord lever is the same: push layout /
dtype / normalization out to the bridge and keep only color-conversion in core
(which is already a torch-free .cu kernel + FFmpeg on CPU). Most of our current
ops are metadata-only views anyway (permute/narrow/select cost nothing on
any device); the ones needing real CPU+kernel work are div/to(dtype) (float32)
and rot90/cat (rotate transform / batching).

10.3 Anything exposed to Python needs both op backends + the pybind op

A core op that the decoder calls from Python flows through ops.py, which
dispatches to one of two modules. Adding a Python-visible op means:

• add it to _torch_ops.py (the torch.ops.torchcodec_ns.* schema + impl,
  and a @register_fake if it must work under torch.compile), and
• add it to _numpy_ops.py (calling the pybind frontend), which means
  exposing it in pybind_ops.cpp too — or make it a _requires_torch
  stub there if it's torch-only.

Forgetting the _numpy_ops/pybind side means the feature silently doesn't exist
without torch; forgetting register_fake breaks torch.compile.

10.4 Never allocate or pick a stream directly

• Allocation goes through tc::empty/tc::full/the allocator hook — never
  new/cudaMalloc in the core. This is what lets torch's caching allocator be
  used when present and cudaMalloc when not.
• CUDA stream/context/guard go through getCurrentCudaStream() (the
  CUDAStreamHook seam) and CudaDeviceGuard — never
  aoti_torch_get_current_cuda_stream or torch's DeviceGuard in the core.

10.5 Interop is from_blob+deleter on the torch side, DLPack on the edge

There is no DLPack in torch's stable ABI (§4.2). To hand a core buffer to torch,
use the TCStableConvert.h zero-copy path. DLPack is only for the torch-free
frontend's numpy/cupy/jax consumers. If you return a new array type to Python,
return it as a self-describing _DLPackFrame (so the device is correct) and let
the bridge convert.

10.6 Output type is device- and bridge-dependent — don't assume torch.Tensor

A decoded frame may be a torch.Tensor, numpy.ndarray, or cupy.ndarray.
Code in _video_decoder.py and friends must not assume torch (no .cuda(),
.cpu(), torch.* on decoder output without a guard). Route outputs through
to_bridge_array. New public surfaces should be tested under all of: torch-CPU,
torch-CUDA, torch-free-CPU (numpy), torch-free-CUDA (cupy).

10.7 Device types are a closed enum

tc::DeviceType is {CPU, CUDA, XPU, PrivateUse1}. Supporting a genuinely new
accelerator means extending that enum and every switch over it
(deviceTypeName), the DLPack mapping (toDLPack/fromDLPack), the stable
conversion (TCStableConvert.h), and providing a DeviceBackend. Out-of-tree
plugins register a DeviceInterface under PrivateUse1 (see
test/third-party-interface) — keep that extension point working.

10.8 Be explicit with includes

The core no longer transitively pulls torch's giant headers, which used to mask
missing #includes. MSVC is strict: a missing <array>/<algorithm>/<cstring>
that "worked" via a torch transitive include will now fail the Windows build.
Include what you use.

10.9 The testing/build matrix grew

A decoder change should be exercised in the four build modes (§9) using the three
envs (codeccuda, codecfree, codecfreecuda). Note the shared-src/.so
gotcha: editable installs copy libs into src/torchcodec/, so building in one
env clobbers another's libs (different Python ABI / FFmpeg version → import
errors); rm -f src/torchcodec/*.so and rebuild when switching envs, and give
each env its own TORCHCODEC_CMAKE_BUILD_DIR.

10.10 What is NOT affected

Samplers, transforms, encoders, audio, and WAV still legitimately require torch.
Extending those carries no torch-free obligation today — but the decoder
does, and anything those share with the decoder path (e.g. FrameBatch,
_core ops) must respect §10.1–10.6.

[pytorch-bot]

🔗 Helpful Links

🧪 See artifacts and rendered test results at hud.pytorch.org/pr/meta-pytorch/torchcodec/1495

• 📄 Preview Python docs built from this PR

Note: Links to docs will display an error until the docs builds have been completed.

✅ No Failures

As of commit 7d68779 with merge base eeb61c7 ():
💚 Looks good so far! There are no failures yet. 💚

This comment was automatically generated by Dr. CI and updates every 15 minutes.