[gareth-morgan-nv]

Summary

Streams SENSING sensors to a LiveKit room as named camera tracks. Each track carries H.264 produced by the plugin's hardware encoder and published through the LiveKit SDK's pre-encoded source, so the SDK forwards the access units and does not encode. Optional, off unless built with -DSENSING_LIVEKIT=ON.

Stacked on the mock backend PR (#1340, gmorgan/sensing-mock-backend). Draft until that merges and the Orin items below are checked. Review this PR's own commits only.

What changed

• Hardware encoder, passed through. livekit=<track> builds a JetsonEncoder for the track (the V4L2 engine on a Jetson, NVENC through libavcodec in the mock) and publishes with EncodedVideoSource and VideoEncoderBackend::PreEncoded. There is no software encoder and no fallback: if the hardware encoder cannot start, construction fails.
• Encoder API. JetsonEncoder gains a per-unit drain() (one unit per frame with its own stamp), request_keyframe(), set_bitrate(), a keyframe flag on each unit and backend_name(). File recording is unchanged.
• Output worker. Per sensor: a GPU snapshot of the borrowed camera frame is finished before update() returns, then an optional integer reduction, the encode, and one SDK call per encoded unit, off the polling loop. Two snapshot slots bound memory; a newer capture replaces the waiting one.
• Feedback. Keyframe requests from the receiving side make the next frame an IDR with SPS/PPS. A frame the SDK refuses is followed by a keyframe, and nothing is published in between. Bitrate targets are clamped to the configured range and applied only for a real change, at most every few seconds.
• Metadata. Each frame carries the camera's sequence as its frame ID, the host capture time as its user timestamp, and the sensor's capture TSC as eight bytes of user data, matched to the encoded unit it came from.
• CLI. --livekit-url, --livekit-token-file (the token never appears on the command line or in output), --livekit-fps, --livekit-scale-div, --livekit-bitrate, and livekit=<track> on --add-stream.
• Jetson encoder tuning. The V4L2 FAST hardware preset is selected explicitly (a control separate from max-performance clocks), and submit() acquires an input buffer before the RGBA-to-NV12 conversion, so a frame that will be dropped costs no conversion.
• Receiver page. livekit/receiver/index.html is a static page using the LiveKit browser SDK (CDN, integrity-checked): server URL and subscriber token, Connect/Disconnect, status, and each subscribed video track labelled with its size in its own aspect ratio. No backend or token generator. Installed under receiver/.
• Packaging. The pinned SDK package (x64 or arm64, SHA-256 checked) is fetched at configure time. cmake --install copies liblivekit.so and liblivekit_ffi.so beside the plugin and the SDK's own LICENSE and NOTICE under licenses/livekit-sdk/.

Validation

On a desktop GPU, against a local livekit-server with a second participant decoding what arrives:

• sensing_mock_tests: 70 cases pass with LIVEKIT_SERVER set (the server cases skip without it).
• Tracks, sizes and picture content; frame IDs and exact capture TSCs; a viewer joining late and rejoining; keyframe requests; recovery after a refused frame; the end-of-stream drain; lower bitrate targets; server loss, a bad token, unpublish on shutdown.
• The transport's stats name the pre-encoded passthrough source, and the output reports the upstream encoder; neither mentions OpenH264.
• The arm64 SDK package matches its pinned hash and ships the same pre-encoded API header as the x64 one.
• clang-format-14 and pre-commit are clean.

Known limitations / follow-ups

• Nothing here ran on an Orin. Not real cameras, not the V4L2 encoder feeding LiveKit, not the arm64 package, not the JETSON build with LiveKit.
• The Jetson encoder's new code was not compiled off the rig. forceIDR(), runtime setBitrate(), the keyframe flag and the drain are written against the Multimedia API but unbuilt here.
• WebRTC acceptance of the Jetson engine's H.264 (High profile, Level 5.2) is unproven. The mock's NVENC output with the same settings decoded in the tests.
• The mock reopens its encoder session to change bitrate, because the libavcodec NVENC wrapper takes its rate control at open; the stream continues from an IDR.
• Notices. The SDK's LICENSE and NOTICE are installed, but liblivekit_ffi.so bundles third-party components (WebRTC among them) whose notices the SDK package does not ship. That needs a compliance review before the binaries are distributed.
• Jetson capacity is calculated, not measured (README has the table): head stereo at native size and 30 fps is a plausible first target; four native images at 30 fps depend on the preset and exceed some modules' listed ceilings; four native at 60 fps is not a supported target. Orin Nano has no NVENC. A stream with both output= and livekit= runs two encoders on the one engine, so their pixel rates add. The FAST preset and the input-buffer reordering are unbuilt off the rig.

Test plan

• sensing_mock_tests with LIVEKIT_SERVER set, on a desktop GPU.
• On an Orin with SENSING cameras: build with -DSENSING_LIVEKIT=ON (arm64 package) and confirm a subscriber decodes the tracks.
• Confirm the V4L2 encoder's preset, keyframe, bitrate and drain paths on the rig, and measure the capacity table.
• Receiver page renders and loads the SDK with its integrity hash (desktop Chrome, headless); media not yet seen in a browser.

🤖 Generated with Claude Code

[coderabbitai]

Important

Draft PR not reviewed

Draft PRs are not automatically reviewed by default.

• Trigger a manual review

To automatically review draft PRs, update your CodeRabbit configuration:

reviews:
  auto_review:
    drafts: true

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

_{Comment @coderabbitai help to get the list of available commands.}