Skip to content

Tag encoded video as BT.709 and convert to match - #173

Draft
ruccho wants to merge 1 commit into
mainfrom
feature/color-primaries-bt709
Draft

Tag encoded video as BT.709 and convert to match#173
ruccho wants to merge 1 commit into
mainfrom
feature/color-primaries-bt709

Conversation

@ruccho

@ruccho ruccho commented Aug 19, 2026

Copy link
Copy Markdown
Collaborator

Closes #163.

Problem

The MP4 files produced by every backend carried no color information at all — no colr box and
no colour description in the SPS VUI. Players have to guess at the color space, and some warn
about it.

Two backends were also converting with the wrong matrix, so this was not purely a tagging gap:

  • VideoFrameBgra32::to_yuv420_planes (the readback path on Android and Windows) used BT.601
    limited-range coefficients.
  • The FFmpeg backend let FFmpeg pick the matrix, which is BT.601 at the resolutions this library
    records. Measured with a pure-red frame at the default 1280x720: Y=81 (BT.601) before,
    Y=63 (BT.709) after.

Change

Tag the output as BT.709 limited range (primaries 1, transfer 1, matrix 1, limited range) on all
backends, and make the conversions actually produce BT.709 so the tags describe the pixels.

  • Apple — color properties on the VTCompressionSession, plus matching format-description
    extensions in the muxer. AVAssetWriter writes the colr box from the format description it
    is handed, and since encoded frames travel back through the caller as opaque buffers, the
    muxer never sees the encoder's own.
  • Androidcolor-standard / color-transfer / color-range on the encoder format,
    restated on the muxer track format because codecs are not guaranteed to echo them back.
  • Windows — color attributes on both input and output media types. The muxer inherits them,
    as media-type attributes are serialized generically.
  • FFmpeg-vf setparams=.... The -color_primaries / -color_trc output options are
    not reliably forwarded to the encoder: on 8.0.1 only the matrix and range reach the VUI.
    setparams also selects the conversion matrix, fixing the BT.601 conversion above.
  • WebCodecs — declare the source frame's color space, the only lever the API offers.

Behavior change

This changes the color of existing output. The CPU conversion coefficients change from
BT.601 to BT.709, and the FFmpeg backend's conversion changes likewise. Footage recorded after
this change will not be pixel-identical to footage recorded before it. The new output is
correct: previously the pixels were BT.601 but untagged, so players defaulted to BT.709 for HD
content and rendered them with a color shift.

Verification

cargo test -p unienc --test integration_test on aarch64-apple-darwin, A/B on the same test:

BASELINE   color_range=unknown color_space=unknown color_transfer=unknown color_primaries=unknown
           no colr atom
AFTER      color_range=tv color_space=bt709 color_transfer=bt709 color_primaries=bt709
           colr nclx primaries=1 transfer=1 matrix=1 full_range_flag=0x00

The FFmpeg path was verified by replicating its exact two-stage argument lists; all four tags
reach the MP4 with no muxer change. cargo check -p unienc_c passes on all five platform
targets; cargo fmt --check and tsc --noEmit are clean.

Android, Windows and WebCodecs are compile-verified only. The GPU blit paths on Apple and
Android rely on the platform encoder honoring the requested matrix when converting BGRA itself;
that is expected but unverified on device and worth a check on real hardware. The BGRA reaching
the encoder is sRGB-encoded in both Unity gamma and linear workflows — is_gamma_workflow
selects the sRGB view formats and render pipeline on Apple and the view format on Android
precisely to normalize the two — so a single transfer tag is correct for both.

Note on merge order

This touches CompressionSession::new in unienc_apple_vt/src/video/mod.rs and the
MediaFormat setup in unienc_android_mc/src/video/mod.rs — the same places as the #166
(configurable IDR interval) branch. Depending on merge order one of the two will need conflict
resolution.

Before merging

The native binaries under Packages/jp.co.cyberagent.instant-replay/UniEnc/Plugins/ must be
rebuilt via the build-unienc.yml workflow (manual dispatch).

The MP4 files produced by every backend carried no color information at
all: no `colr` box and no colour description in the SPS VUI. Players have
to guess at the color space, and some warn about it.

Tag the output as BT.709 limited range on all backends, and make the
conversions actually produce BT.709 so the tags describe the pixels:

- `VideoFrameBgra32::to_yuv420_planes`, used by the readback paths on
  Android and Windows, used BT.601 limited range coefficients. Replace
  them with the BT.709 equivalents.
- The FFmpeg backend let FFmpeg pick the matrix, which is BT.601 at the
  resolutions this library records. Tag the frames so the conversion and
  the VUI both become BT.709.

Per backend:

- Apple: set the color properties on the VTCompressionSession, and add the
  matching format description extensions in the muxer. AVAssetWriter
  writes the `colr` box from the format description it is handed, and
  since encoded frames travel back through the caller as opaque buffers,
  the muxer never sees the encoder's own.
- Android: set `color-standard`, `color-transfer` and `color-range` on the
  encoder format, and restate them on the muxer track format because
  codecs are not guaranteed to echo them back.
- Windows: set the color attributes on both the input and the output media
  type. The muxer inherits them, as media type attributes are serialized
  generically.
- FFmpeg: tag the frames with `setparams`. The `-color_primaries` and
  `-color_trc` output options are not reliably forwarded to the encoder.
- WebCodecs: declare the source frame's color space, the only lever the
  API offers.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Set Color Primaries (sRGB)

1 participant