Tag encoded video as BT.709 and convert to match - #173
Draft
ruccho wants to merge 1 commit into
Draft
Conversation
The MP4 files produced by every backend carried no color information at all: no `colr` box and no colour description in the SPS VUI. Players have to guess at the color space, and some warn about it. Tag the output as BT.709 limited range on all backends, and make the conversions actually produce BT.709 so the tags describe the pixels: - `VideoFrameBgra32::to_yuv420_planes`, used by the readback paths on Android and Windows, used BT.601 limited range coefficients. Replace them with the BT.709 equivalents. - The FFmpeg backend let FFmpeg pick the matrix, which is BT.601 at the resolutions this library records. Tag the frames so the conversion and the VUI both become BT.709. Per backend: - Apple: set the color properties on the VTCompressionSession, and add the matching format description extensions in the muxer. AVAssetWriter writes the `colr` box from the format description it is handed, and since encoded frames travel back through the caller as opaque buffers, the muxer never sees the encoder's own. - Android: set `color-standard`, `color-transfer` and `color-range` on the encoder format, and restate them on the muxer track format because codecs are not guaranteed to echo them back. - Windows: set the color attributes on both the input and the output media type. The muxer inherits them, as media type attributes are serialized generically. - FFmpeg: tag the frames with `setparams`. The `-color_primaries` and `-color_trc` output options are not reliably forwarded to the encoder. - WebCodecs: declare the source frame's color space, the only lever the API offers. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #163.
Problem
The MP4 files produced by every backend carried no color information at all — no
colrbox andno colour description in the SPS VUI. Players have to guess at the color space, and some warn
about it.
Two backends were also converting with the wrong matrix, so this was not purely a tagging gap:
VideoFrameBgra32::to_yuv420_planes(the readback path on Android and Windows) used BT.601limited-range coefficients.
records. Measured with a pure-red frame at the default 1280x720:
Y=81(BT.601) before,Y=63(BT.709) after.Change
Tag the output as BT.709 limited range (primaries 1, transfer 1, matrix 1, limited range) on all
backends, and make the conversions actually produce BT.709 so the tags describe the pixels.
VTCompressionSession, plus matching format-descriptionextensions in the muxer.
AVAssetWriterwrites thecolrbox from the format description itis handed, and since encoded frames travel back through the caller as opaque buffers, the
muxer never sees the encoder's own.
color-standard/color-transfer/color-rangeon the encoder format,restated on the muxer track format because codecs are not guaranteed to echo them back.
as media-type attributes are serialized generically.
-vf setparams=.... The-color_primaries/-color_trcoutput options arenot reliably forwarded to the encoder: on 8.0.1 only the matrix and range reach the VUI.
setparamsalso selects the conversion matrix, fixing the BT.601 conversion above.Behavior change
This changes the color of existing output. The CPU conversion coefficients change from
BT.601 to BT.709, and the FFmpeg backend's conversion changes likewise. Footage recorded after
this change will not be pixel-identical to footage recorded before it. The new output is
correct: previously the pixels were BT.601 but untagged, so players defaulted to BT.709 for HD
content and rendered them with a color shift.
Verification
cargo test -p unienc --test integration_testonaarch64-apple-darwin, A/B on the same test:The FFmpeg path was verified by replicating its exact two-stage argument lists; all four tags
reach the MP4 with no muxer change.
cargo check -p unienc_cpasses on all five platformtargets;
cargo fmt --checkandtsc --noEmitare clean.Android, Windows and WebCodecs are compile-verified only. The GPU blit paths on Apple and
Android rely on the platform encoder honoring the requested matrix when converting BGRA itself;
that is expected but unverified on device and worth a check on real hardware. The BGRA reaching
the encoder is sRGB-encoded in both Unity gamma and linear workflows —
is_gamma_workflowselects the sRGB view formats and render pipeline on Apple and the view format on Android
precisely to normalize the two — so a single transfer tag is correct for both.
Note on merge order
This touches
CompressionSession::newinunienc_apple_vt/src/video/mod.rsand theMediaFormatsetup inunienc_android_mc/src/video/mod.rs— the same places as the #166(configurable IDR interval) branch. Depending on merge order one of the two will need conflict
resolution.
Before merging
The native binaries under
Packages/jp.co.cyberagent.instant-replay/UniEnc/Plugins/must berebuilt via the
build-unienc.ymlworkflow (manual dispatch).