[kyzer1023]

Summary

Same-size packed screen frames with an even width and height (BGRZ, BGRA, RGBA, RGBZ) now convert to limited-range BT.709 YUV420P or NV12 with a fixed-point kernel, skipping swscale. This is 2.5x to 3.1x faster at 720p, 1080p and 4K. The kernel runs only when AVX2 is detected at runtime. Odd sizes, scaling, every other pixel format, and CPUs without AVX2 still go through swscale.

Change

• New private module crates/enc-ffmpeg/src/video/rgb_to_yuv.rs.
• queue_frame_to and queue_frame_reusable in h264.rs, and the two H264PacketEncoder encode paths in h264_packet.rs, now call convert_or_scale. When both dimensions are even and the format matches, it writes into the existing converted frame. Otherwise it calls the existing scaler.
• The matrix is 16.16 fixed-point BT.709: full-range RGB in, limited-range YUV out. Chroma is the 2x2 box average of RGB, run through the same matrix.
• try_convert returns false unless is_x86_feature_detected!("avx2") is true, so swscale runs everywhere else (including aarch64). The AVX2 kernel vectorizes luma and interior chroma, and widths that are not a multiple of 8 finish with a scalar tail using the same formula. The release scalar loop alone is slower than swscale on x86 (1080p 4.104 ms vs 3.100 ms), so it is not used as a standalone path.
• No prefetch. Release medians with no prefetch, add(128) and wrapping_add(128) were within about 1% (1080p 1.135 / 1.130 / 1.128 ms), so it was removed.
• Cargo.toml: [profile.dev.package.cap-enc-ffmpeg] and [profile.ci-verify.package.cap-enc-ffmpeg] set opt-level = 1. [profile.ci-verify.package."*"] does not cover workspace members, so this crate was built at opt-level 0 in the verify job, where the AVX2 kernel takes about 180 ms per 1080p frame. At opt-level 1 it is about 1.7 ms (swscale 3.1 ms).
• open_video_encoder_inner and configure_rgb_converter are unchanged.

Benchmarks

Command:

cargo test -p cap-enc-ffmpeg --release --offline --lib video::rgb_to_yuv::tests::bench_bgrz_to_yuv420p -- --ignored --exact --nocapture

Release build. Each time is the median of 31 runs after 5 warmup runs. Before is swscale SWS_FAST_BILINEAR with the same BT.709 full-to-limited setup the encoder already uses.

Hardware: Intel Xeon, family 6 model 207, 4 vCPU, KVM, AVX2 and AVX-512. Prefetch is not in this build.

Size	Before (swscale)	After (kernel)	Speedup
1280x720	1.398 ms	0.502 ms	2.79x
1920x1080	3.260 ms	1.283 ms	2.54x
3840x2160	13.763 ms	4.491 ms	3.06x

A separate run on the same machine gave 0.491, 1.135 and 4.520 ms for the kernel against 1.381, 3.100 and 12.852 ms for swscale (2.81x, 2.73x, 2.84x).

Tests

Before the review fixes, cargo test -p cap-enc-ffmpeg --offline -- --test-threads=8: 212 passed, 0 failed, 2 ignored. After them, the cloud machine's FFmpeg 7.1 lacked libx264 and libopus, so the full run was 171 passed, 41 failed (all CodecNotFound), 2 ignored. Every rgb_to_yuv test passed. CI has the codecs.

• AVX2 matches the scalar kernel on odd widths, widths that are not multiples of 32, odd heights and padded strides.
• A flat (220, 40, 40) frame gives Y 83, Cb 110, Cr 207.
• Color-bar luma matches the expected values within 1.
• Against swscale, even-size frames stay within chroma max 3 and mean under 1.5, edges included, and within luma max 2 and mean under 1.0.
• Odd frames go through convert_or_scale and match swscale exactly.
• rgb_software_conversion_matches_declared_bt709_through_encode_decode still passes.
• New: queue_frame_to_bgrz_flat_red_decodes_as_bt709.
• New: an NV12 320x180 gradient case in swscale_fast_bilinear_stays_within_tolerance.
• The scalar BT.709 helpers, constants and Job are compiled only for x86_64 and tests, and Job.width/height only for x86_64, so ARM builds have no dead code. cargo fmt --all -- --check passes, and cargo clippy -p cap-enc-ffmpeg --all-targets -- -D warnings is clean on x86_64 Linux and aarch64 Linux, including the lib test target. aarch64-apple-darwin could not be cross-checked on Linux because there is no Apple SDK, so macOS CI confirms that target. rgb_to_yuv tests: 7 passed, 1 ignored (the bench).

Test plan

• cargo fmt -p cap-enc-ffmpeg -- --check
• cargo clippy -p cap-enc-ffmpeg --all-targets -- -D warnings (x86_64 and aarch64)
• cargo test -p cap-enc-ffmpeg -- --test-threads=8
• Run the benchmark command above.
• Record a screen on Linux and on one software-encode path (Windows or macOS without hardware encode), then confirm the colors match a recording from main.

Risk

• Odd sizes keep using swscale.
• Even frames can differ from swscale by a few chroma levels, because the kernel uses a 2x2 box.
• Machines without AVX2 keep swscale.
• This is also the software encode path on Windows and macOS.
• Size changes and other pixel formats are unchanged.
• h264.rs is also edited by feat(rendering): add opt-in macOS software codec mode #2948 and perf(recording): cut Windows readback copies and encode from textures #2940, but in different functions. perf: AVX2 UYVY/YUYV conversion in the H264 encoder #2980 changes the same four converter.run lines, so whichever merges second should chain packed YUV, then RGB, then swscale.
• cap-enc-ffmpeg builds at opt-level 1 in dev and ci-verify. Release is unchanged.

Windows real-capture

The first push failed the windows-2022 real-capture job (run): studio-screen/mp4-holds had 334 missing frames against a budget of 5, and studio-screen/fragmented kept 52 frames instead of about 530. The cause was this PR. Windows capture delivers RGBA, which takes the fast path into software libx264, and the verify profile built this crate at opt-level 0, where the kernel ran at about 5.5 fps at 1080p. The same job passed on #2980 and on main at 2026-10-23 03:50 UTC. The opt-level 1 override above is the fix.

Confidence Score: 5/5

The PR appears safe to merge, with all previous findings addressed and no new blocking issues found.

Summary

This PR adds fixed-point BT.709 conversion for same-size, even-sized packed RGB frames on CPUs with AVX2.

• Eligible RGB frames now convert to YUV faster before H.264 encoding.

_{Reviews (3) · Last reviewed commit: "Gate x86 only conversion helpers" · Reviewed by Greptile}

[vercel]

@cursoragent is attempting to deploy a commit to the Cap Software Inc Team on Vercel.

A member of the Team first needs to authorize it.

[greptile-apps]

Prefetch pointers leave the buffer

kernel8 uses ptr.add(128) for prefetch addresses, but that operation requires the resulting pointer to stay within the allocation. The tight 8×2 test buffer has only 64 bytes, so both addresses exceed it. A non-faulting CPU prefetch does not make this Rust pointer arithmetic valid. Use wrapping_add(128) or bound the prefetch.

Suggested change

	_mm_prefetch(row0.add(128) as *const i8, _MM_HINT_T0);
	_mm_prefetch(row1.add(128) as *const i8, _MM_HINT_T0);
	_mm_prefetch(row0.wrapping_add(128) as *const i8, _MM_HINT_T0);
	_mm_prefetch(row1.wrapping_add(128) as *const i8, _MM_HINT_T0);

Prompt To Fix With AI

This is a comment left during a code review.
Path: crates/enc-ffmpeg/src/video/rgb_to_yuv.rs
Line: 364-365

Comment:
**Prefetch pointers leave the buffer**

`kernel8` uses `ptr.add(128)` for prefetch addresses, but that operation requires the resulting pointer to stay within the allocation. The tight 8×2 test buffer has only 64 bytes, so both addresses exceed it. A non-faulting CPU prefetch does not make this Rust pointer arithmetic valid. Use `wrapping_add(128)` or bound the prefetch.

```suggestion
            _mm_prefetch(row0.wrapping_add(128) as *const i8, _MM_HINT_T0);
            _mm_prefetch(row1.wrapping_add(128) as *const i8, _MM_HINT_T0);
```

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[kyzer1023]

Fixed in 4e205d7 by removing the prefetch. Release medians with no prefetch, add(128) and wrapping_add(128) were within about 1% (1080p 1.135 / 1.130 / 1.128 ms, 4K 4.520 / 4.494 / 4.513 ms), so dropping it also drops the pointer arithmetic.

[greptile-apps]

Scalar speed is unmeasured

The shortcut replaces swscale on Apple ARM and CPUs without AVX2 too. However, the benchmark selects the kernel automatically, so the reported AVX2 machine never measures convert_range. If the scalar loop is slower than swscale, these machines could use more CPU or drop recording frames. Add a scalar-versus-swscale benchmark, including an Apple ARM result, and keep swscale where the scalar loop loses.

Prompt To Fix With AI

This is a comment left during a code review.
Path: crates/enc-ffmpeg/src/video/rgb_to_yuv.rs
Line: 126-127

Comment:
**Scalar speed is unmeasured**

The shortcut replaces swscale on Apple ARM and CPUs without AVX2 too. However, the benchmark selects the kernel automatically, so the reported AVX2 machine never measures `convert_range`. If the scalar loop is slower than swscale, these machines could use more CPU or drop recording frames. Add a scalar-versus-swscale benchmark, including an Apple ARM result, and keep swscale where the scalar loop loses.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

Note: If this suggestion doesn't match your team's coding style, reply to this and let me know. I'll remember it for next time!

[kyzer1023]

Fixed in 4e205d7. Measured on x86, the release scalar loop is slower than swscale (1080p 4.104 ms vs 3.100 ms), and I have no aarch64 numbers. try_convert now returns false unless AVX2 is detected at runtime, so Apple ARM and non-AVX2 CPUs keep swscale.

[richiemcilroy]

Thanks @kyzer1023, nice work. I checked the 16.16 coefficients against BT.709 (Kr 0.2126, Kb 0.0722, scaled by 219/255 and 224/255; each chroma row sums to 0). I also checked the round-half-up biases, the luma and chroma clamps, and that odd sizes and scaling still fall back to swscale. All of that is correct, and the encode/decode test confirms the stream is still tagged BT.709 limited. Two small changes before we merge: 1) In crates/enc-ffmpeg/src/video/rgb_to_yuv.rs:364-365, row0.add(128) / row1.add(128) can point past the end of the source buffer on the last block of the frame. ptr::add out of bounds is undefined behaviour even though the prefetch itself never faults. Please use wrapping_add(128) or drop the prefetch, since the loads are sequential anyway. 2) On CPUs without AVX2 (Windows on ARM, Linux aarch64, older x86), try_convert now replaces swscale with the scalar convert_range loop, and swscale has its own NEON and SSE paths there. The benchmark only measured AVX2. Please either take the fast path only when AVX2 is detected, with try_convert returning false otherwise so swscale keeps running, or add scalar-vs-swscale numbers from an aarch64 machine showing it is not slower. Optional: swscale_fast_bilinear_stays_within_tolerance only covers YUV420P. One NV12 case against swscale would also cover the encoders that take NV12.

[kyzer1023]

Thanks @richiemcilroy. In 4e205d7:

• The prefetch is removed. With no prefetch, add(128) and wrapping_add(128), the release medians were within about 1%.
• The fast path runs only with runtime AVX2. The release scalar loop is slower than swscale on x86 (1080p 4.104 ms vs 3.100 ms), and I have no aarch64 numbers, so everything else stays on swscale.
• There is a new NV12 case in swscale_fast_bilinear_stays_within_tolerance.

The windows-2022 real-capture failure was caused by this PR, not flakiness. Windows capture delivers RGBA, which takes this path into software libx264. [profile.ci-verify.package."*"] does not apply to workspace members, so cap-enc-ffmpeg was built at opt-level 0, where the AVX2 kernel takes about 180 ms per 1080p frame (about 5.5 fps). That matches the 52 kept frames against about 530 on passing runs. The same job passed on #2980 and on main (run 39897180400). dev and ci-verify now set opt-level = 1 for cap-enc-ffmpeg, which brings 1080p to about 1.7 ms against 3.1 ms for swscale. Release is unchanged. If you'd rather not touch the profiles, I can instead gate the fast path off in unoptimized builds.

Release bench: 2.79x at 720p, 2.54x at 1080p, 3.06x at 4K (n=31). The description is updated.

@greptileai review

[greptile-apps]

ARM Clippy builds fail

Removing the scalar call from execute leaves convert_range, luma, chroma_from_sum, load_rgb, and store_chroma unused in non-test ARM builds. The macOS Clippy job builds on aarch64-apple-darwin with -D warnings, so these new dead_code warnings fail CI.

Gate the kernel-only helpers and data behind #[cfg(any(target_arch = "x86_64", test))], while keeping the swscale fallback available everywhere.

Prompt To Fix With AI

This is a comment left during a code review.
Path: crates/enc-ffmpeg/src/video/rgb_to_yuv.rs
Line: 135-136

Comment:
**ARM Clippy builds fail**

Removing the scalar call from `execute` leaves `convert_range`, `luma`, `chroma_from_sum`, `load_rgb`, and `store_chroma` unused in non-test ARM builds. The macOS Clippy job builds on `aarch64-apple-darwin` with `-D warnings`, so these new `dead_code` warnings fail CI.

Gate the kernel-only helpers and data behind `#[cfg(any(target_arch = "x86_64", test))]`, while keeping the swscale fallback available everywhere.

---

For each issue above, determine whether it is valid and should be fixed. If so, fix it directly.

[kyzer1023]

Fixed in 16a5ccd. The scalar helpers, the Y_R..ROUND constants, ChannelOrder and Job are now #[cfg(any(target_arch = "x86_64", test))], and Job.width/height are x86_64 only, since only the AVX2 kernel reads them. Clippy with -D warnings --all-targets is clean on aarch64 Linux (lib test target included) and on x86_64.

[kyzer1023]

@greptileai review