Skip to content

highbd_sad_sse2.asm: fix XMM register count for sad_skip - #5474

Merged
urvangjoshi merged 1 commit into
AOMediaCodec:mainfrom
Z3n1th-1:fix-win64-xmm7
Oct 23, 2026
Merged

urvangjoshi merged 1 commit into
AOMediaCodec:mainfrom
Z3n1th-1:fix-win64-xmm7

Conversation

@Z3n1th-1

@Z3n1th-1 Z3n1th-1 commented Oct 2, 2026

Copy link
Copy Markdown
Contributor

The skip-row declarations in HIGH_SAD_FN hard-code the XMM register count to 7. HIGH_SAD8XN passes 8 and uses xmm7, so the generated functions fail to preserve this nonvolatile register on Win64. This can corrupt the caller's state and cause encoder crashes.

Use the fifth macro argument (%5) in both skip-row declarations, matching the normal and average SAD declarations. Other block widths retain their default count of 7. The SAD calculations are unchanged.

Equivalent to James Zern's libaom fix from 2021.

Manually tested on Windows 11 x64 with MSYS2 CLANG64: a previously crashing 1080p 10-bit single-frame encode completed with IntraBC enabled. A complete 72-frame 856x480 encode produced a byte-identical IVF file compared with the original build. The unit-test suite was not run.

@urvangjoshi

Copy link
Copy Markdown
Contributor

Marking as draft until CLA is signed.

@urvangjoshi
urvangjoshi marked this pull request as draft October 8, 2026 18:33
@Z3n1th-1

Copy link
Copy Markdown
Contributor Author

Hi @urvangjoshi, I signed the AOM Individual CLA on October 2 and received the DocuSign confirmation that all parties had completed signing. However, this PR still has the needs-cla label.

Could you please check whether my CLA is correctly associated with my GitHub account (Z3n1th-1) and re-run the CLA check if needed? Thank you!

@urvangjoshi
urvangjoshi marked this pull request as ready for review October 21, 2026 16:57
@urvangjoshi

Copy link
Copy Markdown
Contributor

Hi @urvangjoshi, I signed the AOM Individual CLA on October 2 and received the DocuSign confirmation that all parties had completed signing. However, this PR still has the needs-cla label.

Could you please check whether my CLA is correctly associated with my GitHub account (Z3n1th-1) and re-run the CLA check if needed? Thank you!

Thanks for taking care of the CLA!
Currently, your email on the commit shows up as *@@users.noreply.github.com, so CLA bot can't verify it matches the email in CLA. Could you amend your commit (to have proper email address) and then force-push (don't create merge commit!).

@Z3n1th-1

Copy link
Copy Markdown
Contributor Author

Hi @urvangjoshi, I amended the commit to use the same email address as my signed CLA and force-pushed it. I also checked that the email and GitHub username in the CLA are correct, but the CLA check still seems to be failing.

@urvangjoshi
urvangjoshi marked this pull request as draft October 23, 2026 17:30
@urvangjoshi
urvangjoshi marked this pull request as ready for review October 23, 2026 17:30
@urvangjoshi

Copy link
Copy Markdown
Contributor

Hi @urvangjoshi, I amended the commit to use the same email address as my signed CLA and force-pushed it. I also checked that the email and GitHub username in the CLA are correct, but the CLA check still seems to be failing.

There is a manual step involved to whitelist the email. I just made that update, so CLA label is now green.

@urvangjoshi

Copy link
Copy Markdown
Contributor

@jjustiss-apple : pls review.

@jjustiss-apple

Copy link
Copy Markdown
Contributor

LGTM on inspection. I'm not able to cross check as I don't have a windows setup.

Thanks for the fix @Z3n1th-1 !

@urvangjoshi
urvangjoshi merged commit e0e3aa8 into AOMediaCodec:main Oct 23, 2026
8 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants