Skip to content

Round forward weights of user-defined QMs - #5591

Open
oddharsh wants to merge 2 commits into
AOMediaCodec:mainfrom
oddharsh:round-qm-forward-weights
Open

oddharsh wants to merge 2 commits into
AOMediaCodec:mainfrom
oddharsh:round-qm-forward-weights

Conversation

@oddharsh

Copy link
Copy Markdown

Currently calc_wt_matrix() truncates 1024 / iwt, while every predefined forward weight is that value rounded, so a user-defined copy of a predefined QM quantizes differently from it: 1342 of 3840 luma and chroma weights at 8x8, 8x4 and 4x8 come out one lower. Found while sending the predefined matrices back through a user-defined QM in an encoder experiment; #5586 covers the 4x4 side of that.

This rounds instead. The decoder only uses iwt, so decoded output stays conformant, and encodes without user-defined QMs don't change.

QmTest.UserDefinedCopyOfPredefinedHasSameWeights fails without the change (weight 26 against 27 at level 0) and passes with it, and the rest of QmTest passes. The QMTest cases that encode with user-defined QMs (QM5 to QM11) pass at speed 6; I didn't run speed 5. av2-enc has the same line; happy to send it there too if that's preferred.

Red-green output
base main b09013498, head e6eac1c4c
red (av2/common/quant_common.c reverted), 3 of 3 runs:
  qm_test.cc:333: Failure
    got[i]  Which is: '\x1A' (26)
    want[i] Which is: '\x1B' (27)
  level 0 plane 0 tx_size 1 index 2
  [  FAILED  ] QmTest.UserDefinedCopyOfPredefinedHasSameWeights
green (head), 3 of 3 runs:
  [       OK ] QmTest.UserDefinedCopyOfPredefinedHasSameWeights
suite on head: --gtest_filter='Qm*', exit 0

AI-assisted. Recon, red-green check, and PR text by houseguest.

calc_wt_matrix() truncated 1024 / iwt, while every predefined forward
weight is that value rounded, so a user-defined copy of a predefined QM
quantized differently from it (1342 of 3840 luma and chroma weights at
8x8, 8x4 and 4x8 came out one lower). Round instead; the decoder only
uses iwt.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants