Skip to content

AGX correctness series — fast-track for upstream submission - #2

Closed
joshuaswarren wants to merge 10000 commits into
omacom:mainfrom
joshuaswarren:upstream/correctness
Closed

joshuaswarren wants to merge 10000 commits into
omacom:mainfrom
joshuaswarren:upstream/correctness

Conversation

@joshuaswarren

Copy link
Copy Markdown

AGX correctness series — fast-track for upstream submission

Six patches, stock-main only, no perf-only changes, no internal receipts: the correctness subset of the [Honeykrisp integration PR](PR # (integration branch, opened alongside)) that Dj asked to have upstreamed, prepared per the agreed sequencing ("send the ubfe mod-32 correctness fix first; hold cooperative-matrix commits until the correctness work is through").

  1. agx: keep the mod-32 shift count when fusing ushr+iand into ubfe — wrong-results fix
  2. agx: represent the fused shift-then-mask as ubfe, not ubitfield_extract
  3. agx: correctly rounded fdiv/frcp, faithful log/log2, exact sin/cos reduction
  4. agx: lower the fdiv that int64 lowering and the perspective divide emit late
  5. agx: handle flush-to-zero in precise fp32 division, log and sine
  6. nir: pass FP_MATH_CTRL to the builder generator — affects every Mesa driver, not just AGX: the builder generator dropped FP_MATH_CTRL, so any driver using it compiled with wrong fast-math settings

Intended flow: merge or cherry-pick to freedesktop mesa/main (an MR-ready format-patch series is at joshuaswarren/mesa:upstream/correctness). The full Honeykrisp integration PR will rebase after this lands.

All six verified on Apple M1 hardware (t8103) against the mlx-omarchy regression battery — the same commits that carry the pinned-inference results the team publishes.

drawoc and others added 30 commits August 31, 2026 23:13
Previously, the offsets for buffer_to_image and image_to_image gfx
copies had opposite signs. Make them match so the code is a bit more
regular.

Reviewed-by: Caterina Shablia <nanokatze@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44067>
We were reading only vertex_input_state->attributes_valid, leaving at 0
the attributes defined by the Vertex Shader but not bound.  Leading to
crashes when the VS tried to read those.

To fix it, we take the max of the two, and fill the unbound attributes
with a constant 0 special format, same thing we are doing in panfrost.

CC: mesa-stable
Signed-off-by: Lorenzo Rossi <lorenzo.rossi@collabora.com>
Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44084>
…h M3+

Reviewed-by: Arcady Goldmints-Orlov <arcady@lunarg.com>
Signed-off-by: Aitor Camacho <aitor@lunarg.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44014>
Reviewed-by: Arcady Goldmints-Orlov <arcady@lunarg.com>
Signed-off-by: Aitor Camacho <aitor@lunarg.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44100>
Doubles FPS with Godot's Third Person Shooter Demo
~1% improvement in aztec_high and ~9% improvement in aztec_normal

Reviewed-by: Arcady Goldmints-Orlov <arcady@lunarg.com>
Signed-off-by: Aitor Camacho <aitor@lunarg.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44100>
Reviewed-by: Valentine Burley <valentine.burley@collabora.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/43983>
Reviewed-by: Valentine Burley <valentine.burley@collabora.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/43983>
The Linux kernel CI-tron uses is independent of the linux-firmware files,
so we can start using the https://gitlab.freedesktop.org/gfx-ci/firmware
repository for firmware ahead of switching to the gfx-ci/linux kernel.
CI-tron jobs will still use the downstream kernel and kernel modules for
the time being.

This also updates the firmware version used in CI-tron, including on
Raphael, which was previously using a very old version
(`mec_fw_version=22`). The new version is `mec_fw_version=121`.

Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/14377
Signed-off-by: Valentine Burley <valentine.burley@collabora.com>
Reviewed-by: Martin Roukala <martin.roukala@mupuf.org>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/43983>
Signed-off-by: Martin Roukala (né Peres) <martin.roukala@mupuf.org>
Reviewed-by: Valentine Burley <valentine.burley@collabora.com>
Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/43983>
Previously cmd_fb_barrier would skip generating a barrier entirely on
inherited secondary cmdbufs, because the layer count is not known at
record-time and so the layer_count state is set to 0.

To handle this, we need to store the layer count to memory in the
primary cmdbuf and load it dynamically in the secondary cmdbuf. There is
a comment about this situation in launch_indirect_draw suggesting we use
some of the reserved words in the TD to store the layer count. I opted
to use the subqueue ctx instead, just because using the reserved words
sounds like a headache to debug if they ever get unreserved in new
hardware.

Signed-off-by: Olivia Lee <olivia.lee@collabora.com>
Fixes: 34c0d9f (panvk: Use a full-screen barrier draw for FB barriers)
Reviewed-by: Eric R. Smith <eric.smith@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/43960>
This brings the indirect draw path up to parity with the direct draw
path. For both direct and indirect draws, we need additional changes to
handle layered draws in secondary command buffers.

Signed-off-by: Olivia Lee <olivia.lee@collabora.com>
Reviewed-by: Eric R. Smith <eric.smith@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/43960>
counter_reg was assigned reg 17, despite none of the registers between 4
and 17 being assigned. This isn't problematic currently, but makes
things awkward when adding more registers in this function.

Signed-off-by: Olivia Lee <olivia.lee@collabora.com>
Reviewed-by: Eric R. Smith <eric.smith@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/43960>
Similar to the FB barrier fix, we don't know the layer count at
record-time and so need to load it dynamically from memory written by
the primary cmdbuf.

Signed-off-by: Olivia Lee <olivia.lee@collabora.com>
Reviewed-by: Eric R. Smith <eric.smith@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/43960>
We can't support the corresponding VK_EXT_shader_viewport_layer_index
extension yet because we don't support multiViewport, but in vulkan 1.2
layer outputs are an independent feature. Unfortunately, most of the CTS
tests that touch this use the extension, and so won't be run.

Signed-off-by: Olivia Lee <olivia.lee@collabora.com>
Reviewed-by: Eric R. Smith <eric.smith@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/43960>
This is a no-op.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44016>
It's possible to hit assertions and potential issues by emitting
invalid cache flushes on compute. Using RADV_CMD_FLUSH_ALL_COMPUTE is
just the opposite.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44016>
Uses compute like other meta operations and move barriers in the
caller, to separate the operation from the synchronization.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44016>
It just can't be implemented on SDMA. After investigating various other
ideas, it seems using a compute fallback is the only solution that
prevents all potential edge cases to happen (like release to a foreign
queue from transfer queue).

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44016>
When the src/dst are either GENERAL/COMPUTE, do the transitions on
the most flexible queue.

This also prevents a very rare issue when an application would try to
release scanout images with display DCC to a foreign queue.

Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44016>
PanVK refuses to load on GPUs where support is experimental unless
this variable is set. Document the opt-in and its caveats, following
the PowerVR docs.

Closes: #16079

Acked-by: Boris Brezillon <boris.brezillon@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/43748>
Signed-off-by: Lorenzo Rossi <lorenzo.rossi@collabora.com>
Reviewed-by: Faith Ekstrand <faith.ekstrand@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44120>
These are broken in hardware and it's been confirmed by Arm.

Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44120>
Lorenzo Rossi and others added 21 commits September 3, 2026 21:24
This commit adds RawTestShaderBuilder, a builder similar to
TestShaderBuilder but specialized for raw hardware tests, it does not
call any compiler passes and expects the user to already write registers,
handle legalization and message assignments.  This is useful when we want
to test the encoder/hardware itself instead of the compiler.

Signed-off-by: Lorenzo Rossi <lorenzo.rossi@collabora.com>
Reviewed-by: Jakob Sinclair <jakob.sinclair@arm.com>
Reviewed-by: Faith Ekstrand <faith.ekstrand@collabora.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44054>
As initially reported by Faith, LD_PKA has problems with .b1 and .b3,
using the low-level hardware tests we can now probe them directly to
confirm the bug

Reported-by: Faith Ekstrand <faith.ekstrand@collabora.com>
Signed-off-by: Lorenzo Rossi <lorenzo.rossi@collabora.com>
Reviewed-by: Jakob Sinclair <jakob.sinclair@arm.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44054>
The next commit will add more callers, and doing this separately makes
that change easier to understand.

Tested-by: Emre Cecanpunar <emreleno@gmail.com>
Reviewed-by: Michael Cheng <michael.cheng@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/43346>
I had previously thought that an early spill or fill would easily
propagate to all the fills that follow it in a block. That is often not
the case. Consider a situation like:

    fill    r30, XYZ
    ...
    fill    r31, XYZ
    ...
    add     r30, ...
    fill    r33, XYZ

The fill to r31 will become a move from r30, but the fill to r33 cannot
see the filled value in r30. Since the fill to r31 was already converted
to a move, the fill to r33 won't see it either.

The easy solution is to propagate each fill as it is encountered. It may
also be possible to achieve similar result by processing blocks in
reverse, but I believe that would be a lot more messy.

v2: Add a note about why block_progress is not set after the added calls
to propagate_fill_to_fill.

v3: Only copy-from-a-copy when the original value could not be used.
This prevents creation of dependency chains, and that slightly helps
post-RA scheduling (On MTL: Cycle count: 75697069507 -> 75693054670
(-0.01%); split: -0.01%, +0.00%). Recursion only occurs if the original
fill was invalidated as this is the only time it should be able to make
progress.

shader-db:

Lunar Lake, Meteor Lake, and DG2 had similar results. (Lunar Lake shown)
total instructions in shared programs: 17188150 -> 17188112 (<.01%)
instructions in affected programs: 2294 -> 2256 (-1.66%)
helped: 1 / HURT: 0

total cycles in shared programs: 856490262 -> 856489736 (<.01%)
cycles in affected programs: 17340 -> 16814 (-3.03%)
helped: 1 / HURT: 0

total fills in shared programs: 1607 -> 1588 (-1.18%)
fills in affected programs: 177 -> 158 (-10.73%)
helped: 1 / HURT: 0

No changes on Panther Lake or any non-LSC platforms.

fossil-db:

Panther Lake
Totals:
Instrs: 909927379 -> 909927225 (-0.00%)
CodeSize: 12376158304 -> 12376155120 (-0.00%)
Cycle count: 227969034447 -> 227967212803 (-0.00%); split: -0.00%, +0.00%
Spill count: 195314 -> 195313 (-0.00%)
Fill count: 34331 -> 34271 (-0.17%)

Totals from 17 (0.00% of 2033556) affected shaders:
Instrs: 379183 -> 379029 (-0.04%)
CodeSize: 5530848 -> 5527664 (-0.06%)
Cycle count: 1820995546 -> 1819173902 (-0.10%); split: -0.10%, +0.00%
Spill count: 11189 -> 11188 (-0.01%)
Fill count: 16958 -> 16898 (-0.35%)

Lunar Lake, Meteor Lake, and DG2 had similar results. (Lunar Lake shown)
Totals:
Instrs: 920856294 -> 920798318 (-0.01%)
CodeSize: 12608004368 -> 12606671760 (-0.01%); split: -0.01%, +0.00%
Cycle count: 240717410401 -> 240658191112 (-0.02%); split: -0.05%, +0.03%
Spill count: 2905016 -> 2904951 (-0.00%)
Fill count: 3923667 -> 3894895 (-0.73%)

Totals from 27581 (1.34% of 2053507) affected shaders:
Instrs: 136774419 -> 136716443 (-0.04%)
CodeSize: 1926081296 -> 1924748688 (-0.07%); split: -0.07%, +0.00%
Cycle count: 56973679209 -> 56914459920 (-0.10%); split: -0.23%, +0.12%
Spill count: 1823361 -> 1823296 (-0.00%)
Fill count: 2890829 -> 2862057 (-1.00%)

No changes on any non-LSC platforms.

Tested-by: Emre Cecanpunar <emreleno@gmail.com>
Reviewed-by: Michael Cheng <michael.cheng@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/43346>
Signed-off-by: Rob Clark <rob.clark@oss.qualcomm.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44153>
These two regs are the same until at least gen8.

Signed-off-by: Rob Clark <rob.clark@oss.qualcomm.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44153>
Similar to fd_pkt7 on the gallium side.

Signed-off-by: Rob Clark <rob.clark@oss.qualcomm.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44153>
We'll need this type further up in the file in the next commit.

Signed-off-by: Rob Clark <rob.clark@oss.qualcomm.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44153>
Make saving/restoring regs to/from scratch ram a bit less open coded.

Signed-off-by: Rob Clark <rob.clark@oss.qualcomm.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44153>
Normally I'd call it reg_rmw() but the "REG" part of the name doesn't
fit as well once the pkt was extended to support scratch-ram (a6xx).

Signed-off-by: Rob Clark <rob.clark@oss.qualcomm.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44153>
I skipped tu_autotune to avoid going down the rabbit hole of fixing it
to use CHIP template param properly.

Signed-off-by: Rob Clark <rob.clark@oss.qualcomm.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44153>
Src/poll types ON_CHIP and SYSTEM_VARIABLE are not implemented yet, but
also not needed yet.

Signed-off-by: Rob Clark <rob.clark@oss.qualcomm.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44153>
Datagrams/packets up to 64 KB are guaranteed to transfer
atomically.  Fixes failures for stream size being small.

Reviewed-by: David Gilhooley <djgilhooley@gmail.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44209>
Bug: (word >> s) & mask with a non-constant s returns zero on Honeykrisp
whenever s has any bit set above bit 4 (s >= 32). mlx-omarchy hit this as
the "dynamic shift-then-mask byte extraction" miscompile family: bool
scatter dropping writes across word lanes, select picking the wrong
operand under broadcast conditions, reduce_general's load_truthy reading
zero, masked_scatter's scan undercounting (s_scan[255] = 32, not 128).

Root cause: agx_nir_fuse_algebraic_late rewrites
   iand(ushr(a, b), (1 << bits) - 1)  ->  ubitfield_extract(a, b, bits)
NIR's ushr uses only the low 5 bits of its count, and by this point the
main nir_opt_algebraic has already relied on that to strip explicit masks
(ushr(a, iand(b, 31)) -> ushr(a, b); ushr(a, ishl(iand(b, 3), 3)) ->
ushr(a, ishl(b, 3))). ubitfield_extract has no such wrap: an offset >= 32
is undefined in NIR and bfeil returns 0 for it on the hardware.
lower_sm5_shift, which masks the count of every surviving shift, runs
after the fusion and never sees the ushr it consumed.

Fix: fuse directly only when the count is a constant below 32; otherwise
fuse with an explicit iand(b, 31) on the offset. Same instruction count as
the unfused shift+and, and the mask CSEs with one the shader already had.

Verification on Apple M1 (G13G B1), standalone Vulkan compute reproducer
(20 variants, 1024 lanes each, host-compared):
  stock Mesa 26.1.7:  9 of 16 base variants FAIL (byte_dyn,
    byte_dyn_const, byte_dyn_gid, scan_dyn, scan_dyn_const, scan_dyn_rand,
    select_bcast, select_dyn, shift_dyn_mask), plus all 4 boundary
    variants FAIL (count exactly 32: 1016/1024 wrong; 63: 516/1024;
    uniform 40: 1020/1024; divergent 8/40: 509/1024, only the 40 lanes).
  this commit:        20 of 20 PASS, 0 mismatches.
mlx-omarchy suites, stock vs this commit (AGX_SIMDMAT=1), identical:
  primitive 98/98 (2699373 assertions), reduce_ops 34/34 (7152),
  runtime 40/40 (6307), indexing_ops 53/53 (4093), select_layout 11/11
  (1908), scatter_determinism 21/21 (127).
Follow-up to 149c44d, addressing Dj's review note on the omacom board.

149c44d fixed the miscompile by masking the offset, but kept
ubitfield_extract as the fused form. An out-of-range offset is undefined
for that opcode, so a later NIR pass is entitled to treat the value as
poison and optimise on that basis. The explicit iand makes the current
code correct without making the representation correct: nothing stops a
future pass from reasoning about the undefined case.

ubfe is the SM5 form, defined to take both its offset and its width
modulo 32, which is exactly ushr's behaviour and exactly what this fusion
means. Fusing to it removes the undefined case rather than defending
against it.

The mask stays, because the hardware is the part that does not wrap:
bfeil reads zero for an offset >= 32. It costs nothing, as it CSEs with
the mask the shader had before the fusion.

has_bfe is left unadvertised, so NIR still never forms ubfe on its own;
agx_nir_fuse_algebraic_late remains the only producer, and instruction
selection maps it to bfeil. Unlike ubitfield_extract, a width of zero
reads as zero rather than meaning "all 32 bits", so it gets its own case
rather than sharing the existing one.

Generated code is unchanged. NIR for byte_dyn_const, fused form only:

  before: %13 = iand %11, 0x1f
          %14 = ubitfield_extract 0x10001, %13, 8
  after:  %13 = iand %11, 0x1f
          %14 = ubfe 0x10001, %13, 8

  both:   and   $r2, r2, 31
          bfeil r2, 0, u8, r2, 8

Verification on Apple M1 (G13G B1), this branch built as
Mesa 26.3.0-devel, against the same tree with this commit reverted:

  20 of 20 reproducer variants PASS on both, 0 mismatches.
  All 20 produce byte-identical disassembly on both, so this cannot
  move performance: it is the same machine code.

mlx-omarchy suites on this commit (AGX_SIMDMAT=1), identical to the
149c44d receipt: primitive 98/98 (2699373 assertions), reduce_ops 34/34
(7152), runtime 40/40 (6307), indexing_ops 53/53 (4093), select_layout
11/11 (1908), scatter_determinism 21/21 (127).
…duction

Three fp32 rounding defects on Honeykrisp (Apple M1 G13G, Mesa 26.1.7)
that mlx-omarchy patches around in its shaders:

  a) GLSL "/" was lowered by nir_opt_algebraic (lower_fdiv) to
     a * frcp(b), and frcp was only Newton-refined when the instruction
     was `precise`. Even with an exact reciprocal a * RN(1/b) is not
     RN(a/b): 80/25 and 10/25 come out one ulp low, 6.55859375/0.030016447
     one ulp high. Measured on 1008 pairs: 299 mismatches (298 at 1 ulp,
     1 at 2 ulp), 1/b: 204 mismatches; 1M random pairs: 305792 mismatches.
     fma() itself is IEEE (0/1000 mismatches), the mlx-omarchy note
     blaming fma contraction was a misdiagnosis.
  b) log(x) was flog2(x) * ln2 on the hardware log2, which is only
     absolutely accurate: log(3) = 1.0986121892929077 instead of
     1.0986123085021973, 2792/4107 mismatches on [0.5, 64] with errors up
     to 357 ulp near 1 (log(1.0000001) = 1.65e-7 vs 1.0e-7); log2 had
     2905/4107 mismatches.
  c) sin/cos reduced the argument as fract(x / 2pi) in fp32, so the
     reduced angle lost ulp(x) of precision: sin(3.0) 14 ulp off,
     sin(1e5) 1.06e-3 absolute error, sin(1e8) = 0.0, sin(-1e-3) 571 ulp
     off, sin(pi_f32) = 0, and the hardware sin_pt_2 core itself is 2 to
     3 ulp off (cos(1e-3) 3 ulp).

The lowering moves to a new src/asahi/compiler/agx_nir_lower_math.c,
built only from the exact fp32 fma and marked no-fast-math so
nir_opt_algebraic cannot reassociate it:

  - lower_fdiv is dropped from agx_nir_options; fdiv32 becomes Markstein
    division (y = refined rcp, q = a y, r = fma(-b, q, a), q + r y) with
    the NaN/zero guard keeping q for b = 0, overflow and signed zero.
    frcp32 is always refined. fdiv16 keeps a * frcp(b).
  - fmul(flog2(x), ln2) as emitted by nir_flog, and any remaining flog2,
    become a double-float evaluation of log(m) = 2s + 2s^3 P(s^2) with
    s = (m-1)/(m+1) kept as a pair, m folded into [sqrt(1/2), sqrt(2)),
    e ln2 split hi/lo and Fast2Sum accumulation (about 40 bits before the
    final rounding). Zero/subnormal/negative/inf/NaN are handled by bit
    tests, not the hardware log2.
  - fsin/fcos: Cody-Waite in radians with pi/2 in three pieces, the k C2
    product and its subtraction as TwoProduct/TwoSum so the reduced
    argument is a pair accurate to an ulp, for |x| < 2^22; an integer
    Payne-Hanek (24-bit mantissa times a 128-bit window of 2/pi, 64
    fraction bits kept) for |x| >= 2^22 up to FLT_MAX. The quadrant then
    selects a Taylor polynomial in quadrant units (sin degree 13, cos
    degree 14, leading terms as pairs, low part of the argument fed to
    the derivative). The hardware sin_pt_1/sin_pt_2 core (nir_op_fsin_agx)
    is no longer used by the lowering. fp16 sin/cos go through the fp32
    path.

Verification (receipts/2026-09-08-precise-math.json, standalone Vulkan
compute reproducer in receipts/precise-math/):

  - division: 1008-pair table incl. 80/25, 10/25, 1000 random: 0
    mismatches (stock 299); 1/b 0 (stock 204); 1M random pairs 0
    mismatches for both a/b and 1/b; b = 0, inf, NaN, overflow, flushed
    subnormal and -0/5 = -0 match host IEEE.
  - log: 4107 points on [0.5, 64] plus pinned exact cases: 0 mismatches
    (stock 2792), log(3) = 1.0986123085021973; log2 0 (stock 2905);
    1M random on [1e-37, 1e38]: log 3 at 1 ulp, log2 14 at 1 ulp,
    rest exact.
  - sin/cos at 1e5, 5e5, 1e6, 5e6, 1e8: all 0 ulp from the double
    reference except cos(5e6) at 1 ulp (stock: 2.8e5 to 1.06e9 ulp);
    1310 mixed points max 1 ulp; 1M random over [1e-3, 1e9] and [-10, 10]:
    sin 42346 and cos 41184 at 1 ulp, rest exact, max 1 ulp (stock
    >= 8 ulp for 627689 / 512816 of them).
  - mlx-omarchy suites on the same Release tree, stock vs fixed:
    primitive 98/98, eq_math 7/7, fast_ops 30/30, runtime 40/40,
    reduce_ops 34/34 both; complex_ops 30/33 both with the same three
    pre-existing cases (failed assertions 31 -> 29). The upstream
    ops_tests log(3) pin through libmlx: FAIL on stock, PASS on fixed,
    as do 80/25, 10/25 and sin/cos up to the 1e5 limit mlx enforces.
  - timing: 16M-element elementwise dispatch is memory bound at ~2.45 ms
    on both drivers (sin 2.474 -> 2.463 ms median, within spread). A
    32-deep in-register chain shows the ALU cost: sin 6.64 -> 39.6 ms,
    log 2.57 -> 34.8 ms, div 2.84 -> 7.24 ms, mul unchanged.
…it late

Symptom: any shader converting fp32 to a 64-bit integer aborts in the
backend with "Unhandled ALU op fdiv" (agx_compile.c agx_emit_alu); the
mlx-omarchy suite omarchy_copy_offset_tests crashes on "dtype converting
copies cover the integer family and bool". Fragment shaders with
perspective-correct varyings hit the same path.

Cause: 2022dde dropped lower_fdiv from the compiler options so
nir_opt_algebraic no longer splits fdiv into fmul/frcp, and only ran
agx_nir_lower_fdiv in agx_preprocess_nir. nir_lower_int64 (f2u64/f2i64:
fdiv by 2^32, plus the frem that lower_fmod turns into a second fdiv) and
agx_nir_lower_interpolation run after preprocessing, so their fdiv reached
the backend, which has no fdiv.

Fix: agx_nir_lower_fdiv_late runs in agx_optimize_nir after the int64
cleanup loop and lowers only fdiv, leaving frcp alone so the reciprocals
built by the first pass and by the log lowering are not refined twice.

Verification: hk/precise-math driver alone reproduced the abort on m1-test-host;
with this commit omarchy_copy_offset_tests passes 25/25 (see the
honeykrisp-omarchy integration receipt).
Three cases where the lowerings assumed IEEE denormal behaviour that the
hardware does not provide. All three were reported by Dj from stress
runs on an M2 Pro, and reproduce on an M1 (G13G B1).

Division refined a * rcp(b) in place, so when the exact quotient sits
just below the normal boundary the GPU zeroes the correction step's
intermediate value and the step is silently skipped, giving a 1 ulp
error: 18,382 wrong results over 1M random pairs. Divide the normalized
mantissas instead and restore the exponent afterwards, so the refinement
always runs on operands near one and the flush happens on the exact
quotient, before rounding. The rounding decision is now integer: the
half-ulp point is exact by construction, and the residual of the second
Markstein step is exact by the fma contract, so the comparison against
it decides the tie without a third refinement.

log() lowered to hardware log2 times ln2 and the multiply was subject to
contraction, so a shader computing precise log2(x) * ln2 got different
bits depending on whether ln2 was a literal or read from a buffer, on
26.8% of inputs. log2 is now computed directly as a double-float series
and log() is the one rounded multiply the source asks for.

sin() returned its argument unchanged for a zero input. The comparison
flushes, so a subnormal argument matched and the unflushed subnormal
leaked out through the select. Return a signed zero.

Verified on an M1 against an exact model over 262,144 pairs covering
edge bit patterns and the binade boundaries: division, reciprocal and
the literal-versus-buffer pair are exact, log and log2 are within 1 ulp.
An independent exact-rational checker agrees over 489M cases. CTS
float_controls goes from one failure to 1,146 pass, 0 fail; arithmetic
is 311/311.
nir_builder_opcodes_h.py renders the intrinsic builders from a mako
template that references FP_MATH_CTRL, but the index was never imported
or passed to render(), so mako resolved the name to UNDEFINED and
silently skipped the builder-defaulting block for every FP_MATH_CTRL
intrinsic. NoContraction on cooperative-matrix operations was therefore
dropped in every driver, not just this one, and the generated header is
missing the twelve fp_math_ctrl defaulting blocks.

Import the index and hand it to the template. The generated header grows
exactly 24 lines, twelve two-line defaulting blocks.
@joshuaswarren

ghost commented Sep 30, 2026

Copy link
Copy Markdown
Author

Superseded by the stack #3 → #4 → #5. Review it in that order.

Where the commits of this pull request went:

Closing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.