Repository navigation
AGX correctness series — fast-track for upstream submission - #2
Closed
joshuaswarren wants to merge 10000 commits into
Closed
joshuaswarren wants to merge 10000 commits into
joshuaswarren wants to merge 10000 commits into
Conversation
Reviewed-by: Karol Herbst <kherbst@redhat.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44067>
Previously, the offsets for buffer_to_image and image_to_image gfx copies had opposite signs. Make them match so the code is a bit more regular. Reviewed-by: Caterina Shablia <nanokatze@gmail.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44067>
Reviewed-by: Caterina Shablia <nanokatze@gmail.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44067>
We were reading only vertex_input_state->attributes_valid, leaving at 0 the attributes defined by the Vertex Shader but not bound. Leading to crashes when the VS tried to read those. To fix it, we take the max of the two, and fill the unbound attributes with a constant 0 special format, same thing we are doing in panfrost. CC: mesa-stable Signed-off-by: Lorenzo Rossi <lorenzo.rossi@collabora.com> Reviewed-by: Boris Brezillon <boris.brezillon@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44084>
…h M3+ Reviewed-by: Arcady Goldmints-Orlov <arcady@lunarg.com> Signed-off-by: Aitor Camacho <aitor@lunarg.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44014>
Reviewed-by: Arcady Goldmints-Orlov <arcady@lunarg.com> Signed-off-by: Aitor Camacho <aitor@lunarg.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44100>
Doubles FPS with Godot's Third Person Shooter Demo ~1% improvement in aztec_high and ~9% improvement in aztec_normal Reviewed-by: Arcady Goldmints-Orlov <arcady@lunarg.com> Signed-off-by: Aitor Camacho <aitor@lunarg.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44100>
https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44027 fixed this too. Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44131>
Reviewed-by: Valentine Burley <valentine.burley@collabora.com> Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/43983>
Reviewed-by: Valentine Burley <valentine.burley@collabora.com> Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/43983>
The Linux kernel CI-tron uses is independent of the linux-firmware files, so we can start using the https://gitlab.freedesktop.org/gfx-ci/firmware repository for firmware ahead of switching to the gfx-ci/linux kernel. CI-tron jobs will still use the downstream kernel and kernel modules for the time being. This also updates the firmware version used in CI-tron, including on Raphael, which was previously using a very old version (`mec_fw_version=22`). The new version is `mec_fw_version=121`. Closes: https://gitlab.freedesktop.org/mesa/mesa/-/issues/14377 Signed-off-by: Valentine Burley <valentine.burley@collabora.com> Reviewed-by: Martin Roukala <martin.roukala@mupuf.org> Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/43983>
Signed-off-by: Martin Roukala (né Peres) <martin.roukala@mupuf.org> Reviewed-by: Valentine Burley <valentine.burley@collabora.com> Reviewed-by: Samuel Pitoiset <samuel.pitoiset@gmail.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/43983>
Previously cmd_fb_barrier would skip generating a barrier entirely on inherited secondary cmdbufs, because the layer count is not known at record-time and so the layer_count state is set to 0. To handle this, we need to store the layer count to memory in the primary cmdbuf and load it dynamically in the secondary cmdbuf. There is a comment about this situation in launch_indirect_draw suggesting we use some of the reserved words in the TD to store the layer count. I opted to use the subqueue ctx instead, just because using the reserved words sounds like a headache to debug if they ever get unreserved in new hardware. Signed-off-by: Olivia Lee <olivia.lee@collabora.com> Fixes: 34c0d9f (panvk: Use a full-screen barrier draw for FB barriers) Reviewed-by: Eric R. Smith <eric.smith@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/43960>
This brings the indirect draw path up to parity with the direct draw path. For both direct and indirect draws, we need additional changes to handle layered draws in secondary command buffers. Signed-off-by: Olivia Lee <olivia.lee@collabora.com> Reviewed-by: Eric R. Smith <eric.smith@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/43960>
counter_reg was assigned reg 17, despite none of the registers between 4 and 17 being assigned. This isn't problematic currently, but makes things awkward when adding more registers in this function. Signed-off-by: Olivia Lee <olivia.lee@collabora.com> Reviewed-by: Eric R. Smith <eric.smith@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/43960>
Similar to the FB barrier fix, we don't know the layer count at record-time and so need to load it dynamically from memory written by the primary cmdbuf. Signed-off-by: Olivia Lee <olivia.lee@collabora.com> Reviewed-by: Eric R. Smith <eric.smith@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/43960>
We can't support the corresponding VK_EXT_shader_viewport_layer_index extension yet because we don't support multiViewport, but in vulkan 1.2 layer outputs are an independent feature. Unfortunately, most of the CTS tests that touch this use the extension, and so won't be run. Signed-off-by: Olivia Lee <olivia.lee@collabora.com> Reviewed-by: Eric R. Smith <eric.smith@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/43960>
This is a no-op. Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44016>
It's possible to hit assertions and potential issues by emitting invalid cache flushes on compute. Using RADV_CMD_FLUSH_ALL_COMPUTE is just the opposite. Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44016>
Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44016>
Uses compute like other meta operations and move barriers in the caller, to separate the operation from the synchronization. Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44016>
It just can't be implemented on SDMA. After investigating various other ideas, it seems using a compute fallback is the only solution that prevents all potential edge cases to happen (like release to a foreign queue from transfer queue). Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44016>
When the src/dst are either GENERAL/COMPUTE, do the transitions on the most flexible queue. This also prevents a very rare issue when an application would try to release scanout images with display DCC to a foreign queue. Signed-off-by: Samuel Pitoiset <samuel.pitoiset@gmail.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44016>
PanVK refuses to load on GPUs where support is experimental unless this variable is set. Document the opt-in and its caveats, following the PowerVR docs. Closes: #16079 Acked-by: Boris Brezillon <boris.brezillon@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/43748>
Signed-off-by: Lorenzo Rossi <lorenzo.rossi@collabora.com> Reviewed-by: Faith Ekstrand <faith.ekstrand@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44120>
These are broken in hardware and it's been confirmed by Arm. Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44120>
This commit adds RawTestShaderBuilder, a builder similar to TestShaderBuilder but specialized for raw hardware tests, it does not call any compiler passes and expects the user to already write registers, handle legalization and message assignments. This is useful when we want to test the encoder/hardware itself instead of the compiler. Signed-off-by: Lorenzo Rossi <lorenzo.rossi@collabora.com> Reviewed-by: Jakob Sinclair <jakob.sinclair@arm.com> Reviewed-by: Faith Ekstrand <faith.ekstrand@collabora.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44054>
As initially reported by Faith, LD_PKA has problems with .b1 and .b3, using the low-level hardware tests we can now probe them directly to confirm the bug Reported-by: Faith Ekstrand <faith.ekstrand@collabora.com> Signed-off-by: Lorenzo Rossi <lorenzo.rossi@collabora.com> Reviewed-by: Jakob Sinclair <jakob.sinclair@arm.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44054>
The next commit will add more callers, and doing this separately makes that change easier to understand. Tested-by: Emre Cecanpunar <emreleno@gmail.com> Reviewed-by: Michael Cheng <michael.cheng@intel.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/43346>
I had previously thought that an early spill or fill would easily
propagate to all the fills that follow it in a block. That is often not
the case. Consider a situation like:
fill r30, XYZ
...
fill r31, XYZ
...
add r30, ...
fill r33, XYZ
The fill to r31 will become a move from r30, but the fill to r33 cannot
see the filled value in r30. Since the fill to r31 was already converted
to a move, the fill to r33 won't see it either.
The easy solution is to propagate each fill as it is encountered. It may
also be possible to achieve similar result by processing blocks in
reverse, but I believe that would be a lot more messy.
v2: Add a note about why block_progress is not set after the added calls
to propagate_fill_to_fill.
v3: Only copy-from-a-copy when the original value could not be used.
This prevents creation of dependency chains, and that slightly helps
post-RA scheduling (On MTL: Cycle count: 75697069507 -> 75693054670
(-0.01%); split: -0.01%, +0.00%). Recursion only occurs if the original
fill was invalidated as this is the only time it should be able to make
progress.
shader-db:
Lunar Lake, Meteor Lake, and DG2 had similar results. (Lunar Lake shown)
total instructions in shared programs: 17188150 -> 17188112 (<.01%)
instructions in affected programs: 2294 -> 2256 (-1.66%)
helped: 1 / HURT: 0
total cycles in shared programs: 856490262 -> 856489736 (<.01%)
cycles in affected programs: 17340 -> 16814 (-3.03%)
helped: 1 / HURT: 0
total fills in shared programs: 1607 -> 1588 (-1.18%)
fills in affected programs: 177 -> 158 (-10.73%)
helped: 1 / HURT: 0
No changes on Panther Lake or any non-LSC platforms.
fossil-db:
Panther Lake
Totals:
Instrs: 909927379 -> 909927225 (-0.00%)
CodeSize: 12376158304 -> 12376155120 (-0.00%)
Cycle count: 227969034447 -> 227967212803 (-0.00%); split: -0.00%, +0.00%
Spill count: 195314 -> 195313 (-0.00%)
Fill count: 34331 -> 34271 (-0.17%)
Totals from 17 (0.00% of 2033556) affected shaders:
Instrs: 379183 -> 379029 (-0.04%)
CodeSize: 5530848 -> 5527664 (-0.06%)
Cycle count: 1820995546 -> 1819173902 (-0.10%); split: -0.10%, +0.00%
Spill count: 11189 -> 11188 (-0.01%)
Fill count: 16958 -> 16898 (-0.35%)
Lunar Lake, Meteor Lake, and DG2 had similar results. (Lunar Lake shown)
Totals:
Instrs: 920856294 -> 920798318 (-0.01%)
CodeSize: 12608004368 -> 12606671760 (-0.01%); split: -0.01%, +0.00%
Cycle count: 240717410401 -> 240658191112 (-0.02%); split: -0.05%, +0.03%
Spill count: 2905016 -> 2904951 (-0.00%)
Fill count: 3923667 -> 3894895 (-0.73%)
Totals from 27581 (1.34% of 2053507) affected shaders:
Instrs: 136774419 -> 136716443 (-0.04%)
CodeSize: 1926081296 -> 1924748688 (-0.07%); split: -0.07%, +0.00%
Cycle count: 56973679209 -> 56914459920 (-0.10%); split: -0.23%, +0.12%
Spill count: 1823361 -> 1823296 (-0.00%)
Fill count: 2890829 -> 2862057 (-1.00%)
No changes on any non-LSC platforms.
Tested-by: Emre Cecanpunar <emreleno@gmail.com>
Reviewed-by: Michael Cheng <michael.cheng@intel.com>
Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/43346>
Signed-off-by: Rob Clark <rob.clark@oss.qualcomm.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44153>
Signed-off-by: Rob Clark <rob.clark@oss.qualcomm.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44153>
These two regs are the same until at least gen8. Signed-off-by: Rob Clark <rob.clark@oss.qualcomm.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44153>
Similar to fd_pkt7 on the gallium side. Signed-off-by: Rob Clark <rob.clark@oss.qualcomm.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44153>
We'll need this type further up in the file in the next commit. Signed-off-by: Rob Clark <rob.clark@oss.qualcomm.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44153>
Make saving/restoring regs to/from scratch ram a bit less open coded. Signed-off-by: Rob Clark <rob.clark@oss.qualcomm.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44153>
Signed-off-by: Rob Clark <rob.clark@oss.qualcomm.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44153>
Normally I'd call it reg_rmw() but the "REG" part of the name doesn't fit as well once the pkt was extended to support scratch-ram (a6xx). Signed-off-by: Rob Clark <rob.clark@oss.qualcomm.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44153>
I skipped tu_autotune to avoid going down the rabbit hole of fixing it to use CHIP template param properly. Signed-off-by: Rob Clark <rob.clark@oss.qualcomm.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44153>
Src/poll types ON_CHIP and SYSTEM_VARIABLE are not implemented yet, but also not needed yet. Signed-off-by: Rob Clark <rob.clark@oss.qualcomm.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44153>
Datagrams/packets up to 64 KB are guaranteed to transfer atomically. Fixes failures for stream size being small. Reviewed-by: David Gilhooley <djgilhooley@gmail.com> Part-of: <https://gitlab.freedesktop.org/mesa/mesa/-/merge_requests/44209>
Bug: (word >> s) & mask with a non-constant s returns zero on Honeykrisp
whenever s has any bit set above bit 4 (s >= 32). mlx-omarchy hit this as
the "dynamic shift-then-mask byte extraction" miscompile family: bool
scatter dropping writes across word lanes, select picking the wrong
operand under broadcast conditions, reduce_general's load_truthy reading
zero, masked_scatter's scan undercounting (s_scan[255] = 32, not 128).
Root cause: agx_nir_fuse_algebraic_late rewrites
iand(ushr(a, b), (1 << bits) - 1) -> ubitfield_extract(a, b, bits)
NIR's ushr uses only the low 5 bits of its count, and by this point the
main nir_opt_algebraic has already relied on that to strip explicit masks
(ushr(a, iand(b, 31)) -> ushr(a, b); ushr(a, ishl(iand(b, 3), 3)) ->
ushr(a, ishl(b, 3))). ubitfield_extract has no such wrap: an offset >= 32
is undefined in NIR and bfeil returns 0 for it on the hardware.
lower_sm5_shift, which masks the count of every surviving shift, runs
after the fusion and never sees the ushr it consumed.
Fix: fuse directly only when the count is a constant below 32; otherwise
fuse with an explicit iand(b, 31) on the offset. Same instruction count as
the unfused shift+and, and the mask CSEs with one the shader already had.
Verification on Apple M1 (G13G B1), standalone Vulkan compute reproducer
(20 variants, 1024 lanes each, host-compared):
stock Mesa 26.1.7: 9 of 16 base variants FAIL (byte_dyn,
byte_dyn_const, byte_dyn_gid, scan_dyn, scan_dyn_const, scan_dyn_rand,
select_bcast, select_dyn, shift_dyn_mask), plus all 4 boundary
variants FAIL (count exactly 32: 1016/1024 wrong; 63: 516/1024;
uniform 40: 1020/1024; divergent 8/40: 509/1024, only the 40 lanes).
this commit: 20 of 20 PASS, 0 mismatches.
mlx-omarchy suites, stock vs this commit (AGX_SIMDMAT=1), identical:
primitive 98/98 (2699373 assertions), reduce_ops 34/34 (7152),
runtime 40/40 (6307), indexing_ops 53/53 (4093), select_layout 11/11
(1908), scatter_determinism 21/21 (127).
Follow-up to 149c44d, addressing Dj's review note on the omacom board. 149c44d fixed the miscompile by masking the offset, but kept ubitfield_extract as the fused form. An out-of-range offset is undefined for that opcode, so a later NIR pass is entitled to treat the value as poison and optimise on that basis. The explicit iand makes the current code correct without making the representation correct: nothing stops a future pass from reasoning about the undefined case. ubfe is the SM5 form, defined to take both its offset and its width modulo 32, which is exactly ushr's behaviour and exactly what this fusion means. Fusing to it removes the undefined case rather than defending against it. The mask stays, because the hardware is the part that does not wrap: bfeil reads zero for an offset >= 32. It costs nothing, as it CSEs with the mask the shader had before the fusion. has_bfe is left unadvertised, so NIR still never forms ubfe on its own; agx_nir_fuse_algebraic_late remains the only producer, and instruction selection maps it to bfeil. Unlike ubitfield_extract, a width of zero reads as zero rather than meaning "all 32 bits", so it gets its own case rather than sharing the existing one. Generated code is unchanged. NIR for byte_dyn_const, fused form only: before: %13 = iand %11, 0x1f %14 = ubitfield_extract 0x10001, %13, 8 after: %13 = iand %11, 0x1f %14 = ubfe 0x10001, %13, 8 both: and $r2, r2, 31 bfeil r2, 0, u8, r2, 8 Verification on Apple M1 (G13G B1), this branch built as Mesa 26.3.0-devel, against the same tree with this commit reverted: 20 of 20 reproducer variants PASS on both, 0 mismatches. All 20 produce byte-identical disassembly on both, so this cannot move performance: it is the same machine code. mlx-omarchy suites on this commit (AGX_SIMDMAT=1), identical to the 149c44d receipt: primitive 98/98 (2699373 assertions), reduce_ops 34/34 (7152), runtime 40/40 (6307), indexing_ops 53/53 (4093), select_layout 11/11 (1908), scatter_determinism 21/21 (127).
…duction
Three fp32 rounding defects on Honeykrisp (Apple M1 G13G, Mesa 26.1.7)
that mlx-omarchy patches around in its shaders:
a) GLSL "/" was lowered by nir_opt_algebraic (lower_fdiv) to
a * frcp(b), and frcp was only Newton-refined when the instruction
was `precise`. Even with an exact reciprocal a * RN(1/b) is not
RN(a/b): 80/25 and 10/25 come out one ulp low, 6.55859375/0.030016447
one ulp high. Measured on 1008 pairs: 299 mismatches (298 at 1 ulp,
1 at 2 ulp), 1/b: 204 mismatches; 1M random pairs: 305792 mismatches.
fma() itself is IEEE (0/1000 mismatches), the mlx-omarchy note
blaming fma contraction was a misdiagnosis.
b) log(x) was flog2(x) * ln2 on the hardware log2, which is only
absolutely accurate: log(3) = 1.0986121892929077 instead of
1.0986123085021973, 2792/4107 mismatches on [0.5, 64] with errors up
to 357 ulp near 1 (log(1.0000001) = 1.65e-7 vs 1.0e-7); log2 had
2905/4107 mismatches.
c) sin/cos reduced the argument as fract(x / 2pi) in fp32, so the
reduced angle lost ulp(x) of precision: sin(3.0) 14 ulp off,
sin(1e5) 1.06e-3 absolute error, sin(1e8) = 0.0, sin(-1e-3) 571 ulp
off, sin(pi_f32) = 0, and the hardware sin_pt_2 core itself is 2 to
3 ulp off (cos(1e-3) 3 ulp).
The lowering moves to a new src/asahi/compiler/agx_nir_lower_math.c,
built only from the exact fp32 fma and marked no-fast-math so
nir_opt_algebraic cannot reassociate it:
- lower_fdiv is dropped from agx_nir_options; fdiv32 becomes Markstein
division (y = refined rcp, q = a y, r = fma(-b, q, a), q + r y) with
the NaN/zero guard keeping q for b = 0, overflow and signed zero.
frcp32 is always refined. fdiv16 keeps a * frcp(b).
- fmul(flog2(x), ln2) as emitted by nir_flog, and any remaining flog2,
become a double-float evaluation of log(m) = 2s + 2s^3 P(s^2) with
s = (m-1)/(m+1) kept as a pair, m folded into [sqrt(1/2), sqrt(2)),
e ln2 split hi/lo and Fast2Sum accumulation (about 40 bits before the
final rounding). Zero/subnormal/negative/inf/NaN are handled by bit
tests, not the hardware log2.
- fsin/fcos: Cody-Waite in radians with pi/2 in three pieces, the k C2
product and its subtraction as TwoProduct/TwoSum so the reduced
argument is a pair accurate to an ulp, for |x| < 2^22; an integer
Payne-Hanek (24-bit mantissa times a 128-bit window of 2/pi, 64
fraction bits kept) for |x| >= 2^22 up to FLT_MAX. The quadrant then
selects a Taylor polynomial in quadrant units (sin degree 13, cos
degree 14, leading terms as pairs, low part of the argument fed to
the derivative). The hardware sin_pt_1/sin_pt_2 core (nir_op_fsin_agx)
is no longer used by the lowering. fp16 sin/cos go through the fp32
path.
Verification (receipts/2026-09-08-precise-math.json, standalone Vulkan
compute reproducer in receipts/precise-math/):
- division: 1008-pair table incl. 80/25, 10/25, 1000 random: 0
mismatches (stock 299); 1/b 0 (stock 204); 1M random pairs 0
mismatches for both a/b and 1/b; b = 0, inf, NaN, overflow, flushed
subnormal and -0/5 = -0 match host IEEE.
- log: 4107 points on [0.5, 64] plus pinned exact cases: 0 mismatches
(stock 2792), log(3) = 1.0986123085021973; log2 0 (stock 2905);
1M random on [1e-37, 1e38]: log 3 at 1 ulp, log2 14 at 1 ulp,
rest exact.
- sin/cos at 1e5, 5e5, 1e6, 5e6, 1e8: all 0 ulp from the double
reference except cos(5e6) at 1 ulp (stock: 2.8e5 to 1.06e9 ulp);
1310 mixed points max 1 ulp; 1M random over [1e-3, 1e9] and [-10, 10]:
sin 42346 and cos 41184 at 1 ulp, rest exact, max 1 ulp (stock
>= 8 ulp for 627689 / 512816 of them).
- mlx-omarchy suites on the same Release tree, stock vs fixed:
primitive 98/98, eq_math 7/7, fast_ops 30/30, runtime 40/40,
reduce_ops 34/34 both; complex_ops 30/33 both with the same three
pre-existing cases (failed assertions 31 -> 29). The upstream
ops_tests log(3) pin through libmlx: FAIL on stock, PASS on fixed,
as do 80/25, 10/25 and sin/cos up to the 1e5 limit mlx enforces.
- timing: 16M-element elementwise dispatch is memory bound at ~2.45 ms
on both drivers (sin 2.474 -> 2.463 ms median, within spread). A
32-deep in-register chain shows the ALU cost: sin 6.64 -> 39.6 ms,
log 2.57 -> 34.8 ms, div 2.84 -> 7.24 ms, mul unchanged.
…it late Symptom: any shader converting fp32 to a 64-bit integer aborts in the backend with "Unhandled ALU op fdiv" (agx_compile.c agx_emit_alu); the mlx-omarchy suite omarchy_copy_offset_tests crashes on "dtype converting copies cover the integer family and bool". Fragment shaders with perspective-correct varyings hit the same path. Cause: 2022dde dropped lower_fdiv from the compiler options so nir_opt_algebraic no longer splits fdiv into fmul/frcp, and only ran agx_nir_lower_fdiv in agx_preprocess_nir. nir_lower_int64 (f2u64/f2i64: fdiv by 2^32, plus the frem that lower_fmod turns into a second fdiv) and agx_nir_lower_interpolation run after preprocessing, so their fdiv reached the backend, which has no fdiv. Fix: agx_nir_lower_fdiv_late runs in agx_optimize_nir after the int64 cleanup loop and lowers only fdiv, leaving frcp alone so the reciprocals built by the first pass and by the log lowering are not refined twice. Verification: hk/precise-math driver alone reproduced the abort on m1-test-host; with this commit omarchy_copy_offset_tests passes 25/25 (see the honeykrisp-omarchy integration receipt).
Three cases where the lowerings assumed IEEE denormal behaviour that the hardware does not provide. All three were reported by Dj from stress runs on an M2 Pro, and reproduce on an M1 (G13G B1). Division refined a * rcp(b) in place, so when the exact quotient sits just below the normal boundary the GPU zeroes the correction step's intermediate value and the step is silently skipped, giving a 1 ulp error: 18,382 wrong results over 1M random pairs. Divide the normalized mantissas instead and restore the exponent afterwards, so the refinement always runs on operands near one and the flush happens on the exact quotient, before rounding. The rounding decision is now integer: the half-ulp point is exact by construction, and the residual of the second Markstein step is exact by the fma contract, so the comparison against it decides the tie without a third refinement. log() lowered to hardware log2 times ln2 and the multiply was subject to contraction, so a shader computing precise log2(x) * ln2 got different bits depending on whether ln2 was a literal or read from a buffer, on 26.8% of inputs. log2 is now computed directly as a double-float series and log() is the one rounded multiply the source asks for. sin() returned its argument unchanged for a zero input. The comparison flushes, so a subnormal argument matched and the unflushed subnormal leaked out through the select. Return a signed zero. Verified on an M1 against an exact model over 262,144 pairs covering edge bit patterns and the binade boundaries: division, reciprocal and the literal-versus-buffer pair are exact, log and log2 are within 1 ulp. An independent exact-rational checker agrees over 489M cases. CTS float_controls goes from one failure to 1,146 pass, 0 fail; arithmetic is 311/311.
nir_builder_opcodes_h.py renders the intrinsic builders from a mako template that references FP_MATH_CTRL, but the index was never imported or passed to render(), so mako resolved the name to UNDEFINED and silently skipped the builder-defaulting block for every FP_MATH_CTRL intrinsic. NoContraction on cooperative-matrix operations was therefore dropped in every driver, not just this one, and the generated header is missing the twelve fp_math_ctrl defaulting blocks. Import the index and hand it to the template. The generated header grows exactly 24 lines, twelve two-line defaulting blocks.
joshuaswarren
force-pushed
the
upstream/correctness
branch
from
September 21, 2026 15:44
aa4fee0 to
728fe70
Compare
commented
Sep 30, 2026
Author
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
AGX correctness series — fast-track for upstream submission
Six patches, stock-main only, no perf-only changes, no internal receipts: the correctness subset of the [Honeykrisp integration PR](PR # (integration branch, opened alongside)) that Dj asked to have upstreamed, prepared per the agreed sequencing ("send the ubfe mod-32 correctness fix first; hold cooperative-matrix commits until the correctness work is through").
agx: keep the mod-32 shift count when fusing ushr+iand into ubfe— wrong-results fixagx: represent the fused shift-then-mask as ubfe, not ubitfield_extractagx: correctly rounded fdiv/frcp, faithful log/log2, exact sin/cos reductionagx: lower the fdiv that int64 lowering and the perspective divide emit lateagx: handle flush-to-zero in precise fp32 division, log and sinenir: pass FP_MATH_CTRL to the builder generator— affects every Mesa driver, not just AGX: the builder generator droppedFP_MATH_CTRL, so any driver using it compiled with wrong fast-math settingsIntended flow: merge or cherry-pick to freedesktop
mesa/main(an MR-readyformat-patchseries is atjoshuaswarren/mesa:upstream/correctness). The full Honeykrisp integration PR will rebase after this lands.All six verified on Apple M1 hardware (t8103) against the mlx-omarchy regression battery — the same commits that carry the pinned-inference results the team publishes.