Skip to content

feat(vulkan): native kCausalConv1dFwd, the GDN prefill conv - #3368

Draft
allenk wants to merge 1 commit into
mudler:mainfrom
allenk:row/BACKEND-VULKAN-CONVFWD
Draft

allenk wants to merge 1 commit into
mudler:mainfrom
allenk:row/BACKEND-VULKAN-CONVFWD

Conversation

@allenk

@allenk allenk commented Oct 1, 2026 •

Copy link
Copy Markdown

Row

BACKEND-VULKAN, branch row/BACKEND-VULKAN-CONVFWD; the same sub-row naming
as row/BACKEND-VULKAN-GDN (#145) and row/BACKEND-VULKAN-GDN-CORE (#152).

Before starting

  • Issue/PR search and existing claim: no open PR, claim or issue names
    kCausalConv1dFwd on Vulkan. Local issue filed and closed in this PR:
    .agents/issues/BACKEND-VULKAN/ISSUE-LOCAL-01M3V01R90QQTQ08BWSBM7AGN2.md
    (Kind: gap).
  • Pull request shape selected at row claim: one op, one shader, its tests and
    docs; helper PR.
  • Roadmap or matrix row: .agents/backend-matrix.md BACKEND-VULKAN; VK-G in
    .agents/specs/vulkan-full-support.md. scripts/ready-for-helper.py --check-local: structure OK, live claims not verified locally.
  • Exact current-code and test/evidence anchors inspected: the GDN-family block
    comment in src/vt/vulkan/vulkan_ops.cpp that left this op "for a follow-up
    rather than guessed at here" and named the two safe dispatch shapes;
    src/vt/cpu/cpu_ops.cpp CausalConv1dFwdKernel; src/vt/ops.cpp
    CausalConv1dFwd / CheckConvCommon; the decode-update and compressed-state
    cases in tests/vt/test_vulkan_backend.cpp; §6.0a of the spec.

What changed

A native Vulkan kernel for vt::CausalConv1dFwd, the GDN PREFILL depthwise
causal conv, which ran on the portable CPU reference tier once per GDN layer.
The default mapping is the first shape the existing comment named: one
invocation per (sequence, channel), serial over its tokens, with the old state
window copied to a private array before the write-back, as the CPU kernel's
old_row does. An opt-in token split (VT_VULKAN_CONV_TARGET_GROUPS) stays in
one dispatch by giving every token that reads the carried state (t < width)
to block 0, which is also the only block that reads or writes conv_state.
Shapes the shader does not serve decline to the reference tier through
GetOpFallback: K = 1, K - 1 > 8, conv_state rows wider than K - 1, storage
dtypes outside f32/f16/bf16, flags neither i8 nor i32, a grid past
maxComputeWorkGroupCount[0], or any index past the shader's uint32
arithmetic. VT_VULKAN_CONV_FWD=0 keeps the op on the reference tier for a
same-binary A/B. Both knobs are documented in docs/ENVIRONMENT.md; the spec
gains §6.0b.

Evidence

New cases in tests/vt/test_vulkan_backend.cpp:

  • the prefill causal conv1d runs NATIVELY on Vulkan, in both token mappings:
    default and a 5-block split on lengths 5, 1, 9, against the CPU oracle; the
    specialization value (|1, |5) is asserted from PipelineKeysFor.
  • the prefill causal conv1d matches the CPU oracle on its edge shapes: i8
    flags at byte offsets 0-3, no bias, silu off, bf16 and f16 operands with bf16
    or f32 output, a zero-length sequence, a length-1 sequence with initial state,
    a padded x row stride.
  • the prefill causal conv1d keeps a bf16 conv_state IN PLACE, bit-exact vs the f32 arm.
  • the prefill causal conv1d DECLINES a width past its window, and stays correct and ... DECLINES a conv_state row wider than K-1: decline counted,
    bytes equal to the reference.

Outputs to the GDN NMSE tolerance; rolled state bit-exact, and spelled out
independently of the oracle for the main case.

Red-before / green-after on llvmpipe (LLVM 20.1.2, 256 bits), the device class
this repository's Vulkan CI uses; five runs per arm of
-tc='*prefill causal conv1d*,*cooperative-matrix capability*':

this PR                                   5/5   6 passed   798/798 assertions
pre-fix behaviour (every block copies the
  state row, plain blk*tb split)          5/5   1 failed   790/798 assertions
    CHECK( nmse <= kGdnNmseTol )            0.0140122 <= 0.0005
    CHECK( memcmp(split, default) == 0 )    115 == 0

Production call site removed (the RegisterOp disabled): 5 of the 6 cases fail
(RanNative, decline counters).

Discrete NVIDIA RTX PRO 6000 (Windows, MSVC 14.44.35207, /W4 /WX, 0
warnings): all new cases pass. test_vulkan_backend 47/51; the four failures
are on main and unchanged by this PR (three TQ2 NMSE cases, the
vt_matmul_tiled pipeline check), same values with and without it.

scripts/gen-vulkan-spirv.py --check   committed SPIR-V is up to date
scripts/check-agent-record.py         agent record OK
scripts/check-env-doc.py              no finding from this PR
  • scripts/agent-preflight.sh passes: not run in full, see Honest gaps.
  • tests that cover this change: listed above.
  • public docs changed only for facts this PR owns: two rows in
    docs/ENVIRONMENT.md.

Speed claims

  • This PR makes NO speed claim.

Honest gaps

  • The race test is a red-before on llvmpipe only. On the discrete NVIDIA card
    the pre-fix behaviour PASSES, most likely because the blocks of one
    (sequence, channel) are adjacent invocations that run together there (not
    separately demonstrated). It also stops failing on llvmpipe if the split uses
    4 blocks instead of 5, which is why the test uses 5 and the block count is
    capped rather than rounded to a power of two.
  • The split and the decision to leave it off by default were looked at on one
    discrete NVIDIA card only. No other device informed them.
  • The shader's query_start_loc guard bounds memory but is not validation: a
    malformed table whose intervals overlap gives unspecified output here, where
    the CPU reference throws. Device-side all-or-nothing validation would need a
    second dispatch and is not attempted.
  • Observed, not changed: the CPU reference addresses conv_state rows with
    stride K - 1, while CheckConvCommon admits wider rows (speculative-decode
    taps) and describes them as using the physical stride. This kernel declines
    those shapes, so it reproduces the reference exactly, including that
    addressing.
  • VT_VULKAN_CONV_FWD=0 is read once at registration and has no unit test.
  • scripts/agent-preflight.sh was not run in full: on Windows one of its tests
    shells out to WSL and hangs. The platform-neutral checkers it calls were run
    individually (above); CI is the full run.
  • Built and run on Windows with local patches that are NOT in this diff: the
    tev1 registration's cross-TU static-init order, and three MSVC C4244
    narrowings in existing lines (test_tev1.cpp, two in vulkan_ops.cpp). The
    Linux llvmpipe build (GCC 13.3) used the same patched tree; whether it needs
    them was not tested.
  • check-env-doc.py fails on main for three existing variables
    (VT_VK_DISABLE, VT_VK_DISABLE_PAGED_ATTN, VT_VK_FENCE_TIMEOUT_MS); not
    touched here.
  • vulkan_spirv.h records glslang 16.5.0 (the version gen-vulkan-spirv.py
    names as CI's pin) where main recorded 16.4.0. --check with 16.5.0 matches
    main's blobs, so the existing 43 modules are byte-identical; only that line
    and the new module change.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5-5 [claude-code]

## Row

`BACKEND-VULKAN`, branch `row/BACKEND-VULKAN-CONVFWD`; the same sub-row naming
as `row/BACKEND-VULKAN-GDN` (mudler#145) and `row/BACKEND-VULKAN-GDN-CORE` (mudler#152).

## Before starting

- Issue/PR search and existing claim: no open PR, claim or issue names
  `kCausalConv1dFwd` on Vulkan. Local issue filed and closed in this PR:
  `.agents/issues/BACKEND-VULKAN/ISSUE-LOCAL-01M3V01R90QQTQ08BWSBM7AGN2.md`
  (Kind: gap).
- Pull request shape selected at row claim: one op, one shader, its tests and
  docs; helper PR.
- Roadmap or matrix row: `.agents/backend-matrix.md` `BACKEND-VULKAN`; VK-G in
  `.agents/specs/vulkan-full-support.md`. `scripts/ready-for-helper.py
  --check-local`: structure OK, live claims not verified locally.
- Exact current-code and test/evidence anchors inspected: the GDN-family block
  comment in `src/vt/vulkan/vulkan_ops.cpp` that left this op "for a follow-up
  rather than guessed at here" and named the two safe dispatch shapes;
  `src/vt/cpu/cpu_ops.cpp` `CausalConv1dFwdKernel`; `src/vt/ops.cpp`
  `CausalConv1dFwd` / `CheckConvCommon`; the decode-update and compressed-state
  cases in `tests/vt/test_vulkan_backend.cpp`; §6.0a of the spec.

## What changed

A native Vulkan kernel for `vt::CausalConv1dFwd`, the GDN PREFILL depthwise
causal conv, which ran on the portable CPU reference tier once per GDN layer.
The default mapping is the first shape the existing comment named: one
invocation per (sequence, channel), serial over its tokens, with the old state
window copied to a private array before the write-back, as the CPU kernel's
`old_row` does. An opt-in token split (`VT_VULKAN_CONV_TARGET_GROUPS`) stays in
one dispatch by giving every token that reads the carried state (`t < width`)
to block 0, which is also the only block that reads or writes `conv_state`.
Shapes the shader does not serve decline to the reference tier through
`GetOpFallback`: K = 1, K - 1 > 8, `conv_state` rows wider than K - 1, storage
dtypes outside f32/f16/bf16, flags neither i8 nor i32, a grid past
`maxComputeWorkGroupCount[0]`, or any index past the shader's uint32
arithmetic. `VT_VULKAN_CONV_FWD=0` keeps the op on the reference tier for a
same-binary A/B. Both knobs are documented in `docs/ENVIRONMENT.md`; the spec
gains §6.0b.

## Evidence

New cases in `tests/vt/test_vulkan_backend.cpp`:

- `the prefill causal conv1d runs NATIVELY on Vulkan, in both token mappings`:
  default and a 5-block split on lengths 5, 1, 9, against the CPU oracle; the
  specialization value (`|1`, `|5`) is asserted from `PipelineKeysFor`.
- `the prefill causal conv1d matches the CPU oracle on its edge shapes`: i8
  flags at byte offsets 0-3, no bias, silu off, bf16 and f16 operands with bf16
  or f32 output, a zero-length sequence, a length-1 sequence with initial state,
  a padded `x` row stride.
- `the prefill causal conv1d keeps a bf16 conv_state IN PLACE, bit-exact vs the
  f32 arm`.
- `the prefill causal conv1d DECLINES a width past its window, and stays
  correct` and `... DECLINES a conv_state row wider than K-1`: decline counted,
  bytes equal to the reference.

Outputs to the GDN NMSE tolerance; rolled state bit-exact, and spelled out
independently of the oracle for the main case.

Red-before / green-after on llvmpipe (LLVM 20.1.2, 256 bits), the device class
this repository's Vulkan CI uses; five runs per arm of
`-tc='*prefill causal conv1d*,*cooperative-matrix capability*'`:

```
this PR                                   5/5   6 passed   798/798 assertions
pre-fix behaviour (every block copies the
  state row, plain blk*tb split)          5/5   1 failed   790/798 assertions
    CHECK( nmse <= kGdnNmseTol )            0.0140122 <= 0.0005
    CHECK( memcmp(split, default) == 0 )    115 == 0
```

Production call site removed (the `RegisterOp` disabled): 5 of the 6 cases fail
(`RanNative`, decline counters).

Discrete NVIDIA RTX PRO 6000 (Windows, MSVC 14.44.35207, `/W4 /WX`, 0
warnings): all new cases pass. `test_vulkan_backend` 47/51; the four failures
are on main and unchanged by this PR (three TQ2 NMSE cases, the
`vt_matmul_tiled` pipeline check), same values with and without it.

```
scripts/gen-vulkan-spirv.py --check   committed SPIR-V is up to date
scripts/check-agent-record.py         agent record OK
scripts/check-env-doc.py              no finding from this PR
```

- [ ] `scripts/agent-preflight.sh` passes: not run in full, see Honest gaps.
- [x] tests that cover this change: listed above.
- [x] public docs changed only for facts this PR owns: two rows in
  `docs/ENVIRONMENT.md`.

## Speed claims

- [x] This PR makes NO speed claim.

## Honest gaps

- The race test is a red-before on llvmpipe only. On the discrete NVIDIA card
  the pre-fix behaviour PASSES, most likely because the blocks of one
  (sequence, channel) are adjacent invocations that run together there (not
  separately demonstrated). It also stops failing on llvmpipe if the split uses
  4 blocks instead of 5, which is why the test uses 5 and the block count is
  capped rather than rounded to a power of two.
- The split and the decision to leave it off by default were looked at on one
  discrete NVIDIA card only. No other device informed them.
- The shader's `query_start_loc` guard bounds memory but is not validation: a
  malformed table whose intervals overlap gives unspecified output here, where
  the CPU reference throws. Device-side all-or-nothing validation would need a
  second dispatch and is not attempted.
- Observed, not changed: the CPU reference addresses `conv_state` rows with
  stride K - 1, while `CheckConvCommon` admits wider rows (speculative-decode
  taps) and describes them as using the physical stride. This kernel declines
  those shapes, so it reproduces the reference exactly, including that
  addressing.
- `VT_VULKAN_CONV_FWD=0` is read once at registration and has no unit test.
- `scripts/agent-preflight.sh` was not run in full: on Windows one of its tests
  shells out to WSL and hangs. The platform-neutral checkers it calls were run
  individually (above); CI is the full run.
- Built and run on Windows with local patches that are NOT in this diff: the
  tev1 registration's cross-TU static-init order, and three MSVC C4244
  narrowings in existing lines (`test_tev1.cpp`, two in `vulkan_ops.cpp`). The
  Linux llvmpipe build (GCC 13.3) used the same patched tree; whether it needs
  them was not tested.
- `check-env-doc.py` fails on main for three existing variables
  (`VT_VK_DISABLE`, `VT_VK_DISABLE_PAGED_ATTN`, `VT_VK_FENCE_TIMEOUT_MS`); not
  touched here.
- `vulkan_spirv.h` records glslang 16.5.0 (the version `gen-vulkan-spirv.py`
  names as CI's pin) where main recorded 16.4.0. `--check` with 16.5.0 matches
  main's blobs, so the existing 43 modules are byte-identical; only that line
  and the new module change.

FOLLOWING_AGENTS_PROTOCOL

Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: AGENT:claude-opus-5-5 [claude-code]

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant