diff --git a/.agents/issues/BACKEND-VULKAN/ISSUE-LOCAL-01M3V01R90QQTQ08BWSBM7AGN2.md b/.agents/issues/BACKEND-VULKAN/ISSUE-LOCAL-01M3V01R90QQTQ08BWSBM7AGN2.md new file mode 100644 index 0000000000..e9d4ca4bd4 --- /dev/null +++ b/.agents/issues/BACKEND-VULKAN/ISSUE-LOCAL-01M3V01R90QQTQ08BWSBM7AGN2.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M3V01R90QQTQ08BWSBM7AGN2 +Title: kCausalConv1dFwd (GDN prefill conv) has no native Vulkan kernel +Row: BACKEND-VULKAN +State: CLOSED +Kind: gap +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-10-01 +Updated: 2026-10-01 +Closed: 2026-10-01 + +## Problem + +vt::CausalConv1dFwd, the GDN PREFILL depthwise causal conv, is not registered for DeviceType::kVULKAN, so every GDN prefill on a Vulkan queue runs it on the portable CPU reference tier, once per GDN layer, behind a full batch drain. src/vt/vulkan/vulkan_ops.cpp left it as a follow-up because the op reads the OLD conv-state window and overwrites it in the same call; the comment names the two safe dispatch shapes (a serial invocation per (sequence, channel) over the whole token range, or a buffered old row). .agents/specs/vulkan-full-support.md section 6.0a lists it as one of the two remaining reference-tier declines. + +## Resolution + +Closed by row/BACKEND-VULKAN-CONVFWD: native vt_causal_conv1d_fwd, default one invocation per (sequence, channel) serial over its tokens with the old window copied before the write-back; opt-in token split (VT_VULKAN_CONV_TARGET_GROUPS) giving every t < width to block 0; unservable shapes decline through GetOpFallback; VT_VULKAN_CONV_FWD=0 keeps the reference tier. Gated by tests/vt/test_vulkan_backend.cpp against the CPU oracle; the split arm is red on llvmpipe with the pre-fix plain split and green with this PR. diff --git a/.agents/specs/vulkan-full-support.md b/.agents/specs/vulkan-full-support.md index 9f2c687206..01533b0f16 100644 --- a/.agents/specs/vulkan-full-support.md +++ b/.agents/specs/vulkan-full-support.md @@ -613,6 +613,54 @@ gate is a memcmp of the two arms in ONE process, plus the specialization VALUES from `PipelineKeys()`, because both arms are the same module and produce identical bytes; a numeric check alone could never see the mechanism. +### 6.0b `VK-G` partial: the PREFILL CONV landed — 2026-10-01 + +`row/BACKEND-VULKAN-CONVFWD`. `kCausalConv1dFwd` is native +(`vt_causal_conv1d_fwd`); module count 43 -> 44. This updates the closing +sentence of §6.0a: of the two ops it named as the reference-tier declines on that +path, `kRopeCosSinCache` remains, by design. + +**Why it had been left.** The op computes every output from the OLD conv-state +window and then overwrites that window in the same call, so a dispatch that +splits a sequence across invocations reads state another invocation is +rewriting. `vulkan_ops.cpp` named the two safe shapes: a serial invocation per +(sequence, channel) over the whole token range, or a buffered old row. + +**What landed.** The first of those shapes is the DEFAULT: one invocation per +(sequence, channel), the CPU kernel's own `ForRows(n * c_dim, ...)` unit, with +the old window copied into a private array before the write-back (the CPU +kernel's `old_row`). Per-element arithmetic is ported 1:1 from +`src/vt/cpu/cpu_ops.cpp` `CausalConv1dFwdKernel`, including its silu spelling. + +An OPT-IN token split, `VT_VULKAN_CONV_TARGET_GROUPS`, divides each sequence +into token blocks in the same dispatch. Only `t < width` reads the carried state; +the shader gives every such token to block 0, which also writes the state back +and is the only block that touches `conv_state`. Without that rule the split +raced: on llvmpipe the wrong elements were exactly the `t < width` tokens that +fell outside block 0. It is off by default; whether a split is worth anything is +a property of the device. + +Shapes the shader does not serve DECLINE to the reference tier through +`GetOpFallback`, the seam §6.0 uses: K = 1, a kernel width past its 8-slot +window, `conv_state` rows wider than K-1 (the CPU reference addresses rows with +stride K-1 while the op layer admits wider rows; declining keeps the reference's +answer instead of choosing), storage dtypes outside f32/f16/bf16, a +`has_initial_state` that is neither i8 nor i32, a grid past the device's +`maxComputeWorkGroupCount[0]`, and any index that would not fit the shader's +uint32 arithmetic. The native path does not read `query_start_loc` on the host; +the shader's guard keeps every access in bounds but does not validate the table, +so a malformed table that the CPU reference would reject gives unspecified +output here. + +**Gates.** `test_vulkan_backend`, against the CPU oracle: the default mapping +and the split on a varlen batch (lengths 5, 1, 9, one sequence without initial +state), with the specialization value asserted; i8 flags at byte offsets 1-3, no +bias, silu off, bf16 and f16 operands with bf16 or f32 output, a zero-length +sequence and a padded `x` row stride; a bf16 +`conv_state` against the f32 arm; and the K = 10 and widened-row declines. +Outputs to the GDN NMSE tolerance, rolled state bit-exact. +`VT_VULKAN_CONV_FWD=0` keeps the op on the reference tier for a same-binary A/B. + ### 6.0a `VK-G` partial: the FUSED ATTN PREAMBLE landed — 2026-08-09 `row/BACKEND-VULKAN-QKNORM`. `kAttnQkNormRopeGate` — gemma-RMSNorm(q) + diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md index d514d76cda..56035fb1c9 100644 --- a/docs/ENVIRONMENT.md +++ b/docs/ENVIRONMENT.md @@ -292,6 +292,8 @@ portable/reference path. In normal operation leave them unset. | `VT_VULKAN_RMSNORM` | auto | Which `vt::RmsNorm` SPIR-V module runs: `wide` forces the 1024-invocation subgroup-reducing one, `base` forces the portable 128-invocation one, unset lets the device capability decide (1024 invocations on the X axis plus compute subgroup BASIC and ARITHMETIC). The wide module exists because `RmsNorm` dispatches ONE WORKGROUP PER ROW and a batch-1 decode step has exactly one row: on Qwen3.6-27B that put 128 invocations on a 5120-wide row, four warps of one SM, with the rest of the GPU idle. MEASURED on GB10 by the two-length GPU-timestamp diff: **0.0611 -> 0.0123 ms/call, 7.88 -> 1.59 ms/token**, and paired decode **241.9 -> 235.6 ms** median TPOT. The tell that it was OCCUPANCY and not the reduction is that the SAME shader costs 0.066 ms/call during PREFILL, where 32 rows give it 32 workgroups and 32x the data. It exists for the same-binary A/B and so the unit gate can exercise the fallback on hardware that would always pick the wide arm. Vulkan-only | | `VT_VULKAN_MATMUL_NCOLS` | 4 | Output columns each lane of the portable scalar GEMM computes, in the `[K,N]` (non-transposed) orientation only. At 1 the kernel is the flat one-invocation-per-output-element body; above 1 a workgroup takes `128*NCOLS` CONSECUTIVE output columns of one row, so at each step of K it reads a contiguous run of that many elements instead of 128. This is the ONE decode GEMM that cannot reach the `vt_matmul_vec` tactic, because in `[K,N]` the lanes are already coalesced and the GEMV shape would make them strided; on the 27B it is the lm_head, `m=1 k=5120 n=248320`, 2.54 GB moved per token. MEASURED on GB10, 27B decode, `ms/call` medians over interleaved replicates: NCOLS 1 = 12.48, 2 = 12.46, **4 = 11.54**, 8 = 12.81, with 4 winning **6 of 6** interleaved pairs against 1. Blocking is a TRADE, not a monotone win: at 8 the dispatch falls to 243 workgroups (~31k threads) and the device runs out of work to hide memory latency with faster than the longer contiguous run buys back. It rides a specialization constant, so every arm is the same committed module and they A/B in one binary. Every arm is BIT-IDENTICAL -- each accumulator owns one output element and sums the whole K sequentially, which is the CPU kernel's order -- so this kernel keeps the byte-exact tier that the coopmat and GEMV tactics gave up; a memcmp gates that. Vulkan-only | | `VT_VULKAN_COOPMAT` | on | `=0` forces the Vulkan GEMM onto the portable SCALAR kernel instead of the cooperative-matrix (tensor-core) tactic. The coopmat path is selected only where the device reports the exact `16x16x16 bf16/bf16/f32/f32 SUBGROUP` configuration, subgroup size is 32, both operands are bf16, and M, N and K are all multiples of 16. The whole-tile requirement on M and N is not a tuning choice: `coopMatLoad` reads a full 16x16 tile with no masking, so a partial tile reads past the operand and can fault the GPU. Ragged shapes fall back to the scalar kernel; this switch bypasses that selection entirely. It exists for the same-binary A/B in `examples/vulkan-gemm-ab` (measured 11.1x-32.9x on NVIDIA Thor) and as the bisect lever if a coopmat result is ever suspect. Vulkan-only | +| `VT_VULKAN_CONV_FWD` | on | `=0` leaves `vt::CausalConv1dFwd` (the GDN PREFILL conv) on the portable CPU reference tier instead of registering the native `vt_causal_conv1d_fwd` kernel, which is where the op ran before that kernel existed. Read once, at backend registration. It exists for the same-binary A/B and as the bisect lever, the same shape as `VT_VULKAN_COOPMAT`. The native kernel matches the CPU reference to a tolerance on the outputs and bit-exactly on the rolled `conv_state`. Vulkan-only | +| `VT_VULKAN_CONV_TARGET_GROUPS` | unset (no split) | Opt-in TOKEN SPLIT for the native prefill conv. Unset, each (sequence, channel) is one invocation serial over its tokens. Set to a workgroup target `g` (1..65536), each sequence is split into `ceil(g / base_groups)` token blocks, where `base_groups` is the default grid in workgroups, capped at the mean sequence length, at what one dispatch can launch, and at 64 (each block count is its own compiled pipeline). Only tokens `t < width` read the carried state; the shader gives all of them to block 0, which is also the only block that writes the state back, so no other block touches `conv_state`. Each element is computed by the same expression whatever the block count; `test_vulkan_backend` compares 1 and 5 blocks bit-for-bit. Whether a split is worth anything depends on the device, which is why there is no default. Read on every call. Vulkan-only | | `VT_GLM5_NEXT_DEVICE_EXPERTS` | **off (opt-in)** | `=1` lets `Glm5NextForConditionalGeneration` (GLM-5.3-Flash) accept a non-CPU queue and route its routed-expert keep-quant GEMM to the device, against banks made resident by `dense_attn::ResidentWeight`. **IT IS OFF BECAUSE THE PATH IT ENABLES IS MEASURED TO CRASH, not because it is unmeasured.** On `dgx:gpu0` against the published 101.24 GiB `UD-Q2_K_XL` artifact, ALL THREE `--device cuda` legs died with SIGSEGV (rc=139) having emitted no token, interleaved against three `--device cpu` legs that all emitted ` Paris.` from the same binary. **The mixed-residency reading of those legs is FALSIFIED and the cause is elsewhere**: the two log lines that suggested it are once-flags, and the process dies in `StoreCaches`, which host-stores into the runner's `cudaMalloc` KV pages after the forward has already returned. That defect is older than this knob and only became reachable when the non-CPU refusal above it was removed; see `.agents/specs/glm5-next-flash.md` O49 and [#2480](https://github.com/mudler/vllm.cpp/issues/2480), which owns the fix. The default is the refusal the tree carried before the arm existed, because turning a clean named error into a segfault is strictly worse for a user. **Set this only to debug that crash; it is not a serving knob.** Parsed strictly (`1` and nothing else, not the usual first-character rule) precisely because it opts into a crashing path. Inert on every other model and on `--device cpu`. See `.agents/specs/glm5-next-flash.md` O46 and [#2464](https://github.com/mudler/vllm.cpp/issues/2464) | | `VT_GLM5_NEXT_DEVICE` | **off (opt-in)** | `=1` routes the entire GLM-5.3-Flash forward through `Glm5NextDeviceForward`, which dispatches embedding, RMSNorm, MoE combine, the k-pool indexer and `lm_head` through `vt::*` device ops on the queue, keeping MLA attention, mHC sites and the dense MLP as host-fallback islands (the `kimi_linear_device.cpp` single-queue pattern). This is a superset of `VT_GLM5_NEXT_DEVICE_EXPERTS`: when on, the whole forward delegates and the per-arm experts split is not reached. On a CPU queue the `vt::*` kernels use float32 accumulation where the host reference uses double, so the output agrees within a float-vs-double envelope rather than byte-exact. Inert on every other model. See `.agents/specs/glm5-next-flash.md` W9c-3 and [#3175](https://github.com/mudler/vllm.cpp/pull/3175) | diff --git a/src/vt/vulkan/shaders/vt_causal_conv1d_fwd.comp b/src/vt/vulkan/shaders/vt_causal_conv1d_fwd.comp new file mode 100644 index 0000000000..5b5f433d16 --- /dev/null +++ b/src/vt/vulkan/shaders/vt_causal_conv1d_fwd.comp @@ -0,0 +1,225 @@ +#version 450 +// vt::CausalConv1dFwd — the GDN PREFILL depthwise causal conv. +// +// WHY THIS EXISTS. The op had no Vulkan kernel, so every GDN prefill fell to the +// portable CPU reference tier once per GDN layer. vulkan_ops.cpp left it as a +// follow-up because of the state write-back hazard described below; this is +// that follow-up, in the serial-per-(sequence, channel) shape it named. +// +// Row: BACKEND-VULKAN, branch row/BACKEND-VULKAN-CONVFWD (one op of the VK-G +// family in .agents/specs/vulkan-full-support.md). +// +// SHAPE OF THE WORK, and why it is NOT llama.cpp's shape. llama.cpp's +// ssm_conv.comp parallelises over tokens too (BLOCK_SIZE=32, TOKENS_PER_WG=16) +// and gets away with it because ggml's dataflow is SSA: its conv op only +// produces outputs, and advancing the state is a separate node. This op does +// both in one kernel -- it reads the OLD state while computing outputs, then +// OVERWRITES that state with the last K-1 input samples. Splitting tokens +// across invocations would put those two against each other across invocation +// boundaries. +// +// So the unit here is ONE INVOCATION PER (sequence, channel), which is exactly +// the CPU kernel's own chunking unit (`ForRows(n * c_dim, ...)`), and the old +// state is copied into a private array first -- again exactly what the CPU +// kernel does with `old_row`. Each (s, c) pair then owns its output column +// slice and its state row outright, so there is no cross-invocation hazard and +// no barrier is needed. +// +// That is the DEFAULT mapping. It gives up token parallelism: the grid is +// n * c_dim invocations, each doing t_len * k multiply-adds. An opt-in token +// split exists (VT_CONV_BLOCKS below) and stays in ONE dispatch, because only +// t < width reads the carried state and all of those go to the block that +// also writes it back. +// +// PORTED 1:1 from src/vt/cpu/cpu_ops.cpp CausalConv1dFwdKernel. The accumulation +// order per output element is identical, the silu spelling is the reference's +// own (`acc / (1 + exp(-acc))`, not `acc * sigmoid(acc)`, which rounds +// differently), and VT_LOAD/VT_STORE keep the same single round on store. +#extension GL_GOOGLE_include_directive : require +#include "vt_common.glsl" + +layout(local_size_x = 128, local_size_y = 1, local_size_z = 1) in; + +layout(binding = 0) buffer Ob32 { uint v[]; } O32; +layout(binding = 1) buffer Ob16 { uint16_t v[]; } O16; +layout(binding = 2) readonly buffer Xb32 { uint v[]; } X32; +layout(binding = 3) readonly buffer Xb16 { uint16_t v[]; } X16; +layout(binding = 4) readonly buffer Wb32 { uint v[]; } W32; +layout(binding = 5) readonly buffer Wb16 { uint16_t v[]; } W16; +// bias; aliases the weight when the caller passed none (p.has_bias == 0). +layout(binding = 6) readonly buffer Bb32 { uint v[]; } B32; +layout(binding = 7) readonly buffer Bb16 { uint16_t v[]; } B16; +// conv_state, read AND written in place; f32 or bf16 (p.st_dt). +layout(binding = 8) buffer Sb32 { uint v[]; } ST32; +layout(binding = 9) buffer Sb16 { uint16_t v[]; } ST16; +// query_start_loc, i32, n+1 entries. +layout(binding = 10) readonly buffer Qb { uint v[]; } QSL; +// has_initial_state, i32 or i8 per sequence (p.his_is_i8 picks the read). +layout(binding = 11) readonly buffer Hb { uint v[]; } HIS; + +layout(push_constant) uniform Params { + uint n; // conv_state.shape[0] — sequence count + uint c_dim; // x.shape[1] + uint k; // weight.shape[1] + uint width; // k - 1 + uint x_rs; // x.stride[0], in elements (x may be a padded-row view) + uint max_t_len; // x.shape[0]: no sequence can be longer than the input + uint has_bias, his_is_i8, silu; + uint out_dt, x_dt, w_dt, bias_dt, st_dt; + uint out_off, x_off, w_off, bias_off, st_off, qsl_off, his_off; +} p; + +// The private window, sized for the largest K this backend accepts. The host +// predicate refuses anything wider, so the loop below never runs past it. +const uint VT_CONV_MAX_WIDTH = 8u; + +// TOKEN BLOCKS PER (sequence, channel). 1 is the default mapping -- one +// invocation walks the whole sequence -- and the host only asks for more when +// VT_VULKAN_CONV_TARGET_GROUPS is set (vulkan_ops.cpp CausalConv1dFwdKernel). +// +// WHY SPLITTING IS CORRECT: this convolution carries NO state along t. Output t +// reads x[t-K+1 .. t] and, only while t < K-1, the pre-roll window. Block 0 owns +// every t < K-1 AND the state write-back, and it is the ONLY block that reads or +// writes conv_state at all; every other block reads x and writes its own output +// rows. No barrier, no cross-workgroup dependency, unchanged arithmetic, so the +// output does not depend on the block count. +// +// The split only changes the grid shape. Whether that is worth anything is a +// property of the device, so the default is 1 and the host exposes the target +// as a knob rather than choosing one. +layout(constant_id = 0) const uint VT_CONV_BLOCKS = 1u; + +void main() { + uint gid = gl_GlobalInvocationID.x; + if (gid >= p.n * p.c_dim * VT_CONV_BLOCKS) { return; } + uint blk = gid % VT_CONV_BLOCKS; + uint sc = gid / VT_CONV_BLOCKS; + uint s = sc / p.c_dim; + uint c = sc % p.c_dim; + + // begin/end come from query_start_loc. + // + // ⚠️ t_len IS A LOOP BOUND AND THE SUBTRACTION IS UNSIGNED. If end < begin -- + // a non-monotonic table, a wrong buffer offset, anything -- `end - begin` + // wraps to nearly 2^32 and the output loop below becomes effectively + // infinite. On a GPU that is not a crash: the dispatch never finishes, the + // fence never signals, and it presents as a HANG with no error -- the same + // failure mode documented for an out-of-bounds coopMatLoad. + // + // The native host path does not read query_start_loc (it is device memory), + // so this guard is what bounds memory: end <= max_t_len with begin <= end + // keeps begin + t inside x and out for every t < t_len. It is NOT validation. + // A malformed table whose intervals overlap still writes overlapping rows, + // and its output is unspecified; the CPU reference rejects such a table. + uint begin = QSL.v[(p.qsl_off >> 2) + s]; + uint end = QSL.v[(p.qsl_off >> 2) + s + 1u]; + if (end < begin || end > p.max_t_len) { return; } + uint t_len = end - begin; + + // has_initial_state is i32 or i8 per sequence. The i8 case is read through the + // u32 view and shifted, since this backend binds byte data through the 32-bit + // window like every other dtype-erased operand. his_off is a BYTE offset and + // an i8 view may start at any byte, so the shift comes from the full byte + // address, not from s alone. The word holding the last flag is always inside + // the buffer: vulkan_context.cpp AllocBuffer rounds every buffer length up to + // a whole 32-bit word, and a view never runs past its allocation. + bool init; + if (p.his_is_i8 != 0u) { + uint addr = p.his_off + s; + uint word = HIS.v[addr >> 2]; + init = ((word >> ((addr & 3u) * 8u)) & 0xFFu) != 0u; + } else { + init = HIS.v[(p.his_off >> 2) + s] != 0u; + } + + uint srow = (s * p.c_dim + c) * p.width; + + // COPY THE OLD STATE FIRST. This is the whole reason the kernel is safe + // without a barrier: the output loop below reads the pre-roll window while the + // final loop overwrites it, and the CPU reference resolves that same conflict + // the same way (`old_row`). ONLY BLOCK 0 COPIES IT: no other block computes a + // t < width, so none needs the window, and a load there would be an + // unordered read of the row block 0 is about to overwrite. + float old_row[VT_CONV_MAX_WIDTH]; + if (blk == 0u) { + for (uint j = 0u; j < p.width; ++j) { + old_row[j] = VT_LOAD(ST32, ST16, p.st_dt, p.st_off, srow + j); + } + } + + float b = p.has_bias != 0u ? VT_LOAD(B32, B16, p.bias_dt, p.bias_off, c) : 0.0; + + // This block's slice of the sequence. tb is derived from t_len rather than + // passed in, so a short sequence simply leaves the later blocks empty instead + // of the host having to know each length. + // Ceiling division without forming t_len + VT_CONV_BLOCKS - 1, which could + // wrap. (The host also bounds t_len by the i32 query_start_loc contract and + // VT_CONV_BLOCKS by 64, so blk * tb + tb stays far below 2^32.) + const uint tb = t_len / VT_CONV_BLOCKS + uint(t_len % VT_CONV_BLOCKS != 0u); + // BLOCK 0 TAKES EVERY t THAT READS THE CARRIED STATE, which is t < width. + // + // It has to, because block 0 is ALSO the block that overwrites that state at + // the end of this same dispatch, and there is no ordering between + // invocations, in or across workgroups (the blocks of one (sequence, channel) + // are adjacent invocations, and the shader has no barrier). + // The plain `blk * tb` split let a t < width land on blk > 0, which then read + // the state row in a race against block 0's write-back: whichever workgroup + // the driver happened to schedule first decided the answer. + // + // Observed with an earlier split, VT_CONV_BLOCKS = 5 on a varlen batch of + // lengths 5, 1, 9 (K 4, width 3), on llvmpipe: the wrong elements were + // exactly the (sequence, t) pairs with t < width that fell outside block 0 -- + // seq0 t=1,2 and seq2 t=2 -- never t=0, and never a t inside block 0. + // tests/vt/test_vulkan_backend.cpp runs the split on that shape: with the + // pre-fix behaviour restored it failed 5 runs of 5 on llvmpipe, and passed on + // the one discrete NVIDIA GPU it was tried on (most likely because the blocks + // of one (sequence, channel) are adjacent invocations that run together + // there; not separately demonstrated). A green run on a GPU therefore says + // nothing about this rule. + // + // The cost is that block 0 may carry up to `width` tokens instead of `tb`. + // width is K-1 and K is small, and the blocks it displaces simply come out + // empty (t_lo >= t_hi), so the grid is unchanged. + const uint t_lo = (blk == 0u) ? 0u : max(blk * tb, p.width); + const uint t_hi = (blk == 0u) ? min(max(tb, p.width), t_len) + : min(blk * tb + tb, t_len); + + for (uint t = t_lo; t < t_hi; ++t) { + float acc = b; + for (uint j = 0u; j < p.k; ++j) { + // ti is the token index of window slot j, and it goes NEGATIVE into the + // carried state, so the arithmetic is signed even though every index that + // reaches a buffer is not. + int ti = int(t) - (int(p.k) - 1 - int(j)); + float v = 0.0; + if (ti >= 0) { + v = VT_LOAD(X32, X16, p.x_dt, p.x_off, (begin + uint(ti)) * p.x_rs + c); + } else if (init) { + v = old_row[uint(int(p.width) + ti)]; // state col (K-1)+(t-i) + } + acc += VT_LOAD(W32, W16, p.w_dt, p.w_off, c * p.k + j) * v; + } + // Silu spelled as the CPU reference spells it (cpu_ops.cpp); + // `acc * sigmoid(acc)` is a different rounding. + VT_STORE(O32, O16, p.out_dt, p.out_off, (begin + t) * p.c_dim + c, + p.silu != 0u ? (acc / (1.0 + exp(-acc))) : acc); + } + + // ADVANCE THE STATE, once per (sequence, channel). Block 0 owns it: it always + // exists even when t_len is shorter than the block count, and everything it + // reads is either x (read-only) or its own copy of the old window. + if (blk != 0u) { return; } + + // new column j holds token (t_len - width + j), falling back into the shifted + // old window when the sequence is shorter than K-1. + for (uint j = 0u; j < p.width; ++j) { + int tj = int(t_len) - int(p.width) + int(j); + float v = 0.0; + if (tj >= 0) { + v = VT_LOAD(X32, X16, p.x_dt, p.x_off, (begin + uint(tj)) * p.x_rs + c); + } else if (init) { + v = old_row[uint(int(p.width) + tj)]; + } + VT_STORE(ST32, ST16, p.st_dt, p.st_off, srow + j, v); + } +} diff --git a/src/vt/vulkan/vulkan_ops.cpp b/src/vt/vulkan/vulkan_ops.cpp index b33d1bec8a..02eaac7f08 100644 --- a/src/vt/vulkan/vulkan_ops.cpp +++ b/src/vt/vulkan/vulkan_ops.cpp @@ -345,7 +345,8 @@ static_assert(sizeof(PagedAttnParams) <= 128, "push constants must fit the guaranteed 128 bytes"); static_assert(sizeof(ConvUpdateParams) <= 128, "push constants must fit the guaranteed 128 bytes"); -// The widest block in the backend at 84 bytes: the fused post-conv carries ten +// The widest block in the backend at 84 bytes (tied with ConvFwdParams): the +// fused post-conv carries ten // operand offsets. If it ever needs an eleventh, the step list has to move to the // scratch buffer the way vt_fused_chain's does. static_assert(sizeof(GdnPostConvParams) <= 128, @@ -1668,12 +1669,11 @@ void QkvSplitKernel(Queue&, Tensor& q_out, Tensor& k_out, Tensor& v_out, const T // __init__). Leaving it on the host MIRRORS upstream; "implementing" it would // be a regression, and the assertion in tests/vt/test_vulkan_backend.cpp says // so out loud. -// * kCausalConv1dFwd — the PREFILL conv. It is the same arithmetic as the -// update below but its state write-back reads the OLD state row while other -// tokens of the same sequence are still reading it, so it needs either a -// per-(sequence, channel) serial invocation over the whole token range or a -// buffered old row; that is a different dispatch shape, not a wider push -// block, and it is left for a follow-up rather than guessed at here. +// * kCausalConv1dFwd — the PREFILL conv — is NATIVE now, in the first of the +// two shapes this comment used to name: one invocation per (sequence, +// channel), serial over the whole token range, with the old state row copied +// into a private array before the write-back. See CausalConv1dFwdKernel; an +// opt-in token-block split exists and its own hazard is documented there. // =========================================================================== // cpu_ops.cpp:2272-2279 SigmoidGateBf16Kernel. Flat, one invocation per element. @@ -1806,6 +1806,181 @@ void GdnStateScatterKernel(Queue&, Tensor& cache, const Tensor& working, Go("vt_gdn_state_scatter", bind, p, FlatGroupCount(g.rows * g.work_row)); } +struct ConvFwdParams { + uint32_t n, c_dim, k, width, x_rs; + uint32_t max_t_len; + uint32_t has_bias, his_is_i8, silu; + uint32_t out_dt, x_dt, w_dt, bias_dt, st_dt; + uint32_t out_off, x_off, w_off, bias_off, st_off, qsl_off, his_off; +}; +static_assert(sizeof(ConvFwdParams) <= 128, + "push constants must fit the guaranteed 128 bytes"); + +// cpu_ops.cpp CausalConv1dFwdKernel, the GDN PREFILL conv. +// +// DEFAULT MAPPING: one invocation per (sequence, channel), serial over that +// sequence's whole token range -- the CPU kernel's own ForRows unit, and the +// first of the two shapes the block comment above this family names for this +// op. The op computes every output from the OLD state window and then +// overwrites that window; one invocation owns both the reads and the +// write-back for its (sequence, channel), and it copies the old window into a +// private array first (the CPU kernel's `old_row`), so no other invocation can +// observe a half-rolled state. +// +// Deliberately NOT llama.cpp's ssm_conv.comp shape by default, which also +// parallelises over tokens: that works there because ggml advances the conv +// state as a separate node, and here the same dispatch reads and advances it. +// +// OPTIONAL TOKEN BLOCKS, VT_VULKAN_CONV_TARGET_GROUPS=: split each sequence +// into VT_CONV_BLOCKS token blocks in the same dispatch. Only tokens t < width +// read the carried state, and the shader gives every one of them to block 0 -- +// the block that also writes the state back -- so block 0 is the only one that +// touches conv_state at all. An earlier split that did not do this raced: see +// the BLOCK 0 comment in vt_causal_conv1d_fwd.comp. The split changes the grid +// shape only, never the output, so it is off unless asked for. +// +// STATE ROWS: the CPU reference addresses conv_state with stride `width` and +// the shader mirrors it. The op layer also admits rows WIDER than K-1 +// (speculative-decode taps, ops.cpp CheckConvCommon), where that stride and +// the row's physical stride disagree. This kernel declines those shapes +// rather than choose between them, so they keep the reference tier's answer. +void CausalConv1dFwdKernel(Queue& q, Tensor& out, const Tensor& x, const Tensor& w, + const Tensor* bias, Tensor& conv_state, const Tensor& qsl, + const Tensor& his, const CausalConv1dArgs& args) { + const int64_t c_dim = x.shape[1], k = w.shape[1], total = x.shape[0]; + const int64_t n = conv_state.shape[0]; + if (n == 0 || c_dim == 0) return; + auto decline = [&] { + auto next = reinterpret_cast( + GetOpFallback(OpId::kCausalConv1dFwd, DeviceType::kVULKAN, kNativeProviderName)); + next(q, out, x, w, bias, conv_state, qsl, his, args); + }; + // PER-CALL REFUSAL rather than a throw, the seam vt_paged_attn and the GDN + // recurrences use: a shape this shader cannot serve forwards to the portable + // reference tier, which is correct for every shape. Reasons to decline: + // * K = 1 (no carried state, so a zero-length state row to bind) or a + // kernel width past the shader's private window (VT_CONV_MAX_WIDTH). + // * conv_state rows wider than K-1 (see STATE ROWS above). + // * A storage dtype outside f32/f16/bf16 (conv_state: f32/bf16). + // * has_initial_state that is neither i8 nor i32. + // * More (sequence, channel) pairs than one dispatch can cover: the grid + // is 1-D and Dispatch refuses a count above the device's + // maxComputeWorkGroupCount[0]. + // * Any index the shader forms that would not fit its uint32 arithmetic + // (checked after binding, below, because it includes the offsets). + const auto is_float = [](DType d) { + return d == DType::kF32 || d == DType::kF16 || d == DType::kBF16; + }; + const int64_t max_invocations = + static_cast(VulkanContext::Get().max_workgroup_count_x()) * kWorkgroupSize; + const bool serve = n * c_dim <= max_invocations && k >= 2 && k - 1 <= 8 && + conv_state.shape[2] == k - 1 && is_float(out.dtype) && + is_float(x.dtype) && is_float(w.dtype) && + (bias == nullptr || is_float(bias->dtype)) && + (conv_state.dtype == DType::kF32 || conv_state.dtype == DType::kBF16) && + (his.dtype == DType::kI8 || his.dtype == DType::kI32); + if (!serve) { + decline(); + return; + } + // query_start_loc is NOT read on the host: it is device memory, and reading + // it here would need the pending batch drained first. The shader guards + // memory, not semantics -- see the guard at the top of its main(). (A + // DECLINED call does reach host code: GetOpFallback drains the batch and the + // reference tier runs over host-visible device storage.) + + Binder bind; + const uint32_t out_off = bind.Add(out, "causal_conv1d_fwd: out"); + const uint32_t x_off = bind.Add(x, "causal_conv1d_fwd: x"); + const uint32_t w_off = bind.Add(w, "causal_conv1d_fwd: weight"); + const uint32_t bias_off = bias != nullptr ? bind.Add(*bias, "causal_conv1d_fwd: bias") + : bind.Add(w, "causal_conv1d_fwd: weight"); + const uint32_t st_off = bind.Add(conv_state, "causal_conv1d_fwd: conv_state"); + const uint32_t qsl_off = bind.AddU32Only(qsl, "causal_conv1d_fwd: query_start_loc"); + // has_initial_state may be i8, which is not 4-byte aligned, so it goes through + // the byte view and the shader unpacks it -- the same reasoning the update + // kernel gives for aliasing a bf16 state through AddByteView. + const uint32_t his_off = bind.AddByteView(his, "causal_conv1d_fwd: has_initial_state"); + + // F5 of the review: the shader forms every index in uint32. Decline when the + // largest one -- the binding's base element plus the farthest element the + // shader can address in it -- would not fit. + { + constexpr int64_t kU32 = 0xFFFFFFFFll; + const auto elem = [](DType d) { return d == DType::kF32 ? 4 : 2; }; + const auto fits = [&](uint32_t off_bytes, int64_t esize, int64_t extent) { + return static_cast(off_bytes) / esize + extent <= kU32; + }; + const int64_t x_extent = (total > 0 ? (total - 1) * x.stride[0] : 0) + c_dim; + const bool ok = + // query_start_loc is i32 and its last entry is `total`, so every + // sequence length the shader sees is below 2^31; asserting it here is + // what makes the shader's block arithmetic provably wrap-free. + total <= 0x7FFFFFFFll && x.stride[0] >= 0 && x.stride[0] <= kU32 && + n * c_dim * k <= kU32 && + fits(out_off, elem(out.dtype), total * c_dim) && + fits(x_off, elem(x.dtype), x_extent) && + fits(w_off, elem(w.dtype), c_dim * k) && + (bias == nullptr || fits(bias_off, elem(bias->dtype), c_dim)) && + fits(st_off, elem(conv_state.dtype), n * c_dim * (k - 1)) && + fits(qsl_off, 4, n + 1) && fits(his_off, 1, n + 4); + if (!ok) { + decline(); + return; + } + } + + ConvFwdParams p{static_cast(n), + static_cast(c_dim), + static_cast(k), + static_cast(k - 1), + static_cast(x.stride[0]), + static_cast(total), + bias != nullptr ? 1u : 0u, + his.dtype == DType::kI8 ? 1u : 0u, + args.silu_activation ? 1u : 0u, + DtypeCode(out.dtype), + DtypeCode(x.dtype), + DtypeCode(w.dtype), + bias != nullptr ? DtypeCode(bias->dtype) : DtypeCode(w.dtype), + DtypeCode(conv_state.dtype), + out_off, + x_off, + w_off, + bias_off, + st_off, + qsl_off, + his_off}; + // TOKEN BLOCKS: 1 unless VT_VULKAN_CONV_TARGET_GROUPS asks for a split. Read + // on every call rather than cached, so one test process can run both + // mappings -- the split path carries the block-0 rule and has to stay under + // test even though it is not the default. + int64_t blocks = 1; + if (const char* v = std::getenv("VT_VULKAN_CONV_TARGET_GROUPS")) { + const long g = std::strtol(v, nullptr, 10); + if (g >= 1 && g <= 65536) { + const int64_t base_groups = (n * c_dim + 127) / 128; + // Never more blocks than there are tokens to give them: past that the + // extra invocations are empty and only cost launch. + const int64_t avg_len = std::max(1, total / n); + blocks = (static_cast(g) + base_groups - 1) / base_groups; + blocks = std::max(1, std::min(blocks, avg_len)); + // ...and never more than one dispatch can launch. + blocks = std::min(blocks, max_invocations / (n * c_dim)); + // VT_CONV_BLOCKS is a specialization constant, so every distinct value is + // a separately compiled pipeline; the cap bounds what this knob can + // create to 64. A cap, NOT power-of-two rounding: with rounding, the + // pre-fix race stopped reproducing on llvmpipe (4 blocks of one + // (sequence, channel) sat inside one 8-lane subgroup in the test's + // layout), so it would have hidden the hazard from the test that guards it. + blocks = std::min(blocks, 64); + } + } + const uint32_t conv_spec[1] = {static_cast(blocks)}; + Go("vt_causal_conv1d_fwd", bind, p, + FlatGroupCount(n * c_dim * blocks), conv_spec, 1); +} + // cpu_ops.cpp:1081-1127 CausalConv1dUpdateKernel. One invocation per // (token, channel) — the CPU kernel's own row-chunking unit, and what makes the // read-old-then-roll safe with no barrier. @@ -2620,9 +2795,8 @@ struct Registrar { reinterpret_cast(static_cast(&RmsNormKernel))); RegisterOp(OpId::kFusedChain, DeviceType::kVULKAN, reinterpret_cast(static_cast(&FusedChainKernel))); - // BACKEND-VULKAN-GDN: the GDN glue family. kCausalConv1dFwd (the prefill - // conv) stays on the portable reference tier; see the block comment above - // these kernels. + // BACKEND-VULKAN-GDN: the GDN glue family, now including kCausalConv1dFwd + // (the prefill conv), registered after the update op below. RegisterOp(OpId::kSigmoidGateBf16, DeviceType::kVULKAN, reinterpret_cast(static_cast(&SigmoidGateBf16Kernel))); RegisterOp(OpId::kRmsNormGated, DeviceType::kVULKAN, @@ -2634,6 +2808,17 @@ struct Registrar { RegisterOp( OpId::kCausalConv1dUpdate, DeviceType::kVULKAN, reinterpret_cast(static_cast(&CausalConv1dUpdateKernel))); + // The PREFILL conv. VT_VULKAN_CONV_FWD=0 leaves it on the portable CPU + // reference tier, where it lived before this kernel existed, so its effect + // can be attributed with a same-binary A/B rather than across two builds -- + // the same shape as VT_VULKAN_COOPMAT. + { + const char* conv_env = std::getenv("VT_VULKAN_CONV_FWD"); + if (conv_env == nullptr || conv_env[0] != '0') { + RegisterOp(OpId::kCausalConv1dFwd, DeviceType::kVULKAN, + reinterpret_cast(static_cast(&CausalConv1dFwdKernel))); + } + } RegisterOp(OpId::kGdnPostConv, DeviceType::kVULKAN, reinterpret_cast(static_cast(&GdnPostConvKernel))); // BACKEND-VULKAN-GDN-CORE: the two recurrences. diff --git a/src/vt/vulkan/vulkan_spirv.cpp b/src/vt/vulkan/vulkan_spirv.cpp index b69772f52f..8a469cfcd5 100644 --- a/src/vt/vulkan/vulkan_spirv.cpp +++ b/src/vt/vulkan/vulkan_spirv.cpp @@ -18,7 +18,7 @@ // and extern declarations; vulkan_spirv.cpp carries the data, so adding shaders // costs one TU's compile time rather than all of them. // -// Produced by: Glslang Version: 11:16.4.0 +// Produced by: Glslang Version: 11:16.5.0 // Target environment: vulkan1.1 #include "vulkan_spirv.h" @@ -1356,6 +1356,667 @@ constexpr uint32_t kSpv_vt_cast[] = { 0x000200feu, 0x0000014au, 0x00010038u, }; +constexpr uint32_t kSpv_vt_causal_conv1d_fwd[] = { + 0x07230203u, 0x00010300u, 0x0008000bu, 0x000003e7u, 0x00000000u, 0x00020011u, 0x00000001u, 0x00020011u, + 0x00001151u, 0x0006000bu, 0x00000001u, 0x4c534c47u, 0x6474732eu, 0x3035342eu, 0x00000000u, 0x0003000eu, + 0x00000000u, 0x00000001u, 0x0006000fu, 0x00000005u, 0x00000004u, 0x6e69616du, 0x00000000u, 0x00000150u, + 0x00060010u, 0x00000004u, 0x00000011u, 0x00000080u, 0x00000001u, 0x00000001u, 0x00040047u, 0x00000150u, + 0x0000000bu, 0x0000001cu, 0x00030047u, 0x00000155u, 0x00000002u, 0x00050048u, 0x00000155u, 0x00000000u, + 0x00000023u, 0x00000000u, 0x00050048u, 0x00000155u, 0x00000001u, 0x00000023u, 0x00000004u, 0x00050048u, + 0x00000155u, 0x00000002u, 0x00000023u, 0x00000008u, 0x00050048u, 0x00000155u, 0x00000003u, 0x00000023u, + 0x0000000cu, 0x00050048u, 0x00000155u, 0x00000004u, 0x00000023u, 0x00000010u, 0x00050048u, 0x00000155u, + 0x00000005u, 0x00000023u, 0x00000014u, 0x00050048u, 0x00000155u, 0x00000006u, 0x00000023u, 0x00000018u, + 0x00050048u, 0x00000155u, 0x00000007u, 0x00000023u, 0x0000001cu, 0x00050048u, 0x00000155u, 0x00000008u, + 0x00000023u, 0x00000020u, 0x00050048u, 0x00000155u, 0x00000009u, 0x00000023u, 0x00000024u, 0x00050048u, + 0x00000155u, 0x0000000au, 0x00000023u, 0x00000028u, 0x00050048u, 0x00000155u, 0x0000000bu, 0x00000023u, + 0x0000002cu, 0x00050048u, 0x00000155u, 0x0000000cu, 0x00000023u, 0x00000030u, 0x00050048u, 0x00000155u, + 0x0000000du, 0x00000023u, 0x00000034u, 0x00050048u, 0x00000155u, 0x0000000eu, 0x00000023u, 0x00000038u, + 0x00050048u, 0x00000155u, 0x0000000fu, 0x00000023u, 0x0000003cu, 0x00050048u, 0x00000155u, 0x00000010u, + 0x00000023u, 0x00000040u, 0x00050048u, 0x00000155u, 0x00000011u, 0x00000023u, 0x00000044u, 0x00050048u, + 0x00000155u, 0x00000012u, 0x00000023u, 0x00000048u, 0x00050048u, 0x00000155u, 0x00000013u, 0x00000023u, + 0x0000004cu, 0x00050048u, 0x00000155u, 0x00000014u, 0x00000023u, 0x00000050u, 0x00040047u, 0x0000015eu, + 0x00000001u, 0x00000000u, 0x00040047u, 0x00000175u, 0x00000006u, 0x00000004u, 0x00030047u, 0x00000176u, + 0x00000002u, 0x00040048u, 0x00000176u, 0x00000000u, 0x00000018u, 0x00050048u, 0x00000176u, 0x00000000u, + 0x00000023u, 0x00000000u, 0x00030047u, 0x00000178u, 0x00000018u, 0x00040047u, 0x00000178u, 0x00000021u, + 0x0000000au, 0x00040047u, 0x00000178u, 0x00000022u, 0x00000000u, 0x00040047u, 0x000001acu, 0x00000006u, + 0x00000004u, 0x00030047u, 0x000001adu, 0x00000002u, 0x00040048u, 0x000001adu, 0x00000000u, 0x00000018u, + 0x00050048u, 0x000001adu, 0x00000000u, 0x00000023u, 0x00000000u, 0x00030047u, 0x000001afu, 0x00000018u, + 0x00040047u, 0x000001afu, 0x00000021u, 0x0000000bu, 0x00040047u, 0x000001afu, 0x00000022u, 0x00000000u, + 0x00040047u, 0x000001ebu, 0x00000006u, 0x00000004u, 0x00030047u, 0x000001ecu, 0x00000002u, 0x00050048u, + 0x000001ecu, 0x00000000u, 0x00000023u, 0x00000000u, 0x00040047u, 0x000001eeu, 0x00000021u, 0x00000008u, + 0x00040047u, 0x000001eeu, 0x00000022u, 0x00000000u, 0x00040047u, 0x000001fcu, 0x00000006u, 0x00000002u, + 0x00030047u, 0x000001fdu, 0x00000002u, 0x00050048u, 0x000001fdu, 0x00000000u, 0x00000023u, 0x00000000u, + 0x00040047u, 0x000001ffu, 0x00000021u, 0x00000009u, 0x00040047u, 0x000001ffu, 0x00000022u, 0x00000000u, + 0x00040047u, 0x00000223u, 0x00000006u, 0x00000004u, 0x00030047u, 0x00000224u, 0x00000002u, 0x00040048u, + 0x00000224u, 0x00000000u, 0x00000018u, 0x00050048u, 0x00000224u, 0x00000000u, 0x00000023u, 0x00000000u, + 0x00030047u, 0x00000226u, 0x00000018u, 0x00040047u, 0x00000226u, 0x00000021u, 0x00000006u, 0x00040047u, + 0x00000226u, 0x00000022u, 0x00000000u, 0x00040047u, 0x00000231u, 0x00000006u, 0x00000002u, 0x00030047u, + 0x00000232u, 0x00000002u, 0x00040048u, 0x00000232u, 0x00000000u, 0x00000018u, 0x00050048u, 0x00000232u, + 0x00000000u, 0x00000023u, 0x00000000u, 0x00030047u, 0x00000234u, 0x00000018u, 0x00040047u, 0x00000234u, + 0x00000021u, 0x00000007u, 0x00040047u, 0x00000234u, 0x00000022u, 0x00000000u, 0x00040047u, 0x0000029du, + 0x00000006u, 0x00000004u, 0x00030047u, 0x0000029eu, 0x00000002u, 0x00040048u, 0x0000029eu, 0x00000000u, + 0x00000018u, 0x00050048u, 0x0000029eu, 0x00000000u, 0x00000023u, 0x00000000u, 0x00030047u, 0x000002a0u, + 0x00000018u, 0x00040047u, 0x000002a0u, 0x00000021u, 0x00000002u, 0x00040047u, 0x000002a0u, 0x00000022u, + 0x00000000u, 0x00040047u, 0x000002b3u, 0x00000006u, 0x00000002u, 0x00030047u, 0x000002b4u, 0x00000002u, + 0x00040048u, 0x000002b4u, 0x00000000u, 0x00000018u, 0x00050048u, 0x000002b4u, 0x00000000u, 0x00000023u, + 0x00000000u, 0x00030047u, 0x000002b6u, 0x00000018u, 0x00040047u, 0x000002b6u, 0x00000021u, 0x00000003u, + 0x00040047u, 0x000002b6u, 0x00000022u, 0x00000000u, 0x00040047u, 0x000002e0u, 0x00000006u, 0x00000004u, + 0x00030047u, 0x000002e1u, 0x00000002u, 0x00040048u, 0x000002e1u, 0x00000000u, 0x00000018u, 0x00050048u, + 0x000002e1u, 0x00000000u, 0x00000023u, 0x00000000u, 0x00030047u, 0x000002e3u, 0x00000018u, 0x00040047u, + 0x000002e3u, 0x00000021u, 0x00000004u, 0x00040047u, 0x000002e3u, 0x00000022u, 0x00000000u, 0x00040047u, + 0x000002f2u, 0x00000006u, 0x00000002u, 0x00030047u, 0x000002f3u, 0x00000002u, 0x00040048u, 0x000002f3u, + 0x00000000u, 0x00000018u, 0x00050048u, 0x000002f3u, 0x00000000u, 0x00000023u, 0x00000000u, 0x00030047u, + 0x000002f5u, 0x00000018u, 0x00040047u, 0x000002f5u, 0x00000021u, 0x00000005u, 0x00040047u, 0x000002f5u, + 0x00000022u, 0x00000000u, 0x00040047u, 0x00000319u, 0x00000006u, 0x00000004u, 0x00030047u, 0x0000031au, + 0x00000002u, 0x00050048u, 0x0000031au, 0x00000000u, 0x00000023u, 0x00000000u, 0x00040047u, 0x0000031cu, + 0x00000021u, 0x00000000u, 0x00040047u, 0x0000031cu, 0x00000022u, 0x00000000u, 0x00040047u, 0x0000033du, + 0x00000006u, 0x00000002u, 0x00030047u, 0x0000033eu, 0x00000002u, 0x00050048u, 0x0000033eu, 0x00000000u, + 0x00000023u, 0x00000000u, 0x00040047u, 0x00000340u, 0x00000021u, 0x00000001u, 0x00040047u, 0x00000340u, + 0x00000022u, 0x00000000u, 0x00040047u, 0x000003e6u, 0x0000000bu, 0x00000019u, 0x00020013u, 0x00000002u, + 0x00030021u, 0x00000003u, 0x00000002u, 0x00040015u, 0x00000006u, 0x00000020u, 0x00000000u, 0x00040020u, + 0x00000007u, 0x00000007u, 0x00000006u, 0x00030016u, 0x00000008u, 0x00000020u, 0x00040021u, 0x00000009u, + 0x00000008u, 0x00000007u, 0x00040020u, 0x0000000du, 0x00000007u, 0x00000008u, 0x00040021u, 0x0000000eu, + 0x00000006u, 0x0000000du, 0x00050021u, 0x00000018u, 0x00000008u, 0x00000007u, 0x00000007u, 0x00050021u, + 0x0000001du, 0x00000006u, 0x0000000du, 0x00000007u, 0x00040015u, 0x00000023u, 0x00000020u, 0x00000001u, + 0x0004002bu, 0x00000023u, 0x00000024u, 0x00000010u, 0x00020014u, 0x0000002cu, 0x0004002bu, 0x00000006u, + 0x0000002eu, 0x7f800000u, 0x0004002bu, 0x00000006u, 0x00000034u, 0x007fffffu, 0x0004002bu, 0x00000006u, + 0x00000036u, 0x00000000u, 0x0004002bu, 0x00000006u, 0x0000003du, 0x00000040u, 0x0004002bu, 0x00000006u, + 0x0000003fu, 0x0000ffffu, 0x0004002bu, 0x00000006u, 0x00000043u, 0x00007fffu, 0x0004002bu, 0x00000006u, + 0x00000046u, 0x00000001u, 0x0004002bu, 0x00000006u, 0x00000052u, 0x00008000u, 0x0004002bu, 0x00000023u, + 0x00000057u, 0x0000000au, 0x0004002bu, 0x00000006u, 0x00000059u, 0x0000001fu, 0x0004002bu, 0x00000006u, + 0x0000005du, 0x000003ffu, 0x0004002bu, 0x00000023u, 0x00000066u, 0x0000000du, 0x0004002bu, 0x00000006u, + 0x0000007du, 0x00000400u, 0x0004002bu, 0x00000023u, 0x00000080u, 0x00000001u, 0x0004002bu, 0x00000006u, + 0x00000088u, 0x00000071u, 0x0004002bu, 0x00000023u, 0x0000008bu, 0x00000017u, 0x0004002bu, 0x00000006u, + 0x00000095u, 0x00000070u, 0x0004002bu, 0x00000006u, 0x000000a9u, 0x000000ffu, 0x00040020u, 0x000000abu, + 0x00000007u, 0x00000023u, 0x0004002bu, 0x00000023u, 0x000000afu, 0x0000007fu, 0x0004002bu, 0x00000023u, + 0x000000b1u, 0x0000000fu, 0x0004002bu, 0x00000006u, 0x000000bbu, 0x00007c00u, 0x0004002bu, 0x00000006u, + 0x000000c2u, 0x00000200u, 0x0004002bu, 0x00000023u, 0x000000cbu, 0x0000001fu, 0x0004002bu, 0x00000023u, + 0x000000d3u, 0x00000000u, 0x0004002bu, 0x00000023u, 0x000000d8u, 0xfffffff6u, 0x0004002bu, 0x00000006u, + 0x000000deu, 0x00800000u, 0x0004002bu, 0x00000023u, 0x000000e2u, 0x0000000eu, 0x0004002bu, 0x00000006u, + 0x00000115u, 0x00001fffu, 0x0004002bu, 0x00000006u, 0x00000118u, 0x00001000u, 0x00040017u, 0x0000014eu, + 0x00000006u, 0x00000003u, 0x00040020u, 0x0000014fu, 0x00000001u, 0x0000014eu, 0x0004003bu, 0x0000014fu, + 0x00000150u, 0x00000001u, 0x00040020u, 0x00000151u, 0x00000001u, 0x00000006u, 0x0017001eu, 0x00000155u, + 0x00000006u, 0x00000006u, 0x00000006u, 0x00000006u, 0x00000006u, 0x00000006u, 0x00000006u, 0x00000006u, + 0x00000006u, 0x00000006u, 0x00000006u, 0x00000006u, 0x00000006u, 0x00000006u, 0x00000006u, 0x00000006u, + 0x00000006u, 0x00000006u, 0x00000006u, 0x00000006u, 0x00000006u, 0x00040020u, 0x00000156u, 0x00000009u, + 0x00000155u, 0x0004003bu, 0x00000156u, 0x00000157u, 0x00000009u, 0x00040020u, 0x00000158u, 0x00000009u, + 0x00000006u, 0x00040032u, 0x00000006u, 0x0000015eu, 0x00000001u, 0x0003001du, 0x00000175u, 0x00000006u, + 0x0003001eu, 0x00000176u, 0x00000175u, 0x00040020u, 0x00000177u, 0x0000000cu, 0x00000176u, 0x0004003bu, + 0x00000177u, 0x00000178u, 0x0000000cu, 0x0004002bu, 0x00000023u, 0x00000179u, 0x00000013u, 0x0004002bu, + 0x00000023u, 0x0000017cu, 0x00000002u, 0x00040020u, 0x00000180u, 0x0000000cu, 0x00000006u, 0x0004002bu, + 0x00000023u, 0x00000193u, 0x00000005u, 0x0004002bu, 0x00000023u, 0x0000019fu, 0x00000007u, 0x0004002bu, + 0x00000023u, 0x000001a6u, 0x00000014u, 0x0003001du, 0x000001acu, 0x00000006u, 0x0003001eu, 0x000001adu, + 0x000001acu, 0x00040020u, 0x000001aeu, 0x0000000cu, 0x000001adu, 0x0004003bu, 0x000001aeu, 0x000001afu, + 0x0000000cu, 0x00040020u, 0x000001b4u, 0x00000007u, 0x0000002cu, 0x0004002bu, 0x00000006u, 0x000001b8u, + 0x00000003u, 0x0004002bu, 0x00000006u, 0x000001bau, 0x00000008u, 0x0004002bu, 0x00000023u, 0x000001cfu, + 0x00000003u, 0x0004001cu, 0x000001e1u, 0x00000008u, 0x000001bau, 0x00040020u, 0x000001e2u, 0x00000007u, + 0x000001e1u, 0x0003001du, 0x000001ebu, 0x00000006u, 0x0003001eu, 0x000001ecu, 0x000001ebu, 0x00040020u, + 0x000001edu, 0x0000000cu, 0x000001ecu, 0x0004003bu, 0x000001edu, 0x000001eeu, 0x0000000cu, 0x0004002bu, + 0x00000023u, 0x000001efu, 0x00000012u, 0x00040015u, 0x000001fbu, 0x00000010u, 0x00000000u, 0x0003001du, + 0x000001fcu, 0x000001fbu, 0x0003001eu, 0x000001fdu, 0x000001fcu, 0x00040020u, 0x000001feu, 0x0000000cu, + 0x000001fdu, 0x0004003bu, 0x000001feu, 0x000001ffu, 0x0000000cu, 0x00040020u, 0x00000207u, 0x0000000cu, + 0x000001fbu, 0x0004002bu, 0x00000023u, 0x00000215u, 0x00000006u, 0x0004002bu, 0x00000023u, 0x0000021cu, + 0x0000000cu, 0x0003001du, 0x00000223u, 0x00000006u, 0x0003001eu, 0x00000224u, 0x00000223u, 0x00040020u, + 0x00000225u, 0x0000000cu, 0x00000224u, 0x0004003bu, 0x00000225u, 0x00000226u, 0x0000000cu, 0x0004002bu, + 0x00000023u, 0x00000227u, 0x00000011u, 0x0003001du, 0x00000231u, 0x000001fbu, 0x0003001eu, 0x00000232u, + 0x00000231u, 0x00040020u, 0x00000233u, 0x0000000cu, 0x00000232u, 0x0004003bu, 0x00000233u, 0x00000234u, + 0x0000000cu, 0x0004002bu, 0x00000008u, 0x00000244u, 0x00000000u, 0x0003001du, 0x0000029du, 0x00000006u, + 0x0003001eu, 0x0000029eu, 0x0000029du, 0x00040020u, 0x0000029fu, 0x0000000cu, 0x0000029eu, 0x0004003bu, + 0x0000029fu, 0x000002a0u, 0x0000000cu, 0x0004002bu, 0x00000023u, 0x000002a8u, 0x00000004u, 0x0003001du, + 0x000002b3u, 0x000001fbu, 0x0003001eu, 0x000002b4u, 0x000002b3u, 0x00040020u, 0x000002b5u, 0x0000000cu, + 0x000002b4u, 0x0004003bu, 0x000002b5u, 0x000002b6u, 0x0000000cu, 0x0004002bu, 0x00000023u, 0x000002d9u, + 0x0000000bu, 0x0003001du, 0x000002e0u, 0x00000006u, 0x0003001eu, 0x000002e1u, 0x000002e0u, 0x00040020u, + 0x000002e2u, 0x0000000cu, 0x000002e1u, 0x0004003bu, 0x000002e2u, 0x000002e3u, 0x0000000cu, 0x0003001du, + 0x000002f2u, 0x000001fbu, 0x0003001eu, 0x000002f3u, 0x000002f2u, 0x00040020u, 0x000002f4u, 0x0000000cu, + 0x000002f3u, 0x0004003bu, 0x000002f4u, 0x000002f5u, 0x0000000cu, 0x0004002bu, 0x00000023u, 0x00000313u, + 0x00000009u, 0x0003001du, 0x00000319u, 0x00000006u, 0x0003001eu, 0x0000031au, 0x00000319u, 0x00040020u, + 0x0000031bu, 0x0000000cu, 0x0000031au, 0x0004003bu, 0x0000031bu, 0x0000031cu, 0x0000000cu, 0x0004002bu, + 0x00000023u, 0x00000329u, 0x00000008u, 0x0004002bu, 0x00000008u, 0x00000331u, 0x3f800000u, 0x0003001du, + 0x0000033du, 0x000001fbu, 0x0003001eu, 0x0000033eu, 0x0000033du, 0x00040020u, 0x0000033fu, 0x0000000cu, + 0x0000033eu, 0x0004003bu, 0x0000033fu, 0x00000340u, 0x0000000cu, 0x0003002au, 0x0000002cu, 0x00000363u, + 0x0004002bu, 0x00000006u, 0x000003e2u, 0x00000080u, 0x0004001cu, 0x000003e3u, 0x00000008u, 0x000003e2u, + 0x00040020u, 0x000003e4u, 0x00000004u, 0x000003e3u, 0x0004003bu, 0x000003e4u, 0x000003e5u, 0x00000004u, + 0x0006002cu, 0x0000014eu, 0x000003e6u, 0x000003e2u, 0x00000046u, 0x00000046u, 0x00050036u, 0x00000002u, + 0x00000004u, 0x00000000u, 0x00000003u, 0x000200f8u, 0x00000005u, 0x0004003bu, 0x00000007u, 0x0000014du, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000164u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000167u, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000016au, 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000016fu, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000174u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000183u, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000019bu, 0x00000007u, 0x0004003bu, 0x00000007u, 0x000001a5u, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x000001abu, 0x00000007u, 0x0004003bu, 0x000001b4u, 0x000001b5u, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x000001c8u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x000001d7u, + 0x00000007u, 0x0004003bu, 0x000001e2u, 0x000001e3u, 0x00000007u, 0x0004003bu, 0x0000000du, 0x000001e8u, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000020bu, 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000020cu, + 0x00000007u, 0x0004003bu, 0x0000000du, 0x00000214u, 0x00000007u, 0x0004003bu, 0x0000000du, 0x00000219u, + 0x00000007u, 0x0004003bu, 0x0000000du, 0x00000220u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000023du, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000023eu, 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000246u, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000024eu, 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000251u, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000025cu, 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000025fu, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000271u, 0x00000007u, 0x0004003bu, 0x0000000du, 0x0000027bu, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000027du, 0x00000007u, 0x0004003bu, 0x000000abu, 0x00000287u, + 0x00000007u, 0x0004003bu, 0x0000000du, 0x00000292u, 0x00000007u, 0x0004003bu, 0x0000000du, 0x0000029au, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x000002c7u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x000002c8u, + 0x00000007u, 0x0004003bu, 0x0000000du, 0x000002ddu, 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000303u, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000304u, 0x00000007u, 0x0004003bu, 0x0000000du, 0x0000032du, + 0x00000007u, 0x0004003bu, 0x0000000du, 0x00000350u, 0x00000007u, 0x0004003bu, 0x0000000du, 0x0000035bu, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000035du, 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000036bu, + 0x00000007u, 0x0004003bu, 0x000000abu, 0x00000375u, 0x00000007u, 0x0004003bu, 0x0000000du, 0x0000037fu, + 0x00000007u, 0x0004003bu, 0x0000000du, 0x00000387u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x000003abu, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x000003acu, 0x00000007u, 0x0004003bu, 0x0000000du, 0x000003d8u, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x000003dau, 0x00000007u, 0x00050041u, 0x00000151u, 0x00000152u, + 0x00000150u, 0x00000036u, 0x0004003du, 0x00000006u, 0x00000153u, 0x00000152u, 0x0003003eu, 0x0000014du, + 0x00000153u, 0x0004003du, 0x00000006u, 0x00000154u, 0x0000014du, 0x00050041u, 0x00000158u, 0x00000159u, + 0x00000157u, 0x000000d3u, 0x0004003du, 0x00000006u, 0x0000015au, 0x00000159u, 0x00050041u, 0x00000158u, + 0x0000015bu, 0x00000157u, 0x00000080u, 0x0004003du, 0x00000006u, 0x0000015cu, 0x0000015bu, 0x00050084u, + 0x00000006u, 0x0000015du, 0x0000015au, 0x0000015cu, 0x00050084u, 0x00000006u, 0x0000015fu, 0x0000015du, + 0x0000015eu, 0x000500aeu, 0x0000002cu, 0x00000160u, 0x00000154u, 0x0000015fu, 0x000300f7u, 0x00000162u, + 0x00000000u, 0x000400fau, 0x00000160u, 0x00000161u, 0x00000162u, 0x000200f8u, 0x00000161u, 0x000100fdu, + 0x000200f8u, 0x00000162u, 0x0004003du, 0x00000006u, 0x00000165u, 0x0000014du, 0x00050089u, 0x00000006u, + 0x00000166u, 0x00000165u, 0x0000015eu, 0x0003003eu, 0x00000164u, 0x00000166u, 0x0004003du, 0x00000006u, + 0x00000168u, 0x0000014du, 0x00050086u, 0x00000006u, 0x00000169u, 0x00000168u, 0x0000015eu, 0x0003003eu, + 0x00000167u, 0x00000169u, 0x0004003du, 0x00000006u, 0x0000016bu, 0x00000167u, 0x00050041u, 0x00000158u, + 0x0000016cu, 0x00000157u, 0x00000080u, 0x0004003du, 0x00000006u, 0x0000016du, 0x0000016cu, 0x00050086u, + 0x00000006u, 0x0000016eu, 0x0000016bu, 0x0000016du, 0x0003003eu, 0x0000016au, 0x0000016eu, 0x0004003du, + 0x00000006u, 0x00000170u, 0x00000167u, 0x00050041u, 0x00000158u, 0x00000171u, 0x00000157u, 0x00000080u, + 0x0004003du, 0x00000006u, 0x00000172u, 0x00000171u, 0x00050089u, 0x00000006u, 0x00000173u, 0x00000170u, + 0x00000172u, 0x0003003eu, 0x0000016fu, 0x00000173u, 0x00050041u, 0x00000158u, 0x0000017au, 0x00000157u, + 0x00000179u, 0x0004003du, 0x00000006u, 0x0000017bu, 0x0000017au, 0x000500c2u, 0x00000006u, 0x0000017du, + 0x0000017bu, 0x0000017cu, 0x0004003du, 0x00000006u, 0x0000017eu, 0x0000016au, 0x00050080u, 0x00000006u, + 0x0000017fu, 0x0000017du, 0x0000017eu, 0x00060041u, 0x00000180u, 0x00000181u, 0x00000178u, 0x000000d3u, + 0x0000017fu, 0x0004003du, 0x00000006u, 0x00000182u, 0x00000181u, 0x0003003eu, 0x00000174u, 0x00000182u, + 0x00050041u, 0x00000158u, 0x00000184u, 0x00000157u, 0x00000179u, 0x0004003du, 0x00000006u, 0x00000185u, + 0x00000184u, 0x000500c2u, 0x00000006u, 0x00000186u, 0x00000185u, 0x0000017cu, 0x0004003du, 0x00000006u, + 0x00000187u, 0x0000016au, 0x00050080u, 0x00000006u, 0x00000188u, 0x00000186u, 0x00000187u, 0x00050080u, + 0x00000006u, 0x00000189u, 0x00000188u, 0x00000046u, 0x00060041u, 0x00000180u, 0x0000018au, 0x00000178u, + 0x000000d3u, 0x00000189u, 0x0004003du, 0x00000006u, 0x0000018bu, 0x0000018au, 0x0003003eu, 0x00000183u, + 0x0000018bu, 0x0004003du, 0x00000006u, 0x0000018cu, 0x00000183u, 0x0004003du, 0x00000006u, 0x0000018du, + 0x00000174u, 0x000500b0u, 0x0000002cu, 0x0000018eu, 0x0000018cu, 0x0000018du, 0x000400a8u, 0x0000002cu, + 0x0000018fu, 0x0000018eu, 0x000300f7u, 0x00000191u, 0x00000000u, 0x000400fau, 0x0000018fu, 0x00000190u, + 0x00000191u, 0x000200f8u, 0x00000190u, 0x0004003du, 0x00000006u, 0x00000192u, 0x00000183u, 0x00050041u, + 0x00000158u, 0x00000194u, 0x00000157u, 0x00000193u, 0x0004003du, 0x00000006u, 0x00000195u, 0x00000194u, + 0x000500acu, 0x0000002cu, 0x00000196u, 0x00000192u, 0x00000195u, 0x000200f9u, 0x00000191u, 0x000200f8u, + 0x00000191u, 0x000700f5u, 0x0000002cu, 0x00000197u, 0x0000018eu, 0x00000162u, 0x00000196u, 0x00000190u, + 0x000300f7u, 0x00000199u, 0x00000000u, 0x000400fau, 0x00000197u, 0x00000198u, 0x00000199u, 0x000200f8u, + 0x00000198u, 0x000100fdu, 0x000200f8u, 0x00000199u, 0x0004003du, 0x00000006u, 0x0000019cu, 0x00000183u, + 0x0004003du, 0x00000006u, 0x0000019du, 0x00000174u, 0x00050082u, 0x00000006u, 0x0000019eu, 0x0000019cu, + 0x0000019du, 0x0003003eu, 0x0000019bu, 0x0000019eu, 0x00050041u, 0x00000158u, 0x000001a0u, 0x00000157u, + 0x0000019fu, 0x0004003du, 0x00000006u, 0x000001a1u, 0x000001a0u, 0x000500abu, 0x0000002cu, 0x000001a2u, + 0x000001a1u, 0x00000036u, 0x000300f7u, 0x000001a4u, 0x00000000u, 0x000400fau, 0x000001a2u, 0x000001a3u, + 0x000001bfu, 0x000200f8u, 0x000001a3u, 0x00050041u, 0x00000158u, 0x000001a7u, 0x00000157u, 0x000001a6u, + 0x0004003du, 0x00000006u, 0x000001a8u, 0x000001a7u, 0x0004003du, 0x00000006u, 0x000001a9u, 0x0000016au, + 0x00050080u, 0x00000006u, 0x000001aau, 0x000001a8u, 0x000001a9u, 0x0003003eu, 0x000001a5u, 0x000001aau, + 0x0004003du, 0x00000006u, 0x000001b0u, 0x000001a5u, 0x000500c2u, 0x00000006u, 0x000001b1u, 0x000001b0u, + 0x0000017cu, 0x00060041u, 0x00000180u, 0x000001b2u, 0x000001afu, 0x000000d3u, 0x000001b1u, 0x0004003du, + 0x00000006u, 0x000001b3u, 0x000001b2u, 0x0003003eu, 0x000001abu, 0x000001b3u, 0x0004003du, 0x00000006u, + 0x000001b6u, 0x000001abu, 0x0004003du, 0x00000006u, 0x000001b7u, 0x000001a5u, 0x000500c7u, 0x00000006u, + 0x000001b9u, 0x000001b7u, 0x000001b8u, 0x00050084u, 0x00000006u, 0x000001bbu, 0x000001b9u, 0x000001bau, + 0x000500c2u, 0x00000006u, 0x000001bcu, 0x000001b6u, 0x000001bbu, 0x000500c7u, 0x00000006u, 0x000001bdu, + 0x000001bcu, 0x000000a9u, 0x000500abu, 0x0000002cu, 0x000001beu, 0x000001bdu, 0x00000036u, 0x0003003eu, + 0x000001b5u, 0x000001beu, 0x000200f9u, 0x000001a4u, 0x000200f8u, 0x000001bfu, 0x00050041u, 0x00000158u, + 0x000001c0u, 0x00000157u, 0x000001a6u, 0x0004003du, 0x00000006u, 0x000001c1u, 0x000001c0u, 0x000500c2u, + 0x00000006u, 0x000001c2u, 0x000001c1u, 0x0000017cu, 0x0004003du, 0x00000006u, 0x000001c3u, 0x0000016au, + 0x00050080u, 0x00000006u, 0x000001c4u, 0x000001c2u, 0x000001c3u, 0x00060041u, 0x00000180u, 0x000001c5u, + 0x000001afu, 0x000000d3u, 0x000001c4u, 0x0004003du, 0x00000006u, 0x000001c6u, 0x000001c5u, 0x000500abu, + 0x0000002cu, 0x000001c7u, 0x000001c6u, 0x00000036u, 0x0003003eu, 0x000001b5u, 0x000001c7u, 0x000200f9u, + 0x000001a4u, 0x000200f8u, 0x000001a4u, 0x0004003du, 0x00000006u, 0x000001c9u, 0x0000016au, 0x00050041u, + 0x00000158u, 0x000001cau, 0x00000157u, 0x00000080u, 0x0004003du, 0x00000006u, 0x000001cbu, 0x000001cau, + 0x00050084u, 0x00000006u, 0x000001ccu, 0x000001c9u, 0x000001cbu, 0x0004003du, 0x00000006u, 0x000001cdu, + 0x0000016fu, 0x00050080u, 0x00000006u, 0x000001ceu, 0x000001ccu, 0x000001cdu, 0x00050041u, 0x00000158u, + 0x000001d0u, 0x00000157u, 0x000001cfu, 0x0004003du, 0x00000006u, 0x000001d1u, 0x000001d0u, 0x00050084u, + 0x00000006u, 0x000001d2u, 0x000001ceu, 0x000001d1u, 0x0003003eu, 0x000001c8u, 0x000001d2u, 0x0004003du, + 0x00000006u, 0x000001d3u, 0x00000164u, 0x000500aau, 0x0000002cu, 0x000001d4u, 0x000001d3u, 0x00000036u, + 0x000300f7u, 0x000001d6u, 0x00000000u, 0x000400fau, 0x000001d4u, 0x000001d5u, 0x000001d6u, 0x000200f8u, + 0x000001d5u, 0x0003003eu, 0x000001d7u, 0x00000036u, 0x000200f9u, 0x000001d8u, 0x000200f8u, 0x000001d8u, + 0x000400f6u, 0x000001dau, 0x000001dbu, 0x00000000u, 0x000200f9u, 0x000001dcu, 0x000200f8u, 0x000001dcu, + 0x0004003du, 0x00000006u, 0x000001ddu, 0x000001d7u, 0x00050041u, 0x00000158u, 0x000001deu, 0x00000157u, + 0x000001cfu, 0x0004003du, 0x00000006u, 0x000001dfu, 0x000001deu, 0x000500b0u, 0x0000002cu, 0x000001e0u, + 0x000001ddu, 0x000001dfu, 0x000400fau, 0x000001e0u, 0x000001d9u, 0x000001dau, 0x000200f8u, 0x000001d9u, + 0x0004003du, 0x00000006u, 0x000001e4u, 0x000001d7u, 0x00050041u, 0x00000158u, 0x000001e5u, 0x00000157u, + 0x00000066u, 0x0004003du, 0x00000006u, 0x000001e6u, 0x000001e5u, 0x000500aau, 0x0000002cu, 0x000001e7u, + 0x000001e6u, 0x00000036u, 0x000300f7u, 0x000001eau, 0x00000000u, 0x000400fau, 0x000001e7u, 0x000001e9u, + 0x000001fau, 0x000200f8u, 0x000001e9u, 0x00050041u, 0x00000158u, 0x000001f0u, 0x00000157u, 0x000001efu, + 0x0004003du, 0x00000006u, 0x000001f1u, 0x000001f0u, 0x000500c2u, 0x00000006u, 0x000001f2u, 0x000001f1u, + 0x0000017cu, 0x0004003du, 0x00000006u, 0x000001f3u, 0x000001c8u, 0x0004003du, 0x00000006u, 0x000001f4u, + 0x000001d7u, 0x00050080u, 0x00000006u, 0x000001f5u, 0x000001f3u, 0x000001f4u, 0x00050080u, 0x00000006u, + 0x000001f6u, 0x000001f2u, 0x000001f5u, 0x00060041u, 0x00000180u, 0x000001f7u, 0x000001eeu, 0x000000d3u, + 0x000001f6u, 0x0004003du, 0x00000006u, 0x000001f8u, 0x000001f7u, 0x0004007cu, 0x00000008u, 0x000001f9u, + 0x000001f8u, 0x0003003eu, 0x000001e8u, 0x000001f9u, 0x000200f9u, 0x000001eau, 0x000200f8u, 0x000001fau, + 0x00050041u, 0x00000158u, 0x00000200u, 0x00000157u, 0x000001efu, 0x0004003du, 0x00000006u, 0x00000201u, + 0x00000200u, 0x000500c2u, 0x00000006u, 0x00000202u, 0x00000201u, 0x00000080u, 0x0004003du, 0x00000006u, + 0x00000203u, 0x000001c8u, 0x0004003du, 0x00000006u, 0x00000204u, 0x000001d7u, 0x00050080u, 0x00000006u, + 0x00000205u, 0x00000203u, 0x00000204u, 0x00050080u, 0x00000006u, 0x00000206u, 0x00000202u, 0x00000205u, + 0x00060041u, 0x00000207u, 0x00000208u, 0x000001ffu, 0x000000d3u, 0x00000206u, 0x0004003du, 0x000001fbu, + 0x00000209u, 0x00000208u, 0x00040071u, 0x00000006u, 0x0000020au, 0x00000209u, 0x0003003eu, 0x0000020bu, + 0x0000020au, 0x00050041u, 0x00000158u, 0x0000020du, 0x00000157u, 0x00000066u, 0x0004003du, 0x00000006u, + 0x0000020eu, 0x0000020du, 0x0003003eu, 0x0000020cu, 0x0000020eu, 0x00060039u, 0x00000008u, 0x0000020fu, + 0x0000001bu, 0x0000020bu, 0x0000020cu, 0x0003003eu, 0x000001e8u, 0x0000020fu, 0x000200f9u, 0x000001eau, + 0x000200f8u, 0x000001eau, 0x0004003du, 0x00000008u, 0x00000210u, 0x000001e8u, 0x00050041u, 0x0000000du, + 0x00000211u, 0x000001e3u, 0x000001e4u, 0x0003003eu, 0x00000211u, 0x00000210u, 0x000200f9u, 0x000001dbu, + 0x000200f8u, 0x000001dbu, 0x0004003du, 0x00000006u, 0x00000212u, 0x000001d7u, 0x00050080u, 0x00000006u, + 0x00000213u, 0x00000212u, 0x00000080u, 0x0003003eu, 0x000001d7u, 0x00000213u, 0x000200f9u, 0x000001d8u, + 0x000200f8u, 0x000001dau, 0x000200f9u, 0x000001d6u, 0x000200f8u, 0x000001d6u, 0x00050041u, 0x00000158u, + 0x00000216u, 0x00000157u, 0x00000215u, 0x0004003du, 0x00000006u, 0x00000217u, 0x00000216u, 0x000500abu, + 0x0000002cu, 0x00000218u, 0x00000217u, 0x00000036u, 0x000300f7u, 0x0000021bu, 0x00000000u, 0x000400fau, + 0x00000218u, 0x0000021au, 0x00000243u, 0x000200f8u, 0x0000021au, 0x00050041u, 0x00000158u, 0x0000021du, + 0x00000157u, 0x0000021cu, 0x0004003du, 0x00000006u, 0x0000021eu, 0x0000021du, 0x000500aau, 0x0000002cu, + 0x0000021fu, 0x0000021eu, 0x00000036u, 0x000300f7u, 0x00000222u, 0x00000000u, 0x000400fau, 0x0000021fu, + 0x00000221u, 0x00000230u, 0x000200f8u, 0x00000221u, 0x00050041u, 0x00000158u, 0x00000228u, 0x00000157u, + 0x00000227u, 0x0004003du, 0x00000006u, 0x00000229u, 0x00000228u, 0x000500c2u, 0x00000006u, 0x0000022au, + 0x00000229u, 0x0000017cu, 0x0004003du, 0x00000006u, 0x0000022bu, 0x0000016fu, 0x00050080u, 0x00000006u, + 0x0000022cu, 0x0000022au, 0x0000022bu, 0x00060041u, 0x00000180u, 0x0000022du, 0x00000226u, 0x000000d3u, + 0x0000022cu, 0x0004003du, 0x00000006u, 0x0000022eu, 0x0000022du, 0x0004007cu, 0x00000008u, 0x0000022fu, + 0x0000022eu, 0x0003003eu, 0x00000220u, 0x0000022fu, 0x000200f9u, 0x00000222u, 0x000200f8u, 0x00000230u, + 0x00050041u, 0x00000158u, 0x00000235u, 0x00000157u, 0x00000227u, 0x0004003du, 0x00000006u, 0x00000236u, + 0x00000235u, 0x000500c2u, 0x00000006u, 0x00000237u, 0x00000236u, 0x00000080u, 0x0004003du, 0x00000006u, + 0x00000238u, 0x0000016fu, 0x00050080u, 0x00000006u, 0x00000239u, 0x00000237u, 0x00000238u, 0x00060041u, + 0x00000207u, 0x0000023au, 0x00000234u, 0x000000d3u, 0x00000239u, 0x0004003du, 0x000001fbu, 0x0000023bu, + 0x0000023au, 0x00040071u, 0x00000006u, 0x0000023cu, 0x0000023bu, 0x0003003eu, 0x0000023du, 0x0000023cu, + 0x00050041u, 0x00000158u, 0x0000023fu, 0x00000157u, 0x0000021cu, 0x0004003du, 0x00000006u, 0x00000240u, + 0x0000023fu, 0x0003003eu, 0x0000023eu, 0x00000240u, 0x00060039u, 0x00000008u, 0x00000241u, 0x0000001bu, + 0x0000023du, 0x0000023eu, 0x0003003eu, 0x00000220u, 0x00000241u, 0x000200f9u, 0x00000222u, 0x000200f8u, + 0x00000222u, 0x0004003du, 0x00000008u, 0x00000242u, 0x00000220u, 0x0003003eu, 0x00000219u, 0x00000242u, + 0x000200f9u, 0x0000021bu, 0x000200f8u, 0x00000243u, 0x0003003eu, 0x00000219u, 0x00000244u, 0x000200f9u, + 0x0000021bu, 0x000200f8u, 0x0000021bu, 0x0004003du, 0x00000008u, 0x00000245u, 0x00000219u, 0x0003003eu, + 0x00000214u, 0x00000245u, 0x0004003du, 0x00000006u, 0x00000247u, 0x0000019bu, 0x00050086u, 0x00000006u, + 0x00000248u, 0x00000247u, 0x0000015eu, 0x0004003du, 0x00000006u, 0x00000249u, 0x0000019bu, 0x00050089u, + 0x00000006u, 0x0000024au, 0x00000249u, 0x0000015eu, 0x000500abu, 0x0000002cu, 0x0000024bu, 0x0000024au, + 0x00000036u, 0x000600a9u, 0x00000006u, 0x0000024cu, 0x0000024bu, 0x00000046u, 0x00000036u, 0x00050080u, + 0x00000006u, 0x0000024du, 0x00000248u, 0x0000024cu, 0x0003003eu, 0x00000246u, 0x0000024du, 0x0004003du, + 0x00000006u, 0x0000024fu, 0x00000164u, 0x000500aau, 0x0000002cu, 0x00000250u, 0x0000024fu, 0x00000036u, + 0x000300f7u, 0x00000253u, 0x00000000u, 0x000400fau, 0x00000250u, 0x00000252u, 0x00000254u, 0x000200f8u, + 0x00000252u, 0x0003003eu, 0x00000251u, 0x00000036u, 0x000200f9u, 0x00000253u, 0x000200f8u, 0x00000254u, + 0x0004003du, 0x00000006u, 0x00000255u, 0x00000164u, 0x0004003du, 0x00000006u, 0x00000256u, 0x00000246u, + 0x00050084u, 0x00000006u, 0x00000257u, 0x00000255u, 0x00000256u, 0x00050041u, 0x00000158u, 0x00000258u, + 0x00000157u, 0x000001cfu, 0x0004003du, 0x00000006u, 0x00000259u, 0x00000258u, 0x0007000cu, 0x00000006u, + 0x0000025au, 0x00000001u, 0x00000029u, 0x00000257u, 0x00000259u, 0x0003003eu, 0x00000251u, 0x0000025au, + 0x000200f9u, 0x00000253u, 0x000200f8u, 0x00000253u, 0x0004003du, 0x00000006u, 0x0000025bu, 0x00000251u, + 0x0003003eu, 0x0000024eu, 0x0000025bu, 0x0004003du, 0x00000006u, 0x0000025du, 0x00000164u, 0x000500aau, + 0x0000002cu, 0x0000025eu, 0x0000025du, 0x00000036u, 0x000300f7u, 0x00000261u, 0x00000000u, 0x000400fau, + 0x0000025eu, 0x00000260u, 0x00000268u, 0x000200f8u, 0x00000260u, 0x0004003du, 0x00000006u, 0x00000262u, + 0x00000246u, 0x00050041u, 0x00000158u, 0x00000263u, 0x00000157u, 0x000001cfu, 0x0004003du, 0x00000006u, + 0x00000264u, 0x00000263u, 0x0007000cu, 0x00000006u, 0x00000265u, 0x00000001u, 0x00000029u, 0x00000262u, + 0x00000264u, 0x0004003du, 0x00000006u, 0x00000266u, 0x0000019bu, 0x0007000cu, 0x00000006u, 0x00000267u, + 0x00000001u, 0x00000026u, 0x00000265u, 0x00000266u, 0x0003003eu, 0x0000025fu, 0x00000267u, 0x000200f9u, + 0x00000261u, 0x000200f8u, 0x00000268u, 0x0004003du, 0x00000006u, 0x00000269u, 0x00000164u, 0x0004003du, + 0x00000006u, 0x0000026au, 0x00000246u, 0x00050084u, 0x00000006u, 0x0000026bu, 0x00000269u, 0x0000026au, + 0x0004003du, 0x00000006u, 0x0000026cu, 0x00000246u, 0x00050080u, 0x00000006u, 0x0000026du, 0x0000026bu, + 0x0000026cu, 0x0004003du, 0x00000006u, 0x0000026eu, 0x0000019bu, 0x0007000cu, 0x00000006u, 0x0000026fu, + 0x00000001u, 0x00000026u, 0x0000026du, 0x0000026eu, 0x0003003eu, 0x0000025fu, 0x0000026fu, 0x000200f9u, + 0x00000261u, 0x000200f8u, 0x00000261u, 0x0004003du, 0x00000006u, 0x00000270u, 0x0000025fu, 0x0003003eu, + 0x0000025cu, 0x00000270u, 0x0004003du, 0x00000006u, 0x00000272u, 0x0000024eu, 0x0003003eu, 0x00000271u, + 0x00000272u, 0x000200f9u, 0x00000273u, 0x000200f8u, 0x00000273u, 0x000400f6u, 0x00000275u, 0x00000276u, + 0x00000000u, 0x000200f9u, 0x00000277u, 0x000200f8u, 0x00000277u, 0x0004003du, 0x00000006u, 0x00000278u, + 0x00000271u, 0x0004003du, 0x00000006u, 0x00000279u, 0x0000025cu, 0x000500b0u, 0x0000002cu, 0x0000027au, + 0x00000278u, 0x00000279u, 0x000400fau, 0x0000027au, 0x00000274u, 0x00000275u, 0x000200f8u, 0x00000274u, + 0x0004003du, 0x00000008u, 0x0000027cu, 0x00000214u, 0x0003003eu, 0x0000027bu, 0x0000027cu, 0x0003003eu, + 0x0000027du, 0x00000036u, 0x000200f9u, 0x0000027eu, 0x000200f8u, 0x0000027eu, 0x000400f6u, 0x00000280u, + 0x00000281u, 0x00000000u, 0x000200f9u, 0x00000282u, 0x000200f8u, 0x00000282u, 0x0004003du, 0x00000006u, + 0x00000283u, 0x0000027du, 0x00050041u, 0x00000158u, 0x00000284u, 0x00000157u, 0x0000017cu, 0x0004003du, + 0x00000006u, 0x00000285u, 0x00000284u, 0x000500b0u, 0x0000002cu, 0x00000286u, 0x00000283u, 0x00000285u, + 0x000400fau, 0x00000286u, 0x0000027fu, 0x00000280u, 0x000200f8u, 0x0000027fu, 0x0004003du, 0x00000006u, + 0x00000288u, 0x00000271u, 0x0004007cu, 0x00000023u, 0x00000289u, 0x00000288u, 0x00050041u, 0x00000158u, + 0x0000028au, 0x00000157u, 0x0000017cu, 0x0004003du, 0x00000006u, 0x0000028bu, 0x0000028au, 0x0004007cu, + 0x00000023u, 0x0000028cu, 0x0000028bu, 0x00050082u, 0x00000023u, 0x0000028du, 0x0000028cu, 0x00000080u, + 0x0004003du, 0x00000006u, 0x0000028eu, 0x0000027du, 0x0004007cu, 0x00000023u, 0x0000028fu, 0x0000028eu, + 0x00050082u, 0x00000023u, 0x00000290u, 0x0000028du, 0x0000028fu, 0x00050082u, 0x00000023u, 0x00000291u, + 0x00000289u, 0x00000290u, 0x0003003eu, 0x00000287u, 0x00000291u, 0x0003003eu, 0x00000292u, 0x00000244u, + 0x0004003du, 0x00000023u, 0x00000293u, 0x00000287u, 0x000500afu, 0x0000002cu, 0x00000294u, 0x00000293u, + 0x000000d3u, 0x000300f7u, 0x00000296u, 0x00000000u, 0x000400fau, 0x00000294u, 0x00000295u, 0x000002cdu, + 0x000200f8u, 0x00000295u, 0x00050041u, 0x00000158u, 0x00000297u, 0x00000157u, 0x00000057u, 0x0004003du, + 0x00000006u, 0x00000298u, 0x00000297u, 0x000500aau, 0x0000002cu, 0x00000299u, 0x00000298u, 0x00000036u, + 0x000300f7u, 0x0000029cu, 0x00000000u, 0x000400fau, 0x00000299u, 0x0000029bu, 0x000002b2u, 0x000200f8u, + 0x0000029bu, 0x00050041u, 0x00000158u, 0x000002a1u, 0x00000157u, 0x000000b1u, 0x0004003du, 0x00000006u, + 0x000002a2u, 0x000002a1u, 0x000500c2u, 0x00000006u, 0x000002a3u, 0x000002a2u, 0x0000017cu, 0x0004003du, + 0x00000006u, 0x000002a4u, 0x00000174u, 0x0004003du, 0x00000023u, 0x000002a5u, 0x00000287u, 0x0004007cu, + 0x00000006u, 0x000002a6u, 0x000002a5u, 0x00050080u, 0x00000006u, 0x000002a7u, 0x000002a4u, 0x000002a6u, + 0x00050041u, 0x00000158u, 0x000002a9u, 0x00000157u, 0x000002a8u, 0x0004003du, 0x00000006u, 0x000002aau, + 0x000002a9u, 0x00050084u, 0x00000006u, 0x000002abu, 0x000002a7u, 0x000002aau, 0x0004003du, 0x00000006u, + 0x000002acu, 0x0000016fu, 0x00050080u, 0x00000006u, 0x000002adu, 0x000002abu, 0x000002acu, 0x00050080u, + 0x00000006u, 0x000002aeu, 0x000002a3u, 0x000002adu, 0x00060041u, 0x00000180u, 0x000002afu, 0x000002a0u, + 0x000000d3u, 0x000002aeu, 0x0004003du, 0x00000006u, 0x000002b0u, 0x000002afu, 0x0004007cu, 0x00000008u, + 0x000002b1u, 0x000002b0u, 0x0003003eu, 0x0000029au, 0x000002b1u, 0x000200f9u, 0x0000029cu, 0x000200f8u, + 0x000002b2u, 0x00050041u, 0x00000158u, 0x000002b7u, 0x00000157u, 0x000000b1u, 0x0004003du, 0x00000006u, + 0x000002b8u, 0x000002b7u, 0x000500c2u, 0x00000006u, 0x000002b9u, 0x000002b8u, 0x00000080u, 0x0004003du, + 0x00000006u, 0x000002bau, 0x00000174u, 0x0004003du, 0x00000023u, 0x000002bbu, 0x00000287u, 0x0004007cu, + 0x00000006u, 0x000002bcu, 0x000002bbu, 0x00050080u, 0x00000006u, 0x000002bdu, 0x000002bau, 0x000002bcu, + 0x00050041u, 0x00000158u, 0x000002beu, 0x00000157u, 0x000002a8u, 0x0004003du, 0x00000006u, 0x000002bfu, + 0x000002beu, 0x00050084u, 0x00000006u, 0x000002c0u, 0x000002bdu, 0x000002bfu, 0x0004003du, 0x00000006u, + 0x000002c1u, 0x0000016fu, 0x00050080u, 0x00000006u, 0x000002c2u, 0x000002c0u, 0x000002c1u, 0x00050080u, + 0x00000006u, 0x000002c3u, 0x000002b9u, 0x000002c2u, 0x00060041u, 0x00000207u, 0x000002c4u, 0x000002b6u, + 0x000000d3u, 0x000002c3u, 0x0004003du, 0x000001fbu, 0x000002c5u, 0x000002c4u, 0x00040071u, 0x00000006u, + 0x000002c6u, 0x000002c5u, 0x0003003eu, 0x000002c7u, 0x000002c6u, 0x00050041u, 0x00000158u, 0x000002c9u, + 0x00000157u, 0x00000057u, 0x0004003du, 0x00000006u, 0x000002cau, 0x000002c9u, 0x0003003eu, 0x000002c8u, + 0x000002cau, 0x00060039u, 0x00000008u, 0x000002cbu, 0x0000001bu, 0x000002c7u, 0x000002c8u, 0x0003003eu, + 0x0000029au, 0x000002cbu, 0x000200f9u, 0x0000029cu, 0x000200f8u, 0x0000029cu, 0x0004003du, 0x00000008u, + 0x000002ccu, 0x0000029au, 0x0003003eu, 0x00000292u, 0x000002ccu, 0x000200f9u, 0x00000296u, 0x000200f8u, + 0x000002cdu, 0x0004003du, 0x0000002cu, 0x000002ceu, 0x000001b5u, 0x000300f7u, 0x000002d0u, 0x00000000u, + 0x000400fau, 0x000002ceu, 0x000002cfu, 0x000002d0u, 0x000200f8u, 0x000002cfu, 0x00050041u, 0x00000158u, + 0x000002d1u, 0x00000157u, 0x000001cfu, 0x0004003du, 0x00000006u, 0x000002d2u, 0x000002d1u, 0x0004007cu, + 0x00000023u, 0x000002d3u, 0x000002d2u, 0x0004003du, 0x00000023u, 0x000002d4u, 0x00000287u, 0x00050080u, + 0x00000023u, 0x000002d5u, 0x000002d3u, 0x000002d4u, 0x0004007cu, 0x00000006u, 0x000002d6u, 0x000002d5u, + 0x00050041u, 0x0000000du, 0x000002d7u, 0x000001e3u, 0x000002d6u, 0x0004003du, 0x00000008u, 0x000002d8u, + 0x000002d7u, 0x0003003eu, 0x00000292u, 0x000002d8u, 0x000200f9u, 0x000002d0u, 0x000200f8u, 0x000002d0u, + 0x000200f9u, 0x00000296u, 0x000200f8u, 0x00000296u, 0x00050041u, 0x00000158u, 0x000002dau, 0x00000157u, + 0x000002d9u, 0x0004003du, 0x00000006u, 0x000002dbu, 0x000002dau, 0x000500aau, 0x0000002cu, 0x000002dcu, + 0x000002dbu, 0x00000036u, 0x000300f7u, 0x000002dfu, 0x00000000u, 0x000400fau, 0x000002dcu, 0x000002deu, + 0x000002f1u, 0x000200f8u, 0x000002deu, 0x00050041u, 0x00000158u, 0x000002e4u, 0x00000157u, 0x00000024u, + 0x0004003du, 0x00000006u, 0x000002e5u, 0x000002e4u, 0x000500c2u, 0x00000006u, 0x000002e6u, 0x000002e5u, + 0x0000017cu, 0x0004003du, 0x00000006u, 0x000002e7u, 0x0000016fu, 0x00050041u, 0x00000158u, 0x000002e8u, + 0x00000157u, 0x0000017cu, 0x0004003du, 0x00000006u, 0x000002e9u, 0x000002e8u, 0x00050084u, 0x00000006u, + 0x000002eau, 0x000002e7u, 0x000002e9u, 0x0004003du, 0x00000006u, 0x000002ebu, 0x0000027du, 0x00050080u, + 0x00000006u, 0x000002ecu, 0x000002eau, 0x000002ebu, 0x00050080u, 0x00000006u, 0x000002edu, 0x000002e6u, + 0x000002ecu, 0x00060041u, 0x00000180u, 0x000002eeu, 0x000002e3u, 0x000000d3u, 0x000002edu, 0x0004003du, + 0x00000006u, 0x000002efu, 0x000002eeu, 0x0004007cu, 0x00000008u, 0x000002f0u, 0x000002efu, 0x0003003eu, + 0x000002ddu, 0x000002f0u, 0x000200f9u, 0x000002dfu, 0x000200f8u, 0x000002f1u, 0x00050041u, 0x00000158u, + 0x000002f6u, 0x00000157u, 0x00000024u, 0x0004003du, 0x00000006u, 0x000002f7u, 0x000002f6u, 0x000500c2u, + 0x00000006u, 0x000002f8u, 0x000002f7u, 0x00000080u, 0x0004003du, 0x00000006u, 0x000002f9u, 0x0000016fu, + 0x00050041u, 0x00000158u, 0x000002fau, 0x00000157u, 0x0000017cu, 0x0004003du, 0x00000006u, 0x000002fbu, + 0x000002fau, 0x00050084u, 0x00000006u, 0x000002fcu, 0x000002f9u, 0x000002fbu, 0x0004003du, 0x00000006u, + 0x000002fdu, 0x0000027du, 0x00050080u, 0x00000006u, 0x000002feu, 0x000002fcu, 0x000002fdu, 0x00050080u, + 0x00000006u, 0x000002ffu, 0x000002f8u, 0x000002feu, 0x00060041u, 0x00000207u, 0x00000300u, 0x000002f5u, + 0x000000d3u, 0x000002ffu, 0x0004003du, 0x000001fbu, 0x00000301u, 0x00000300u, 0x00040071u, 0x00000006u, + 0x00000302u, 0x00000301u, 0x0003003eu, 0x00000303u, 0x00000302u, 0x00050041u, 0x00000158u, 0x00000305u, + 0x00000157u, 0x000002d9u, 0x0004003du, 0x00000006u, 0x00000306u, 0x00000305u, 0x0003003eu, 0x00000304u, + 0x00000306u, 0x00060039u, 0x00000008u, 0x00000307u, 0x0000001bu, 0x00000303u, 0x00000304u, 0x0003003eu, + 0x000002ddu, 0x00000307u, 0x000200f9u, 0x000002dfu, 0x000200f8u, 0x000002dfu, 0x0004003du, 0x00000008u, + 0x00000308u, 0x000002ddu, 0x0004003du, 0x00000008u, 0x00000309u, 0x00000292u, 0x00050085u, 0x00000008u, + 0x0000030au, 0x00000308u, 0x00000309u, 0x0004003du, 0x00000008u, 0x0000030bu, 0x0000027bu, 0x00050081u, + 0x00000008u, 0x0000030cu, 0x0000030bu, 0x0000030au, 0x0003003eu, 0x0000027bu, 0x0000030cu, 0x000200f9u, + 0x00000281u, 0x000200f8u, 0x00000281u, 0x0004003du, 0x00000006u, 0x0000030du, 0x0000027du, 0x00050080u, + 0x00000006u, 0x0000030eu, 0x0000030du, 0x00000080u, 0x0003003eu, 0x0000027du, 0x0000030eu, 0x000200f9u, + 0x0000027eu, 0x000200f8u, 0x00000280u, 0x000200f9u, 0x0000030fu, 0x000200f8u, 0x0000030fu, 0x000400f6u, + 0x00000311u, 0x00000312u, 0x00000000u, 0x000200f9u, 0x00000310u, 0x000200f8u, 0x00000310u, 0x00050041u, + 0x00000158u, 0x00000314u, 0x00000157u, 0x00000313u, 0x0004003du, 0x00000006u, 0x00000315u, 0x00000314u, + 0x000500aau, 0x0000002cu, 0x00000316u, 0x00000315u, 0x00000036u, 0x000300f7u, 0x00000318u, 0x00000000u, + 0x000400fau, 0x00000316u, 0x00000317u, 0x0000033cu, 0x000200f8u, 0x00000317u, 0x00050041u, 0x00000158u, + 0x0000031du, 0x00000157u, 0x000000e2u, 0x0004003du, 0x00000006u, 0x0000031eu, 0x0000031du, 0x000500c2u, + 0x00000006u, 0x0000031fu, 0x0000031eu, 0x0000017cu, 0x0004003du, 0x00000006u, 0x00000320u, 0x00000174u, + 0x0004003du, 0x00000006u, 0x00000321u, 0x00000271u, 0x00050080u, 0x00000006u, 0x00000322u, 0x00000320u, + 0x00000321u, 0x00050041u, 0x00000158u, 0x00000323u, 0x00000157u, 0x00000080u, 0x0004003du, 0x00000006u, + 0x00000324u, 0x00000323u, 0x00050084u, 0x00000006u, 0x00000325u, 0x00000322u, 0x00000324u, 0x0004003du, + 0x00000006u, 0x00000326u, 0x0000016fu, 0x00050080u, 0x00000006u, 0x00000327u, 0x00000325u, 0x00000326u, + 0x00050080u, 0x00000006u, 0x00000328u, 0x0000031fu, 0x00000327u, 0x00050041u, 0x00000158u, 0x0000032au, + 0x00000157u, 0x00000329u, 0x0004003du, 0x00000006u, 0x0000032bu, 0x0000032au, 0x000500abu, 0x0000002cu, + 0x0000032cu, 0x0000032bu, 0x00000036u, 0x000300f7u, 0x0000032fu, 0x00000000u, 0x000400fau, 0x0000032cu, + 0x0000032eu, 0x00000337u, 0x000200f8u, 0x0000032eu, 0x0004003du, 0x00000008u, 0x00000330u, 0x0000027bu, + 0x0004003du, 0x00000008u, 0x00000332u, 0x0000027bu, 0x0004007fu, 0x00000008u, 0x00000333u, 0x00000332u, + 0x0006000cu, 0x00000008u, 0x00000334u, 0x00000001u, 0x0000001bu, 0x00000333u, 0x00050081u, 0x00000008u, + 0x00000335u, 0x00000331u, 0x00000334u, 0x00050088u, 0x00000008u, 0x00000336u, 0x00000330u, 0x00000335u, + 0x0003003eu, 0x0000032du, 0x00000336u, 0x000200f9u, 0x0000032fu, 0x000200f8u, 0x00000337u, 0x0004003du, + 0x00000008u, 0x00000338u, 0x0000027bu, 0x0003003eu, 0x0000032du, 0x00000338u, 0x000200f9u, 0x0000032fu, + 0x000200f8u, 0x0000032fu, 0x0004003du, 0x00000008u, 0x00000339u, 0x0000032du, 0x0004007cu, 0x00000006u, + 0x0000033au, 0x00000339u, 0x00060041u, 0x00000180u, 0x0000033bu, 0x0000031cu, 0x000000d3u, 0x00000328u, + 0x0003003eu, 0x0000033bu, 0x0000033au, 0x000200f9u, 0x00000318u, 0x000200f8u, 0x0000033cu, 0x00050041u, + 0x00000158u, 0x00000341u, 0x00000157u, 0x000000e2u, 0x0004003du, 0x00000006u, 0x00000342u, 0x00000341u, + 0x000500c2u, 0x00000006u, 0x00000343u, 0x00000342u, 0x00000080u, 0x0004003du, 0x00000006u, 0x00000344u, + 0x00000174u, 0x0004003du, 0x00000006u, 0x00000345u, 0x00000271u, 0x00050080u, 0x00000006u, 0x00000346u, + 0x00000344u, 0x00000345u, 0x00050041u, 0x00000158u, 0x00000347u, 0x00000157u, 0x00000080u, 0x0004003du, + 0x00000006u, 0x00000348u, 0x00000347u, 0x00050084u, 0x00000006u, 0x00000349u, 0x00000346u, 0x00000348u, + 0x0004003du, 0x00000006u, 0x0000034au, 0x0000016fu, 0x00050080u, 0x00000006u, 0x0000034bu, 0x00000349u, + 0x0000034au, 0x00050080u, 0x00000006u, 0x0000034cu, 0x00000343u, 0x0000034bu, 0x00050041u, 0x00000158u, + 0x0000034du, 0x00000157u, 0x00000329u, 0x0004003du, 0x00000006u, 0x0000034eu, 0x0000034du, 0x000500abu, + 0x0000002cu, 0x0000034fu, 0x0000034eu, 0x00000036u, 0x000300f7u, 0x00000352u, 0x00000000u, 0x000400fau, + 0x0000034fu, 0x00000351u, 0x00000359u, 0x000200f8u, 0x00000351u, 0x0004003du, 0x00000008u, 0x00000353u, + 0x0000027bu, 0x0004003du, 0x00000008u, 0x00000354u, 0x0000027bu, 0x0004007fu, 0x00000008u, 0x00000355u, + 0x00000354u, 0x0006000cu, 0x00000008u, 0x00000356u, 0x00000001u, 0x0000001bu, 0x00000355u, 0x00050081u, + 0x00000008u, 0x00000357u, 0x00000331u, 0x00000356u, 0x00050088u, 0x00000008u, 0x00000358u, 0x00000353u, + 0x00000357u, 0x0003003eu, 0x00000350u, 0x00000358u, 0x000200f9u, 0x00000352u, 0x000200f8u, 0x00000359u, + 0x0004003du, 0x00000008u, 0x0000035au, 0x0000027bu, 0x0003003eu, 0x00000350u, 0x0000035au, 0x000200f9u, + 0x00000352u, 0x000200f8u, 0x00000352u, 0x0004003du, 0x00000008u, 0x0000035cu, 0x00000350u, 0x0003003eu, + 0x0000035bu, 0x0000035cu, 0x00050041u, 0x00000158u, 0x0000035eu, 0x00000157u, 0x00000313u, 0x0004003du, + 0x00000006u, 0x0000035fu, 0x0000035eu, 0x0003003eu, 0x0000035du, 0x0000035fu, 0x00060039u, 0x00000006u, + 0x00000360u, 0x00000020u, 0x0000035bu, 0x0000035du, 0x00040071u, 0x000001fbu, 0x00000361u, 0x00000360u, + 0x00060041u, 0x00000207u, 0x00000362u, 0x00000340u, 0x000000d3u, 0x0000034cu, 0x0003003eu, 0x00000362u, + 0x00000361u, 0x000200f9u, 0x00000318u, 0x000200f8u, 0x00000318u, 0x000200f9u, 0x00000312u, 0x000200f8u, + 0x00000312u, 0x000400fau, 0x00000363u, 0x0000030fu, 0x00000311u, 0x000200f8u, 0x00000311u, 0x000200f9u, + 0x00000276u, 0x000200f8u, 0x00000276u, 0x0004003du, 0x00000006u, 0x00000364u, 0x00000271u, 0x00050080u, + 0x00000006u, 0x00000365u, 0x00000364u, 0x00000080u, 0x0003003eu, 0x00000271u, 0x00000365u, 0x000200f9u, + 0x00000273u, 0x000200f8u, 0x00000275u, 0x0004003du, 0x00000006u, 0x00000366u, 0x00000164u, 0x000500abu, + 0x0000002cu, 0x00000367u, 0x00000366u, 0x00000036u, 0x000300f7u, 0x00000369u, 0x00000000u, 0x000400fau, + 0x00000367u, 0x00000368u, 0x00000369u, 0x000200f8u, 0x00000368u, 0x000100fdu, 0x000200f8u, 0x00000369u, + 0x0003003eu, 0x0000036bu, 0x00000036u, 0x000200f9u, 0x0000036cu, 0x000200f8u, 0x0000036cu, 0x000400f6u, + 0x0000036eu, 0x0000036fu, 0x00000000u, 0x000200f9u, 0x00000370u, 0x000200f8u, 0x00000370u, 0x0004003du, + 0x00000006u, 0x00000371u, 0x0000036bu, 0x00050041u, 0x00000158u, 0x00000372u, 0x00000157u, 0x000001cfu, + 0x0004003du, 0x00000006u, 0x00000373u, 0x00000372u, 0x000500b0u, 0x0000002cu, 0x00000374u, 0x00000371u, + 0x00000373u, 0x000400fau, 0x00000374u, 0x0000036du, 0x0000036eu, 0x000200f8u, 0x0000036du, 0x0004003du, + 0x00000006u, 0x00000376u, 0x0000019bu, 0x0004007cu, 0x00000023u, 0x00000377u, 0x00000376u, 0x00050041u, + 0x00000158u, 0x00000378u, 0x00000157u, 0x000001cfu, 0x0004003du, 0x00000006u, 0x00000379u, 0x00000378u, + 0x0004007cu, 0x00000023u, 0x0000037au, 0x00000379u, 0x00050082u, 0x00000023u, 0x0000037bu, 0x00000377u, + 0x0000037au, 0x0004003du, 0x00000006u, 0x0000037cu, 0x0000036bu, 0x0004007cu, 0x00000023u, 0x0000037du, + 0x0000037cu, 0x00050080u, 0x00000023u, 0x0000037eu, 0x0000037bu, 0x0000037du, 0x0003003eu, 0x00000375u, + 0x0000037eu, 0x0003003eu, 0x0000037fu, 0x00000244u, 0x0004003du, 0x00000023u, 0x00000380u, 0x00000375u, + 0x000500afu, 0x0000002cu, 0x00000381u, 0x00000380u, 0x000000d3u, 0x000300f7u, 0x00000383u, 0x00000000u, + 0x000400fau, 0x00000381u, 0x00000382u, 0x000003b1u, 0x000200f8u, 0x00000382u, 0x00050041u, 0x00000158u, + 0x00000384u, 0x00000157u, 0x00000057u, 0x0004003du, 0x00000006u, 0x00000385u, 0x00000384u, 0x000500aau, + 0x0000002cu, 0x00000386u, 0x00000385u, 0x00000036u, 0x000300f7u, 0x00000389u, 0x00000000u, 0x000400fau, + 0x00000386u, 0x00000388u, 0x0000039au, 0x000200f8u, 0x00000388u, 0x00050041u, 0x00000158u, 0x0000038au, + 0x00000157u, 0x000000b1u, 0x0004003du, 0x00000006u, 0x0000038bu, 0x0000038au, 0x000500c2u, 0x00000006u, + 0x0000038cu, 0x0000038bu, 0x0000017cu, 0x0004003du, 0x00000006u, 0x0000038du, 0x00000174u, 0x0004003du, + 0x00000023u, 0x0000038eu, 0x00000375u, 0x0004007cu, 0x00000006u, 0x0000038fu, 0x0000038eu, 0x00050080u, + 0x00000006u, 0x00000390u, 0x0000038du, 0x0000038fu, 0x00050041u, 0x00000158u, 0x00000391u, 0x00000157u, + 0x000002a8u, 0x0004003du, 0x00000006u, 0x00000392u, 0x00000391u, 0x00050084u, 0x00000006u, 0x00000393u, + 0x00000390u, 0x00000392u, 0x0004003du, 0x00000006u, 0x00000394u, 0x0000016fu, 0x00050080u, 0x00000006u, + 0x00000395u, 0x00000393u, 0x00000394u, 0x00050080u, 0x00000006u, 0x00000396u, 0x0000038cu, 0x00000395u, + 0x00060041u, 0x00000180u, 0x00000397u, 0x000002a0u, 0x000000d3u, 0x00000396u, 0x0004003du, 0x00000006u, + 0x00000398u, 0x00000397u, 0x0004007cu, 0x00000008u, 0x00000399u, 0x00000398u, 0x0003003eu, 0x00000387u, + 0x00000399u, 0x000200f9u, 0x00000389u, 0x000200f8u, 0x0000039au, 0x00050041u, 0x00000158u, 0x0000039bu, + 0x00000157u, 0x000000b1u, 0x0004003du, 0x00000006u, 0x0000039cu, 0x0000039bu, 0x000500c2u, 0x00000006u, + 0x0000039du, 0x0000039cu, 0x00000080u, 0x0004003du, 0x00000006u, 0x0000039eu, 0x00000174u, 0x0004003du, + 0x00000023u, 0x0000039fu, 0x00000375u, 0x0004007cu, 0x00000006u, 0x000003a0u, 0x0000039fu, 0x00050080u, + 0x00000006u, 0x000003a1u, 0x0000039eu, 0x000003a0u, 0x00050041u, 0x00000158u, 0x000003a2u, 0x00000157u, + 0x000002a8u, 0x0004003du, 0x00000006u, 0x000003a3u, 0x000003a2u, 0x00050084u, 0x00000006u, 0x000003a4u, + 0x000003a1u, 0x000003a3u, 0x0004003du, 0x00000006u, 0x000003a5u, 0x0000016fu, 0x00050080u, 0x00000006u, + 0x000003a6u, 0x000003a4u, 0x000003a5u, 0x00050080u, 0x00000006u, 0x000003a7u, 0x0000039du, 0x000003a6u, + 0x00060041u, 0x00000207u, 0x000003a8u, 0x000002b6u, 0x000000d3u, 0x000003a7u, 0x0004003du, 0x000001fbu, + 0x000003a9u, 0x000003a8u, 0x00040071u, 0x00000006u, 0x000003aau, 0x000003a9u, 0x0003003eu, 0x000003abu, + 0x000003aau, 0x00050041u, 0x00000158u, 0x000003adu, 0x00000157u, 0x00000057u, 0x0004003du, 0x00000006u, + 0x000003aeu, 0x000003adu, 0x0003003eu, 0x000003acu, 0x000003aeu, 0x00060039u, 0x00000008u, 0x000003afu, + 0x0000001bu, 0x000003abu, 0x000003acu, 0x0003003eu, 0x00000387u, 0x000003afu, 0x000200f9u, 0x00000389u, + 0x000200f8u, 0x00000389u, 0x0004003du, 0x00000008u, 0x000003b0u, 0x00000387u, 0x0003003eu, 0x0000037fu, + 0x000003b0u, 0x000200f9u, 0x00000383u, 0x000200f8u, 0x000003b1u, 0x0004003du, 0x0000002cu, 0x000003b2u, + 0x000001b5u, 0x000300f7u, 0x000003b4u, 0x00000000u, 0x000400fau, 0x000003b2u, 0x000003b3u, 0x000003b4u, + 0x000200f8u, 0x000003b3u, 0x00050041u, 0x00000158u, 0x000003b5u, 0x00000157u, 0x000001cfu, 0x0004003du, + 0x00000006u, 0x000003b6u, 0x000003b5u, 0x0004007cu, 0x00000023u, 0x000003b7u, 0x000003b6u, 0x0004003du, + 0x00000023u, 0x000003b8u, 0x00000375u, 0x00050080u, 0x00000023u, 0x000003b9u, 0x000003b7u, 0x000003b8u, + 0x0004007cu, 0x00000006u, 0x000003bau, 0x000003b9u, 0x00050041u, 0x0000000du, 0x000003bbu, 0x000001e3u, + 0x000003bau, 0x0004003du, 0x00000008u, 0x000003bcu, 0x000003bbu, 0x0003003eu, 0x0000037fu, 0x000003bcu, + 0x000200f9u, 0x000003b4u, 0x000200f8u, 0x000003b4u, 0x000200f9u, 0x00000383u, 0x000200f8u, 0x00000383u, + 0x000200f9u, 0x000003bdu, 0x000200f8u, 0x000003bdu, 0x000400f6u, 0x000003bfu, 0x000003c0u, 0x00000000u, + 0x000200f9u, 0x000003beu, 0x000200f8u, 0x000003beu, 0x00050041u, 0x00000158u, 0x000003c1u, 0x00000157u, + 0x00000066u, 0x0004003du, 0x00000006u, 0x000003c2u, 0x000003c1u, 0x000500aau, 0x0000002cu, 0x000003c3u, + 0x000003c2u, 0x00000036u, 0x000300f7u, 0x000003c5u, 0x00000000u, 0x000400fau, 0x000003c3u, 0x000003c4u, + 0x000003d0u, 0x000200f8u, 0x000003c4u, 0x00050041u, 0x00000158u, 0x000003c6u, 0x00000157u, 0x000001efu, + 0x0004003du, 0x00000006u, 0x000003c7u, 0x000003c6u, 0x000500c2u, 0x00000006u, 0x000003c8u, 0x000003c7u, + 0x0000017cu, 0x0004003du, 0x00000006u, 0x000003c9u, 0x000001c8u, 0x0004003du, 0x00000006u, 0x000003cau, + 0x0000036bu, 0x00050080u, 0x00000006u, 0x000003cbu, 0x000003c9u, 0x000003cau, 0x00050080u, 0x00000006u, + 0x000003ccu, 0x000003c8u, 0x000003cbu, 0x0004003du, 0x00000008u, 0x000003cdu, 0x0000037fu, 0x0004007cu, + 0x00000006u, 0x000003ceu, 0x000003cdu, 0x00060041u, 0x00000180u, 0x000003cfu, 0x000001eeu, 0x000000d3u, + 0x000003ccu, 0x0003003eu, 0x000003cfu, 0x000003ceu, 0x000200f9u, 0x000003c5u, 0x000200f8u, 0x000003d0u, + 0x00050041u, 0x00000158u, 0x000003d1u, 0x00000157u, 0x000001efu, 0x0004003du, 0x00000006u, 0x000003d2u, + 0x000003d1u, 0x000500c2u, 0x00000006u, 0x000003d3u, 0x000003d2u, 0x00000080u, 0x0004003du, 0x00000006u, + 0x000003d4u, 0x000001c8u, 0x0004003du, 0x00000006u, 0x000003d5u, 0x0000036bu, 0x00050080u, 0x00000006u, + 0x000003d6u, 0x000003d4u, 0x000003d5u, 0x00050080u, 0x00000006u, 0x000003d7u, 0x000003d3u, 0x000003d6u, + 0x0004003du, 0x00000008u, 0x000003d9u, 0x0000037fu, 0x0003003eu, 0x000003d8u, 0x000003d9u, 0x00050041u, + 0x00000158u, 0x000003dbu, 0x00000157u, 0x00000066u, 0x0004003du, 0x00000006u, 0x000003dcu, 0x000003dbu, + 0x0003003eu, 0x000003dau, 0x000003dcu, 0x00060039u, 0x00000006u, 0x000003ddu, 0x00000020u, 0x000003d8u, + 0x000003dau, 0x00040071u, 0x000001fbu, 0x000003deu, 0x000003ddu, 0x00060041u, 0x00000207u, 0x000003dfu, + 0x000001ffu, 0x000000d3u, 0x000003d7u, 0x0003003eu, 0x000003dfu, 0x000003deu, 0x000200f9u, 0x000003c5u, + 0x000200f8u, 0x000003c5u, 0x000200f9u, 0x000003c0u, 0x000200f8u, 0x000003c0u, 0x000400fau, 0x00000363u, + 0x000003bdu, 0x000003bfu, 0x000200f8u, 0x000003bfu, 0x000200f9u, 0x0000036fu, 0x000200f8u, 0x0000036fu, + 0x0004003du, 0x00000006u, 0x000003e0u, 0x0000036bu, 0x00050080u, 0x00000006u, 0x000003e1u, 0x000003e0u, + 0x00000080u, 0x0003003eu, 0x0000036bu, 0x000003e1u, 0x000200f9u, 0x0000036cu, 0x000200f8u, 0x0000036eu, + 0x000100fdu, 0x00010038u, 0x00050036u, 0x00000008u, 0x0000000bu, 0x00000000u, 0x00000009u, 0x00030037u, + 0x00000007u, 0x0000000au, 0x000200f8u, 0x0000000cu, 0x0004003du, 0x00000006u, 0x00000022u, 0x0000000au, + 0x000500c4u, 0x00000006u, 0x00000025u, 0x00000022u, 0x00000024u, 0x0004007cu, 0x00000008u, 0x00000026u, + 0x00000025u, 0x000200feu, 0x00000026u, 0x00010038u, 0x00050036u, 0x00000006u, 0x00000010u, 0x00000000u, + 0x0000000eu, 0x00030037u, 0x0000000du, 0x0000000fu, 0x000200f8u, 0x00000011u, 0x0004003bu, 0x00000007u, + 0x00000029u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000042u, 0x00000007u, 0x0004003du, 0x00000008u, + 0x0000002au, 0x0000000fu, 0x0004007cu, 0x00000006u, 0x0000002bu, 0x0000002au, 0x0003003eu, 0x00000029u, + 0x0000002bu, 0x0004003du, 0x00000006u, 0x0000002du, 0x00000029u, 0x000500c7u, 0x00000006u, 0x0000002fu, + 0x0000002du, 0x0000002eu, 0x000500aau, 0x0000002cu, 0x00000030u, 0x0000002fu, 0x0000002eu, 0x000300f7u, + 0x00000032u, 0x00000000u, 0x000400fau, 0x00000030u, 0x00000031u, 0x00000032u, 0x000200f8u, 0x00000031u, + 0x0004003du, 0x00000006u, 0x00000033u, 0x00000029u, 0x000500c7u, 0x00000006u, 0x00000035u, 0x00000033u, + 0x00000034u, 0x000500abu, 0x0000002cu, 0x00000037u, 0x00000035u, 0x00000036u, 0x000200f9u, 0x00000032u, + 0x000200f8u, 0x00000032u, 0x000700f5u, 0x0000002cu, 0x00000038u, 0x00000030u, 0x00000011u, 0x00000037u, + 0x00000031u, 0x000300f7u, 0x0000003au, 0x00000000u, 0x000400fau, 0x00000038u, 0x00000039u, 0x0000003au, + 0x000200f8u, 0x00000039u, 0x0004003du, 0x00000006u, 0x0000003bu, 0x00000029u, 0x000500c2u, 0x00000006u, + 0x0000003cu, 0x0000003bu, 0x00000024u, 0x000500c5u, 0x00000006u, 0x0000003eu, 0x0000003cu, 0x0000003du, + 0x000500c7u, 0x00000006u, 0x00000040u, 0x0000003eu, 0x0000003fu, 0x000200feu, 0x00000040u, 0x000200f8u, + 0x0000003au, 0x0004003du, 0x00000006u, 0x00000044u, 0x00000029u, 0x000500c2u, 0x00000006u, 0x00000045u, + 0x00000044u, 0x00000024u, 0x000500c7u, 0x00000006u, 0x00000047u, 0x00000045u, 0x00000046u, 0x00050080u, + 0x00000006u, 0x00000048u, 0x00000043u, 0x00000047u, 0x0003003eu, 0x00000042u, 0x00000048u, 0x0004003du, + 0x00000006u, 0x00000049u, 0x00000029u, 0x0004003du, 0x00000006u, 0x0000004au, 0x00000042u, 0x00050080u, + 0x00000006u, 0x0000004bu, 0x00000049u, 0x0000004au, 0x000500c2u, 0x00000006u, 0x0000004cu, 0x0000004bu, + 0x00000024u, 0x000500c7u, 0x00000006u, 0x0000004du, 0x0000004cu, 0x0000003fu, 0x000200feu, 0x0000004du, + 0x00010038u, 0x00050036u, 0x00000008u, 0x00000013u, 0x00000000u, 0x00000009u, 0x00030037u, 0x00000007u, + 0x00000012u, 0x000200f8u, 0x00000014u, 0x0004003bu, 0x00000007u, 0x00000050u, 0x00000007u, 0x0004003bu, + 0x00000007u, 0x00000055u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000005bu, 0x00000007u, 0x0004003bu, + 0x00000007u, 0x00000076u, 0x00000007u, 0x0004003du, 0x00000006u, 0x00000051u, 0x00000012u, 0x000500c7u, + 0x00000006u, 0x00000053u, 0x00000051u, 0x00000052u, 0x000500c4u, 0x00000006u, 0x00000054u, 0x00000053u, + 0x00000024u, 0x0003003eu, 0x00000050u, 0x00000054u, 0x0004003du, 0x00000006u, 0x00000056u, 0x00000012u, + 0x000500c2u, 0x00000006u, 0x00000058u, 0x00000056u, 0x00000057u, 0x000500c7u, 0x00000006u, 0x0000005au, + 0x00000058u, 0x00000059u, 0x0003003eu, 0x00000055u, 0x0000005au, 0x0004003du, 0x00000006u, 0x0000005cu, + 0x00000012u, 0x000500c7u, 0x00000006u, 0x0000005eu, 0x0000005cu, 0x0000005du, 0x0003003eu, 0x0000005bu, + 0x0000005eu, 0x0004003du, 0x00000006u, 0x0000005fu, 0x00000055u, 0x000500aau, 0x0000002cu, 0x00000060u, + 0x0000005fu, 0x00000059u, 0x000300f7u, 0x00000062u, 0x00000000u, 0x000400fau, 0x00000060u, 0x00000061u, + 0x00000062u, 0x000200f8u, 0x00000061u, 0x0004003du, 0x00000006u, 0x00000063u, 0x00000050u, 0x000500c5u, + 0x00000006u, 0x00000064u, 0x00000063u, 0x0000002eu, 0x0004003du, 0x00000006u, 0x00000065u, 0x0000005bu, + 0x000500c4u, 0x00000006u, 0x00000067u, 0x00000065u, 0x00000066u, 0x000500c5u, 0x00000006u, 0x00000068u, + 0x00000064u, 0x00000067u, 0x0004007cu, 0x00000008u, 0x00000069u, 0x00000068u, 0x000200feu, 0x00000069u, + 0x000200f8u, 0x00000062u, 0x0004003du, 0x00000006u, 0x0000006bu, 0x00000055u, 0x000500aau, 0x0000002cu, + 0x0000006cu, 0x0000006bu, 0x00000036u, 0x000300f7u, 0x0000006eu, 0x00000000u, 0x000400fau, 0x0000006cu, + 0x0000006du, 0x0000006eu, 0x000200f8u, 0x0000006du, 0x0004003du, 0x00000006u, 0x0000006fu, 0x0000005bu, + 0x000500aau, 0x0000002cu, 0x00000070u, 0x0000006fu, 0x00000036u, 0x000300f7u, 0x00000072u, 0x00000000u, + 0x000400fau, 0x00000070u, 0x00000071u, 0x00000072u, 0x000200f8u, 0x00000071u, 0x0004003du, 0x00000006u, + 0x00000073u, 0x00000050u, 0x0004007cu, 0x00000008u, 0x00000074u, 0x00000073u, 0x000200feu, 0x00000074u, + 0x000200f8u, 0x00000072u, 0x0003003eu, 0x00000076u, 0x00000036u, 0x000200f9u, 0x00000077u, 0x000200f8u, + 0x00000077u, 0x000400f6u, 0x00000079u, 0x0000007au, 0x00000000u, 0x000200f9u, 0x0000007bu, 0x000200f8u, + 0x0000007bu, 0x0004003du, 0x00000006u, 0x0000007cu, 0x0000005bu, 0x000500c7u, 0x00000006u, 0x0000007eu, + 0x0000007cu, 0x0000007du, 0x000500aau, 0x0000002cu, 0x0000007fu, 0x0000007eu, 0x00000036u, 0x000400fau, + 0x0000007fu, 0x00000078u, 0x00000079u, 0x000200f8u, 0x00000078u, 0x0004003du, 0x00000006u, 0x00000081u, + 0x0000005bu, 0x000500c4u, 0x00000006u, 0x00000082u, 0x00000081u, 0x00000080u, 0x0003003eu, 0x0000005bu, + 0x00000082u, 0x0004003du, 0x00000006u, 0x00000083u, 0x00000076u, 0x00050080u, 0x00000006u, 0x00000084u, + 0x00000083u, 0x00000046u, 0x0003003eu, 0x00000076u, 0x00000084u, 0x000200f9u, 0x0000007au, 0x000200f8u, + 0x0000007au, 0x000200f9u, 0x00000077u, 0x000200f8u, 0x00000079u, 0x0004003du, 0x00000006u, 0x00000085u, + 0x0000005bu, 0x000500c7u, 0x00000006u, 0x00000086u, 0x00000085u, 0x0000005du, 0x0003003eu, 0x0000005bu, + 0x00000086u, 0x0004003du, 0x00000006u, 0x00000087u, 0x00000050u, 0x0004003du, 0x00000006u, 0x00000089u, + 0x00000076u, 0x00050082u, 0x00000006u, 0x0000008au, 0x00000088u, 0x00000089u, 0x000500c4u, 0x00000006u, + 0x0000008cu, 0x0000008au, 0x0000008bu, 0x000500c5u, 0x00000006u, 0x0000008du, 0x00000087u, 0x0000008cu, + 0x0004003du, 0x00000006u, 0x0000008eu, 0x0000005bu, 0x000500c4u, 0x00000006u, 0x0000008fu, 0x0000008eu, + 0x00000066u, 0x000500c5u, 0x00000006u, 0x00000090u, 0x0000008du, 0x0000008fu, 0x0004007cu, 0x00000008u, + 0x00000091u, 0x00000090u, 0x000200feu, 0x00000091u, 0x000200f8u, 0x0000006eu, 0x0004003du, 0x00000006u, + 0x00000093u, 0x00000050u, 0x0004003du, 0x00000006u, 0x00000094u, 0x00000055u, 0x00050080u, 0x00000006u, + 0x00000096u, 0x00000094u, 0x00000095u, 0x000500c4u, 0x00000006u, 0x00000097u, 0x00000096u, 0x0000008bu, + 0x000500c5u, 0x00000006u, 0x00000098u, 0x00000093u, 0x00000097u, 0x0004003du, 0x00000006u, 0x00000099u, + 0x0000005bu, 0x000500c4u, 0x00000006u, 0x0000009au, 0x00000099u, 0x00000066u, 0x000500c5u, 0x00000006u, + 0x0000009bu, 0x00000098u, 0x0000009au, 0x0004007cu, 0x00000008u, 0x0000009cu, 0x0000009bu, 0x000200feu, + 0x0000009cu, 0x00010038u, 0x00050036u, 0x00000006u, 0x00000016u, 0x00000000u, 0x0000000eu, 0x00030037u, + 0x0000000du, 0x00000015u, 0x000200f8u, 0x00000017u, 0x0004003bu, 0x00000007u, 0x0000009fu, 0x00000007u, + 0x0004003bu, 0x00000007u, 0x000000a2u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x000000a6u, 0x00000007u, + 0x0004003bu, 0x000000abu, 0x000000acu, 0x00000007u, 0x0004003bu, 0x00000007u, 0x000000b3u, 0x00000007u, + 0x0004003bu, 0x00000007u, 0x000000bfu, 0x00000007u, 0x0004003bu, 0x00000007u, 0x000000e1u, 0x00000007u, + 0x0004003bu, 0x00000007u, 0x000000e6u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x000000eau, 0x00000007u, + 0x0004003bu, 0x00000007u, 0x000000f0u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000010cu, 0x00000007u, + 0x0004003bu, 0x00000007u, 0x00000113u, 0x00000007u, 0x0004003du, 0x00000008u, 0x000000a0u, 0x00000015u, + 0x0004007cu, 0x00000006u, 0x000000a1u, 0x000000a0u, 0x0003003eu, 0x0000009fu, 0x000000a1u, 0x0004003du, + 0x00000006u, 0x000000a3u, 0x0000009fu, 0x000500c2u, 0x00000006u, 0x000000a4u, 0x000000a3u, 0x00000024u, + 0x000500c7u, 0x00000006u, 0x000000a5u, 0x000000a4u, 0x00000052u, 0x0003003eu, 0x000000a2u, 0x000000a5u, + 0x0004003du, 0x00000006u, 0x000000a7u, 0x0000009fu, 0x000500c2u, 0x00000006u, 0x000000a8u, 0x000000a7u, + 0x0000008bu, 0x000500c7u, 0x00000006u, 0x000000aau, 0x000000a8u, 0x000000a9u, 0x0003003eu, 0x000000a6u, + 0x000000aau, 0x0004003du, 0x00000006u, 0x000000adu, 0x000000a6u, 0x0004007cu, 0x00000023u, 0x000000aeu, + 0x000000adu, 0x00050082u, 0x00000023u, 0x000000b0u, 0x000000aeu, 0x000000afu, 0x00050080u, 0x00000023u, + 0x000000b2u, 0x000000b0u, 0x000000b1u, 0x0003003eu, 0x000000acu, 0x000000b2u, 0x0004003du, 0x00000006u, + 0x000000b4u, 0x0000009fu, 0x000500c7u, 0x00000006u, 0x000000b5u, 0x000000b4u, 0x00000034u, 0x0003003eu, + 0x000000b3u, 0x000000b5u, 0x0004003du, 0x00000006u, 0x000000b6u, 0x000000a6u, 0x000500aau, 0x0000002cu, + 0x000000b7u, 0x000000b6u, 0x000000a9u, 0x000300f7u, 0x000000b9u, 0x00000000u, 0x000400fau, 0x000000b7u, + 0x000000b8u, 0x000000b9u, 0x000200f8u, 0x000000b8u, 0x0004003du, 0x00000006u, 0x000000bau, 0x000000a2u, + 0x000500c5u, 0x00000006u, 0x000000bcu, 0x000000bau, 0x000000bbu, 0x0004003du, 0x00000006u, 0x000000bdu, + 0x000000b3u, 0x000500abu, 0x0000002cu, 0x000000beu, 0x000000bdu, 0x00000036u, 0x000300f7u, 0x000000c1u, + 0x00000000u, 0x000400fau, 0x000000beu, 0x000000c0u, 0x000000c6u, 0x000200f8u, 0x000000c0u, 0x0004003du, + 0x00000006u, 0x000000c3u, 0x000000b3u, 0x000500c2u, 0x00000006u, 0x000000c4u, 0x000000c3u, 0x00000066u, + 0x000500c5u, 0x00000006u, 0x000000c5u, 0x000000c2u, 0x000000c4u, 0x0003003eu, 0x000000bfu, 0x000000c5u, + 0x000200f9u, 0x000000c1u, 0x000200f8u, 0x000000c6u, 0x0003003eu, 0x000000bfu, 0x00000036u, 0x000200f9u, + 0x000000c1u, 0x000200f8u, 0x000000c1u, 0x0004003du, 0x00000006u, 0x000000c7u, 0x000000bfu, 0x000500c5u, + 0x00000006u, 0x000000c8u, 0x000000bcu, 0x000000c7u, 0x000200feu, 0x000000c8u, 0x000200f8u, 0x000000b9u, + 0x0004003du, 0x00000023u, 0x000000cau, 0x000000acu, 0x000500afu, 0x0000002cu, 0x000000ccu, 0x000000cau, + 0x000000cbu, 0x000300f7u, 0x000000ceu, 0x00000000u, 0x000400fau, 0x000000ccu, 0x000000cdu, 0x000000ceu, + 0x000200f8u, 0x000000cdu, 0x0004003du, 0x00000006u, 0x000000cfu, 0x000000a2u, 0x000500c5u, 0x00000006u, + 0x000000d0u, 0x000000cfu, 0x000000bbu, 0x000200feu, 0x000000d0u, 0x000200f8u, 0x000000ceu, 0x0004003du, + 0x00000023u, 0x000000d2u, 0x000000acu, 0x000500b3u, 0x0000002cu, 0x000000d4u, 0x000000d2u, 0x000000d3u, + 0x000300f7u, 0x000000d6u, 0x00000000u, 0x000400fau, 0x000000d4u, 0x000000d5u, 0x000000d6u, 0x000200f8u, + 0x000000d5u, 0x0004003du, 0x00000023u, 0x000000d7u, 0x000000acu, 0x000500b1u, 0x0000002cu, 0x000000d9u, + 0x000000d7u, 0x000000d8u, 0x000300f7u, 0x000000dbu, 0x00000000u, 0x000400fau, 0x000000d9u, 0x000000dau, + 0x000000dbu, 0x000200f8u, 0x000000dau, 0x0004003du, 0x00000006u, 0x000000dcu, 0x000000a2u, 0x000200feu, + 0x000000dcu, 0x000200f8u, 0x000000dbu, 0x0004003du, 0x00000006u, 0x000000dfu, 0x000000b3u, 0x000500c5u, + 0x00000006u, 0x000000e0u, 0x000000dfu, 0x000000deu, 0x0003003eu, 0x000000b3u, 0x000000e0u, 0x0004003du, + 0x00000023u, 0x000000e3u, 0x000000acu, 0x00050082u, 0x00000023u, 0x000000e4u, 0x000000e2u, 0x000000e3u, + 0x0004007cu, 0x00000006u, 0x000000e5u, 0x000000e4u, 0x0003003eu, 0x000000e1u, 0x000000e5u, 0x0004003du, + 0x00000006u, 0x000000e7u, 0x000000b3u, 0x0004003du, 0x00000006u, 0x000000e8u, 0x000000e1u, 0x000500c2u, + 0x00000006u, 0x000000e9u, 0x000000e7u, 0x000000e8u, 0x0003003eu, 0x000000e6u, 0x000000e9u, 0x0004003du, + 0x00000006u, 0x000000ebu, 0x000000b3u, 0x0004003du, 0x00000006u, 0x000000ecu, 0x000000e1u, 0x000500c4u, + 0x00000006u, 0x000000edu, 0x00000046u, 0x000000ecu, 0x00050082u, 0x00000006u, 0x000000eeu, 0x000000edu, + 0x00000046u, 0x000500c7u, 0x00000006u, 0x000000efu, 0x000000ebu, 0x000000eeu, 0x0003003eu, 0x000000eau, + 0x000000efu, 0x0004003du, 0x00000006u, 0x000000f1u, 0x000000e1u, 0x00050082u, 0x00000006u, 0x000000f2u, + 0x000000f1u, 0x00000046u, 0x000500c4u, 0x00000006u, 0x000000f3u, 0x00000046u, 0x000000f2u, 0x0003003eu, + 0x000000f0u, 0x000000f3u, 0x0004003du, 0x00000006u, 0x000000f4u, 0x000000eau, 0x0004003du, 0x00000006u, + 0x000000f5u, 0x000000f0u, 0x000500acu, 0x0000002cu, 0x000000f6u, 0x000000f4u, 0x000000f5u, 0x000400a8u, + 0x0000002cu, 0x000000f7u, 0x000000f6u, 0x000300f7u, 0x000000f9u, 0x00000000u, 0x000400fau, 0x000000f7u, + 0x000000f8u, 0x000000f9u, 0x000200f8u, 0x000000f8u, 0x0004003du, 0x00000006u, 0x000000fau, 0x000000eau, + 0x0004003du, 0x00000006u, 0x000000fbu, 0x000000f0u, 0x000500aau, 0x0000002cu, 0x000000fcu, 0x000000fau, + 0x000000fbu, 0x000300f7u, 0x000000feu, 0x00000000u, 0x000400fau, 0x000000fcu, 0x000000fdu, 0x000000feu, + 0x000200f8u, 0x000000fdu, 0x0004003du, 0x00000006u, 0x000000ffu, 0x000000e6u, 0x000500c7u, 0x00000006u, + 0x00000100u, 0x000000ffu, 0x00000046u, 0x000500abu, 0x0000002cu, 0x00000101u, 0x00000100u, 0x00000036u, + 0x000200f9u, 0x000000feu, 0x000200f8u, 0x000000feu, 0x000700f5u, 0x0000002cu, 0x00000102u, 0x000000fcu, + 0x000000f8u, 0x00000101u, 0x000000fdu, 0x000200f9u, 0x000000f9u, 0x000200f8u, 0x000000f9u, 0x000700f5u, + 0x0000002cu, 0x00000103u, 0x000000f6u, 0x000000dbu, 0x00000102u, 0x000000feu, 0x000300f7u, 0x00000105u, + 0x00000000u, 0x000400fau, 0x00000103u, 0x00000104u, 0x00000105u, 0x000200f8u, 0x00000104u, 0x0004003du, + 0x00000006u, 0x00000106u, 0x000000e6u, 0x00050080u, 0x00000006u, 0x00000107u, 0x00000106u, 0x00000046u, + 0x0003003eu, 0x000000e6u, 0x00000107u, 0x000200f9u, 0x00000105u, 0x000200f8u, 0x00000105u, 0x0004003du, + 0x00000006u, 0x00000108u, 0x000000a2u, 0x0004003du, 0x00000006u, 0x00000109u, 0x000000e6u, 0x000500c5u, + 0x00000006u, 0x0000010au, 0x00000108u, 0x00000109u, 0x000200feu, 0x0000010au, 0x000200f8u, 0x000000d6u, + 0x0004003du, 0x00000023u, 0x0000010du, 0x000000acu, 0x0004007cu, 0x00000006u, 0x0000010eu, 0x0000010du, + 0x000500c4u, 0x00000006u, 0x0000010fu, 0x0000010eu, 0x00000057u, 0x0004003du, 0x00000006u, 0x00000110u, + 0x000000b3u, 0x000500c2u, 0x00000006u, 0x00000111u, 0x00000110u, 0x00000066u, 0x000500c5u, 0x00000006u, + 0x00000112u, 0x0000010fu, 0x00000111u, 0x0003003eu, 0x0000010cu, 0x00000112u, 0x0004003du, 0x00000006u, + 0x00000114u, 0x000000b3u, 0x000500c7u, 0x00000006u, 0x00000116u, 0x00000114u, 0x00000115u, 0x0003003eu, + 0x00000113u, 0x00000116u, 0x0004003du, 0x00000006u, 0x00000117u, 0x00000113u, 0x000500acu, 0x0000002cu, + 0x00000119u, 0x00000117u, 0x00000118u, 0x000400a8u, 0x0000002cu, 0x0000011au, 0x00000119u, 0x000300f7u, + 0x0000011cu, 0x00000000u, 0x000400fau, 0x0000011au, 0x0000011bu, 0x0000011cu, 0x000200f8u, 0x0000011bu, + 0x0004003du, 0x00000006u, 0x0000011du, 0x00000113u, 0x000500aau, 0x0000002cu, 0x0000011eu, 0x0000011du, + 0x00000118u, 0x000300f7u, 0x00000120u, 0x00000000u, 0x000400fau, 0x0000011eu, 0x0000011fu, 0x00000120u, + 0x000200f8u, 0x0000011fu, 0x0004003du, 0x00000006u, 0x00000121u, 0x0000010cu, 0x000500c7u, 0x00000006u, + 0x00000122u, 0x00000121u, 0x00000046u, 0x000500abu, 0x0000002cu, 0x00000123u, 0x00000122u, 0x00000036u, + 0x000200f9u, 0x00000120u, 0x000200f8u, 0x00000120u, 0x000700f5u, 0x0000002cu, 0x00000124u, 0x0000011eu, + 0x0000011bu, 0x00000123u, 0x0000011fu, 0x000200f9u, 0x0000011cu, 0x000200f8u, 0x0000011cu, 0x000700f5u, + 0x0000002cu, 0x00000125u, 0x00000119u, 0x000000d6u, 0x00000124u, 0x00000120u, 0x000300f7u, 0x00000127u, + 0x00000000u, 0x000400fau, 0x00000125u, 0x00000126u, 0x00000127u, 0x000200f8u, 0x00000126u, 0x0004003du, + 0x00000006u, 0x00000128u, 0x0000010cu, 0x00050080u, 0x00000006u, 0x00000129u, 0x00000128u, 0x00000046u, + 0x0003003eu, 0x0000010cu, 0x00000129u, 0x000200f9u, 0x00000127u, 0x000200f8u, 0x00000127u, 0x0004003du, + 0x00000006u, 0x0000012au, 0x000000a2u, 0x0004003du, 0x00000006u, 0x0000012bu, 0x0000010cu, 0x000500c5u, + 0x00000006u, 0x0000012cu, 0x0000012au, 0x0000012bu, 0x000200feu, 0x0000012cu, 0x00010038u, 0x00050036u, + 0x00000008u, 0x0000001bu, 0x00000000u, 0x00000018u, 0x00030037u, 0x00000007u, 0x00000019u, 0x00030037u, + 0x00000007u, 0x0000001au, 0x000200f8u, 0x0000001cu, 0x0004003bu, 0x0000000du, 0x00000131u, 0x00000007u, + 0x0004003bu, 0x00000007u, 0x00000134u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000138u, 0x00000007u, + 0x0004003du, 0x00000006u, 0x0000012fu, 0x0000001au, 0x000500aau, 0x0000002cu, 0x00000130u, 0x0000012fu, + 0x00000046u, 0x000300f7u, 0x00000133u, 0x00000000u, 0x000400fau, 0x00000130u, 0x00000132u, 0x00000137u, + 0x000200f8u, 0x00000132u, 0x0004003du, 0x00000006u, 0x00000135u, 0x00000019u, 0x0003003eu, 0x00000134u, + 0x00000135u, 0x00050039u, 0x00000008u, 0x00000136u, 0x00000013u, 0x00000134u, 0x0003003eu, 0x00000131u, + 0x00000136u, 0x000200f9u, 0x00000133u, 0x000200f8u, 0x00000137u, 0x0004003du, 0x00000006u, 0x00000139u, + 0x00000019u, 0x0003003eu, 0x00000138u, 0x00000139u, 0x00050039u, 0x00000008u, 0x0000013au, 0x0000000bu, + 0x00000138u, 0x0003003eu, 0x00000131u, 0x0000013au, 0x000200f9u, 0x00000133u, 0x000200f8u, 0x00000133u, + 0x0004003du, 0x00000008u, 0x0000013bu, 0x00000131u, 0x000200feu, 0x0000013bu, 0x00010038u, 0x00050036u, + 0x00000006u, 0x00000020u, 0x00000000u, 0x0000001du, 0x00030037u, 0x0000000du, 0x0000001eu, 0x00030037u, + 0x00000007u, 0x0000001fu, 0x000200f8u, 0x00000021u, 0x0004003bu, 0x00000007u, 0x00000140u, 0x00000007u, + 0x0004003bu, 0x0000000du, 0x00000143u, 0x00000007u, 0x0004003bu, 0x0000000du, 0x00000147u, 0x00000007u, + 0x0004003du, 0x00000006u, 0x0000013eu, 0x0000001fu, 0x000500aau, 0x0000002cu, 0x0000013fu, 0x0000013eu, + 0x00000046u, 0x000300f7u, 0x00000142u, 0x00000000u, 0x000400fau, 0x0000013fu, 0x00000141u, 0x00000146u, + 0x000200f8u, 0x00000141u, 0x0004003du, 0x00000008u, 0x00000144u, 0x0000001eu, 0x0003003eu, 0x00000143u, + 0x00000144u, 0x00050039u, 0x00000006u, 0x00000145u, 0x00000016u, 0x00000143u, 0x0003003eu, 0x00000140u, + 0x00000145u, 0x000200f9u, 0x00000142u, 0x000200f8u, 0x00000146u, 0x0004003du, 0x00000008u, 0x00000148u, + 0x0000001eu, 0x0003003eu, 0x00000147u, 0x00000148u, 0x00050039u, 0x00000006u, 0x00000149u, 0x00000010u, + 0x00000147u, 0x0003003eu, 0x00000140u, 0x00000149u, 0x000200f9u, 0x00000142u, 0x000200f8u, 0x00000142u, + 0x0004003du, 0x00000006u, 0x0000014au, 0x00000140u, 0x000200feu, 0x0000014au, 0x00010038u, +}; + constexpr uint32_t kSpv_vt_causal_conv1d_update[] = { 0x07230203u, 0x00010300u, 0x0008000bu, 0x00000391u, 0x00000000u, 0x00020011u, 0x00000001u, 0x00020011u, 0x00001151u, 0x0006000bu, 0x00000001u, 0x4c534c47u, 0x6474732eu, 0x3035342eu, 0x00000000u, 0x0003000eu, @@ -24116,6 +24777,10 @@ constexpr uint32_t kSpecIds_vt_cast[] = { 0u, 1u, }; +constexpr uint32_t kSpecIds_vt_causal_conv1d_fwd[] = { + 0u, +}; + constexpr uint32_t kSpecIds_vt_embedding[] = { 0u, 1u, 2u, }; @@ -24234,6 +24899,7 @@ const SpirvModule kSpirvModules[] = { {"vt_add", kSpv_vt_add, sizeof(kSpv_vt_add) / sizeof(uint32_t), nullptr, 0, 6u, 0x00000030u}, {"vt_attn_qk_norm_rope_gate", kSpv_vt_attn_qk_norm_rope_gate, sizeof(kSpv_vt_attn_qk_norm_rope_gate) / sizeof(uint32_t), kSpecIds_vt_attn_qk_norm_rope_gate, 3, 13u, 0x000003f0u}, {"vt_cast", kSpv_vt_cast, sizeof(kSpv_vt_cast) / sizeof(uint32_t), kSpecIds_vt_cast, 2, 4u, 0x0000000cu}, + {"vt_causal_conv1d_fwd", kSpv_vt_causal_conv1d_fwd, sizeof(kSpv_vt_causal_conv1d_fwd) / sizeof(uint32_t), kSpecIds_vt_causal_conv1d_fwd, 1, 12u, 0x00000303u}, {"vt_causal_conv1d_update", kSpv_vt_causal_conv1d_update, sizeof(kSpv_vt_causal_conv1d_update) / sizeof(uint32_t), nullptr, 0, 11u, 0x00000303u}, {"vt_embedding", kSpv_vt_embedding, sizeof(kSpv_vt_embedding) / sizeof(uint32_t), kSpecIds_vt_embedding, 3, 6u, 0x00000030u}, {"vt_exl3_gemm", kSpv_vt_exl3_gemm, sizeof(kSpv_vt_exl3_gemm) / sizeof(uint32_t), nullptr, 0, 3u, 0x00000001u}, diff --git a/src/vt/vulkan/vulkan_spirv.h b/src/vt/vulkan/vulkan_spirv.h index d7369eaedf..e084815580 100644 --- a/src/vt/vulkan/vulkan_spirv.h +++ b/src/vt/vulkan/vulkan_spirv.h @@ -18,7 +18,7 @@ // and extern declarations; vulkan_spirv.cpp carries the data, so adding shaders // costs one TU's compile time rather than all of them. // -// Produced by: Glslang Version: 11:16.4.0 +// Produced by: Glslang Version: 11:16.5.0 // Target environment: vulkan1.1 #ifndef VT_VULKAN_VULKAN_SPIRV_H_ #define VT_VULKAN_VULKAN_SPIRV_H_ diff --git a/tests/vt/test_vulkan_backend.cpp b/tests/vt/test_vulkan_backend.cpp index 2c19f594a2..5c7b34b25e 100644 --- a/tests/vt/test_vulkan_backend.cpp +++ b/tests/vt/test_vulkan_backend.cpp @@ -33,6 +33,7 @@ #include "vt/quant.h" #include "vt/vulkan/vulkan_context.h" #include "vt/vulkan/vulkan_spirv.h" +#include "support/test_env.h" // SetEnv/UnsetEnv -- MSVC has no setenv (#603) using vt::Backend; using vt::Device; @@ -60,7 +61,8 @@ TEST_CASE("the committed SPIR-V table is present and well-formed") { // point of the split: at the target shader surface the words must not be // re-parsed by every TU that merely needs the table. const size_t n = vt::vulkan::kSpirvModuleCount; - CHECK(n == 43); // +2: BACKEND-VULKAN-EXL3 (#2530); +15: BACKEND-VULKAN-TQ1_0 + CHECK(n == 44); // +1: BACKEND-VULKAN-GDN (vt_causal_conv1d_fwd); + // +2: BACKEND-VULKAN-EXL3 (#2530); +15: BACKEND-VULKAN-TQ1_0 // (vt_matmul_bt_tq2, vt_matmul_bt_tq2_grouped, vt_matmul_bt_tq2_dev, // vt_moe_gate_up_swiglu_grouped_tq2, vt_matmul_bt_tq2_grouped_dev, // vt_matmul_bt_tq2_dev; VK4 rope/moe) @@ -81,7 +83,8 @@ TEST_CASE("the committed SPIR-V table is present and well-formed") { "vt_rms_norm_wide", "vt_rope_from_cache", "vt_silu_and_mul", // BACKEND-VULKAN-GDN: the GDN / conv1d glue family. - "vt_causal_conv1d_update", "vt_gdn_post_conv", + "vt_causal_conv1d_update", "vt_causal_conv1d_fwd", + "vt_gdn_post_conv", "vt_gdn_state_gather", "vt_gdn_state_scatter", "vt_rms_norm_gated", "vt_sigmoid_gate_bf16", // BACKEND-VULKAN-GDN-CORE: the two recurrences. @@ -285,6 +288,13 @@ TEST_CASE("the committed SPIR-V table records each module's specialization const // q/k dtype, position dtype, llama3 flag, q/k head-dim overrides. REQUIRE(m.spec_id_count == 6); for (uint32_t want = 0; want < 6; ++want) CHECK(m.spec_ids[want] == want); + } else if (std::strcmp(m.name, "vt_causal_conv1d_fwd") == 0) { + // ONE axis, and it is not a dtype: VT_CONV_BLOCKS, the token blocks per + // (sequence, channel). Every dtype is a push-constant code, because the + // five tensors vary independently and a spec axis per tensor would be a + // module explosion for a kernel that runs once per GDN layer. + REQUIRE(m.spec_id_count == 1); + CHECK(m.spec_ids[0] == 0u); } else if (std::strcmp(m.name, "vt_moe_router_topk") == 0) { // E, K, renormalize flag, logits dtype. REQUIRE(m.spec_id_count == 4); @@ -590,6 +600,8 @@ TEST_CASE("Vulkan registers the W0 op set and NOT the unimplemented rest") { vt::OpId::kSigmoidGateBf16, vt::OpId::kRmsNormGated, vt::OpId::kGdnStateGather, vt::OpId::kGdnStateScatter, vt::OpId::kCausalConv1dUpdate, vt::OpId::kGdnPostConv, + // ...and the PREFILL conv, which closes the family. + vt::OpId::kCausalConv1dFwd, // BACKEND-VULKAN-GDN-CORE: the two gated-delta recurrences // themselves, which are where a GDN hybrid's prefill time // actually was. @@ -606,10 +618,9 @@ TEST_CASE("Vulkan registers the W0 op set and NOT the unimplemented rest") { // the two MoE router/combine ops (rope_cos_sin_cache, rope_neox, // moe_router_topk, moe_combine); the reference tier likewise serves // MatmulBTQuant's keep-quant path. Still genuinely unimplemented: the sampler - // beyond greedy argmax (kApplyTemperature) and the PRE-FILL conv - // (kCausalConv1dFwd -- its state write-back needs a different dispatch shape - // than the decode update, see src/vt/vulkan/vulkan_ops.cpp). - for (vt::OpId op : {vt::OpId::kApplyTemperature, vt::OpId::kCausalConv1dFwd}) { + // beyond greedy argmax (kApplyTemperature). The PRE-FILL conv used to be listed + // here and is native now (BACKEND-VULKAN-GDN, vt_causal_conv1d_fwd). + for (vt::OpId op : {vt::OpId::kApplyTemperature}) { CHECK_FALSE(vt::OpRegistered(op, DeviceType::kVULKAN)); } // ...but they no longer THROW, and this assertion used to say they did. @@ -2396,6 +2407,464 @@ TEST_CASE("the decode causal conv1d update runs NATIVELY on Vulkan, state roll i cpu.DestroyQueue(cq); } +TEST_CASE("the prefill causal conv1d runs NATIVELY on Vulkan, in both token mappings") { + if (!VulkanPresent()) return; + auto& ctx = vt::vulkan::VulkanContext::Get(); + Backend& vk = vt::GetBackend(DeviceType::kVULKAN); + Backend& cpu = vt::GetBackend(DeviceType::kCPU); + Queue vq = vk.CreateQueue(); + Queue cq = cpu.CreateQueue(); + const Device vd{DeviceType::kVULKAN, 0}; + const Device cd{DeviceType::kCPU, 0}; + + // A varlen batch of three sequences, lengths 5, 1, 9, one of them with no + // initial state. The shape is chosen for the SPLIT arm below: with three + // sequences of 24 channels the default grid is a single workgroup, a target + // of 5 groups therefore asks for 5 token blocks, and sequence 0 (length 5) + // then gets one token per block -- so t = 1 and t = 2, both < width and both + // reading the carried state, land OUTSIDE block 0 unless the shader gives + // them to it. That is the race the block-0 rule exists for. + // + // FIVE, NOT FOUR, deliberately. The blocks of one (sequence, channel) are + // adjacent invocations; with 4 they sat inside one of llvmpipe's 8-lane + // subgroups in this layout and the pre-fix shader PASSED 5 runs of 5. With 5 + // they straddle a subgroup and it fails 5 of 5. + constexpr int64_t kN = 3, kT = 15, kC = 24, kK = 4, kWidth = kK - 1; + const std::vector qsl = {0, 5, 6, 15}; + const std::vector has = {1, 0, 1}; + const std::vector x = Spread(kT * kC, 2.0f, 71u); + const std::vector w = Spread(kC * kK, 0.5f, 73u); + const std::vector bias = Spread(kC, 0.2f, 79u); + const std::vector state0 = Spread(kN * kC * kWidth, 0.5f, 83u); + + // CPU oracle, once. + Buf cx(cpu, kT * kC, 4), cw(cpu, kC * kK, 4), cb(cpu, kC, 4), co(cpu, kT * kC, 4), + cs(cpu, kN * kC * kWidth, 4), cqsl(cpu, kN + 1, 4), chas(cpu, kN, 4); + std::memcpy(cx.p(), x.data(), x.size() * 4); + std::memcpy(cw.p(), w.data(), w.size() * 4); + std::memcpy(cb.p(), bias.data(), bias.size() * 4); + std::memcpy(cs.p(), state0.data(), state0.size() * 4); + std::memcpy(cqsl.p(), qsl.data(), qsl.size() * 4); + std::memcpy(chas.p(), has.data(), has.size() * 4); + std::memset(co.p(), 0, kT * kC * 4); + { + Tensor xt = Tensor::Contiguous(cx.p(), vt::DType::kF32, cd, {kT, kC}); + Tensor wt = Tensor::Contiguous(cw.p(), vt::DType::kF32, cd, {kC, kK}); + Tensor bt = Tensor::Contiguous(cb.p(), vt::DType::kF32, cd, {kC}); + Tensor ot = Tensor::Contiguous(co.p(), vt::DType::kF32, cd, {kT, kC}); + Tensor st = Tensor::Contiguous(cs.p(), vt::DType::kF32, cd, {kN, kC, kWidth}); + Tensor qt = Tensor::Contiguous(cqsl.p(), vt::DType::kI32, cd, {kN + 1}); + Tensor ht = Tensor::Contiguous(chas.p(), vt::DType::kI32, cd, {kN}); + vt::CausalConv1dFwd(cq, ot, xt, wt, &bt, st, qt, ht, vt::CausalConv1dArgs{}); + } + const std::vector out_ref(co.as(), co.as() + kT * kC); + const std::vector state_ref(cs.as(), cs.as() + kN * kC * kWidth); + + // Restores the knob even when a CHECK below throws out of the case. + struct EnvRestore { + ~EnvRestore() { vllm_test::UnsetEnv("VT_VULKAN_CONV_TARGET_GROUPS"); } + } restore; + + // Arm "" is the default mapping (one invocation per (sequence, channel)); arm + // "5" is the opt-in split. The split runs several times because a race is a + // scheduling outcome: one green dispatch would not show it is absent. + std::vector out_default; + for (const char* target : {"", "5", "5", "5", "5"}) { + CAPTURE(std::string(target)); + vllm_test::SetEnv("VT_VULKAN_CONV_TARGET_GROUPS", target); + + Buf vx(vk, kT * kC, 4), vw(vk, kC * kK, 4), vb(vk, kC, 4), vo(vk, kT * kC, 4), + vs(vk, kN * kC * kWidth, 4), vqsl(vk, kN + 1, 4), vhas(vk, kN, 4); + vk.Copy(vq, vx.p(), x.data(), x.size() * 4); + vk.Copy(vq, vw.p(), w.data(), w.size() * 4); + vk.Copy(vq, vb.p(), bias.data(), bias.size() * 4); + vk.Copy(vq, vs.p(), state0.data(), state0.size() * 4); + vk.Copy(vq, vqsl.p(), qsl.data(), qsl.size() * 4); + vk.Copy(vq, vhas.p(), has.data(), has.size() * 4); + vk.Synchronize(vq); + Tensor xt = Tensor::Contiguous(vx.p(), vt::DType::kF32, vd, {kT, kC}); + Tensor wt = Tensor::Contiguous(vw.p(), vt::DType::kF32, vd, {kC, kK}); + Tensor bt = Tensor::Contiguous(vb.p(), vt::DType::kF32, vd, {kC}); + Tensor ot = Tensor::Contiguous(vo.p(), vt::DType::kF32, vd, {kT, kC}); + Tensor st = Tensor::Contiguous(vs.p(), vt::DType::kF32, vd, {kN, kC, kWidth}); + Tensor qt = Tensor::Contiguous(vqsl.p(), vt::DType::kI32, vd, {kN + 1}); + Tensor ht = Tensor::Contiguous(vhas.p(), vt::DType::kI32, vd, {kN}); + vt::CausalConv1dFwd(vq, ot, xt, wt, &bt, st, qt, ht, vt::CausalConv1dArgs{}); + vk.Synchronize(vq); + + CHECK(RanNative(vt::OpId::kCausalConv1dFwd)); + // The block count is the mechanism, and it is invisible in the numbers by + // design, so the specialization VALUE is asserted, not just the module. + { + const std::string want = std::string("vt_causal_conv1d_fwd|") + + (target[0] == '\0' ? "1" : "5"); + const std::vector keys = ctx.PipelineKeysFor("vt_causal_conv1d_fwd"); + std::string joined; + for (const std::string& k : keys) joined += k + " "; + CAPTURE(joined); + CHECK(std::find(keys.begin(), keys.end(), want) != keys.end()); + } + + std::vector out_got(kT * kC), state_got(kN * kC * kWidth); + vk.Copy(vq, out_got.data(), vo.p(), out_got.size() * 4); + vk.Copy(vq, state_got.data(), vs.p(), state_got.size() * 4); + vk.Synchronize(vq); + + const double nmse = NmseOf(out_ref, out_got); + MESSAGE("causal_conv1d_fwd NMSE vs the CPU oracle: " << nmse); + CHECK(nmse <= kGdnNmseTol); + // The state write-back moves RAW x samples, so it is exact, not a tolerance. + CHECK(std::memcmp(state_got.data(), state_ref.data(), state_got.size() * 4) == 0); + // Spelled out independently of the oracle for the two sequences at least + // `width` long: their new state is their own last `width` input rows. + for (int64_t s : {int64_t{0}, int64_t{2}}) { + const int64_t end = qsl[static_cast(s) + 1]; + for (int64_t c = 0; c < kC; ++c) { + for (int64_t j = 0; j < kWidth; ++j) { + CAPTURE(s); + CAPTURE(c); + CAPTURE(j); + CHECK(state_got[static_cast((s * kC + c) * kWidth + j)] == + x[static_cast((end - kWidth + j) * kC + c)]); + } + } + } + // The split changes which invocation computes an element, never the order + // of its own k-tap sum, so both mappings must agree to the bit. + if (target[0] == '\0') { + out_default = out_got; + } else { + CHECK(std::memcmp(out_got.data(), out_default.data(), out_got.size() * 4) == 0); + } + } + + vk.DestroyQueue(vq); + cpu.DestroyQueue(cq); +} + +namespace { + +// One prefill-conv call on one backend, everything uploaded from host vectors +// and downloaded back. Shared by the edge-case and compressed-state cases below +// so the CPU oracle and the Vulkan arm run the SAME setup code. +struct ConvFwdCase { + std::vector qsl; + std::vector has; // one flag per sequence + bool his_i8 = false; + int64_t his_byte_off = 0; // i8 view start within its allocation + int64_t c = 8, k = 4; + int64_t x_pad = 0; // extra elements per x row (padded stride) + int64_t state_extra = 0; // conv_state.shape[2] = k - 1 + state_extra + bool with_bias = true; + bool silu = true; + vt::DType state_dt = vt::DType::kF32; + vt::DType op_dt = vt::DType::kF32; // x, weight and bias storage + vt::DType out_dt = vt::DType::kF32; + std::vector x, w, bias, state0; // state0 is [n, c, k-1+state_extra] +}; + +struct ConvFwdResult { + std::vector out, state; // state as f32 whatever its storage dtype +}; + +// f32 values to the bytes of `dt` (f32, f16 or bf16), and back. +std::vector ConvEncode(const std::vector& v, vt::DType dt) { + std::vector out(v.size() * (dt == vt::DType::kF32 ? 4 : 2)); + for (size_t i = 0; i < v.size(); ++i) { + if (dt == vt::DType::kF32) { + std::memcpy(out.data() + i * 4, &v[i], 4); + } else { + const uint16_t h = dt == vt::DType::kBF16 ? vt::F32ToBF16(v[i]) : vt::F32ToF16(v[i]); + std::memcpy(out.data() + i * 2, &h, 2); + } + } + return out; +} + +std::vector ConvDecode(const std::vector& b, vt::DType dt) { + const size_t es = dt == vt::DType::kF32 ? 4 : 2; + std::vector out(b.size() / es); + for (size_t i = 0; i < out.size(); ++i) { + if (dt == vt::DType::kF32) { + std::memcpy(&out[i], b.data() + i * 4, 4); + } else { + uint16_t h; + std::memcpy(&h, b.data() + i * 2, 2); + out[i] = dt == vt::DType::kBF16 ? vt::BF16ToF32(h) : vt::F16ToF32(h); + } + } + return out; +} + +ConvFwdResult RunConvFwd(Backend& b, Queue q, Device d, const ConvFwdCase& cs) { + const int64_t n = static_cast(cs.qsl.size()) - 1; + const int64_t t = cs.qsl.back(); + const int64_t rs = cs.c + cs.x_pad; + const int64_t sw = cs.k - 1 + cs.state_extra; + const int64_t sn = n * cs.c * sw; + const size_t st_bytes = cs.state_dt == vt::DType::kF32 ? 4 : 2; + const std::vector xe = ConvEncode(cs.x, cs.op_dt); + const std::vector we = ConvEncode(cs.w, cs.op_dt); + const std::vector be = ConvEncode(cs.bias, cs.op_dt); + const size_t oes = cs.out_dt == vt::DType::kF32 ? 4 : 2; + Buf bx(b, xe.size() / 4 + 1, 4), bw(b, we.size() / 4 + 1, 4), bb(b, be.size() / 4 + 1, 4), + bo(b, static_cast(std::max(1, t) * cs.c), oes), + bs(b, static_cast(sn), st_bytes), bq(b, static_cast(n + 1), 4), + bh(b, static_cast(n + 8), 4); + b.Copy(q, bx.p(), xe.data(), xe.size()); + b.Copy(q, bw.p(), we.data(), we.size()); + if (cs.with_bias) b.Copy(q, bb.p(), be.data(), be.size()); + if (cs.state_dt == vt::DType::kF32) { + b.Copy(q, bs.p(), cs.state0.data(), cs.state0.size() * 4); + } else { + std::vector s16(cs.state0.size()); + for (size_t i = 0; i < s16.size(); ++i) s16[i] = vt::F32ToBF16(cs.state0[i]); + b.Copy(q, bs.p(), s16.data(), s16.size() * 2); + } + b.Copy(q, bq.p(), cs.qsl.data(), cs.qsl.size() * 4); + // Flags: i32 at offset 0, or i8 bytes starting at his_byte_off of a buffer + // whose other bytes are a pattern that would flip the answer if read. + std::vector hb(static_cast((n + 8) * 4), 0xA5); + if (cs.his_i8) { + for (int64_t s = 0; s < n; ++s) hb[static_cast(cs.his_byte_off + s)] = cs.has[s]; + } else { + for (int64_t s = 0; s < n; ++s) { + const int32_t v = cs.has[static_cast(s)]; + std::memcpy(hb.data() + s * 4, &v, 4); + } + } + b.Copy(q, bh.p(), hb.data(), hb.size()); + const std::vector seed = ConvEncode( + std::vector(static_cast(std::max(1, t) * cs.c), -7.0f), cs.out_dt); + b.Copy(q, bo.p(), seed.data(), seed.size()); + b.Synchronize(q); + + Tensor xt = Tensor::Contiguous(bx.p(), cs.op_dt, d, {t, cs.c}); + xt.stride[0] = rs; + Tensor wt = Tensor::Contiguous(bw.p(), cs.op_dt, d, {cs.c, cs.k}); + Tensor bt = Tensor::Contiguous(bb.p(), cs.op_dt, d, {cs.c}); + Tensor ot = Tensor::Contiguous(bo.p(), cs.out_dt, d, {t, cs.c}); + Tensor st = Tensor::Contiguous(bs.p(), cs.state_dt, d, {n, cs.c, sw}); + Tensor qt = Tensor::Contiguous(bq.p(), vt::DType::kI32, d, {n + 1}); + Tensor ht = cs.his_i8 + ? Tensor::Contiguous(static_cast(bh.p()) + cs.his_byte_off, + vt::DType::kI8, d, {n}) + : Tensor::Contiguous(bh.p(), vt::DType::kI32, d, {n}); + vt::CausalConv1dArgs args; + args.silu_activation = cs.silu; + vt::CausalConv1dFwd(q, ot, xt, wt, cs.with_bias ? &bt : nullptr, st, qt, ht, args); + b.Synchronize(q); + + ConvFwdResult r; + std::vector ob(static_cast(t * cs.c) * oes); + b.Copy(q, ob.data(), bo.p(), ob.size()); + r.state.resize(static_cast(sn)); + if (cs.state_dt == vt::DType::kF32) { + b.Copy(q, r.state.data(), bs.p(), r.state.size() * 4); + } else { + std::vector s16(r.state.size()); + b.Copy(q, s16.data(), bs.p(), s16.size() * 2); + for (size_t i = 0; i < s16.size(); ++i) r.state[i] = vt::BF16ToF32(s16[i]); + } + b.Synchronize(q); + r.out = ConvDecode(ob, cs.out_dt); + return r; +} + +ConvFwdCase MakeConvFwdCase(std::vector qsl, std::vector has, int64_t c, + int64_t k, int64_t x_pad, uint32_t seed) { + ConvFwdCase cs; + cs.qsl = std::move(qsl); + cs.has = std::move(has); + cs.c = c; + cs.k = k; + cs.x_pad = x_pad; + const int64_t n = static_cast(cs.qsl.size()) - 1; + const int64_t t = cs.qsl.back(); + cs.x = Spread(static_cast(std::max(1, t) * (c + x_pad)), 2.0f, seed); + cs.w = Spread(static_cast(c * k), 0.5f, seed + 1); + cs.bias = Spread(static_cast(c), 0.2f, seed + 2); + cs.state0 = Spread(static_cast(n * c * (k - 1)), 0.5f, seed + 3); + return cs; +} + +} // namespace + +TEST_CASE("the prefill causal conv1d matches the CPU oracle on its edge shapes") { + if (!VulkanPresent()) return; + Backend& vk = vt::GetBackend(DeviceType::kVULKAN); + Backend& cpu = vt::GetBackend(DeviceType::kCPU); + Queue vq = vk.CreateQueue(); + Queue cq = cpu.CreateQueue(); + const Device vd{DeviceType::kVULKAN, 0}; + const Device cd{DeviceType::kCPU, 0}; + + // Lengths 5, 0, 1, 7: an EMPTY sequence, and a length-1 sequence WITH initial + // state, whose new state is two shifted old taps plus one x sample. Flags + // 1,1,1,0. x rows are padded by 3 elements, so the row stride is not c. + ConvFwdCase base = MakeConvFwdCase({0, 5, 5, 6, 13}, {1, 1, 1, 0}, 40, 4, 3, 211u); + + struct Variant { + const char* name; + bool his_i8; + int64_t his_byte_off; + bool with_bias, silu; + vt::DType op_dt = vt::DType::kF32, out_dt = vt::DType::kF32; + }; + // i8 flags at every byte phase of a 32-bit word, the read a 4-aligned + // assumption gets wrong; and bias/silu both ways. + for (const Variant& v : {Variant{"i32 flags, bias, silu", false, 0, true, true}, + Variant{"i8 flags @0, no bias, no silu", true, 0, false, false}, + Variant{"i8 flags @1", true, 1, true, true}, + Variant{"i8 flags @2, no bias", true, 2, false, true}, + Variant{"i8 flags @3, no silu", true, 3, true, false}, + // Reduced-precision operands and output: the shader + // reads them through the 16-bit views and its own + // dtype codes, the CPU reference through LoadF32. + Variant{"bf16 x/w/bias, bf16 out", false, 0, true, true, + vt::DType::kBF16, vt::DType::kBF16}, + Variant{"f16 x/w/bias, f32 out", true, 1, true, true, + vt::DType::kF16, vt::DType::kF32}}) { + CAPTURE(std::string(v.name)); + ConvFwdCase cs = base; + cs.his_i8 = v.his_i8; + cs.his_byte_off = v.his_byte_off; + cs.with_bias = v.with_bias; + cs.silu = v.silu; + cs.op_dt = v.op_dt; + cs.out_dt = v.out_dt; + const auto before = vt::GetOpProviderStats(vt::OpId::kCausalConv1dFwd, DeviceType::kVULKAN); + const ConvFwdResult ref = RunConvFwd(cpu, cq, cd, cs); + const ConvFwdResult got = RunConvFwd(vk, vq, vd, cs); + const auto after = vt::GetOpProviderStats(vt::OpId::kCausalConv1dFwd, DeviceType::kVULKAN); + CHECK(RanNative(vt::OpId::kCausalConv1dFwd)); + CHECK(after.declines == before.declines); + CHECK(NmseOf(ref.out, got.out) <= kGdnNmseTol); + CHECK(std::memcmp(got.state.data(), ref.state.data(), ref.state.size() * 4) == 0); + } + + vk.DestroyQueue(vq); + cpu.DestroyQueue(cq); +} + +TEST_CASE("the prefill causal conv1d keeps a bf16 conv_state IN PLACE, bit-exact vs the f32 arm") { + if (!VulkanPresent()) return; + REQUIRE(vt::GetBackend(DeviceType::kVULKAN).SupportsCompressedConvState()); + Backend& vk = vt::GetBackend(DeviceType::kVULKAN); + Queue vq = vk.CreateQueue(); + const Device vd{DeviceType::kVULKAN, 0}; + + // The CPU reference reads f32 state only, so the oracle here is the native f32 + // arm on bf16-REPRESENTABLE inputs -- the same shape as the compressed-state + // update case above. With every x and state value exactly a bf16, the + // compressed arm must read the same taps, produce the same outputs, and store + // the same raw samples back. + ConvFwdCase cs = MakeConvFwdCase({0, 5, 6, 15}, {1, 0, 1}, 24, 4, 0, 307u); + for (float& v : cs.x) v = vt::BF16ToF32(vt::F32ToBF16(v)); + for (float& v : cs.state0) v = vt::BF16ToF32(vt::F32ToBF16(v)); + const ConvFwdResult f32_arm = RunConvFwd(vk, vq, vd, cs); + cs.state_dt = vt::DType::kBF16; + const ConvFwdResult bf16_arm = RunConvFwd(vk, vq, vd, cs); + CHECK(RanNative(vt::OpId::kCausalConv1dFwd)); + CHECK(std::memcmp(bf16_arm.out.data(), f32_arm.out.data(), f32_arm.out.size() * 4) == 0); + CHECK(std::memcmp(bf16_arm.state.data(), f32_arm.state.data(), f32_arm.state.size() * 4) == 0); + + vk.DestroyQueue(vq); +} + +TEST_CASE("the prefill causal conv1d DECLINES a conv_state row wider than K-1") { + if (!VulkanPresent()) return; + REQUIRE(vt::ReferenceTierEligible(DeviceType::kVULKAN)); + Backend& vk = vt::GetBackend(DeviceType::kVULKAN); + Backend& cpu = vt::GetBackend(DeviceType::kCPU); + Queue vq = vk.CreateQueue(); + Queue cq = cpu.CreateQueue(); + const Device vd{DeviceType::kVULKAN, 0}; + const Device cd{DeviceType::kCPU, 0}; + + // A speculative-decode-widened row (K-1 + 2). The CPU reference addresses + // rows with stride K-1; the op layer admits the wider row; the native kernel + // declines rather than pick one, so the answer is the reference's, byte for + // byte, and the decline is counted. Whether the reference's stride is right + // for widened rows is a question about the CPU kernel, not tested here. + ConvFwdCase cs = MakeConvFwdCase({0, 4, 9}, {1, 1}, 8, 4, 0, 401u); + cs.state_extra = 2; + cs.state0 = Spread(static_cast(2 * 8 * (3 + 2)), 0.5f, 404u); + const ConvFwdResult ref = RunConvFwd(cpu, cq, cd, cs); + const auto before = vt::GetOpProviderStats(vt::OpId::kCausalConv1dFwd, DeviceType::kVULKAN); + const ConvFwdResult got = RunConvFwd(vk, vq, vd, cs); + const auto after = vt::GetOpProviderStats(vt::OpId::kCausalConv1dFwd, DeviceType::kVULKAN); + CHECK(after.declines == before.declines + 1); + CHECK(std::memcmp(got.out.data(), ref.out.data(), ref.out.size() * 4) == 0); + CHECK(std::memcmp(got.state.data(), ref.state.data(), ref.state.size() * 4) == 0); + + vk.DestroyQueue(vq); + cpu.DestroyQueue(cq); +} + +TEST_CASE("the prefill causal conv1d DECLINES a width past its window, and stays correct") { + if (!VulkanPresent()) return; + REQUIRE(vt::ReferenceTierEligible(DeviceType::kVULKAN)); + Backend& vk = vt::GetBackend(DeviceType::kVULKAN); + Backend& cpu = vt::GetBackend(DeviceType::kCPU); + Queue vq = vk.CreateQueue(); + Queue cq = cpu.CreateQueue(); + const Device vd{DeviceType::kVULKAN, 0}; + const Device cd{DeviceType::kCPU, 0}; + + // K = 10 -> width 9, one past the shader's private window. The native kernel + // must hand the call to the next provider, not throw and not read past the + // array -- and the answer must still be the CPU oracle's. + constexpr int64_t kN = 1, kT = 12, kC = 8, kK = 10, kWidth = kK - 1; + const std::vector qsl = {0, 12}; + const std::vector has = {1}; + const std::vector x = Spread(kT * kC, 2.0f, 89u); + const std::vector w = Spread(kC * kK, 0.5f, 97u); + const std::vector state0 = Spread(kN * kC * kWidth, 0.5f, 101u); + + struct Arm { + Backend& b; + Queue qu; + Device d; + }; + std::vector outs[2], states[2]; + for (int arm = 0; arm < 2; ++arm) { + Arm a = arm == 0 ? Arm{cpu, cq, cd} : Arm{vk, vq, vd}; + Buf bx(a.b, kT * kC, 4), bw(a.b, kC * kK, 4), bo(a.b, kT * kC, 4), + bs(a.b, kN * kC * kWidth, 4), bq(a.b, kN + 1, 4), bh(a.b, kN, 4); + a.b.Copy(a.qu, bx.p(), x.data(), x.size() * 4); + a.b.Copy(a.qu, bw.p(), w.data(), w.size() * 4); + a.b.Copy(a.qu, bs.p(), state0.data(), state0.size() * 4); + a.b.Copy(a.qu, bq.p(), qsl.data(), qsl.size() * 4); + a.b.Copy(a.qu, bh.p(), has.data(), has.size() * 4); + a.b.Synchronize(a.qu); + Tensor xt = Tensor::Contiguous(bx.p(), vt::DType::kF32, a.d, {kT, kC}); + Tensor wt = Tensor::Contiguous(bw.p(), vt::DType::kF32, a.d, {kC, kK}); + Tensor ot = Tensor::Contiguous(bo.p(), vt::DType::kF32, a.d, {kT, kC}); + Tensor st = Tensor::Contiguous(bs.p(), vt::DType::kF32, a.d, {kN, kC, kWidth}); + Tensor qt = Tensor::Contiguous(bq.p(), vt::DType::kI32, a.d, {kN + 1}); + Tensor ht = Tensor::Contiguous(bh.p(), vt::DType::kI32, a.d, {kN}); + const auto before = vt::GetOpProviderStats(vt::OpId::kCausalConv1dFwd, DeviceType::kVULKAN); + vt::CausalConv1dFwd(a.qu, ot, xt, wt, nullptr, st, qt, ht, vt::CausalConv1dArgs{}); + a.b.Synchronize(a.qu); + if (arm == 1) { + const auto after = vt::GetOpProviderStats(vt::OpId::kCausalConv1dFwd, DeviceType::kVULKAN); + CHECK(after.declines == before.declines + 1); + } + outs[arm].resize(kT * kC); + states[arm].resize(kN * kC * kWidth); + a.b.Copy(a.qu, outs[arm].data(), bo.p(), outs[arm].size() * 4); + a.b.Copy(a.qu, states[arm].data(), bs.p(), states[arm].size() * 4); + a.b.Synchronize(a.qu); + } + // The fallback IS the CPU kernel, so this is exact, not a tolerance. + CHECK(std::memcmp(outs[1].data(), outs[0].data(), outs[0].size() * 4) == 0); + CHECK(std::memcmp(states[1].data(), states[0].data(), states[0].size() * 4) == 0); + + vk.DestroyQueue(vq); + cpu.DestroyQueue(cq); +} + // BACKEND-VULKAN-DEVICE-RESIDENT. The capability the model's indexed state-I/O // path needs: a COMPRESSED (bf16) conv_state addressed IN PLACE, which is what // Backend::SupportsCompressedConvState() advertises. The oracle is the arm this