From a34c2ccc77c20270d0119556ca5cb527ef33e682 Mon Sep 17 00:00:00 2001 From: AllenKuo Date: Thu, 1 Oct 2026 13:23:13 +0800 Subject: [PATCH] feat(vulkan): native kCausalConv1dFwd, the GDN prefill conv MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ## Row `BACKEND-VULKAN`, branch `row/BACKEND-VULKAN-CONVFWD`; the same sub-row naming as `row/BACKEND-VULKAN-GDN` (#145) and `row/BACKEND-VULKAN-GDN-CORE` (#152). ## Before starting - Issue/PR search and existing claim: no open PR, claim or issue names `kCausalConv1dFwd` on Vulkan. Local issue filed and closed in this PR: `.agents/issues/BACKEND-VULKAN/ISSUE-LOCAL-01M3V01R90QQTQ08BWSBM7AGN2.md` (Kind: gap). - Pull request shape selected at row claim: one op, one shader, its tests and docs; helper PR. - Roadmap or matrix row: `.agents/backend-matrix.md` `BACKEND-VULKAN`; VK-G in `.agents/specs/vulkan-full-support.md`. `scripts/ready-for-helper.py --check-local`: structure OK, live claims not verified locally. - Exact current-code and test/evidence anchors inspected: the GDN-family block comment in `src/vt/vulkan/vulkan_ops.cpp` that left this op "for a follow-up rather than guessed at here" and named the two safe dispatch shapes; `src/vt/cpu/cpu_ops.cpp` `CausalConv1dFwdKernel`; `src/vt/ops.cpp` `CausalConv1dFwd` / `CheckConvCommon`; the decode-update and compressed-state cases in `tests/vt/test_vulkan_backend.cpp`; §6.0a of the spec. ## What changed A native Vulkan kernel for `vt::CausalConv1dFwd`, the GDN PREFILL depthwise causal conv, which ran on the portable CPU reference tier once per GDN layer. The default mapping is the first shape the existing comment named: one invocation per (sequence, channel), serial over its tokens, with the old state window copied to a private array before the write-back, as the CPU kernel's `old_row` does. An opt-in token split (`VT_VULKAN_CONV_TARGET_GROUPS`) stays in one dispatch by giving every token that reads the carried state (`t < width`) to block 0, which is also the only block that reads or writes `conv_state`. Shapes the shader does not serve decline to the reference tier through `GetOpFallback`: K = 1, K - 1 > 8, `conv_state` rows wider than K - 1, storage dtypes outside f32/f16/bf16, flags neither i8 nor i32, a grid past `maxComputeWorkGroupCount[0]`, or any index past the shader's uint32 arithmetic. `VT_VULKAN_CONV_FWD=0` keeps the op on the reference tier for a same-binary A/B. Both knobs are documented in `docs/ENVIRONMENT.md`; the spec gains §6.0b. ## Evidence New cases in `tests/vt/test_vulkan_backend.cpp`: - `the prefill causal conv1d runs NATIVELY on Vulkan, in both token mappings`: default and a 5-block split on lengths 5, 1, 9, against the CPU oracle; the specialization value (`|1`, `|5`) is asserted from `PipelineKeysFor`. - `the prefill causal conv1d matches the CPU oracle on its edge shapes`: i8 flags at byte offsets 0-3, no bias, silu off, bf16 and f16 operands with bf16 or f32 output, a zero-length sequence, a length-1 sequence with initial state, a padded `x` row stride. - `the prefill causal conv1d keeps a bf16 conv_state IN PLACE, bit-exact vs the f32 arm`. - `the prefill causal conv1d DECLINES a width past its window, and stays correct` and `... DECLINES a conv_state row wider than K-1`: decline counted, bytes equal to the reference. Outputs to the GDN NMSE tolerance; rolled state bit-exact, and spelled out independently of the oracle for the main case. Red-before / green-after on llvmpipe (LLVM 20.1.2, 256 bits), the device class this repository's Vulkan CI uses; five runs per arm of `-tc='*prefill causal conv1d*,*cooperative-matrix capability*'`: ``` this PR 5/5 6 passed 798/798 assertions pre-fix behaviour (every block copies the state row, plain blk*tb split) 5/5 1 failed 790/798 assertions CHECK( nmse <= kGdnNmseTol ) 0.0140122 <= 0.0005 CHECK( memcmp(split, default) == 0 ) 115 == 0 ``` Production call site removed (the `RegisterOp` disabled): 5 of the 6 cases fail (`RanNative`, decline counters). Discrete NVIDIA RTX PRO 6000 (Windows, MSVC 14.44.35207, `/W4 /WX`, 0 warnings): all new cases pass. `test_vulkan_backend` 47/51; the four failures are on main and unchanged by this PR (three TQ2 NMSE cases, the `vt_matmul_tiled` pipeline check), same values with and without it. ``` scripts/gen-vulkan-spirv.py --check committed SPIR-V is up to date scripts/check-agent-record.py agent record OK scripts/check-env-doc.py no finding from this PR ``` - [ ] `scripts/agent-preflight.sh` passes: not run in full, see Honest gaps. - [x] tests that cover this change: listed above. - [x] public docs changed only for facts this PR owns: two rows in `docs/ENVIRONMENT.md`. ## Speed claims - [x] This PR makes NO speed claim. ## Honest gaps - The race test is a red-before on llvmpipe only. On the discrete NVIDIA card the pre-fix behaviour PASSES, most likely because the blocks of one (sequence, channel) are adjacent invocations that run together there (not separately demonstrated). It also stops failing on llvmpipe if the split uses 4 blocks instead of 5, which is why the test uses 5 and the block count is capped rather than rounded to a power of two. - The split and the decision to leave it off by default were looked at on one discrete NVIDIA card only. No other device informed them. - The shader's `query_start_loc` guard bounds memory but is not validation: a malformed table whose intervals overlap gives unspecified output here, where the CPU reference throws. Device-side all-or-nothing validation would need a second dispatch and is not attempted. - Observed, not changed: the CPU reference addresses `conv_state` rows with stride K - 1, while `CheckConvCommon` admits wider rows (speculative-decode taps) and describes them as using the physical stride. This kernel declines those shapes, so it reproduces the reference exactly, including that addressing. - `VT_VULKAN_CONV_FWD=0` is read once at registration and has no unit test. - `scripts/agent-preflight.sh` was not run in full: on Windows one of its tests shells out to WSL and hangs. The platform-neutral checkers it calls were run individually (above); CI is the full run. - Built and run on Windows with local patches that are NOT in this diff: the tev1 registration's cross-TU static-init order, and three MSVC C4244 narrowings in existing lines (`test_tev1.cpp`, two in `vulkan_ops.cpp`). The Linux llvmpipe build (GCC 13.3) used the same patched tree; whether it needs them was not tested. - `check-env-doc.py` fails on main for three existing variables (`VT_VK_DISABLE`, `VT_VK_DISABLE_PAGED_ATTN`, `VT_VK_FENCE_TIMEOUT_MS`); not touched here. - `vulkan_spirv.h` records glslang 16.5.0 (the version `gen-vulkan-spirv.py` names as CI's pin) where main recorded 16.4.0. `--check` with 16.5.0 matches main's blobs, so the existing 43 modules are byte-identical; only that line and the new module change. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: AGENT:claude-opus-5-5 [claude-code] --- .../ISSUE-LOCAL-01M3V01R90QQTQ08BWSBM7AGN2.md | 19 + .agents/specs/vulkan-full-support.md | 48 ++ docs/ENVIRONMENT.md | 2 + .../vulkan/shaders/vt_causal_conv1d_fwd.comp | 225 ++++++ src/vt/vulkan/vulkan_ops.cpp | 205 +++++- src/vt/vulkan/vulkan_spirv.cpp | 668 +++++++++++++++++- src/vt/vulkan/vulkan_spirv.h | 2 +- tests/vt/test_vulkan_backend.cpp | 481 ++++++++++++- 8 files changed, 1632 insertions(+), 18 deletions(-) create mode 100644 .agents/issues/BACKEND-VULKAN/ISSUE-LOCAL-01M3V01R90QQTQ08BWSBM7AGN2.md create mode 100644 src/vt/vulkan/shaders/vt_causal_conv1d_fwd.comp diff --git a/.agents/issues/BACKEND-VULKAN/ISSUE-LOCAL-01M3V01R90QQTQ08BWSBM7AGN2.md b/.agents/issues/BACKEND-VULKAN/ISSUE-LOCAL-01M3V01R90QQTQ08BWSBM7AGN2.md new file mode 100644 index 0000000000..e9d4ca4bd4 --- /dev/null +++ b/.agents/issues/BACKEND-VULKAN/ISSUE-LOCAL-01M3V01R90QQTQ08BWSBM7AGN2.md @@ -0,0 +1,19 @@ +ID: ISSUE-LOCAL-01M3V01R90QQTQ08BWSBM7AGN2 +Title: kCausalConv1dFwd (GDN prefill conv) has no native Vulkan kernel +Row: BACKEND-VULKAN +State: CLOSED +Kind: gap +GitHub: - +Mirror: PENDING +Availability: FULL +Created: 2026-10-01 +Updated: 2026-10-01 +Closed: 2026-10-01 + +## Problem + +vt::CausalConv1dFwd, the GDN PREFILL depthwise causal conv, is not registered for DeviceType::kVULKAN, so every GDN prefill on a Vulkan queue runs it on the portable CPU reference tier, once per GDN layer, behind a full batch drain. src/vt/vulkan/vulkan_ops.cpp left it as a follow-up because the op reads the OLD conv-state window and overwrites it in the same call; the comment names the two safe dispatch shapes (a serial invocation per (sequence, channel) over the whole token range, or a buffered old row). .agents/specs/vulkan-full-support.md section 6.0a lists it as one of the two remaining reference-tier declines. + +## Resolution + +Closed by row/BACKEND-VULKAN-CONVFWD: native vt_causal_conv1d_fwd, default one invocation per (sequence, channel) serial over its tokens with the old window copied before the write-back; opt-in token split (VT_VULKAN_CONV_TARGET_GROUPS) giving every t < width to block 0; unservable shapes decline through GetOpFallback; VT_VULKAN_CONV_FWD=0 keeps the reference tier. Gated by tests/vt/test_vulkan_backend.cpp against the CPU oracle; the split arm is red on llvmpipe with the pre-fix plain split and green with this PR. diff --git a/.agents/specs/vulkan-full-support.md b/.agents/specs/vulkan-full-support.md index 9f2c687206..01533b0f16 100644 --- a/.agents/specs/vulkan-full-support.md +++ b/.agents/specs/vulkan-full-support.md @@ -613,6 +613,54 @@ gate is a memcmp of the two arms in ONE process, plus the specialization VALUES from `PipelineKeys()`, because both arms are the same module and produce identical bytes; a numeric check alone could never see the mechanism. +### 6.0b `VK-G` partial: the PREFILL CONV landed — 2026-10-01 + +`row/BACKEND-VULKAN-CONVFWD`. `kCausalConv1dFwd` is native +(`vt_causal_conv1d_fwd`); module count 43 -> 44. This updates the closing +sentence of §6.0a: of the two ops it named as the reference-tier declines on that +path, `kRopeCosSinCache` remains, by design. + +**Why it had been left.** The op computes every output from the OLD conv-state +window and then overwrites that window in the same call, so a dispatch that +splits a sequence across invocations reads state another invocation is +rewriting. `vulkan_ops.cpp` named the two safe shapes: a serial invocation per +(sequence, channel) over the whole token range, or a buffered old row. + +**What landed.** The first of those shapes is the DEFAULT: one invocation per +(sequence, channel), the CPU kernel's own `ForRows(n * c_dim, ...)` unit, with +the old window copied into a private array before the write-back (the CPU +kernel's `old_row`). Per-element arithmetic is ported 1:1 from +`src/vt/cpu/cpu_ops.cpp` `CausalConv1dFwdKernel`, including its silu spelling. + +An OPT-IN token split, `VT_VULKAN_CONV_TARGET_GROUPS`, divides each sequence +into token blocks in the same dispatch. Only `t < width` reads the carried state; +the shader gives every such token to block 0, which also writes the state back +and is the only block that touches `conv_state`. Without that rule the split +raced: on llvmpipe the wrong elements were exactly the `t < width` tokens that +fell outside block 0. It is off by default; whether a split is worth anything is +a property of the device. + +Shapes the shader does not serve DECLINE to the reference tier through +`GetOpFallback`, the seam §6.0 uses: K = 1, a kernel width past its 8-slot +window, `conv_state` rows wider than K-1 (the CPU reference addresses rows with +stride K-1 while the op layer admits wider rows; declining keeps the reference's +answer instead of choosing), storage dtypes outside f32/f16/bf16, a +`has_initial_state` that is neither i8 nor i32, a grid past the device's +`maxComputeWorkGroupCount[0]`, and any index that would not fit the shader's +uint32 arithmetic. The native path does not read `query_start_loc` on the host; +the shader's guard keeps every access in bounds but does not validate the table, +so a malformed table that the CPU reference would reject gives unspecified +output here. + +**Gates.** `test_vulkan_backend`, against the CPU oracle: the default mapping +and the split on a varlen batch (lengths 5, 1, 9, one sequence without initial +state), with the specialization value asserted; i8 flags at byte offsets 1-3, no +bias, silu off, bf16 and f16 operands with bf16 or f32 output, a zero-length +sequence and a padded `x` row stride; a bf16 +`conv_state` against the f32 arm; and the K = 10 and widened-row declines. +Outputs to the GDN NMSE tolerance, rolled state bit-exact. +`VT_VULKAN_CONV_FWD=0` keeps the op on the reference tier for a same-binary A/B. + ### 6.0a `VK-G` partial: the FUSED ATTN PREAMBLE landed — 2026-08-09 `row/BACKEND-VULKAN-QKNORM`. `kAttnQkNormRopeGate` — gemma-RMSNorm(q) + diff --git a/docs/ENVIRONMENT.md b/docs/ENVIRONMENT.md index d514d76cda..56035fb1c9 100644 --- a/docs/ENVIRONMENT.md +++ b/docs/ENVIRONMENT.md @@ -292,6 +292,8 @@ portable/reference path. In normal operation leave them unset. | `VT_VULKAN_RMSNORM` | auto | Which `vt::RmsNorm` SPIR-V module runs: `wide` forces the 1024-invocation subgroup-reducing one, `base` forces the portable 128-invocation one, unset lets the device capability decide (1024 invocations on the X axis plus compute subgroup BASIC and ARITHMETIC). The wide module exists because `RmsNorm` dispatches ONE WORKGROUP PER ROW and a batch-1 decode step has exactly one row: on Qwen3.6-27B that put 128 invocations on a 5120-wide row, four warps of one SM, with the rest of the GPU idle. MEASURED on GB10 by the two-length GPU-timestamp diff: **0.0611 -> 0.0123 ms/call, 7.88 -> 1.59 ms/token**, and paired decode **241.9 -> 235.6 ms** median TPOT. The tell that it was OCCUPANCY and not the reduction is that the SAME shader costs 0.066 ms/call during PREFILL, where 32 rows give it 32 workgroups and 32x the data. It exists for the same-binary A/B and so the unit gate can exercise the fallback on hardware that would always pick the wide arm. Vulkan-only | | `VT_VULKAN_MATMUL_NCOLS` | 4 | Output columns each lane of the portable scalar GEMM computes, in the `[K,N]` (non-transposed) orientation only. At 1 the kernel is the flat one-invocation-per-output-element body; above 1 a workgroup takes `128*NCOLS` CONSECUTIVE output columns of one row, so at each step of K it reads a contiguous run of that many elements instead of 128. This is the ONE decode GEMM that cannot reach the `vt_matmul_vec` tactic, because in `[K,N]` the lanes are already coalesced and the GEMV shape would make them strided; on the 27B it is the lm_head, `m=1 k=5120 n=248320`, 2.54 GB moved per token. MEASURED on GB10, 27B decode, `ms/call` medians over interleaved replicates: NCOLS 1 = 12.48, 2 = 12.46, **4 = 11.54**, 8 = 12.81, with 4 winning **6 of 6** interleaved pairs against 1. Blocking is a TRADE, not a monotone win: at 8 the dispatch falls to 243 workgroups (~31k threads) and the device runs out of work to hide memory latency with faster than the longer contiguous run buys back. It rides a specialization constant, so every arm is the same committed module and they A/B in one binary. Every arm is BIT-IDENTICAL -- each accumulator owns one output element and sums the whole K sequentially, which is the CPU kernel's order -- so this kernel keeps the byte-exact tier that the coopmat and GEMV tactics gave up; a memcmp gates that. Vulkan-only | | `VT_VULKAN_COOPMAT` | on | `=0` forces the Vulkan GEMM onto the portable SCALAR kernel instead of the cooperative-matrix (tensor-core) tactic. The coopmat path is selected only where the device reports the exact `16x16x16 bf16/bf16/f32/f32 SUBGROUP` configuration, subgroup size is 32, both operands are bf16, and M, N and K are all multiples of 16. The whole-tile requirement on M and N is not a tuning choice: `coopMatLoad` reads a full 16x16 tile with no masking, so a partial tile reads past the operand and can fault the GPU. Ragged shapes fall back to the scalar kernel; this switch bypasses that selection entirely. It exists for the same-binary A/B in `examples/vulkan-gemm-ab` (measured 11.1x-32.9x on NVIDIA Thor) and as the bisect lever if a coopmat result is ever suspect. Vulkan-only | +| `VT_VULKAN_CONV_FWD` | on | `=0` leaves `vt::CausalConv1dFwd` (the GDN PREFILL conv) on the portable CPU reference tier instead of registering the native `vt_causal_conv1d_fwd` kernel, which is where the op ran before that kernel existed. Read once, at backend registration. It exists for the same-binary A/B and as the bisect lever, the same shape as `VT_VULKAN_COOPMAT`. The native kernel matches the CPU reference to a tolerance on the outputs and bit-exactly on the rolled `conv_state`. Vulkan-only | +| `VT_VULKAN_CONV_TARGET_GROUPS` | unset (no split) | Opt-in TOKEN SPLIT for the native prefill conv. Unset, each (sequence, channel) is one invocation serial over its tokens. Set to a workgroup target `g` (1..65536), each sequence is split into `ceil(g / base_groups)` token blocks, where `base_groups` is the default grid in workgroups, capped at the mean sequence length, at what one dispatch can launch, and at 64 (each block count is its own compiled pipeline). Only tokens `t < width` read the carried state; the shader gives all of them to block 0, which is also the only block that writes the state back, so no other block touches `conv_state`. Each element is computed by the same expression whatever the block count; `test_vulkan_backend` compares 1 and 5 blocks bit-for-bit. Whether a split is worth anything depends on the device, which is why there is no default. Read on every call. Vulkan-only | | `VT_GLM5_NEXT_DEVICE_EXPERTS` | **off (opt-in)** | `=1` lets `Glm5NextForConditionalGeneration` (GLM-5.3-Flash) accept a non-CPU queue and route its routed-expert keep-quant GEMM to the device, against banks made resident by `dense_attn::ResidentWeight`. **IT IS OFF BECAUSE THE PATH IT ENABLES IS MEASURED TO CRASH, not because it is unmeasured.** On `dgx:gpu0` against the published 101.24 GiB `UD-Q2_K_XL` artifact, ALL THREE `--device cuda` legs died with SIGSEGV (rc=139) having emitted no token, interleaved against three `--device cpu` legs that all emitted ` Paris.` from the same binary. **The mixed-residency reading of those legs is FALSIFIED and the cause is elsewhere**: the two log lines that suggested it are once-flags, and the process dies in `StoreCaches`, which host-stores into the runner's `cudaMalloc` KV pages after the forward has already returned. That defect is older than this knob and only became reachable when the non-CPU refusal above it was removed; see `.agents/specs/glm5-next-flash.md` O49 and [#2480](https://github.com/mudler/vllm.cpp/issues/2480), which owns the fix. The default is the refusal the tree carried before the arm existed, because turning a clean named error into a segfault is strictly worse for a user. **Set this only to debug that crash; it is not a serving knob.** Parsed strictly (`1` and nothing else, not the usual first-character rule) precisely because it opts into a crashing path. Inert on every other model and on `--device cpu`. See `.agents/specs/glm5-next-flash.md` O46 and [#2464](https://github.com/mudler/vllm.cpp/issues/2464) | | `VT_GLM5_NEXT_DEVICE` | **off (opt-in)** | `=1` routes the entire GLM-5.3-Flash forward through `Glm5NextDeviceForward`, which dispatches embedding, RMSNorm, MoE combine, the k-pool indexer and `lm_head` through `vt::*` device ops on the queue, keeping MLA attention, mHC sites and the dense MLP as host-fallback islands (the `kimi_linear_device.cpp` single-queue pattern). This is a superset of `VT_GLM5_NEXT_DEVICE_EXPERTS`: when on, the whole forward delegates and the per-arm experts split is not reached. On a CPU queue the `vt::*` kernels use float32 accumulation where the host reference uses double, so the output agrees within a float-vs-double envelope rather than byte-exact. Inert on every other model. See `.agents/specs/glm5-next-flash.md` W9c-3 and [#3175](https://github.com/mudler/vllm.cpp/pull/3175) | diff --git a/src/vt/vulkan/shaders/vt_causal_conv1d_fwd.comp b/src/vt/vulkan/shaders/vt_causal_conv1d_fwd.comp new file mode 100644 index 0000000000..5b5f433d16 --- /dev/null +++ b/src/vt/vulkan/shaders/vt_causal_conv1d_fwd.comp @@ -0,0 +1,225 @@ +#version 450 +// vt::CausalConv1dFwd — the GDN PREFILL depthwise causal conv. +// +// WHY THIS EXISTS. The op had no Vulkan kernel, so every GDN prefill fell to the +// portable CPU reference tier once per GDN layer. vulkan_ops.cpp left it as a +// follow-up because of the state write-back hazard described below; this is +// that follow-up, in the serial-per-(sequence, channel) shape it named. +// +// Row: BACKEND-VULKAN, branch row/BACKEND-VULKAN-CONVFWD (one op of the VK-G +// family in .agents/specs/vulkan-full-support.md). +// +// SHAPE OF THE WORK, and why it is NOT llama.cpp's shape. llama.cpp's +// ssm_conv.comp parallelises over tokens too (BLOCK_SIZE=32, TOKENS_PER_WG=16) +// and gets away with it because ggml's dataflow is SSA: its conv op only +// produces outputs, and advancing the state is a separate node. This op does +// both in one kernel -- it reads the OLD state while computing outputs, then +// OVERWRITES that state with the last K-1 input samples. Splitting tokens +// across invocations would put those two against each other across invocation +// boundaries. +// +// So the unit here is ONE INVOCATION PER (sequence, channel), which is exactly +// the CPU kernel's own chunking unit (`ForRows(n * c_dim, ...)`), and the old +// state is copied into a private array first -- again exactly what the CPU +// kernel does with `old_row`. Each (s, c) pair then owns its output column +// slice and its state row outright, so there is no cross-invocation hazard and +// no barrier is needed. +// +// That is the DEFAULT mapping. It gives up token parallelism: the grid is +// n * c_dim invocations, each doing t_len * k multiply-adds. An opt-in token +// split exists (VT_CONV_BLOCKS below) and stays in ONE dispatch, because only +// t < width reads the carried state and all of those go to the block that +// also writes it back. +// +// PORTED 1:1 from src/vt/cpu/cpu_ops.cpp CausalConv1dFwdKernel. The accumulation +// order per output element is identical, the silu spelling is the reference's +// own (`acc / (1 + exp(-acc))`, not `acc * sigmoid(acc)`, which rounds +// differently), and VT_LOAD/VT_STORE keep the same single round on store. +#extension GL_GOOGLE_include_directive : require +#include "vt_common.glsl" + +layout(local_size_x = 128, local_size_y = 1, local_size_z = 1) in; + +layout(binding = 0) buffer Ob32 { uint v[]; } O32; +layout(binding = 1) buffer Ob16 { uint16_t v[]; } O16; +layout(binding = 2) readonly buffer Xb32 { uint v[]; } X32; +layout(binding = 3) readonly buffer Xb16 { uint16_t v[]; } X16; +layout(binding = 4) readonly buffer Wb32 { uint v[]; } W32; +layout(binding = 5) readonly buffer Wb16 { uint16_t v[]; } W16; +// bias; aliases the weight when the caller passed none (p.has_bias == 0). +layout(binding = 6) readonly buffer Bb32 { uint v[]; } B32; +layout(binding = 7) readonly buffer Bb16 { uint16_t v[]; } B16; +// conv_state, read AND written in place; f32 or bf16 (p.st_dt). +layout(binding = 8) buffer Sb32 { uint v[]; } ST32; +layout(binding = 9) buffer Sb16 { uint16_t v[]; } ST16; +// query_start_loc, i32, n+1 entries. +layout(binding = 10) readonly buffer Qb { uint v[]; } QSL; +// has_initial_state, i32 or i8 per sequence (p.his_is_i8 picks the read). +layout(binding = 11) readonly buffer Hb { uint v[]; } HIS; + +layout(push_constant) uniform Params { + uint n; // conv_state.shape[0] — sequence count + uint c_dim; // x.shape[1] + uint k; // weight.shape[1] + uint width; // k - 1 + uint x_rs; // x.stride[0], in elements (x may be a padded-row view) + uint max_t_len; // x.shape[0]: no sequence can be longer than the input + uint has_bias, his_is_i8, silu; + uint out_dt, x_dt, w_dt, bias_dt, st_dt; + uint out_off, x_off, w_off, bias_off, st_off, qsl_off, his_off; +} p; + +// The private window, sized for the largest K this backend accepts. The host +// predicate refuses anything wider, so the loop below never runs past it. +const uint VT_CONV_MAX_WIDTH = 8u; + +// TOKEN BLOCKS PER (sequence, channel). 1 is the default mapping -- one +// invocation walks the whole sequence -- and the host only asks for more when +// VT_VULKAN_CONV_TARGET_GROUPS is set (vulkan_ops.cpp CausalConv1dFwdKernel). +// +// WHY SPLITTING IS CORRECT: this convolution carries NO state along t. Output t +// reads x[t-K+1 .. t] and, only while t < K-1, the pre-roll window. Block 0 owns +// every t < K-1 AND the state write-back, and it is the ONLY block that reads or +// writes conv_state at all; every other block reads x and writes its own output +// rows. No barrier, no cross-workgroup dependency, unchanged arithmetic, so the +// output does not depend on the block count. +// +// The split only changes the grid shape. Whether that is worth anything is a +// property of the device, so the default is 1 and the host exposes the target +// as a knob rather than choosing one. +layout(constant_id = 0) const uint VT_CONV_BLOCKS = 1u; + +void main() { + uint gid = gl_GlobalInvocationID.x; + if (gid >= p.n * p.c_dim * VT_CONV_BLOCKS) { return; } + uint blk = gid % VT_CONV_BLOCKS; + uint sc = gid / VT_CONV_BLOCKS; + uint s = sc / p.c_dim; + uint c = sc % p.c_dim; + + // begin/end come from query_start_loc. + // + // ⚠️ t_len IS A LOOP BOUND AND THE SUBTRACTION IS UNSIGNED. If end < begin -- + // a non-monotonic table, a wrong buffer offset, anything -- `end - begin` + // wraps to nearly 2^32 and the output loop below becomes effectively + // infinite. On a GPU that is not a crash: the dispatch never finishes, the + // fence never signals, and it presents as a HANG with no error -- the same + // failure mode documented for an out-of-bounds coopMatLoad. + // + // The native host path does not read query_start_loc (it is device memory), + // so this guard is what bounds memory: end <= max_t_len with begin <= end + // keeps begin + t inside x and out for every t < t_len. It is NOT validation. + // A malformed table whose intervals overlap still writes overlapping rows, + // and its output is unspecified; the CPU reference rejects such a table. + uint begin = QSL.v[(p.qsl_off >> 2) + s]; + uint end = QSL.v[(p.qsl_off >> 2) + s + 1u]; + if (end < begin || end > p.max_t_len) { return; } + uint t_len = end - begin; + + // has_initial_state is i32 or i8 per sequence. The i8 case is read through the + // u32 view and shifted, since this backend binds byte data through the 32-bit + // window like every other dtype-erased operand. his_off is a BYTE offset and + // an i8 view may start at any byte, so the shift comes from the full byte + // address, not from s alone. The word holding the last flag is always inside + // the buffer: vulkan_context.cpp AllocBuffer rounds every buffer length up to + // a whole 32-bit word, and a view never runs past its allocation. + bool init; + if (p.his_is_i8 != 0u) { + uint addr = p.his_off + s; + uint word = HIS.v[addr >> 2]; + init = ((word >> ((addr & 3u) * 8u)) & 0xFFu) != 0u; + } else { + init = HIS.v[(p.his_off >> 2) + s] != 0u; + } + + uint srow = (s * p.c_dim + c) * p.width; + + // COPY THE OLD STATE FIRST. This is the whole reason the kernel is safe + // without a barrier: the output loop below reads the pre-roll window while the + // final loop overwrites it, and the CPU reference resolves that same conflict + // the same way (`old_row`). ONLY BLOCK 0 COPIES IT: no other block computes a + // t < width, so none needs the window, and a load there would be an + // unordered read of the row block 0 is about to overwrite. + float old_row[VT_CONV_MAX_WIDTH]; + if (blk == 0u) { + for (uint j = 0u; j < p.width; ++j) { + old_row[j] = VT_LOAD(ST32, ST16, p.st_dt, p.st_off, srow + j); + } + } + + float b = p.has_bias != 0u ? VT_LOAD(B32, B16, p.bias_dt, p.bias_off, c) : 0.0; + + // This block's slice of the sequence. tb is derived from t_len rather than + // passed in, so a short sequence simply leaves the later blocks empty instead + // of the host having to know each length. + // Ceiling division without forming t_len + VT_CONV_BLOCKS - 1, which could + // wrap. (The host also bounds t_len by the i32 query_start_loc contract and + // VT_CONV_BLOCKS by 64, so blk * tb + tb stays far below 2^32.) + const uint tb = t_len / VT_CONV_BLOCKS + uint(t_len % VT_CONV_BLOCKS != 0u); + // BLOCK 0 TAKES EVERY t THAT READS THE CARRIED STATE, which is t < width. + // + // It has to, because block 0 is ALSO the block that overwrites that state at + // the end of this same dispatch, and there is no ordering between + // invocations, in or across workgroups (the blocks of one (sequence, channel) + // are adjacent invocations, and the shader has no barrier). + // The plain `blk * tb` split let a t < width land on blk > 0, which then read + // the state row in a race against block 0's write-back: whichever workgroup + // the driver happened to schedule first decided the answer. + // + // Observed with an earlier split, VT_CONV_BLOCKS = 5 on a varlen batch of + // lengths 5, 1, 9 (K 4, width 3), on llvmpipe: the wrong elements were + // exactly the (sequence, t) pairs with t < width that fell outside block 0 -- + // seq0 t=1,2 and seq2 t=2 -- never t=0, and never a t inside block 0. + // tests/vt/test_vulkan_backend.cpp runs the split on that shape: with the + // pre-fix behaviour restored it failed 5 runs of 5 on llvmpipe, and passed on + // the one discrete NVIDIA GPU it was tried on (most likely because the blocks + // of one (sequence, channel) are adjacent invocations that run together + // there; not separately demonstrated). A green run on a GPU therefore says + // nothing about this rule. + // + // The cost is that block 0 may carry up to `width` tokens instead of `tb`. + // width is K-1 and K is small, and the blocks it displaces simply come out + // empty (t_lo >= t_hi), so the grid is unchanged. + const uint t_lo = (blk == 0u) ? 0u : max(blk * tb, p.width); + const uint t_hi = (blk == 0u) ? min(max(tb, p.width), t_len) + : min(blk * tb + tb, t_len); + + for (uint t = t_lo; t < t_hi; ++t) { + float acc = b; + for (uint j = 0u; j < p.k; ++j) { + // ti is the token index of window slot j, and it goes NEGATIVE into the + // carried state, so the arithmetic is signed even though every index that + // reaches a buffer is not. + int ti = int(t) - (int(p.k) - 1 - int(j)); + float v = 0.0; + if (ti >= 0) { + v = VT_LOAD(X32, X16, p.x_dt, p.x_off, (begin + uint(ti)) * p.x_rs + c); + } else if (init) { + v = old_row[uint(int(p.width) + ti)]; // state col (K-1)+(t-i) + } + acc += VT_LOAD(W32, W16, p.w_dt, p.w_off, c * p.k + j) * v; + } + // Silu spelled as the CPU reference spells it (cpu_ops.cpp); + // `acc * sigmoid(acc)` is a different rounding. + VT_STORE(O32, O16, p.out_dt, p.out_off, (begin + t) * p.c_dim + c, + p.silu != 0u ? (acc / (1.0 + exp(-acc))) : acc); + } + + // ADVANCE THE STATE, once per (sequence, channel). Block 0 owns it: it always + // exists even when t_len is shorter than the block count, and everything it + // reads is either x (read-only) or its own copy of the old window. + if (blk != 0u) { return; } + + // new column j holds token (t_len - width + j), falling back into the shifted + // old window when the sequence is shorter than K-1. + for (uint j = 0u; j < p.width; ++j) { + int tj = int(t_len) - int(p.width) + int(j); + float v = 0.0; + if (tj >= 0) { + v = VT_LOAD(X32, X16, p.x_dt, p.x_off, (begin + uint(tj)) * p.x_rs + c); + } else if (init) { + v = old_row[uint(int(p.width) + tj)]; + } + VT_STORE(ST32, ST16, p.st_dt, p.st_off, srow + j, v); + } +} diff --git a/src/vt/vulkan/vulkan_ops.cpp b/src/vt/vulkan/vulkan_ops.cpp index b33d1bec8a..02eaac7f08 100644 --- a/src/vt/vulkan/vulkan_ops.cpp +++ b/src/vt/vulkan/vulkan_ops.cpp @@ -345,7 +345,8 @@ static_assert(sizeof(PagedAttnParams) <= 128, "push constants must fit the guaranteed 128 bytes"); static_assert(sizeof(ConvUpdateParams) <= 128, "push constants must fit the guaranteed 128 bytes"); -// The widest block in the backend at 84 bytes: the fused post-conv carries ten +// The widest block in the backend at 84 bytes (tied with ConvFwdParams): the +// fused post-conv carries ten // operand offsets. If it ever needs an eleventh, the step list has to move to the // scratch buffer the way vt_fused_chain's does. static_assert(sizeof(GdnPostConvParams) <= 128, @@ -1668,12 +1669,11 @@ void QkvSplitKernel(Queue&, Tensor& q_out, Tensor& k_out, Tensor& v_out, const T // __init__). Leaving it on the host MIRRORS upstream; "implementing" it would // be a regression, and the assertion in tests/vt/test_vulkan_backend.cpp says // so out loud. -// * kCausalConv1dFwd — the PREFILL conv. It is the same arithmetic as the -// update below but its state write-back reads the OLD state row while other -// tokens of the same sequence are still reading it, so it needs either a -// per-(sequence, channel) serial invocation over the whole token range or a -// buffered old row; that is a different dispatch shape, not a wider push -// block, and it is left for a follow-up rather than guessed at here. +// * kCausalConv1dFwd — the PREFILL conv — is NATIVE now, in the first of the +// two shapes this comment used to name: one invocation per (sequence, +// channel), serial over the whole token range, with the old state row copied +// into a private array before the write-back. See CausalConv1dFwdKernel; an +// opt-in token-block split exists and its own hazard is documented there. // =========================================================================== // cpu_ops.cpp:2272-2279 SigmoidGateBf16Kernel. Flat, one invocation per element. @@ -1806,6 +1806,181 @@ void GdnStateScatterKernel(Queue&, Tensor& cache, const Tensor& working, Go("vt_gdn_state_scatter", bind, p, FlatGroupCount(g.rows * g.work_row)); } +struct ConvFwdParams { + uint32_t n, c_dim, k, width, x_rs; + uint32_t max_t_len; + uint32_t has_bias, his_is_i8, silu; + uint32_t out_dt, x_dt, w_dt, bias_dt, st_dt; + uint32_t out_off, x_off, w_off, bias_off, st_off, qsl_off, his_off; +}; +static_assert(sizeof(ConvFwdParams) <= 128, + "push constants must fit the guaranteed 128 bytes"); + +// cpu_ops.cpp CausalConv1dFwdKernel, the GDN PREFILL conv. +// +// DEFAULT MAPPING: one invocation per (sequence, channel), serial over that +// sequence's whole token range -- the CPU kernel's own ForRows unit, and the +// first of the two shapes the block comment above this family names for this +// op. The op computes every output from the OLD state window and then +// overwrites that window; one invocation owns both the reads and the +// write-back for its (sequence, channel), and it copies the old window into a +// private array first (the CPU kernel's `old_row`), so no other invocation can +// observe a half-rolled state. +// +// Deliberately NOT llama.cpp's ssm_conv.comp shape by default, which also +// parallelises over tokens: that works there because ggml advances the conv +// state as a separate node, and here the same dispatch reads and advances it. +// +// OPTIONAL TOKEN BLOCKS, VT_VULKAN_CONV_TARGET_GROUPS=: split each sequence +// into VT_CONV_BLOCKS token blocks in the same dispatch. Only tokens t < width +// read the carried state, and the shader gives every one of them to block 0 -- +// the block that also writes the state back -- so block 0 is the only one that +// touches conv_state at all. An earlier split that did not do this raced: see +// the BLOCK 0 comment in vt_causal_conv1d_fwd.comp. The split changes the grid +// shape only, never the output, so it is off unless asked for. +// +// STATE ROWS: the CPU reference addresses conv_state with stride `width` and +// the shader mirrors it. The op layer also admits rows WIDER than K-1 +// (speculative-decode taps, ops.cpp CheckConvCommon), where that stride and +// the row's physical stride disagree. This kernel declines those shapes +// rather than choose between them, so they keep the reference tier's answer. +void CausalConv1dFwdKernel(Queue& q, Tensor& out, const Tensor& x, const Tensor& w, + const Tensor* bias, Tensor& conv_state, const Tensor& qsl, + const Tensor& his, const CausalConv1dArgs& args) { + const int64_t c_dim = x.shape[1], k = w.shape[1], total = x.shape[0]; + const int64_t n = conv_state.shape[0]; + if (n == 0 || c_dim == 0) return; + auto decline = [&] { + auto next = reinterpret_cast( + GetOpFallback(OpId::kCausalConv1dFwd, DeviceType::kVULKAN, kNativeProviderName)); + next(q, out, x, w, bias, conv_state, qsl, his, args); + }; + // PER-CALL REFUSAL rather than a throw, the seam vt_paged_attn and the GDN + // recurrences use: a shape this shader cannot serve forwards to the portable + // reference tier, which is correct for every shape. Reasons to decline: + // * K = 1 (no carried state, so a zero-length state row to bind) or a + // kernel width past the shader's private window (VT_CONV_MAX_WIDTH). + // * conv_state rows wider than K-1 (see STATE ROWS above). + // * A storage dtype outside f32/f16/bf16 (conv_state: f32/bf16). + // * has_initial_state that is neither i8 nor i32. + // * More (sequence, channel) pairs than one dispatch can cover: the grid + // is 1-D and Dispatch refuses a count above the device's + // maxComputeWorkGroupCount[0]. + // * Any index the shader forms that would not fit its uint32 arithmetic + // (checked after binding, below, because it includes the offsets). + const auto is_float = [](DType d) { + return d == DType::kF32 || d == DType::kF16 || d == DType::kBF16; + }; + const int64_t max_invocations = + static_cast(VulkanContext::Get().max_workgroup_count_x()) * kWorkgroupSize; + const bool serve = n * c_dim <= max_invocations && k >= 2 && k - 1 <= 8 && + conv_state.shape[2] == k - 1 && is_float(out.dtype) && + is_float(x.dtype) && is_float(w.dtype) && + (bias == nullptr || is_float(bias->dtype)) && + (conv_state.dtype == DType::kF32 || conv_state.dtype == DType::kBF16) && + (his.dtype == DType::kI8 || his.dtype == DType::kI32); + if (!serve) { + decline(); + return; + } + // query_start_loc is NOT read on the host: it is device memory, and reading + // it here would need the pending batch drained first. The shader guards + // memory, not semantics -- see the guard at the top of its main(). (A + // DECLINED call does reach host code: GetOpFallback drains the batch and the + // reference tier runs over host-visible device storage.) + + Binder bind; + const uint32_t out_off = bind.Add(out, "causal_conv1d_fwd: out"); + const uint32_t x_off = bind.Add(x, "causal_conv1d_fwd: x"); + const uint32_t w_off = bind.Add(w, "causal_conv1d_fwd: weight"); + const uint32_t bias_off = bias != nullptr ? bind.Add(*bias, "causal_conv1d_fwd: bias") + : bind.Add(w, "causal_conv1d_fwd: weight"); + const uint32_t st_off = bind.Add(conv_state, "causal_conv1d_fwd: conv_state"); + const uint32_t qsl_off = bind.AddU32Only(qsl, "causal_conv1d_fwd: query_start_loc"); + // has_initial_state may be i8, which is not 4-byte aligned, so it goes through + // the byte view and the shader unpacks it -- the same reasoning the update + // kernel gives for aliasing a bf16 state through AddByteView. + const uint32_t his_off = bind.AddByteView(his, "causal_conv1d_fwd: has_initial_state"); + + // F5 of the review: the shader forms every index in uint32. Decline when the + // largest one -- the binding's base element plus the farthest element the + // shader can address in it -- would not fit. + { + constexpr int64_t kU32 = 0xFFFFFFFFll; + const auto elem = [](DType d) { return d == DType::kF32 ? 4 : 2; }; + const auto fits = [&](uint32_t off_bytes, int64_t esize, int64_t extent) { + return static_cast(off_bytes) / esize + extent <= kU32; + }; + const int64_t x_extent = (total > 0 ? (total - 1) * x.stride[0] : 0) + c_dim; + const bool ok = + // query_start_loc is i32 and its last entry is `total`, so every + // sequence length the shader sees is below 2^31; asserting it here is + // what makes the shader's block arithmetic provably wrap-free. + total <= 0x7FFFFFFFll && x.stride[0] >= 0 && x.stride[0] <= kU32 && + n * c_dim * k <= kU32 && + fits(out_off, elem(out.dtype), total * c_dim) && + fits(x_off, elem(x.dtype), x_extent) && + fits(w_off, elem(w.dtype), c_dim * k) && + (bias == nullptr || fits(bias_off, elem(bias->dtype), c_dim)) && + fits(st_off, elem(conv_state.dtype), n * c_dim * (k - 1)) && + fits(qsl_off, 4, n + 1) && fits(his_off, 1, n + 4); + if (!ok) { + decline(); + return; + } + } + + ConvFwdParams p{static_cast(n), + static_cast(c_dim), + static_cast(k), + static_cast(k - 1), + static_cast(x.stride[0]), + static_cast(total), + bias != nullptr ? 1u : 0u, + his.dtype == DType::kI8 ? 1u : 0u, + args.silu_activation ? 1u : 0u, + DtypeCode(out.dtype), + DtypeCode(x.dtype), + DtypeCode(w.dtype), + bias != nullptr ? DtypeCode(bias->dtype) : DtypeCode(w.dtype), + DtypeCode(conv_state.dtype), + out_off, + x_off, + w_off, + bias_off, + st_off, + qsl_off, + his_off}; + // TOKEN BLOCKS: 1 unless VT_VULKAN_CONV_TARGET_GROUPS asks for a split. Read + // on every call rather than cached, so one test process can run both + // mappings -- the split path carries the block-0 rule and has to stay under + // test even though it is not the default. + int64_t blocks = 1; + if (const char* v = std::getenv("VT_VULKAN_CONV_TARGET_GROUPS")) { + const long g = std::strtol(v, nullptr, 10); + if (g >= 1 && g <= 65536) { + const int64_t base_groups = (n * c_dim + 127) / 128; + // Never more blocks than there are tokens to give them: past that the + // extra invocations are empty and only cost launch. + const int64_t avg_len = std::max(1, total / n); + blocks = (static_cast(g) + base_groups - 1) / base_groups; + blocks = std::max(1, std::min(blocks, avg_len)); + // ...and never more than one dispatch can launch. + blocks = std::min(blocks, max_invocations / (n * c_dim)); + // VT_CONV_BLOCKS is a specialization constant, so every distinct value is + // a separately compiled pipeline; the cap bounds what this knob can + // create to 64. A cap, NOT power-of-two rounding: with rounding, the + // pre-fix race stopped reproducing on llvmpipe (4 blocks of one + // (sequence, channel) sat inside one 8-lane subgroup in the test's + // layout), so it would have hidden the hazard from the test that guards it. + blocks = std::min(blocks, 64); + } + } + const uint32_t conv_spec[1] = {static_cast(blocks)}; + Go("vt_causal_conv1d_fwd", bind, p, + FlatGroupCount(n * c_dim * blocks), conv_spec, 1); +} + // cpu_ops.cpp:1081-1127 CausalConv1dUpdateKernel. One invocation per // (token, channel) — the CPU kernel's own row-chunking unit, and what makes the // read-old-then-roll safe with no barrier. @@ -2620,9 +2795,8 @@ struct Registrar { reinterpret_cast(static_cast(&RmsNormKernel))); RegisterOp(OpId::kFusedChain, DeviceType::kVULKAN, reinterpret_cast(static_cast(&FusedChainKernel))); - // BACKEND-VULKAN-GDN: the GDN glue family. kCausalConv1dFwd (the prefill - // conv) stays on the portable reference tier; see the block comment above - // these kernels. + // BACKEND-VULKAN-GDN: the GDN glue family, now including kCausalConv1dFwd + // (the prefill conv), registered after the update op below. RegisterOp(OpId::kSigmoidGateBf16, DeviceType::kVULKAN, reinterpret_cast(static_cast(&SigmoidGateBf16Kernel))); RegisterOp(OpId::kRmsNormGated, DeviceType::kVULKAN, @@ -2634,6 +2808,17 @@ struct Registrar { RegisterOp( OpId::kCausalConv1dUpdate, DeviceType::kVULKAN, reinterpret_cast(static_cast(&CausalConv1dUpdateKernel))); + // The PREFILL conv. VT_VULKAN_CONV_FWD=0 leaves it on the portable CPU + // reference tier, where it lived before this kernel existed, so its effect + // can be attributed with a same-binary A/B rather than across two builds -- + // the same shape as VT_VULKAN_COOPMAT. + { + const char* conv_env = std::getenv("VT_VULKAN_CONV_FWD"); + if (conv_env == nullptr || conv_env[0] != '0') { + RegisterOp(OpId::kCausalConv1dFwd, DeviceType::kVULKAN, + reinterpret_cast(static_cast(&CausalConv1dFwdKernel))); + } + } RegisterOp(OpId::kGdnPostConv, DeviceType::kVULKAN, reinterpret_cast(static_cast(&GdnPostConvKernel))); // BACKEND-VULKAN-GDN-CORE: the two recurrences. diff --git a/src/vt/vulkan/vulkan_spirv.cpp b/src/vt/vulkan/vulkan_spirv.cpp index b69772f52f..8a469cfcd5 100644 --- a/src/vt/vulkan/vulkan_spirv.cpp +++ b/src/vt/vulkan/vulkan_spirv.cpp @@ -18,7 +18,7 @@ // and extern declarations; vulkan_spirv.cpp carries the data, so adding shaders // costs one TU's compile time rather than all of them. // -// Produced by: Glslang Version: 11:16.4.0 +// Produced by: Glslang Version: 11:16.5.0 // Target environment: vulkan1.1 #include "vulkan_spirv.h" @@ -1356,6 +1356,667 @@ constexpr uint32_t kSpv_vt_cast[] = { 0x000200feu, 0x0000014au, 0x00010038u, }; +constexpr uint32_t kSpv_vt_causal_conv1d_fwd[] = { + 0x07230203u, 0x00010300u, 0x0008000bu, 0x000003e7u, 0x00000000u, 0x00020011u, 0x00000001u, 0x00020011u, + 0x00001151u, 0x0006000bu, 0x00000001u, 0x4c534c47u, 0x6474732eu, 0x3035342eu, 0x00000000u, 0x0003000eu, + 0x00000000u, 0x00000001u, 0x0006000fu, 0x00000005u, 0x00000004u, 0x6e69616du, 0x00000000u, 0x00000150u, + 0x00060010u, 0x00000004u, 0x00000011u, 0x00000080u, 0x00000001u, 0x00000001u, 0x00040047u, 0x00000150u, + 0x0000000bu, 0x0000001cu, 0x00030047u, 0x00000155u, 0x00000002u, 0x00050048u, 0x00000155u, 0x00000000u, + 0x00000023u, 0x00000000u, 0x00050048u, 0x00000155u, 0x00000001u, 0x00000023u, 0x00000004u, 0x00050048u, + 0x00000155u, 0x00000002u, 0x00000023u, 0x00000008u, 0x00050048u, 0x00000155u, 0x00000003u, 0x00000023u, + 0x0000000cu, 0x00050048u, 0x00000155u, 0x00000004u, 0x00000023u, 0x00000010u, 0x00050048u, 0x00000155u, + 0x00000005u, 0x00000023u, 0x00000014u, 0x00050048u, 0x00000155u, 0x00000006u, 0x00000023u, 0x00000018u, + 0x00050048u, 0x00000155u, 0x00000007u, 0x00000023u, 0x0000001cu, 0x00050048u, 0x00000155u, 0x00000008u, + 0x00000023u, 0x00000020u, 0x00050048u, 0x00000155u, 0x00000009u, 0x00000023u, 0x00000024u, 0x00050048u, + 0x00000155u, 0x0000000au, 0x00000023u, 0x00000028u, 0x00050048u, 0x00000155u, 0x0000000bu, 0x00000023u, + 0x0000002cu, 0x00050048u, 0x00000155u, 0x0000000cu, 0x00000023u, 0x00000030u, 0x00050048u, 0x00000155u, + 0x0000000du, 0x00000023u, 0x00000034u, 0x00050048u, 0x00000155u, 0x0000000eu, 0x00000023u, 0x00000038u, + 0x00050048u, 0x00000155u, 0x0000000fu, 0x00000023u, 0x0000003cu, 0x00050048u, 0x00000155u, 0x00000010u, + 0x00000023u, 0x00000040u, 0x00050048u, 0x00000155u, 0x00000011u, 0x00000023u, 0x00000044u, 0x00050048u, + 0x00000155u, 0x00000012u, 0x00000023u, 0x00000048u, 0x00050048u, 0x00000155u, 0x00000013u, 0x00000023u, + 0x0000004cu, 0x00050048u, 0x00000155u, 0x00000014u, 0x00000023u, 0x00000050u, 0x00040047u, 0x0000015eu, + 0x00000001u, 0x00000000u, 0x00040047u, 0x00000175u, 0x00000006u, 0x00000004u, 0x00030047u, 0x00000176u, + 0x00000002u, 0x00040048u, 0x00000176u, 0x00000000u, 0x00000018u, 0x00050048u, 0x00000176u, 0x00000000u, + 0x00000023u, 0x00000000u, 0x00030047u, 0x00000178u, 0x00000018u, 0x00040047u, 0x00000178u, 0x00000021u, + 0x0000000au, 0x00040047u, 0x00000178u, 0x00000022u, 0x00000000u, 0x00040047u, 0x000001acu, 0x00000006u, + 0x00000004u, 0x00030047u, 0x000001adu, 0x00000002u, 0x00040048u, 0x000001adu, 0x00000000u, 0x00000018u, + 0x00050048u, 0x000001adu, 0x00000000u, 0x00000023u, 0x00000000u, 0x00030047u, 0x000001afu, 0x00000018u, + 0x00040047u, 0x000001afu, 0x00000021u, 0x0000000bu, 0x00040047u, 0x000001afu, 0x00000022u, 0x00000000u, + 0x00040047u, 0x000001ebu, 0x00000006u, 0x00000004u, 0x00030047u, 0x000001ecu, 0x00000002u, 0x00050048u, + 0x000001ecu, 0x00000000u, 0x00000023u, 0x00000000u, 0x00040047u, 0x000001eeu, 0x00000021u, 0x00000008u, + 0x00040047u, 0x000001eeu, 0x00000022u, 0x00000000u, 0x00040047u, 0x000001fcu, 0x00000006u, 0x00000002u, + 0x00030047u, 0x000001fdu, 0x00000002u, 0x00050048u, 0x000001fdu, 0x00000000u, 0x00000023u, 0x00000000u, + 0x00040047u, 0x000001ffu, 0x00000021u, 0x00000009u, 0x00040047u, 0x000001ffu, 0x00000022u, 0x00000000u, + 0x00040047u, 0x00000223u, 0x00000006u, 0x00000004u, 0x00030047u, 0x00000224u, 0x00000002u, 0x00040048u, + 0x00000224u, 0x00000000u, 0x00000018u, 0x00050048u, 0x00000224u, 0x00000000u, 0x00000023u, 0x00000000u, + 0x00030047u, 0x00000226u, 0x00000018u, 0x00040047u, 0x00000226u, 0x00000021u, 0x00000006u, 0x00040047u, + 0x00000226u, 0x00000022u, 0x00000000u, 0x00040047u, 0x00000231u, 0x00000006u, 0x00000002u, 0x00030047u, + 0x00000232u, 0x00000002u, 0x00040048u, 0x00000232u, 0x00000000u, 0x00000018u, 0x00050048u, 0x00000232u, + 0x00000000u, 0x00000023u, 0x00000000u, 0x00030047u, 0x00000234u, 0x00000018u, 0x00040047u, 0x00000234u, + 0x00000021u, 0x00000007u, 0x00040047u, 0x00000234u, 0x00000022u, 0x00000000u, 0x00040047u, 0x0000029du, + 0x00000006u, 0x00000004u, 0x00030047u, 0x0000029eu, 0x00000002u, 0x00040048u, 0x0000029eu, 0x00000000u, + 0x00000018u, 0x00050048u, 0x0000029eu, 0x00000000u, 0x00000023u, 0x00000000u, 0x00030047u, 0x000002a0u, + 0x00000018u, 0x00040047u, 0x000002a0u, 0x00000021u, 0x00000002u, 0x00040047u, 0x000002a0u, 0x00000022u, + 0x00000000u, 0x00040047u, 0x000002b3u, 0x00000006u, 0x00000002u, 0x00030047u, 0x000002b4u, 0x00000002u, + 0x00040048u, 0x000002b4u, 0x00000000u, 0x00000018u, 0x00050048u, 0x000002b4u, 0x00000000u, 0x00000023u, + 0x00000000u, 0x00030047u, 0x000002b6u, 0x00000018u, 0x00040047u, 0x000002b6u, 0x00000021u, 0x00000003u, + 0x00040047u, 0x000002b6u, 0x00000022u, 0x00000000u, 0x00040047u, 0x000002e0u, 0x00000006u, 0x00000004u, + 0x00030047u, 0x000002e1u, 0x00000002u, 0x00040048u, 0x000002e1u, 0x00000000u, 0x00000018u, 0x00050048u, + 0x000002e1u, 0x00000000u, 0x00000023u, 0x00000000u, 0x00030047u, 0x000002e3u, 0x00000018u, 0x00040047u, + 0x000002e3u, 0x00000021u, 0x00000004u, 0x00040047u, 0x000002e3u, 0x00000022u, 0x00000000u, 0x00040047u, + 0x000002f2u, 0x00000006u, 0x00000002u, 0x00030047u, 0x000002f3u, 0x00000002u, 0x00040048u, 0x000002f3u, + 0x00000000u, 0x00000018u, 0x00050048u, 0x000002f3u, 0x00000000u, 0x00000023u, 0x00000000u, 0x00030047u, + 0x000002f5u, 0x00000018u, 0x00040047u, 0x000002f5u, 0x00000021u, 0x00000005u, 0x00040047u, 0x000002f5u, + 0x00000022u, 0x00000000u, 0x00040047u, 0x00000319u, 0x00000006u, 0x00000004u, 0x00030047u, 0x0000031au, + 0x00000002u, 0x00050048u, 0x0000031au, 0x00000000u, 0x00000023u, 0x00000000u, 0x00040047u, 0x0000031cu, + 0x00000021u, 0x00000000u, 0x00040047u, 0x0000031cu, 0x00000022u, 0x00000000u, 0x00040047u, 0x0000033du, + 0x00000006u, 0x00000002u, 0x00030047u, 0x0000033eu, 0x00000002u, 0x00050048u, 0x0000033eu, 0x00000000u, + 0x00000023u, 0x00000000u, 0x00040047u, 0x00000340u, 0x00000021u, 0x00000001u, 0x00040047u, 0x00000340u, + 0x00000022u, 0x00000000u, 0x00040047u, 0x000003e6u, 0x0000000bu, 0x00000019u, 0x00020013u, 0x00000002u, + 0x00030021u, 0x00000003u, 0x00000002u, 0x00040015u, 0x00000006u, 0x00000020u, 0x00000000u, 0x00040020u, + 0x00000007u, 0x00000007u, 0x00000006u, 0x00030016u, 0x00000008u, 0x00000020u, 0x00040021u, 0x00000009u, + 0x00000008u, 0x00000007u, 0x00040020u, 0x0000000du, 0x00000007u, 0x00000008u, 0x00040021u, 0x0000000eu, + 0x00000006u, 0x0000000du, 0x00050021u, 0x00000018u, 0x00000008u, 0x00000007u, 0x00000007u, 0x00050021u, + 0x0000001du, 0x00000006u, 0x0000000du, 0x00000007u, 0x00040015u, 0x00000023u, 0x00000020u, 0x00000001u, + 0x0004002bu, 0x00000023u, 0x00000024u, 0x00000010u, 0x00020014u, 0x0000002cu, 0x0004002bu, 0x00000006u, + 0x0000002eu, 0x7f800000u, 0x0004002bu, 0x00000006u, 0x00000034u, 0x007fffffu, 0x0004002bu, 0x00000006u, + 0x00000036u, 0x00000000u, 0x0004002bu, 0x00000006u, 0x0000003du, 0x00000040u, 0x0004002bu, 0x00000006u, + 0x0000003fu, 0x0000ffffu, 0x0004002bu, 0x00000006u, 0x00000043u, 0x00007fffu, 0x0004002bu, 0x00000006u, + 0x00000046u, 0x00000001u, 0x0004002bu, 0x00000006u, 0x00000052u, 0x00008000u, 0x0004002bu, 0x00000023u, + 0x00000057u, 0x0000000au, 0x0004002bu, 0x00000006u, 0x00000059u, 0x0000001fu, 0x0004002bu, 0x00000006u, + 0x0000005du, 0x000003ffu, 0x0004002bu, 0x00000023u, 0x00000066u, 0x0000000du, 0x0004002bu, 0x00000006u, + 0x0000007du, 0x00000400u, 0x0004002bu, 0x00000023u, 0x00000080u, 0x00000001u, 0x0004002bu, 0x00000006u, + 0x00000088u, 0x00000071u, 0x0004002bu, 0x00000023u, 0x0000008bu, 0x00000017u, 0x0004002bu, 0x00000006u, + 0x00000095u, 0x00000070u, 0x0004002bu, 0x00000006u, 0x000000a9u, 0x000000ffu, 0x00040020u, 0x000000abu, + 0x00000007u, 0x00000023u, 0x0004002bu, 0x00000023u, 0x000000afu, 0x0000007fu, 0x0004002bu, 0x00000023u, + 0x000000b1u, 0x0000000fu, 0x0004002bu, 0x00000006u, 0x000000bbu, 0x00007c00u, 0x0004002bu, 0x00000006u, + 0x000000c2u, 0x00000200u, 0x0004002bu, 0x00000023u, 0x000000cbu, 0x0000001fu, 0x0004002bu, 0x00000023u, + 0x000000d3u, 0x00000000u, 0x0004002bu, 0x00000023u, 0x000000d8u, 0xfffffff6u, 0x0004002bu, 0x00000006u, + 0x000000deu, 0x00800000u, 0x0004002bu, 0x00000023u, 0x000000e2u, 0x0000000eu, 0x0004002bu, 0x00000006u, + 0x00000115u, 0x00001fffu, 0x0004002bu, 0x00000006u, 0x00000118u, 0x00001000u, 0x00040017u, 0x0000014eu, + 0x00000006u, 0x00000003u, 0x00040020u, 0x0000014fu, 0x00000001u, 0x0000014eu, 0x0004003bu, 0x0000014fu, + 0x00000150u, 0x00000001u, 0x00040020u, 0x00000151u, 0x00000001u, 0x00000006u, 0x0017001eu, 0x00000155u, + 0x00000006u, 0x00000006u, 0x00000006u, 0x00000006u, 0x00000006u, 0x00000006u, 0x00000006u, 0x00000006u, + 0x00000006u, 0x00000006u, 0x00000006u, 0x00000006u, 0x00000006u, 0x00000006u, 0x00000006u, 0x00000006u, + 0x00000006u, 0x00000006u, 0x00000006u, 0x00000006u, 0x00000006u, 0x00040020u, 0x00000156u, 0x00000009u, + 0x00000155u, 0x0004003bu, 0x00000156u, 0x00000157u, 0x00000009u, 0x00040020u, 0x00000158u, 0x00000009u, + 0x00000006u, 0x00040032u, 0x00000006u, 0x0000015eu, 0x00000001u, 0x0003001du, 0x00000175u, 0x00000006u, + 0x0003001eu, 0x00000176u, 0x00000175u, 0x00040020u, 0x00000177u, 0x0000000cu, 0x00000176u, 0x0004003bu, + 0x00000177u, 0x00000178u, 0x0000000cu, 0x0004002bu, 0x00000023u, 0x00000179u, 0x00000013u, 0x0004002bu, + 0x00000023u, 0x0000017cu, 0x00000002u, 0x00040020u, 0x00000180u, 0x0000000cu, 0x00000006u, 0x0004002bu, + 0x00000023u, 0x00000193u, 0x00000005u, 0x0004002bu, 0x00000023u, 0x0000019fu, 0x00000007u, 0x0004002bu, + 0x00000023u, 0x000001a6u, 0x00000014u, 0x0003001du, 0x000001acu, 0x00000006u, 0x0003001eu, 0x000001adu, + 0x000001acu, 0x00040020u, 0x000001aeu, 0x0000000cu, 0x000001adu, 0x0004003bu, 0x000001aeu, 0x000001afu, + 0x0000000cu, 0x00040020u, 0x000001b4u, 0x00000007u, 0x0000002cu, 0x0004002bu, 0x00000006u, 0x000001b8u, + 0x00000003u, 0x0004002bu, 0x00000006u, 0x000001bau, 0x00000008u, 0x0004002bu, 0x00000023u, 0x000001cfu, + 0x00000003u, 0x0004001cu, 0x000001e1u, 0x00000008u, 0x000001bau, 0x00040020u, 0x000001e2u, 0x00000007u, + 0x000001e1u, 0x0003001du, 0x000001ebu, 0x00000006u, 0x0003001eu, 0x000001ecu, 0x000001ebu, 0x00040020u, + 0x000001edu, 0x0000000cu, 0x000001ecu, 0x0004003bu, 0x000001edu, 0x000001eeu, 0x0000000cu, 0x0004002bu, + 0x00000023u, 0x000001efu, 0x00000012u, 0x00040015u, 0x000001fbu, 0x00000010u, 0x00000000u, 0x0003001du, + 0x000001fcu, 0x000001fbu, 0x0003001eu, 0x000001fdu, 0x000001fcu, 0x00040020u, 0x000001feu, 0x0000000cu, + 0x000001fdu, 0x0004003bu, 0x000001feu, 0x000001ffu, 0x0000000cu, 0x00040020u, 0x00000207u, 0x0000000cu, + 0x000001fbu, 0x0004002bu, 0x00000023u, 0x00000215u, 0x00000006u, 0x0004002bu, 0x00000023u, 0x0000021cu, + 0x0000000cu, 0x0003001du, 0x00000223u, 0x00000006u, 0x0003001eu, 0x00000224u, 0x00000223u, 0x00040020u, + 0x00000225u, 0x0000000cu, 0x00000224u, 0x0004003bu, 0x00000225u, 0x00000226u, 0x0000000cu, 0x0004002bu, + 0x00000023u, 0x00000227u, 0x00000011u, 0x0003001du, 0x00000231u, 0x000001fbu, 0x0003001eu, 0x00000232u, + 0x00000231u, 0x00040020u, 0x00000233u, 0x0000000cu, 0x00000232u, 0x0004003bu, 0x00000233u, 0x00000234u, + 0x0000000cu, 0x0004002bu, 0x00000008u, 0x00000244u, 0x00000000u, 0x0003001du, 0x0000029du, 0x00000006u, + 0x0003001eu, 0x0000029eu, 0x0000029du, 0x00040020u, 0x0000029fu, 0x0000000cu, 0x0000029eu, 0x0004003bu, + 0x0000029fu, 0x000002a0u, 0x0000000cu, 0x0004002bu, 0x00000023u, 0x000002a8u, 0x00000004u, 0x0003001du, + 0x000002b3u, 0x000001fbu, 0x0003001eu, 0x000002b4u, 0x000002b3u, 0x00040020u, 0x000002b5u, 0x0000000cu, + 0x000002b4u, 0x0004003bu, 0x000002b5u, 0x000002b6u, 0x0000000cu, 0x0004002bu, 0x00000023u, 0x000002d9u, + 0x0000000bu, 0x0003001du, 0x000002e0u, 0x00000006u, 0x0003001eu, 0x000002e1u, 0x000002e0u, 0x00040020u, + 0x000002e2u, 0x0000000cu, 0x000002e1u, 0x0004003bu, 0x000002e2u, 0x000002e3u, 0x0000000cu, 0x0003001du, + 0x000002f2u, 0x000001fbu, 0x0003001eu, 0x000002f3u, 0x000002f2u, 0x00040020u, 0x000002f4u, 0x0000000cu, + 0x000002f3u, 0x0004003bu, 0x000002f4u, 0x000002f5u, 0x0000000cu, 0x0004002bu, 0x00000023u, 0x00000313u, + 0x00000009u, 0x0003001du, 0x00000319u, 0x00000006u, 0x0003001eu, 0x0000031au, 0x00000319u, 0x00040020u, + 0x0000031bu, 0x0000000cu, 0x0000031au, 0x0004003bu, 0x0000031bu, 0x0000031cu, 0x0000000cu, 0x0004002bu, + 0x00000023u, 0x00000329u, 0x00000008u, 0x0004002bu, 0x00000008u, 0x00000331u, 0x3f800000u, 0x0003001du, + 0x0000033du, 0x000001fbu, 0x0003001eu, 0x0000033eu, 0x0000033du, 0x00040020u, 0x0000033fu, 0x0000000cu, + 0x0000033eu, 0x0004003bu, 0x0000033fu, 0x00000340u, 0x0000000cu, 0x0003002au, 0x0000002cu, 0x00000363u, + 0x0004002bu, 0x00000006u, 0x000003e2u, 0x00000080u, 0x0004001cu, 0x000003e3u, 0x00000008u, 0x000003e2u, + 0x00040020u, 0x000003e4u, 0x00000004u, 0x000003e3u, 0x0004003bu, 0x000003e4u, 0x000003e5u, 0x00000004u, + 0x0006002cu, 0x0000014eu, 0x000003e6u, 0x000003e2u, 0x00000046u, 0x00000046u, 0x00050036u, 0x00000002u, + 0x00000004u, 0x00000000u, 0x00000003u, 0x000200f8u, 0x00000005u, 0x0004003bu, 0x00000007u, 0x0000014du, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000164u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000167u, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000016au, 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000016fu, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000174u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000183u, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000019bu, 0x00000007u, 0x0004003bu, 0x00000007u, 0x000001a5u, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x000001abu, 0x00000007u, 0x0004003bu, 0x000001b4u, 0x000001b5u, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x000001c8u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x000001d7u, + 0x00000007u, 0x0004003bu, 0x000001e2u, 0x000001e3u, 0x00000007u, 0x0004003bu, 0x0000000du, 0x000001e8u, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000020bu, 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000020cu, + 0x00000007u, 0x0004003bu, 0x0000000du, 0x00000214u, 0x00000007u, 0x0004003bu, 0x0000000du, 0x00000219u, + 0x00000007u, 0x0004003bu, 0x0000000du, 0x00000220u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000023du, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000023eu, 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000246u, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000024eu, 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000251u, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000025cu, 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000025fu, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000271u, 0x00000007u, 0x0004003bu, 0x0000000du, 0x0000027bu, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000027du, 0x00000007u, 0x0004003bu, 0x000000abu, 0x00000287u, + 0x00000007u, 0x0004003bu, 0x0000000du, 0x00000292u, 0x00000007u, 0x0004003bu, 0x0000000du, 0x0000029au, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x000002c7u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x000002c8u, + 0x00000007u, 0x0004003bu, 0x0000000du, 0x000002ddu, 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000303u, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000304u, 0x00000007u, 0x0004003bu, 0x0000000du, 0x0000032du, + 0x00000007u, 0x0004003bu, 0x0000000du, 0x00000350u, 0x00000007u, 0x0004003bu, 0x0000000du, 0x0000035bu, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000035du, 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000036bu, + 0x00000007u, 0x0004003bu, 0x000000abu, 0x00000375u, 0x00000007u, 0x0004003bu, 0x0000000du, 0x0000037fu, + 0x00000007u, 0x0004003bu, 0x0000000du, 0x00000387u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x000003abu, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x000003acu, 0x00000007u, 0x0004003bu, 0x0000000du, 0x000003d8u, + 0x00000007u, 0x0004003bu, 0x00000007u, 0x000003dau, 0x00000007u, 0x00050041u, 0x00000151u, 0x00000152u, + 0x00000150u, 0x00000036u, 0x0004003du, 0x00000006u, 0x00000153u, 0x00000152u, 0x0003003eu, 0x0000014du, + 0x00000153u, 0x0004003du, 0x00000006u, 0x00000154u, 0x0000014du, 0x00050041u, 0x00000158u, 0x00000159u, + 0x00000157u, 0x000000d3u, 0x0004003du, 0x00000006u, 0x0000015au, 0x00000159u, 0x00050041u, 0x00000158u, + 0x0000015bu, 0x00000157u, 0x00000080u, 0x0004003du, 0x00000006u, 0x0000015cu, 0x0000015bu, 0x00050084u, + 0x00000006u, 0x0000015du, 0x0000015au, 0x0000015cu, 0x00050084u, 0x00000006u, 0x0000015fu, 0x0000015du, + 0x0000015eu, 0x000500aeu, 0x0000002cu, 0x00000160u, 0x00000154u, 0x0000015fu, 0x000300f7u, 0x00000162u, + 0x00000000u, 0x000400fau, 0x00000160u, 0x00000161u, 0x00000162u, 0x000200f8u, 0x00000161u, 0x000100fdu, + 0x000200f8u, 0x00000162u, 0x0004003du, 0x00000006u, 0x00000165u, 0x0000014du, 0x00050089u, 0x00000006u, + 0x00000166u, 0x00000165u, 0x0000015eu, 0x0003003eu, 0x00000164u, 0x00000166u, 0x0004003du, 0x00000006u, + 0x00000168u, 0x0000014du, 0x00050086u, 0x00000006u, 0x00000169u, 0x00000168u, 0x0000015eu, 0x0003003eu, + 0x00000167u, 0x00000169u, 0x0004003du, 0x00000006u, 0x0000016bu, 0x00000167u, 0x00050041u, 0x00000158u, + 0x0000016cu, 0x00000157u, 0x00000080u, 0x0004003du, 0x00000006u, 0x0000016du, 0x0000016cu, 0x00050086u, + 0x00000006u, 0x0000016eu, 0x0000016bu, 0x0000016du, 0x0003003eu, 0x0000016au, 0x0000016eu, 0x0004003du, + 0x00000006u, 0x00000170u, 0x00000167u, 0x00050041u, 0x00000158u, 0x00000171u, 0x00000157u, 0x00000080u, + 0x0004003du, 0x00000006u, 0x00000172u, 0x00000171u, 0x00050089u, 0x00000006u, 0x00000173u, 0x00000170u, + 0x00000172u, 0x0003003eu, 0x0000016fu, 0x00000173u, 0x00050041u, 0x00000158u, 0x0000017au, 0x00000157u, + 0x00000179u, 0x0004003du, 0x00000006u, 0x0000017bu, 0x0000017au, 0x000500c2u, 0x00000006u, 0x0000017du, + 0x0000017bu, 0x0000017cu, 0x0004003du, 0x00000006u, 0x0000017eu, 0x0000016au, 0x00050080u, 0x00000006u, + 0x0000017fu, 0x0000017du, 0x0000017eu, 0x00060041u, 0x00000180u, 0x00000181u, 0x00000178u, 0x000000d3u, + 0x0000017fu, 0x0004003du, 0x00000006u, 0x00000182u, 0x00000181u, 0x0003003eu, 0x00000174u, 0x00000182u, + 0x00050041u, 0x00000158u, 0x00000184u, 0x00000157u, 0x00000179u, 0x0004003du, 0x00000006u, 0x00000185u, + 0x00000184u, 0x000500c2u, 0x00000006u, 0x00000186u, 0x00000185u, 0x0000017cu, 0x0004003du, 0x00000006u, + 0x00000187u, 0x0000016au, 0x00050080u, 0x00000006u, 0x00000188u, 0x00000186u, 0x00000187u, 0x00050080u, + 0x00000006u, 0x00000189u, 0x00000188u, 0x00000046u, 0x00060041u, 0x00000180u, 0x0000018au, 0x00000178u, + 0x000000d3u, 0x00000189u, 0x0004003du, 0x00000006u, 0x0000018bu, 0x0000018au, 0x0003003eu, 0x00000183u, + 0x0000018bu, 0x0004003du, 0x00000006u, 0x0000018cu, 0x00000183u, 0x0004003du, 0x00000006u, 0x0000018du, + 0x00000174u, 0x000500b0u, 0x0000002cu, 0x0000018eu, 0x0000018cu, 0x0000018du, 0x000400a8u, 0x0000002cu, + 0x0000018fu, 0x0000018eu, 0x000300f7u, 0x00000191u, 0x00000000u, 0x000400fau, 0x0000018fu, 0x00000190u, + 0x00000191u, 0x000200f8u, 0x00000190u, 0x0004003du, 0x00000006u, 0x00000192u, 0x00000183u, 0x00050041u, + 0x00000158u, 0x00000194u, 0x00000157u, 0x00000193u, 0x0004003du, 0x00000006u, 0x00000195u, 0x00000194u, + 0x000500acu, 0x0000002cu, 0x00000196u, 0x00000192u, 0x00000195u, 0x000200f9u, 0x00000191u, 0x000200f8u, + 0x00000191u, 0x000700f5u, 0x0000002cu, 0x00000197u, 0x0000018eu, 0x00000162u, 0x00000196u, 0x00000190u, + 0x000300f7u, 0x00000199u, 0x00000000u, 0x000400fau, 0x00000197u, 0x00000198u, 0x00000199u, 0x000200f8u, + 0x00000198u, 0x000100fdu, 0x000200f8u, 0x00000199u, 0x0004003du, 0x00000006u, 0x0000019cu, 0x00000183u, + 0x0004003du, 0x00000006u, 0x0000019du, 0x00000174u, 0x00050082u, 0x00000006u, 0x0000019eu, 0x0000019cu, + 0x0000019du, 0x0003003eu, 0x0000019bu, 0x0000019eu, 0x00050041u, 0x00000158u, 0x000001a0u, 0x00000157u, + 0x0000019fu, 0x0004003du, 0x00000006u, 0x000001a1u, 0x000001a0u, 0x000500abu, 0x0000002cu, 0x000001a2u, + 0x000001a1u, 0x00000036u, 0x000300f7u, 0x000001a4u, 0x00000000u, 0x000400fau, 0x000001a2u, 0x000001a3u, + 0x000001bfu, 0x000200f8u, 0x000001a3u, 0x00050041u, 0x00000158u, 0x000001a7u, 0x00000157u, 0x000001a6u, + 0x0004003du, 0x00000006u, 0x000001a8u, 0x000001a7u, 0x0004003du, 0x00000006u, 0x000001a9u, 0x0000016au, + 0x00050080u, 0x00000006u, 0x000001aau, 0x000001a8u, 0x000001a9u, 0x0003003eu, 0x000001a5u, 0x000001aau, + 0x0004003du, 0x00000006u, 0x000001b0u, 0x000001a5u, 0x000500c2u, 0x00000006u, 0x000001b1u, 0x000001b0u, + 0x0000017cu, 0x00060041u, 0x00000180u, 0x000001b2u, 0x000001afu, 0x000000d3u, 0x000001b1u, 0x0004003du, + 0x00000006u, 0x000001b3u, 0x000001b2u, 0x0003003eu, 0x000001abu, 0x000001b3u, 0x0004003du, 0x00000006u, + 0x000001b6u, 0x000001abu, 0x0004003du, 0x00000006u, 0x000001b7u, 0x000001a5u, 0x000500c7u, 0x00000006u, + 0x000001b9u, 0x000001b7u, 0x000001b8u, 0x00050084u, 0x00000006u, 0x000001bbu, 0x000001b9u, 0x000001bau, + 0x000500c2u, 0x00000006u, 0x000001bcu, 0x000001b6u, 0x000001bbu, 0x000500c7u, 0x00000006u, 0x000001bdu, + 0x000001bcu, 0x000000a9u, 0x000500abu, 0x0000002cu, 0x000001beu, 0x000001bdu, 0x00000036u, 0x0003003eu, + 0x000001b5u, 0x000001beu, 0x000200f9u, 0x000001a4u, 0x000200f8u, 0x000001bfu, 0x00050041u, 0x00000158u, + 0x000001c0u, 0x00000157u, 0x000001a6u, 0x0004003du, 0x00000006u, 0x000001c1u, 0x000001c0u, 0x000500c2u, + 0x00000006u, 0x000001c2u, 0x000001c1u, 0x0000017cu, 0x0004003du, 0x00000006u, 0x000001c3u, 0x0000016au, + 0x00050080u, 0x00000006u, 0x000001c4u, 0x000001c2u, 0x000001c3u, 0x00060041u, 0x00000180u, 0x000001c5u, + 0x000001afu, 0x000000d3u, 0x000001c4u, 0x0004003du, 0x00000006u, 0x000001c6u, 0x000001c5u, 0x000500abu, + 0x0000002cu, 0x000001c7u, 0x000001c6u, 0x00000036u, 0x0003003eu, 0x000001b5u, 0x000001c7u, 0x000200f9u, + 0x000001a4u, 0x000200f8u, 0x000001a4u, 0x0004003du, 0x00000006u, 0x000001c9u, 0x0000016au, 0x00050041u, + 0x00000158u, 0x000001cau, 0x00000157u, 0x00000080u, 0x0004003du, 0x00000006u, 0x000001cbu, 0x000001cau, + 0x00050084u, 0x00000006u, 0x000001ccu, 0x000001c9u, 0x000001cbu, 0x0004003du, 0x00000006u, 0x000001cdu, + 0x0000016fu, 0x00050080u, 0x00000006u, 0x000001ceu, 0x000001ccu, 0x000001cdu, 0x00050041u, 0x00000158u, + 0x000001d0u, 0x00000157u, 0x000001cfu, 0x0004003du, 0x00000006u, 0x000001d1u, 0x000001d0u, 0x00050084u, + 0x00000006u, 0x000001d2u, 0x000001ceu, 0x000001d1u, 0x0003003eu, 0x000001c8u, 0x000001d2u, 0x0004003du, + 0x00000006u, 0x000001d3u, 0x00000164u, 0x000500aau, 0x0000002cu, 0x000001d4u, 0x000001d3u, 0x00000036u, + 0x000300f7u, 0x000001d6u, 0x00000000u, 0x000400fau, 0x000001d4u, 0x000001d5u, 0x000001d6u, 0x000200f8u, + 0x000001d5u, 0x0003003eu, 0x000001d7u, 0x00000036u, 0x000200f9u, 0x000001d8u, 0x000200f8u, 0x000001d8u, + 0x000400f6u, 0x000001dau, 0x000001dbu, 0x00000000u, 0x000200f9u, 0x000001dcu, 0x000200f8u, 0x000001dcu, + 0x0004003du, 0x00000006u, 0x000001ddu, 0x000001d7u, 0x00050041u, 0x00000158u, 0x000001deu, 0x00000157u, + 0x000001cfu, 0x0004003du, 0x00000006u, 0x000001dfu, 0x000001deu, 0x000500b0u, 0x0000002cu, 0x000001e0u, + 0x000001ddu, 0x000001dfu, 0x000400fau, 0x000001e0u, 0x000001d9u, 0x000001dau, 0x000200f8u, 0x000001d9u, + 0x0004003du, 0x00000006u, 0x000001e4u, 0x000001d7u, 0x00050041u, 0x00000158u, 0x000001e5u, 0x00000157u, + 0x00000066u, 0x0004003du, 0x00000006u, 0x000001e6u, 0x000001e5u, 0x000500aau, 0x0000002cu, 0x000001e7u, + 0x000001e6u, 0x00000036u, 0x000300f7u, 0x000001eau, 0x00000000u, 0x000400fau, 0x000001e7u, 0x000001e9u, + 0x000001fau, 0x000200f8u, 0x000001e9u, 0x00050041u, 0x00000158u, 0x000001f0u, 0x00000157u, 0x000001efu, + 0x0004003du, 0x00000006u, 0x000001f1u, 0x000001f0u, 0x000500c2u, 0x00000006u, 0x000001f2u, 0x000001f1u, + 0x0000017cu, 0x0004003du, 0x00000006u, 0x000001f3u, 0x000001c8u, 0x0004003du, 0x00000006u, 0x000001f4u, + 0x000001d7u, 0x00050080u, 0x00000006u, 0x000001f5u, 0x000001f3u, 0x000001f4u, 0x00050080u, 0x00000006u, + 0x000001f6u, 0x000001f2u, 0x000001f5u, 0x00060041u, 0x00000180u, 0x000001f7u, 0x000001eeu, 0x000000d3u, + 0x000001f6u, 0x0004003du, 0x00000006u, 0x000001f8u, 0x000001f7u, 0x0004007cu, 0x00000008u, 0x000001f9u, + 0x000001f8u, 0x0003003eu, 0x000001e8u, 0x000001f9u, 0x000200f9u, 0x000001eau, 0x000200f8u, 0x000001fau, + 0x00050041u, 0x00000158u, 0x00000200u, 0x00000157u, 0x000001efu, 0x0004003du, 0x00000006u, 0x00000201u, + 0x00000200u, 0x000500c2u, 0x00000006u, 0x00000202u, 0x00000201u, 0x00000080u, 0x0004003du, 0x00000006u, + 0x00000203u, 0x000001c8u, 0x0004003du, 0x00000006u, 0x00000204u, 0x000001d7u, 0x00050080u, 0x00000006u, + 0x00000205u, 0x00000203u, 0x00000204u, 0x00050080u, 0x00000006u, 0x00000206u, 0x00000202u, 0x00000205u, + 0x00060041u, 0x00000207u, 0x00000208u, 0x000001ffu, 0x000000d3u, 0x00000206u, 0x0004003du, 0x000001fbu, + 0x00000209u, 0x00000208u, 0x00040071u, 0x00000006u, 0x0000020au, 0x00000209u, 0x0003003eu, 0x0000020bu, + 0x0000020au, 0x00050041u, 0x00000158u, 0x0000020du, 0x00000157u, 0x00000066u, 0x0004003du, 0x00000006u, + 0x0000020eu, 0x0000020du, 0x0003003eu, 0x0000020cu, 0x0000020eu, 0x00060039u, 0x00000008u, 0x0000020fu, + 0x0000001bu, 0x0000020bu, 0x0000020cu, 0x0003003eu, 0x000001e8u, 0x0000020fu, 0x000200f9u, 0x000001eau, + 0x000200f8u, 0x000001eau, 0x0004003du, 0x00000008u, 0x00000210u, 0x000001e8u, 0x00050041u, 0x0000000du, + 0x00000211u, 0x000001e3u, 0x000001e4u, 0x0003003eu, 0x00000211u, 0x00000210u, 0x000200f9u, 0x000001dbu, + 0x000200f8u, 0x000001dbu, 0x0004003du, 0x00000006u, 0x00000212u, 0x000001d7u, 0x00050080u, 0x00000006u, + 0x00000213u, 0x00000212u, 0x00000080u, 0x0003003eu, 0x000001d7u, 0x00000213u, 0x000200f9u, 0x000001d8u, + 0x000200f8u, 0x000001dau, 0x000200f9u, 0x000001d6u, 0x000200f8u, 0x000001d6u, 0x00050041u, 0x00000158u, + 0x00000216u, 0x00000157u, 0x00000215u, 0x0004003du, 0x00000006u, 0x00000217u, 0x00000216u, 0x000500abu, + 0x0000002cu, 0x00000218u, 0x00000217u, 0x00000036u, 0x000300f7u, 0x0000021bu, 0x00000000u, 0x000400fau, + 0x00000218u, 0x0000021au, 0x00000243u, 0x000200f8u, 0x0000021au, 0x00050041u, 0x00000158u, 0x0000021du, + 0x00000157u, 0x0000021cu, 0x0004003du, 0x00000006u, 0x0000021eu, 0x0000021du, 0x000500aau, 0x0000002cu, + 0x0000021fu, 0x0000021eu, 0x00000036u, 0x000300f7u, 0x00000222u, 0x00000000u, 0x000400fau, 0x0000021fu, + 0x00000221u, 0x00000230u, 0x000200f8u, 0x00000221u, 0x00050041u, 0x00000158u, 0x00000228u, 0x00000157u, + 0x00000227u, 0x0004003du, 0x00000006u, 0x00000229u, 0x00000228u, 0x000500c2u, 0x00000006u, 0x0000022au, + 0x00000229u, 0x0000017cu, 0x0004003du, 0x00000006u, 0x0000022bu, 0x0000016fu, 0x00050080u, 0x00000006u, + 0x0000022cu, 0x0000022au, 0x0000022bu, 0x00060041u, 0x00000180u, 0x0000022du, 0x00000226u, 0x000000d3u, + 0x0000022cu, 0x0004003du, 0x00000006u, 0x0000022eu, 0x0000022du, 0x0004007cu, 0x00000008u, 0x0000022fu, + 0x0000022eu, 0x0003003eu, 0x00000220u, 0x0000022fu, 0x000200f9u, 0x00000222u, 0x000200f8u, 0x00000230u, + 0x00050041u, 0x00000158u, 0x00000235u, 0x00000157u, 0x00000227u, 0x0004003du, 0x00000006u, 0x00000236u, + 0x00000235u, 0x000500c2u, 0x00000006u, 0x00000237u, 0x00000236u, 0x00000080u, 0x0004003du, 0x00000006u, + 0x00000238u, 0x0000016fu, 0x00050080u, 0x00000006u, 0x00000239u, 0x00000237u, 0x00000238u, 0x00060041u, + 0x00000207u, 0x0000023au, 0x00000234u, 0x000000d3u, 0x00000239u, 0x0004003du, 0x000001fbu, 0x0000023bu, + 0x0000023au, 0x00040071u, 0x00000006u, 0x0000023cu, 0x0000023bu, 0x0003003eu, 0x0000023du, 0x0000023cu, + 0x00050041u, 0x00000158u, 0x0000023fu, 0x00000157u, 0x0000021cu, 0x0004003du, 0x00000006u, 0x00000240u, + 0x0000023fu, 0x0003003eu, 0x0000023eu, 0x00000240u, 0x00060039u, 0x00000008u, 0x00000241u, 0x0000001bu, + 0x0000023du, 0x0000023eu, 0x0003003eu, 0x00000220u, 0x00000241u, 0x000200f9u, 0x00000222u, 0x000200f8u, + 0x00000222u, 0x0004003du, 0x00000008u, 0x00000242u, 0x00000220u, 0x0003003eu, 0x00000219u, 0x00000242u, + 0x000200f9u, 0x0000021bu, 0x000200f8u, 0x00000243u, 0x0003003eu, 0x00000219u, 0x00000244u, 0x000200f9u, + 0x0000021bu, 0x000200f8u, 0x0000021bu, 0x0004003du, 0x00000008u, 0x00000245u, 0x00000219u, 0x0003003eu, + 0x00000214u, 0x00000245u, 0x0004003du, 0x00000006u, 0x00000247u, 0x0000019bu, 0x00050086u, 0x00000006u, + 0x00000248u, 0x00000247u, 0x0000015eu, 0x0004003du, 0x00000006u, 0x00000249u, 0x0000019bu, 0x00050089u, + 0x00000006u, 0x0000024au, 0x00000249u, 0x0000015eu, 0x000500abu, 0x0000002cu, 0x0000024bu, 0x0000024au, + 0x00000036u, 0x000600a9u, 0x00000006u, 0x0000024cu, 0x0000024bu, 0x00000046u, 0x00000036u, 0x00050080u, + 0x00000006u, 0x0000024du, 0x00000248u, 0x0000024cu, 0x0003003eu, 0x00000246u, 0x0000024du, 0x0004003du, + 0x00000006u, 0x0000024fu, 0x00000164u, 0x000500aau, 0x0000002cu, 0x00000250u, 0x0000024fu, 0x00000036u, + 0x000300f7u, 0x00000253u, 0x00000000u, 0x000400fau, 0x00000250u, 0x00000252u, 0x00000254u, 0x000200f8u, + 0x00000252u, 0x0003003eu, 0x00000251u, 0x00000036u, 0x000200f9u, 0x00000253u, 0x000200f8u, 0x00000254u, + 0x0004003du, 0x00000006u, 0x00000255u, 0x00000164u, 0x0004003du, 0x00000006u, 0x00000256u, 0x00000246u, + 0x00050084u, 0x00000006u, 0x00000257u, 0x00000255u, 0x00000256u, 0x00050041u, 0x00000158u, 0x00000258u, + 0x00000157u, 0x000001cfu, 0x0004003du, 0x00000006u, 0x00000259u, 0x00000258u, 0x0007000cu, 0x00000006u, + 0x0000025au, 0x00000001u, 0x00000029u, 0x00000257u, 0x00000259u, 0x0003003eu, 0x00000251u, 0x0000025au, + 0x000200f9u, 0x00000253u, 0x000200f8u, 0x00000253u, 0x0004003du, 0x00000006u, 0x0000025bu, 0x00000251u, + 0x0003003eu, 0x0000024eu, 0x0000025bu, 0x0004003du, 0x00000006u, 0x0000025du, 0x00000164u, 0x000500aau, + 0x0000002cu, 0x0000025eu, 0x0000025du, 0x00000036u, 0x000300f7u, 0x00000261u, 0x00000000u, 0x000400fau, + 0x0000025eu, 0x00000260u, 0x00000268u, 0x000200f8u, 0x00000260u, 0x0004003du, 0x00000006u, 0x00000262u, + 0x00000246u, 0x00050041u, 0x00000158u, 0x00000263u, 0x00000157u, 0x000001cfu, 0x0004003du, 0x00000006u, + 0x00000264u, 0x00000263u, 0x0007000cu, 0x00000006u, 0x00000265u, 0x00000001u, 0x00000029u, 0x00000262u, + 0x00000264u, 0x0004003du, 0x00000006u, 0x00000266u, 0x0000019bu, 0x0007000cu, 0x00000006u, 0x00000267u, + 0x00000001u, 0x00000026u, 0x00000265u, 0x00000266u, 0x0003003eu, 0x0000025fu, 0x00000267u, 0x000200f9u, + 0x00000261u, 0x000200f8u, 0x00000268u, 0x0004003du, 0x00000006u, 0x00000269u, 0x00000164u, 0x0004003du, + 0x00000006u, 0x0000026au, 0x00000246u, 0x00050084u, 0x00000006u, 0x0000026bu, 0x00000269u, 0x0000026au, + 0x0004003du, 0x00000006u, 0x0000026cu, 0x00000246u, 0x00050080u, 0x00000006u, 0x0000026du, 0x0000026bu, + 0x0000026cu, 0x0004003du, 0x00000006u, 0x0000026eu, 0x0000019bu, 0x0007000cu, 0x00000006u, 0x0000026fu, + 0x00000001u, 0x00000026u, 0x0000026du, 0x0000026eu, 0x0003003eu, 0x0000025fu, 0x0000026fu, 0x000200f9u, + 0x00000261u, 0x000200f8u, 0x00000261u, 0x0004003du, 0x00000006u, 0x00000270u, 0x0000025fu, 0x0003003eu, + 0x0000025cu, 0x00000270u, 0x0004003du, 0x00000006u, 0x00000272u, 0x0000024eu, 0x0003003eu, 0x00000271u, + 0x00000272u, 0x000200f9u, 0x00000273u, 0x000200f8u, 0x00000273u, 0x000400f6u, 0x00000275u, 0x00000276u, + 0x00000000u, 0x000200f9u, 0x00000277u, 0x000200f8u, 0x00000277u, 0x0004003du, 0x00000006u, 0x00000278u, + 0x00000271u, 0x0004003du, 0x00000006u, 0x00000279u, 0x0000025cu, 0x000500b0u, 0x0000002cu, 0x0000027au, + 0x00000278u, 0x00000279u, 0x000400fau, 0x0000027au, 0x00000274u, 0x00000275u, 0x000200f8u, 0x00000274u, + 0x0004003du, 0x00000008u, 0x0000027cu, 0x00000214u, 0x0003003eu, 0x0000027bu, 0x0000027cu, 0x0003003eu, + 0x0000027du, 0x00000036u, 0x000200f9u, 0x0000027eu, 0x000200f8u, 0x0000027eu, 0x000400f6u, 0x00000280u, + 0x00000281u, 0x00000000u, 0x000200f9u, 0x00000282u, 0x000200f8u, 0x00000282u, 0x0004003du, 0x00000006u, + 0x00000283u, 0x0000027du, 0x00050041u, 0x00000158u, 0x00000284u, 0x00000157u, 0x0000017cu, 0x0004003du, + 0x00000006u, 0x00000285u, 0x00000284u, 0x000500b0u, 0x0000002cu, 0x00000286u, 0x00000283u, 0x00000285u, + 0x000400fau, 0x00000286u, 0x0000027fu, 0x00000280u, 0x000200f8u, 0x0000027fu, 0x0004003du, 0x00000006u, + 0x00000288u, 0x00000271u, 0x0004007cu, 0x00000023u, 0x00000289u, 0x00000288u, 0x00050041u, 0x00000158u, + 0x0000028au, 0x00000157u, 0x0000017cu, 0x0004003du, 0x00000006u, 0x0000028bu, 0x0000028au, 0x0004007cu, + 0x00000023u, 0x0000028cu, 0x0000028bu, 0x00050082u, 0x00000023u, 0x0000028du, 0x0000028cu, 0x00000080u, + 0x0004003du, 0x00000006u, 0x0000028eu, 0x0000027du, 0x0004007cu, 0x00000023u, 0x0000028fu, 0x0000028eu, + 0x00050082u, 0x00000023u, 0x00000290u, 0x0000028du, 0x0000028fu, 0x00050082u, 0x00000023u, 0x00000291u, + 0x00000289u, 0x00000290u, 0x0003003eu, 0x00000287u, 0x00000291u, 0x0003003eu, 0x00000292u, 0x00000244u, + 0x0004003du, 0x00000023u, 0x00000293u, 0x00000287u, 0x000500afu, 0x0000002cu, 0x00000294u, 0x00000293u, + 0x000000d3u, 0x000300f7u, 0x00000296u, 0x00000000u, 0x000400fau, 0x00000294u, 0x00000295u, 0x000002cdu, + 0x000200f8u, 0x00000295u, 0x00050041u, 0x00000158u, 0x00000297u, 0x00000157u, 0x00000057u, 0x0004003du, + 0x00000006u, 0x00000298u, 0x00000297u, 0x000500aau, 0x0000002cu, 0x00000299u, 0x00000298u, 0x00000036u, + 0x000300f7u, 0x0000029cu, 0x00000000u, 0x000400fau, 0x00000299u, 0x0000029bu, 0x000002b2u, 0x000200f8u, + 0x0000029bu, 0x00050041u, 0x00000158u, 0x000002a1u, 0x00000157u, 0x000000b1u, 0x0004003du, 0x00000006u, + 0x000002a2u, 0x000002a1u, 0x000500c2u, 0x00000006u, 0x000002a3u, 0x000002a2u, 0x0000017cu, 0x0004003du, + 0x00000006u, 0x000002a4u, 0x00000174u, 0x0004003du, 0x00000023u, 0x000002a5u, 0x00000287u, 0x0004007cu, + 0x00000006u, 0x000002a6u, 0x000002a5u, 0x00050080u, 0x00000006u, 0x000002a7u, 0x000002a4u, 0x000002a6u, + 0x00050041u, 0x00000158u, 0x000002a9u, 0x00000157u, 0x000002a8u, 0x0004003du, 0x00000006u, 0x000002aau, + 0x000002a9u, 0x00050084u, 0x00000006u, 0x000002abu, 0x000002a7u, 0x000002aau, 0x0004003du, 0x00000006u, + 0x000002acu, 0x0000016fu, 0x00050080u, 0x00000006u, 0x000002adu, 0x000002abu, 0x000002acu, 0x00050080u, + 0x00000006u, 0x000002aeu, 0x000002a3u, 0x000002adu, 0x00060041u, 0x00000180u, 0x000002afu, 0x000002a0u, + 0x000000d3u, 0x000002aeu, 0x0004003du, 0x00000006u, 0x000002b0u, 0x000002afu, 0x0004007cu, 0x00000008u, + 0x000002b1u, 0x000002b0u, 0x0003003eu, 0x0000029au, 0x000002b1u, 0x000200f9u, 0x0000029cu, 0x000200f8u, + 0x000002b2u, 0x00050041u, 0x00000158u, 0x000002b7u, 0x00000157u, 0x000000b1u, 0x0004003du, 0x00000006u, + 0x000002b8u, 0x000002b7u, 0x000500c2u, 0x00000006u, 0x000002b9u, 0x000002b8u, 0x00000080u, 0x0004003du, + 0x00000006u, 0x000002bau, 0x00000174u, 0x0004003du, 0x00000023u, 0x000002bbu, 0x00000287u, 0x0004007cu, + 0x00000006u, 0x000002bcu, 0x000002bbu, 0x00050080u, 0x00000006u, 0x000002bdu, 0x000002bau, 0x000002bcu, + 0x00050041u, 0x00000158u, 0x000002beu, 0x00000157u, 0x000002a8u, 0x0004003du, 0x00000006u, 0x000002bfu, + 0x000002beu, 0x00050084u, 0x00000006u, 0x000002c0u, 0x000002bdu, 0x000002bfu, 0x0004003du, 0x00000006u, + 0x000002c1u, 0x0000016fu, 0x00050080u, 0x00000006u, 0x000002c2u, 0x000002c0u, 0x000002c1u, 0x00050080u, + 0x00000006u, 0x000002c3u, 0x000002b9u, 0x000002c2u, 0x00060041u, 0x00000207u, 0x000002c4u, 0x000002b6u, + 0x000000d3u, 0x000002c3u, 0x0004003du, 0x000001fbu, 0x000002c5u, 0x000002c4u, 0x00040071u, 0x00000006u, + 0x000002c6u, 0x000002c5u, 0x0003003eu, 0x000002c7u, 0x000002c6u, 0x00050041u, 0x00000158u, 0x000002c9u, + 0x00000157u, 0x00000057u, 0x0004003du, 0x00000006u, 0x000002cau, 0x000002c9u, 0x0003003eu, 0x000002c8u, + 0x000002cau, 0x00060039u, 0x00000008u, 0x000002cbu, 0x0000001bu, 0x000002c7u, 0x000002c8u, 0x0003003eu, + 0x0000029au, 0x000002cbu, 0x000200f9u, 0x0000029cu, 0x000200f8u, 0x0000029cu, 0x0004003du, 0x00000008u, + 0x000002ccu, 0x0000029au, 0x0003003eu, 0x00000292u, 0x000002ccu, 0x000200f9u, 0x00000296u, 0x000200f8u, + 0x000002cdu, 0x0004003du, 0x0000002cu, 0x000002ceu, 0x000001b5u, 0x000300f7u, 0x000002d0u, 0x00000000u, + 0x000400fau, 0x000002ceu, 0x000002cfu, 0x000002d0u, 0x000200f8u, 0x000002cfu, 0x00050041u, 0x00000158u, + 0x000002d1u, 0x00000157u, 0x000001cfu, 0x0004003du, 0x00000006u, 0x000002d2u, 0x000002d1u, 0x0004007cu, + 0x00000023u, 0x000002d3u, 0x000002d2u, 0x0004003du, 0x00000023u, 0x000002d4u, 0x00000287u, 0x00050080u, + 0x00000023u, 0x000002d5u, 0x000002d3u, 0x000002d4u, 0x0004007cu, 0x00000006u, 0x000002d6u, 0x000002d5u, + 0x00050041u, 0x0000000du, 0x000002d7u, 0x000001e3u, 0x000002d6u, 0x0004003du, 0x00000008u, 0x000002d8u, + 0x000002d7u, 0x0003003eu, 0x00000292u, 0x000002d8u, 0x000200f9u, 0x000002d0u, 0x000200f8u, 0x000002d0u, + 0x000200f9u, 0x00000296u, 0x000200f8u, 0x00000296u, 0x00050041u, 0x00000158u, 0x000002dau, 0x00000157u, + 0x000002d9u, 0x0004003du, 0x00000006u, 0x000002dbu, 0x000002dau, 0x000500aau, 0x0000002cu, 0x000002dcu, + 0x000002dbu, 0x00000036u, 0x000300f7u, 0x000002dfu, 0x00000000u, 0x000400fau, 0x000002dcu, 0x000002deu, + 0x000002f1u, 0x000200f8u, 0x000002deu, 0x00050041u, 0x00000158u, 0x000002e4u, 0x00000157u, 0x00000024u, + 0x0004003du, 0x00000006u, 0x000002e5u, 0x000002e4u, 0x000500c2u, 0x00000006u, 0x000002e6u, 0x000002e5u, + 0x0000017cu, 0x0004003du, 0x00000006u, 0x000002e7u, 0x0000016fu, 0x00050041u, 0x00000158u, 0x000002e8u, + 0x00000157u, 0x0000017cu, 0x0004003du, 0x00000006u, 0x000002e9u, 0x000002e8u, 0x00050084u, 0x00000006u, + 0x000002eau, 0x000002e7u, 0x000002e9u, 0x0004003du, 0x00000006u, 0x000002ebu, 0x0000027du, 0x00050080u, + 0x00000006u, 0x000002ecu, 0x000002eau, 0x000002ebu, 0x00050080u, 0x00000006u, 0x000002edu, 0x000002e6u, + 0x000002ecu, 0x00060041u, 0x00000180u, 0x000002eeu, 0x000002e3u, 0x000000d3u, 0x000002edu, 0x0004003du, + 0x00000006u, 0x000002efu, 0x000002eeu, 0x0004007cu, 0x00000008u, 0x000002f0u, 0x000002efu, 0x0003003eu, + 0x000002ddu, 0x000002f0u, 0x000200f9u, 0x000002dfu, 0x000200f8u, 0x000002f1u, 0x00050041u, 0x00000158u, + 0x000002f6u, 0x00000157u, 0x00000024u, 0x0004003du, 0x00000006u, 0x000002f7u, 0x000002f6u, 0x000500c2u, + 0x00000006u, 0x000002f8u, 0x000002f7u, 0x00000080u, 0x0004003du, 0x00000006u, 0x000002f9u, 0x0000016fu, + 0x00050041u, 0x00000158u, 0x000002fau, 0x00000157u, 0x0000017cu, 0x0004003du, 0x00000006u, 0x000002fbu, + 0x000002fau, 0x00050084u, 0x00000006u, 0x000002fcu, 0x000002f9u, 0x000002fbu, 0x0004003du, 0x00000006u, + 0x000002fdu, 0x0000027du, 0x00050080u, 0x00000006u, 0x000002feu, 0x000002fcu, 0x000002fdu, 0x00050080u, + 0x00000006u, 0x000002ffu, 0x000002f8u, 0x000002feu, 0x00060041u, 0x00000207u, 0x00000300u, 0x000002f5u, + 0x000000d3u, 0x000002ffu, 0x0004003du, 0x000001fbu, 0x00000301u, 0x00000300u, 0x00040071u, 0x00000006u, + 0x00000302u, 0x00000301u, 0x0003003eu, 0x00000303u, 0x00000302u, 0x00050041u, 0x00000158u, 0x00000305u, + 0x00000157u, 0x000002d9u, 0x0004003du, 0x00000006u, 0x00000306u, 0x00000305u, 0x0003003eu, 0x00000304u, + 0x00000306u, 0x00060039u, 0x00000008u, 0x00000307u, 0x0000001bu, 0x00000303u, 0x00000304u, 0x0003003eu, + 0x000002ddu, 0x00000307u, 0x000200f9u, 0x000002dfu, 0x000200f8u, 0x000002dfu, 0x0004003du, 0x00000008u, + 0x00000308u, 0x000002ddu, 0x0004003du, 0x00000008u, 0x00000309u, 0x00000292u, 0x00050085u, 0x00000008u, + 0x0000030au, 0x00000308u, 0x00000309u, 0x0004003du, 0x00000008u, 0x0000030bu, 0x0000027bu, 0x00050081u, + 0x00000008u, 0x0000030cu, 0x0000030bu, 0x0000030au, 0x0003003eu, 0x0000027bu, 0x0000030cu, 0x000200f9u, + 0x00000281u, 0x000200f8u, 0x00000281u, 0x0004003du, 0x00000006u, 0x0000030du, 0x0000027du, 0x00050080u, + 0x00000006u, 0x0000030eu, 0x0000030du, 0x00000080u, 0x0003003eu, 0x0000027du, 0x0000030eu, 0x000200f9u, + 0x0000027eu, 0x000200f8u, 0x00000280u, 0x000200f9u, 0x0000030fu, 0x000200f8u, 0x0000030fu, 0x000400f6u, + 0x00000311u, 0x00000312u, 0x00000000u, 0x000200f9u, 0x00000310u, 0x000200f8u, 0x00000310u, 0x00050041u, + 0x00000158u, 0x00000314u, 0x00000157u, 0x00000313u, 0x0004003du, 0x00000006u, 0x00000315u, 0x00000314u, + 0x000500aau, 0x0000002cu, 0x00000316u, 0x00000315u, 0x00000036u, 0x000300f7u, 0x00000318u, 0x00000000u, + 0x000400fau, 0x00000316u, 0x00000317u, 0x0000033cu, 0x000200f8u, 0x00000317u, 0x00050041u, 0x00000158u, + 0x0000031du, 0x00000157u, 0x000000e2u, 0x0004003du, 0x00000006u, 0x0000031eu, 0x0000031du, 0x000500c2u, + 0x00000006u, 0x0000031fu, 0x0000031eu, 0x0000017cu, 0x0004003du, 0x00000006u, 0x00000320u, 0x00000174u, + 0x0004003du, 0x00000006u, 0x00000321u, 0x00000271u, 0x00050080u, 0x00000006u, 0x00000322u, 0x00000320u, + 0x00000321u, 0x00050041u, 0x00000158u, 0x00000323u, 0x00000157u, 0x00000080u, 0x0004003du, 0x00000006u, + 0x00000324u, 0x00000323u, 0x00050084u, 0x00000006u, 0x00000325u, 0x00000322u, 0x00000324u, 0x0004003du, + 0x00000006u, 0x00000326u, 0x0000016fu, 0x00050080u, 0x00000006u, 0x00000327u, 0x00000325u, 0x00000326u, + 0x00050080u, 0x00000006u, 0x00000328u, 0x0000031fu, 0x00000327u, 0x00050041u, 0x00000158u, 0x0000032au, + 0x00000157u, 0x00000329u, 0x0004003du, 0x00000006u, 0x0000032bu, 0x0000032au, 0x000500abu, 0x0000002cu, + 0x0000032cu, 0x0000032bu, 0x00000036u, 0x000300f7u, 0x0000032fu, 0x00000000u, 0x000400fau, 0x0000032cu, + 0x0000032eu, 0x00000337u, 0x000200f8u, 0x0000032eu, 0x0004003du, 0x00000008u, 0x00000330u, 0x0000027bu, + 0x0004003du, 0x00000008u, 0x00000332u, 0x0000027bu, 0x0004007fu, 0x00000008u, 0x00000333u, 0x00000332u, + 0x0006000cu, 0x00000008u, 0x00000334u, 0x00000001u, 0x0000001bu, 0x00000333u, 0x00050081u, 0x00000008u, + 0x00000335u, 0x00000331u, 0x00000334u, 0x00050088u, 0x00000008u, 0x00000336u, 0x00000330u, 0x00000335u, + 0x0003003eu, 0x0000032du, 0x00000336u, 0x000200f9u, 0x0000032fu, 0x000200f8u, 0x00000337u, 0x0004003du, + 0x00000008u, 0x00000338u, 0x0000027bu, 0x0003003eu, 0x0000032du, 0x00000338u, 0x000200f9u, 0x0000032fu, + 0x000200f8u, 0x0000032fu, 0x0004003du, 0x00000008u, 0x00000339u, 0x0000032du, 0x0004007cu, 0x00000006u, + 0x0000033au, 0x00000339u, 0x00060041u, 0x00000180u, 0x0000033bu, 0x0000031cu, 0x000000d3u, 0x00000328u, + 0x0003003eu, 0x0000033bu, 0x0000033au, 0x000200f9u, 0x00000318u, 0x000200f8u, 0x0000033cu, 0x00050041u, + 0x00000158u, 0x00000341u, 0x00000157u, 0x000000e2u, 0x0004003du, 0x00000006u, 0x00000342u, 0x00000341u, + 0x000500c2u, 0x00000006u, 0x00000343u, 0x00000342u, 0x00000080u, 0x0004003du, 0x00000006u, 0x00000344u, + 0x00000174u, 0x0004003du, 0x00000006u, 0x00000345u, 0x00000271u, 0x00050080u, 0x00000006u, 0x00000346u, + 0x00000344u, 0x00000345u, 0x00050041u, 0x00000158u, 0x00000347u, 0x00000157u, 0x00000080u, 0x0004003du, + 0x00000006u, 0x00000348u, 0x00000347u, 0x00050084u, 0x00000006u, 0x00000349u, 0x00000346u, 0x00000348u, + 0x0004003du, 0x00000006u, 0x0000034au, 0x0000016fu, 0x00050080u, 0x00000006u, 0x0000034bu, 0x00000349u, + 0x0000034au, 0x00050080u, 0x00000006u, 0x0000034cu, 0x00000343u, 0x0000034bu, 0x00050041u, 0x00000158u, + 0x0000034du, 0x00000157u, 0x00000329u, 0x0004003du, 0x00000006u, 0x0000034eu, 0x0000034du, 0x000500abu, + 0x0000002cu, 0x0000034fu, 0x0000034eu, 0x00000036u, 0x000300f7u, 0x00000352u, 0x00000000u, 0x000400fau, + 0x0000034fu, 0x00000351u, 0x00000359u, 0x000200f8u, 0x00000351u, 0x0004003du, 0x00000008u, 0x00000353u, + 0x0000027bu, 0x0004003du, 0x00000008u, 0x00000354u, 0x0000027bu, 0x0004007fu, 0x00000008u, 0x00000355u, + 0x00000354u, 0x0006000cu, 0x00000008u, 0x00000356u, 0x00000001u, 0x0000001bu, 0x00000355u, 0x00050081u, + 0x00000008u, 0x00000357u, 0x00000331u, 0x00000356u, 0x00050088u, 0x00000008u, 0x00000358u, 0x00000353u, + 0x00000357u, 0x0003003eu, 0x00000350u, 0x00000358u, 0x000200f9u, 0x00000352u, 0x000200f8u, 0x00000359u, + 0x0004003du, 0x00000008u, 0x0000035au, 0x0000027bu, 0x0003003eu, 0x00000350u, 0x0000035au, 0x000200f9u, + 0x00000352u, 0x000200f8u, 0x00000352u, 0x0004003du, 0x00000008u, 0x0000035cu, 0x00000350u, 0x0003003eu, + 0x0000035bu, 0x0000035cu, 0x00050041u, 0x00000158u, 0x0000035eu, 0x00000157u, 0x00000313u, 0x0004003du, + 0x00000006u, 0x0000035fu, 0x0000035eu, 0x0003003eu, 0x0000035du, 0x0000035fu, 0x00060039u, 0x00000006u, + 0x00000360u, 0x00000020u, 0x0000035bu, 0x0000035du, 0x00040071u, 0x000001fbu, 0x00000361u, 0x00000360u, + 0x00060041u, 0x00000207u, 0x00000362u, 0x00000340u, 0x000000d3u, 0x0000034cu, 0x0003003eu, 0x00000362u, + 0x00000361u, 0x000200f9u, 0x00000318u, 0x000200f8u, 0x00000318u, 0x000200f9u, 0x00000312u, 0x000200f8u, + 0x00000312u, 0x000400fau, 0x00000363u, 0x0000030fu, 0x00000311u, 0x000200f8u, 0x00000311u, 0x000200f9u, + 0x00000276u, 0x000200f8u, 0x00000276u, 0x0004003du, 0x00000006u, 0x00000364u, 0x00000271u, 0x00050080u, + 0x00000006u, 0x00000365u, 0x00000364u, 0x00000080u, 0x0003003eu, 0x00000271u, 0x00000365u, 0x000200f9u, + 0x00000273u, 0x000200f8u, 0x00000275u, 0x0004003du, 0x00000006u, 0x00000366u, 0x00000164u, 0x000500abu, + 0x0000002cu, 0x00000367u, 0x00000366u, 0x00000036u, 0x000300f7u, 0x00000369u, 0x00000000u, 0x000400fau, + 0x00000367u, 0x00000368u, 0x00000369u, 0x000200f8u, 0x00000368u, 0x000100fdu, 0x000200f8u, 0x00000369u, + 0x0003003eu, 0x0000036bu, 0x00000036u, 0x000200f9u, 0x0000036cu, 0x000200f8u, 0x0000036cu, 0x000400f6u, + 0x0000036eu, 0x0000036fu, 0x00000000u, 0x000200f9u, 0x00000370u, 0x000200f8u, 0x00000370u, 0x0004003du, + 0x00000006u, 0x00000371u, 0x0000036bu, 0x00050041u, 0x00000158u, 0x00000372u, 0x00000157u, 0x000001cfu, + 0x0004003du, 0x00000006u, 0x00000373u, 0x00000372u, 0x000500b0u, 0x0000002cu, 0x00000374u, 0x00000371u, + 0x00000373u, 0x000400fau, 0x00000374u, 0x0000036du, 0x0000036eu, 0x000200f8u, 0x0000036du, 0x0004003du, + 0x00000006u, 0x00000376u, 0x0000019bu, 0x0004007cu, 0x00000023u, 0x00000377u, 0x00000376u, 0x00050041u, + 0x00000158u, 0x00000378u, 0x00000157u, 0x000001cfu, 0x0004003du, 0x00000006u, 0x00000379u, 0x00000378u, + 0x0004007cu, 0x00000023u, 0x0000037au, 0x00000379u, 0x00050082u, 0x00000023u, 0x0000037bu, 0x00000377u, + 0x0000037au, 0x0004003du, 0x00000006u, 0x0000037cu, 0x0000036bu, 0x0004007cu, 0x00000023u, 0x0000037du, + 0x0000037cu, 0x00050080u, 0x00000023u, 0x0000037eu, 0x0000037bu, 0x0000037du, 0x0003003eu, 0x00000375u, + 0x0000037eu, 0x0003003eu, 0x0000037fu, 0x00000244u, 0x0004003du, 0x00000023u, 0x00000380u, 0x00000375u, + 0x000500afu, 0x0000002cu, 0x00000381u, 0x00000380u, 0x000000d3u, 0x000300f7u, 0x00000383u, 0x00000000u, + 0x000400fau, 0x00000381u, 0x00000382u, 0x000003b1u, 0x000200f8u, 0x00000382u, 0x00050041u, 0x00000158u, + 0x00000384u, 0x00000157u, 0x00000057u, 0x0004003du, 0x00000006u, 0x00000385u, 0x00000384u, 0x000500aau, + 0x0000002cu, 0x00000386u, 0x00000385u, 0x00000036u, 0x000300f7u, 0x00000389u, 0x00000000u, 0x000400fau, + 0x00000386u, 0x00000388u, 0x0000039au, 0x000200f8u, 0x00000388u, 0x00050041u, 0x00000158u, 0x0000038au, + 0x00000157u, 0x000000b1u, 0x0004003du, 0x00000006u, 0x0000038bu, 0x0000038au, 0x000500c2u, 0x00000006u, + 0x0000038cu, 0x0000038bu, 0x0000017cu, 0x0004003du, 0x00000006u, 0x0000038du, 0x00000174u, 0x0004003du, + 0x00000023u, 0x0000038eu, 0x00000375u, 0x0004007cu, 0x00000006u, 0x0000038fu, 0x0000038eu, 0x00050080u, + 0x00000006u, 0x00000390u, 0x0000038du, 0x0000038fu, 0x00050041u, 0x00000158u, 0x00000391u, 0x00000157u, + 0x000002a8u, 0x0004003du, 0x00000006u, 0x00000392u, 0x00000391u, 0x00050084u, 0x00000006u, 0x00000393u, + 0x00000390u, 0x00000392u, 0x0004003du, 0x00000006u, 0x00000394u, 0x0000016fu, 0x00050080u, 0x00000006u, + 0x00000395u, 0x00000393u, 0x00000394u, 0x00050080u, 0x00000006u, 0x00000396u, 0x0000038cu, 0x00000395u, + 0x00060041u, 0x00000180u, 0x00000397u, 0x000002a0u, 0x000000d3u, 0x00000396u, 0x0004003du, 0x00000006u, + 0x00000398u, 0x00000397u, 0x0004007cu, 0x00000008u, 0x00000399u, 0x00000398u, 0x0003003eu, 0x00000387u, + 0x00000399u, 0x000200f9u, 0x00000389u, 0x000200f8u, 0x0000039au, 0x00050041u, 0x00000158u, 0x0000039bu, + 0x00000157u, 0x000000b1u, 0x0004003du, 0x00000006u, 0x0000039cu, 0x0000039bu, 0x000500c2u, 0x00000006u, + 0x0000039du, 0x0000039cu, 0x00000080u, 0x0004003du, 0x00000006u, 0x0000039eu, 0x00000174u, 0x0004003du, + 0x00000023u, 0x0000039fu, 0x00000375u, 0x0004007cu, 0x00000006u, 0x000003a0u, 0x0000039fu, 0x00050080u, + 0x00000006u, 0x000003a1u, 0x0000039eu, 0x000003a0u, 0x00050041u, 0x00000158u, 0x000003a2u, 0x00000157u, + 0x000002a8u, 0x0004003du, 0x00000006u, 0x000003a3u, 0x000003a2u, 0x00050084u, 0x00000006u, 0x000003a4u, + 0x000003a1u, 0x000003a3u, 0x0004003du, 0x00000006u, 0x000003a5u, 0x0000016fu, 0x00050080u, 0x00000006u, + 0x000003a6u, 0x000003a4u, 0x000003a5u, 0x00050080u, 0x00000006u, 0x000003a7u, 0x0000039du, 0x000003a6u, + 0x00060041u, 0x00000207u, 0x000003a8u, 0x000002b6u, 0x000000d3u, 0x000003a7u, 0x0004003du, 0x000001fbu, + 0x000003a9u, 0x000003a8u, 0x00040071u, 0x00000006u, 0x000003aau, 0x000003a9u, 0x0003003eu, 0x000003abu, + 0x000003aau, 0x00050041u, 0x00000158u, 0x000003adu, 0x00000157u, 0x00000057u, 0x0004003du, 0x00000006u, + 0x000003aeu, 0x000003adu, 0x0003003eu, 0x000003acu, 0x000003aeu, 0x00060039u, 0x00000008u, 0x000003afu, + 0x0000001bu, 0x000003abu, 0x000003acu, 0x0003003eu, 0x00000387u, 0x000003afu, 0x000200f9u, 0x00000389u, + 0x000200f8u, 0x00000389u, 0x0004003du, 0x00000008u, 0x000003b0u, 0x00000387u, 0x0003003eu, 0x0000037fu, + 0x000003b0u, 0x000200f9u, 0x00000383u, 0x000200f8u, 0x000003b1u, 0x0004003du, 0x0000002cu, 0x000003b2u, + 0x000001b5u, 0x000300f7u, 0x000003b4u, 0x00000000u, 0x000400fau, 0x000003b2u, 0x000003b3u, 0x000003b4u, + 0x000200f8u, 0x000003b3u, 0x00050041u, 0x00000158u, 0x000003b5u, 0x00000157u, 0x000001cfu, 0x0004003du, + 0x00000006u, 0x000003b6u, 0x000003b5u, 0x0004007cu, 0x00000023u, 0x000003b7u, 0x000003b6u, 0x0004003du, + 0x00000023u, 0x000003b8u, 0x00000375u, 0x00050080u, 0x00000023u, 0x000003b9u, 0x000003b7u, 0x000003b8u, + 0x0004007cu, 0x00000006u, 0x000003bau, 0x000003b9u, 0x00050041u, 0x0000000du, 0x000003bbu, 0x000001e3u, + 0x000003bau, 0x0004003du, 0x00000008u, 0x000003bcu, 0x000003bbu, 0x0003003eu, 0x0000037fu, 0x000003bcu, + 0x000200f9u, 0x000003b4u, 0x000200f8u, 0x000003b4u, 0x000200f9u, 0x00000383u, 0x000200f8u, 0x00000383u, + 0x000200f9u, 0x000003bdu, 0x000200f8u, 0x000003bdu, 0x000400f6u, 0x000003bfu, 0x000003c0u, 0x00000000u, + 0x000200f9u, 0x000003beu, 0x000200f8u, 0x000003beu, 0x00050041u, 0x00000158u, 0x000003c1u, 0x00000157u, + 0x00000066u, 0x0004003du, 0x00000006u, 0x000003c2u, 0x000003c1u, 0x000500aau, 0x0000002cu, 0x000003c3u, + 0x000003c2u, 0x00000036u, 0x000300f7u, 0x000003c5u, 0x00000000u, 0x000400fau, 0x000003c3u, 0x000003c4u, + 0x000003d0u, 0x000200f8u, 0x000003c4u, 0x00050041u, 0x00000158u, 0x000003c6u, 0x00000157u, 0x000001efu, + 0x0004003du, 0x00000006u, 0x000003c7u, 0x000003c6u, 0x000500c2u, 0x00000006u, 0x000003c8u, 0x000003c7u, + 0x0000017cu, 0x0004003du, 0x00000006u, 0x000003c9u, 0x000001c8u, 0x0004003du, 0x00000006u, 0x000003cau, + 0x0000036bu, 0x00050080u, 0x00000006u, 0x000003cbu, 0x000003c9u, 0x000003cau, 0x00050080u, 0x00000006u, + 0x000003ccu, 0x000003c8u, 0x000003cbu, 0x0004003du, 0x00000008u, 0x000003cdu, 0x0000037fu, 0x0004007cu, + 0x00000006u, 0x000003ceu, 0x000003cdu, 0x00060041u, 0x00000180u, 0x000003cfu, 0x000001eeu, 0x000000d3u, + 0x000003ccu, 0x0003003eu, 0x000003cfu, 0x000003ceu, 0x000200f9u, 0x000003c5u, 0x000200f8u, 0x000003d0u, + 0x00050041u, 0x00000158u, 0x000003d1u, 0x00000157u, 0x000001efu, 0x0004003du, 0x00000006u, 0x000003d2u, + 0x000003d1u, 0x000500c2u, 0x00000006u, 0x000003d3u, 0x000003d2u, 0x00000080u, 0x0004003du, 0x00000006u, + 0x000003d4u, 0x000001c8u, 0x0004003du, 0x00000006u, 0x000003d5u, 0x0000036bu, 0x00050080u, 0x00000006u, + 0x000003d6u, 0x000003d4u, 0x000003d5u, 0x00050080u, 0x00000006u, 0x000003d7u, 0x000003d3u, 0x000003d6u, + 0x0004003du, 0x00000008u, 0x000003d9u, 0x0000037fu, 0x0003003eu, 0x000003d8u, 0x000003d9u, 0x00050041u, + 0x00000158u, 0x000003dbu, 0x00000157u, 0x00000066u, 0x0004003du, 0x00000006u, 0x000003dcu, 0x000003dbu, + 0x0003003eu, 0x000003dau, 0x000003dcu, 0x00060039u, 0x00000006u, 0x000003ddu, 0x00000020u, 0x000003d8u, + 0x000003dau, 0x00040071u, 0x000001fbu, 0x000003deu, 0x000003ddu, 0x00060041u, 0x00000207u, 0x000003dfu, + 0x000001ffu, 0x000000d3u, 0x000003d7u, 0x0003003eu, 0x000003dfu, 0x000003deu, 0x000200f9u, 0x000003c5u, + 0x000200f8u, 0x000003c5u, 0x000200f9u, 0x000003c0u, 0x000200f8u, 0x000003c0u, 0x000400fau, 0x00000363u, + 0x000003bdu, 0x000003bfu, 0x000200f8u, 0x000003bfu, 0x000200f9u, 0x0000036fu, 0x000200f8u, 0x0000036fu, + 0x0004003du, 0x00000006u, 0x000003e0u, 0x0000036bu, 0x00050080u, 0x00000006u, 0x000003e1u, 0x000003e0u, + 0x00000080u, 0x0003003eu, 0x0000036bu, 0x000003e1u, 0x000200f9u, 0x0000036cu, 0x000200f8u, 0x0000036eu, + 0x000100fdu, 0x00010038u, 0x00050036u, 0x00000008u, 0x0000000bu, 0x00000000u, 0x00000009u, 0x00030037u, + 0x00000007u, 0x0000000au, 0x000200f8u, 0x0000000cu, 0x0004003du, 0x00000006u, 0x00000022u, 0x0000000au, + 0x000500c4u, 0x00000006u, 0x00000025u, 0x00000022u, 0x00000024u, 0x0004007cu, 0x00000008u, 0x00000026u, + 0x00000025u, 0x000200feu, 0x00000026u, 0x00010038u, 0x00050036u, 0x00000006u, 0x00000010u, 0x00000000u, + 0x0000000eu, 0x00030037u, 0x0000000du, 0x0000000fu, 0x000200f8u, 0x00000011u, 0x0004003bu, 0x00000007u, + 0x00000029u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000042u, 0x00000007u, 0x0004003du, 0x00000008u, + 0x0000002au, 0x0000000fu, 0x0004007cu, 0x00000006u, 0x0000002bu, 0x0000002au, 0x0003003eu, 0x00000029u, + 0x0000002bu, 0x0004003du, 0x00000006u, 0x0000002du, 0x00000029u, 0x000500c7u, 0x00000006u, 0x0000002fu, + 0x0000002du, 0x0000002eu, 0x000500aau, 0x0000002cu, 0x00000030u, 0x0000002fu, 0x0000002eu, 0x000300f7u, + 0x00000032u, 0x00000000u, 0x000400fau, 0x00000030u, 0x00000031u, 0x00000032u, 0x000200f8u, 0x00000031u, + 0x0004003du, 0x00000006u, 0x00000033u, 0x00000029u, 0x000500c7u, 0x00000006u, 0x00000035u, 0x00000033u, + 0x00000034u, 0x000500abu, 0x0000002cu, 0x00000037u, 0x00000035u, 0x00000036u, 0x000200f9u, 0x00000032u, + 0x000200f8u, 0x00000032u, 0x000700f5u, 0x0000002cu, 0x00000038u, 0x00000030u, 0x00000011u, 0x00000037u, + 0x00000031u, 0x000300f7u, 0x0000003au, 0x00000000u, 0x000400fau, 0x00000038u, 0x00000039u, 0x0000003au, + 0x000200f8u, 0x00000039u, 0x0004003du, 0x00000006u, 0x0000003bu, 0x00000029u, 0x000500c2u, 0x00000006u, + 0x0000003cu, 0x0000003bu, 0x00000024u, 0x000500c5u, 0x00000006u, 0x0000003eu, 0x0000003cu, 0x0000003du, + 0x000500c7u, 0x00000006u, 0x00000040u, 0x0000003eu, 0x0000003fu, 0x000200feu, 0x00000040u, 0x000200f8u, + 0x0000003au, 0x0004003du, 0x00000006u, 0x00000044u, 0x00000029u, 0x000500c2u, 0x00000006u, 0x00000045u, + 0x00000044u, 0x00000024u, 0x000500c7u, 0x00000006u, 0x00000047u, 0x00000045u, 0x00000046u, 0x00050080u, + 0x00000006u, 0x00000048u, 0x00000043u, 0x00000047u, 0x0003003eu, 0x00000042u, 0x00000048u, 0x0004003du, + 0x00000006u, 0x00000049u, 0x00000029u, 0x0004003du, 0x00000006u, 0x0000004au, 0x00000042u, 0x00050080u, + 0x00000006u, 0x0000004bu, 0x00000049u, 0x0000004au, 0x000500c2u, 0x00000006u, 0x0000004cu, 0x0000004bu, + 0x00000024u, 0x000500c7u, 0x00000006u, 0x0000004du, 0x0000004cu, 0x0000003fu, 0x000200feu, 0x0000004du, + 0x00010038u, 0x00050036u, 0x00000008u, 0x00000013u, 0x00000000u, 0x00000009u, 0x00030037u, 0x00000007u, + 0x00000012u, 0x000200f8u, 0x00000014u, 0x0004003bu, 0x00000007u, 0x00000050u, 0x00000007u, 0x0004003bu, + 0x00000007u, 0x00000055u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000005bu, 0x00000007u, 0x0004003bu, + 0x00000007u, 0x00000076u, 0x00000007u, 0x0004003du, 0x00000006u, 0x00000051u, 0x00000012u, 0x000500c7u, + 0x00000006u, 0x00000053u, 0x00000051u, 0x00000052u, 0x000500c4u, 0x00000006u, 0x00000054u, 0x00000053u, + 0x00000024u, 0x0003003eu, 0x00000050u, 0x00000054u, 0x0004003du, 0x00000006u, 0x00000056u, 0x00000012u, + 0x000500c2u, 0x00000006u, 0x00000058u, 0x00000056u, 0x00000057u, 0x000500c7u, 0x00000006u, 0x0000005au, + 0x00000058u, 0x00000059u, 0x0003003eu, 0x00000055u, 0x0000005au, 0x0004003du, 0x00000006u, 0x0000005cu, + 0x00000012u, 0x000500c7u, 0x00000006u, 0x0000005eu, 0x0000005cu, 0x0000005du, 0x0003003eu, 0x0000005bu, + 0x0000005eu, 0x0004003du, 0x00000006u, 0x0000005fu, 0x00000055u, 0x000500aau, 0x0000002cu, 0x00000060u, + 0x0000005fu, 0x00000059u, 0x000300f7u, 0x00000062u, 0x00000000u, 0x000400fau, 0x00000060u, 0x00000061u, + 0x00000062u, 0x000200f8u, 0x00000061u, 0x0004003du, 0x00000006u, 0x00000063u, 0x00000050u, 0x000500c5u, + 0x00000006u, 0x00000064u, 0x00000063u, 0x0000002eu, 0x0004003du, 0x00000006u, 0x00000065u, 0x0000005bu, + 0x000500c4u, 0x00000006u, 0x00000067u, 0x00000065u, 0x00000066u, 0x000500c5u, 0x00000006u, 0x00000068u, + 0x00000064u, 0x00000067u, 0x0004007cu, 0x00000008u, 0x00000069u, 0x00000068u, 0x000200feu, 0x00000069u, + 0x000200f8u, 0x00000062u, 0x0004003du, 0x00000006u, 0x0000006bu, 0x00000055u, 0x000500aau, 0x0000002cu, + 0x0000006cu, 0x0000006bu, 0x00000036u, 0x000300f7u, 0x0000006eu, 0x00000000u, 0x000400fau, 0x0000006cu, + 0x0000006du, 0x0000006eu, 0x000200f8u, 0x0000006du, 0x0004003du, 0x00000006u, 0x0000006fu, 0x0000005bu, + 0x000500aau, 0x0000002cu, 0x00000070u, 0x0000006fu, 0x00000036u, 0x000300f7u, 0x00000072u, 0x00000000u, + 0x000400fau, 0x00000070u, 0x00000071u, 0x00000072u, 0x000200f8u, 0x00000071u, 0x0004003du, 0x00000006u, + 0x00000073u, 0x00000050u, 0x0004007cu, 0x00000008u, 0x00000074u, 0x00000073u, 0x000200feu, 0x00000074u, + 0x000200f8u, 0x00000072u, 0x0003003eu, 0x00000076u, 0x00000036u, 0x000200f9u, 0x00000077u, 0x000200f8u, + 0x00000077u, 0x000400f6u, 0x00000079u, 0x0000007au, 0x00000000u, 0x000200f9u, 0x0000007bu, 0x000200f8u, + 0x0000007bu, 0x0004003du, 0x00000006u, 0x0000007cu, 0x0000005bu, 0x000500c7u, 0x00000006u, 0x0000007eu, + 0x0000007cu, 0x0000007du, 0x000500aau, 0x0000002cu, 0x0000007fu, 0x0000007eu, 0x00000036u, 0x000400fau, + 0x0000007fu, 0x00000078u, 0x00000079u, 0x000200f8u, 0x00000078u, 0x0004003du, 0x00000006u, 0x00000081u, + 0x0000005bu, 0x000500c4u, 0x00000006u, 0x00000082u, 0x00000081u, 0x00000080u, 0x0003003eu, 0x0000005bu, + 0x00000082u, 0x0004003du, 0x00000006u, 0x00000083u, 0x00000076u, 0x00050080u, 0x00000006u, 0x00000084u, + 0x00000083u, 0x00000046u, 0x0003003eu, 0x00000076u, 0x00000084u, 0x000200f9u, 0x0000007au, 0x000200f8u, + 0x0000007au, 0x000200f9u, 0x00000077u, 0x000200f8u, 0x00000079u, 0x0004003du, 0x00000006u, 0x00000085u, + 0x0000005bu, 0x000500c7u, 0x00000006u, 0x00000086u, 0x00000085u, 0x0000005du, 0x0003003eu, 0x0000005bu, + 0x00000086u, 0x0004003du, 0x00000006u, 0x00000087u, 0x00000050u, 0x0004003du, 0x00000006u, 0x00000089u, + 0x00000076u, 0x00050082u, 0x00000006u, 0x0000008au, 0x00000088u, 0x00000089u, 0x000500c4u, 0x00000006u, + 0x0000008cu, 0x0000008au, 0x0000008bu, 0x000500c5u, 0x00000006u, 0x0000008du, 0x00000087u, 0x0000008cu, + 0x0004003du, 0x00000006u, 0x0000008eu, 0x0000005bu, 0x000500c4u, 0x00000006u, 0x0000008fu, 0x0000008eu, + 0x00000066u, 0x000500c5u, 0x00000006u, 0x00000090u, 0x0000008du, 0x0000008fu, 0x0004007cu, 0x00000008u, + 0x00000091u, 0x00000090u, 0x000200feu, 0x00000091u, 0x000200f8u, 0x0000006eu, 0x0004003du, 0x00000006u, + 0x00000093u, 0x00000050u, 0x0004003du, 0x00000006u, 0x00000094u, 0x00000055u, 0x00050080u, 0x00000006u, + 0x00000096u, 0x00000094u, 0x00000095u, 0x000500c4u, 0x00000006u, 0x00000097u, 0x00000096u, 0x0000008bu, + 0x000500c5u, 0x00000006u, 0x00000098u, 0x00000093u, 0x00000097u, 0x0004003du, 0x00000006u, 0x00000099u, + 0x0000005bu, 0x000500c4u, 0x00000006u, 0x0000009au, 0x00000099u, 0x00000066u, 0x000500c5u, 0x00000006u, + 0x0000009bu, 0x00000098u, 0x0000009au, 0x0004007cu, 0x00000008u, 0x0000009cu, 0x0000009bu, 0x000200feu, + 0x0000009cu, 0x00010038u, 0x00050036u, 0x00000006u, 0x00000016u, 0x00000000u, 0x0000000eu, 0x00030037u, + 0x0000000du, 0x00000015u, 0x000200f8u, 0x00000017u, 0x0004003bu, 0x00000007u, 0x0000009fu, 0x00000007u, + 0x0004003bu, 0x00000007u, 0x000000a2u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x000000a6u, 0x00000007u, + 0x0004003bu, 0x000000abu, 0x000000acu, 0x00000007u, 0x0004003bu, 0x00000007u, 0x000000b3u, 0x00000007u, + 0x0004003bu, 0x00000007u, 0x000000bfu, 0x00000007u, 0x0004003bu, 0x00000007u, 0x000000e1u, 0x00000007u, + 0x0004003bu, 0x00000007u, 0x000000e6u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x000000eau, 0x00000007u, + 0x0004003bu, 0x00000007u, 0x000000f0u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x0000010cu, 0x00000007u, + 0x0004003bu, 0x00000007u, 0x00000113u, 0x00000007u, 0x0004003du, 0x00000008u, 0x000000a0u, 0x00000015u, + 0x0004007cu, 0x00000006u, 0x000000a1u, 0x000000a0u, 0x0003003eu, 0x0000009fu, 0x000000a1u, 0x0004003du, + 0x00000006u, 0x000000a3u, 0x0000009fu, 0x000500c2u, 0x00000006u, 0x000000a4u, 0x000000a3u, 0x00000024u, + 0x000500c7u, 0x00000006u, 0x000000a5u, 0x000000a4u, 0x00000052u, 0x0003003eu, 0x000000a2u, 0x000000a5u, + 0x0004003du, 0x00000006u, 0x000000a7u, 0x0000009fu, 0x000500c2u, 0x00000006u, 0x000000a8u, 0x000000a7u, + 0x0000008bu, 0x000500c7u, 0x00000006u, 0x000000aau, 0x000000a8u, 0x000000a9u, 0x0003003eu, 0x000000a6u, + 0x000000aau, 0x0004003du, 0x00000006u, 0x000000adu, 0x000000a6u, 0x0004007cu, 0x00000023u, 0x000000aeu, + 0x000000adu, 0x00050082u, 0x00000023u, 0x000000b0u, 0x000000aeu, 0x000000afu, 0x00050080u, 0x00000023u, + 0x000000b2u, 0x000000b0u, 0x000000b1u, 0x0003003eu, 0x000000acu, 0x000000b2u, 0x0004003du, 0x00000006u, + 0x000000b4u, 0x0000009fu, 0x000500c7u, 0x00000006u, 0x000000b5u, 0x000000b4u, 0x00000034u, 0x0003003eu, + 0x000000b3u, 0x000000b5u, 0x0004003du, 0x00000006u, 0x000000b6u, 0x000000a6u, 0x000500aau, 0x0000002cu, + 0x000000b7u, 0x000000b6u, 0x000000a9u, 0x000300f7u, 0x000000b9u, 0x00000000u, 0x000400fau, 0x000000b7u, + 0x000000b8u, 0x000000b9u, 0x000200f8u, 0x000000b8u, 0x0004003du, 0x00000006u, 0x000000bau, 0x000000a2u, + 0x000500c5u, 0x00000006u, 0x000000bcu, 0x000000bau, 0x000000bbu, 0x0004003du, 0x00000006u, 0x000000bdu, + 0x000000b3u, 0x000500abu, 0x0000002cu, 0x000000beu, 0x000000bdu, 0x00000036u, 0x000300f7u, 0x000000c1u, + 0x00000000u, 0x000400fau, 0x000000beu, 0x000000c0u, 0x000000c6u, 0x000200f8u, 0x000000c0u, 0x0004003du, + 0x00000006u, 0x000000c3u, 0x000000b3u, 0x000500c2u, 0x00000006u, 0x000000c4u, 0x000000c3u, 0x00000066u, + 0x000500c5u, 0x00000006u, 0x000000c5u, 0x000000c2u, 0x000000c4u, 0x0003003eu, 0x000000bfu, 0x000000c5u, + 0x000200f9u, 0x000000c1u, 0x000200f8u, 0x000000c6u, 0x0003003eu, 0x000000bfu, 0x00000036u, 0x000200f9u, + 0x000000c1u, 0x000200f8u, 0x000000c1u, 0x0004003du, 0x00000006u, 0x000000c7u, 0x000000bfu, 0x000500c5u, + 0x00000006u, 0x000000c8u, 0x000000bcu, 0x000000c7u, 0x000200feu, 0x000000c8u, 0x000200f8u, 0x000000b9u, + 0x0004003du, 0x00000023u, 0x000000cau, 0x000000acu, 0x000500afu, 0x0000002cu, 0x000000ccu, 0x000000cau, + 0x000000cbu, 0x000300f7u, 0x000000ceu, 0x00000000u, 0x000400fau, 0x000000ccu, 0x000000cdu, 0x000000ceu, + 0x000200f8u, 0x000000cdu, 0x0004003du, 0x00000006u, 0x000000cfu, 0x000000a2u, 0x000500c5u, 0x00000006u, + 0x000000d0u, 0x000000cfu, 0x000000bbu, 0x000200feu, 0x000000d0u, 0x000200f8u, 0x000000ceu, 0x0004003du, + 0x00000023u, 0x000000d2u, 0x000000acu, 0x000500b3u, 0x0000002cu, 0x000000d4u, 0x000000d2u, 0x000000d3u, + 0x000300f7u, 0x000000d6u, 0x00000000u, 0x000400fau, 0x000000d4u, 0x000000d5u, 0x000000d6u, 0x000200f8u, + 0x000000d5u, 0x0004003du, 0x00000023u, 0x000000d7u, 0x000000acu, 0x000500b1u, 0x0000002cu, 0x000000d9u, + 0x000000d7u, 0x000000d8u, 0x000300f7u, 0x000000dbu, 0x00000000u, 0x000400fau, 0x000000d9u, 0x000000dau, + 0x000000dbu, 0x000200f8u, 0x000000dau, 0x0004003du, 0x00000006u, 0x000000dcu, 0x000000a2u, 0x000200feu, + 0x000000dcu, 0x000200f8u, 0x000000dbu, 0x0004003du, 0x00000006u, 0x000000dfu, 0x000000b3u, 0x000500c5u, + 0x00000006u, 0x000000e0u, 0x000000dfu, 0x000000deu, 0x0003003eu, 0x000000b3u, 0x000000e0u, 0x0004003du, + 0x00000023u, 0x000000e3u, 0x000000acu, 0x00050082u, 0x00000023u, 0x000000e4u, 0x000000e2u, 0x000000e3u, + 0x0004007cu, 0x00000006u, 0x000000e5u, 0x000000e4u, 0x0003003eu, 0x000000e1u, 0x000000e5u, 0x0004003du, + 0x00000006u, 0x000000e7u, 0x000000b3u, 0x0004003du, 0x00000006u, 0x000000e8u, 0x000000e1u, 0x000500c2u, + 0x00000006u, 0x000000e9u, 0x000000e7u, 0x000000e8u, 0x0003003eu, 0x000000e6u, 0x000000e9u, 0x0004003du, + 0x00000006u, 0x000000ebu, 0x000000b3u, 0x0004003du, 0x00000006u, 0x000000ecu, 0x000000e1u, 0x000500c4u, + 0x00000006u, 0x000000edu, 0x00000046u, 0x000000ecu, 0x00050082u, 0x00000006u, 0x000000eeu, 0x000000edu, + 0x00000046u, 0x000500c7u, 0x00000006u, 0x000000efu, 0x000000ebu, 0x000000eeu, 0x0003003eu, 0x000000eau, + 0x000000efu, 0x0004003du, 0x00000006u, 0x000000f1u, 0x000000e1u, 0x00050082u, 0x00000006u, 0x000000f2u, + 0x000000f1u, 0x00000046u, 0x000500c4u, 0x00000006u, 0x000000f3u, 0x00000046u, 0x000000f2u, 0x0003003eu, + 0x000000f0u, 0x000000f3u, 0x0004003du, 0x00000006u, 0x000000f4u, 0x000000eau, 0x0004003du, 0x00000006u, + 0x000000f5u, 0x000000f0u, 0x000500acu, 0x0000002cu, 0x000000f6u, 0x000000f4u, 0x000000f5u, 0x000400a8u, + 0x0000002cu, 0x000000f7u, 0x000000f6u, 0x000300f7u, 0x000000f9u, 0x00000000u, 0x000400fau, 0x000000f7u, + 0x000000f8u, 0x000000f9u, 0x000200f8u, 0x000000f8u, 0x0004003du, 0x00000006u, 0x000000fau, 0x000000eau, + 0x0004003du, 0x00000006u, 0x000000fbu, 0x000000f0u, 0x000500aau, 0x0000002cu, 0x000000fcu, 0x000000fau, + 0x000000fbu, 0x000300f7u, 0x000000feu, 0x00000000u, 0x000400fau, 0x000000fcu, 0x000000fdu, 0x000000feu, + 0x000200f8u, 0x000000fdu, 0x0004003du, 0x00000006u, 0x000000ffu, 0x000000e6u, 0x000500c7u, 0x00000006u, + 0x00000100u, 0x000000ffu, 0x00000046u, 0x000500abu, 0x0000002cu, 0x00000101u, 0x00000100u, 0x00000036u, + 0x000200f9u, 0x000000feu, 0x000200f8u, 0x000000feu, 0x000700f5u, 0x0000002cu, 0x00000102u, 0x000000fcu, + 0x000000f8u, 0x00000101u, 0x000000fdu, 0x000200f9u, 0x000000f9u, 0x000200f8u, 0x000000f9u, 0x000700f5u, + 0x0000002cu, 0x00000103u, 0x000000f6u, 0x000000dbu, 0x00000102u, 0x000000feu, 0x000300f7u, 0x00000105u, + 0x00000000u, 0x000400fau, 0x00000103u, 0x00000104u, 0x00000105u, 0x000200f8u, 0x00000104u, 0x0004003du, + 0x00000006u, 0x00000106u, 0x000000e6u, 0x00050080u, 0x00000006u, 0x00000107u, 0x00000106u, 0x00000046u, + 0x0003003eu, 0x000000e6u, 0x00000107u, 0x000200f9u, 0x00000105u, 0x000200f8u, 0x00000105u, 0x0004003du, + 0x00000006u, 0x00000108u, 0x000000a2u, 0x0004003du, 0x00000006u, 0x00000109u, 0x000000e6u, 0x000500c5u, + 0x00000006u, 0x0000010au, 0x00000108u, 0x00000109u, 0x000200feu, 0x0000010au, 0x000200f8u, 0x000000d6u, + 0x0004003du, 0x00000023u, 0x0000010du, 0x000000acu, 0x0004007cu, 0x00000006u, 0x0000010eu, 0x0000010du, + 0x000500c4u, 0x00000006u, 0x0000010fu, 0x0000010eu, 0x00000057u, 0x0004003du, 0x00000006u, 0x00000110u, + 0x000000b3u, 0x000500c2u, 0x00000006u, 0x00000111u, 0x00000110u, 0x00000066u, 0x000500c5u, 0x00000006u, + 0x00000112u, 0x0000010fu, 0x00000111u, 0x0003003eu, 0x0000010cu, 0x00000112u, 0x0004003du, 0x00000006u, + 0x00000114u, 0x000000b3u, 0x000500c7u, 0x00000006u, 0x00000116u, 0x00000114u, 0x00000115u, 0x0003003eu, + 0x00000113u, 0x00000116u, 0x0004003du, 0x00000006u, 0x00000117u, 0x00000113u, 0x000500acu, 0x0000002cu, + 0x00000119u, 0x00000117u, 0x00000118u, 0x000400a8u, 0x0000002cu, 0x0000011au, 0x00000119u, 0x000300f7u, + 0x0000011cu, 0x00000000u, 0x000400fau, 0x0000011au, 0x0000011bu, 0x0000011cu, 0x000200f8u, 0x0000011bu, + 0x0004003du, 0x00000006u, 0x0000011du, 0x00000113u, 0x000500aau, 0x0000002cu, 0x0000011eu, 0x0000011du, + 0x00000118u, 0x000300f7u, 0x00000120u, 0x00000000u, 0x000400fau, 0x0000011eu, 0x0000011fu, 0x00000120u, + 0x000200f8u, 0x0000011fu, 0x0004003du, 0x00000006u, 0x00000121u, 0x0000010cu, 0x000500c7u, 0x00000006u, + 0x00000122u, 0x00000121u, 0x00000046u, 0x000500abu, 0x0000002cu, 0x00000123u, 0x00000122u, 0x00000036u, + 0x000200f9u, 0x00000120u, 0x000200f8u, 0x00000120u, 0x000700f5u, 0x0000002cu, 0x00000124u, 0x0000011eu, + 0x0000011bu, 0x00000123u, 0x0000011fu, 0x000200f9u, 0x0000011cu, 0x000200f8u, 0x0000011cu, 0x000700f5u, + 0x0000002cu, 0x00000125u, 0x00000119u, 0x000000d6u, 0x00000124u, 0x00000120u, 0x000300f7u, 0x00000127u, + 0x00000000u, 0x000400fau, 0x00000125u, 0x00000126u, 0x00000127u, 0x000200f8u, 0x00000126u, 0x0004003du, + 0x00000006u, 0x00000128u, 0x0000010cu, 0x00050080u, 0x00000006u, 0x00000129u, 0x00000128u, 0x00000046u, + 0x0003003eu, 0x0000010cu, 0x00000129u, 0x000200f9u, 0x00000127u, 0x000200f8u, 0x00000127u, 0x0004003du, + 0x00000006u, 0x0000012au, 0x000000a2u, 0x0004003du, 0x00000006u, 0x0000012bu, 0x0000010cu, 0x000500c5u, + 0x00000006u, 0x0000012cu, 0x0000012au, 0x0000012bu, 0x000200feu, 0x0000012cu, 0x00010038u, 0x00050036u, + 0x00000008u, 0x0000001bu, 0x00000000u, 0x00000018u, 0x00030037u, 0x00000007u, 0x00000019u, 0x00030037u, + 0x00000007u, 0x0000001au, 0x000200f8u, 0x0000001cu, 0x0004003bu, 0x0000000du, 0x00000131u, 0x00000007u, + 0x0004003bu, 0x00000007u, 0x00000134u, 0x00000007u, 0x0004003bu, 0x00000007u, 0x00000138u, 0x00000007u, + 0x0004003du, 0x00000006u, 0x0000012fu, 0x0000001au, 0x000500aau, 0x0000002cu, 0x00000130u, 0x0000012fu, + 0x00000046u, 0x000300f7u, 0x00000133u, 0x00000000u, 0x000400fau, 0x00000130u, 0x00000132u, 0x00000137u, + 0x000200f8u, 0x00000132u, 0x0004003du, 0x00000006u, 0x00000135u, 0x00000019u, 0x0003003eu, 0x00000134u, + 0x00000135u, 0x00050039u, 0x00000008u, 0x00000136u, 0x00000013u, 0x00000134u, 0x0003003eu, 0x00000131u, + 0x00000136u, 0x000200f9u, 0x00000133u, 0x000200f8u, 0x00000137u, 0x0004003du, 0x00000006u, 0x00000139u, + 0x00000019u, 0x0003003eu, 0x00000138u, 0x00000139u, 0x00050039u, 0x00000008u, 0x0000013au, 0x0000000bu, + 0x00000138u, 0x0003003eu, 0x00000131u, 0x0000013au, 0x000200f9u, 0x00000133u, 0x000200f8u, 0x00000133u, + 0x0004003du, 0x00000008u, 0x0000013bu, 0x00000131u, 0x000200feu, 0x0000013bu, 0x00010038u, 0x00050036u, + 0x00000006u, 0x00000020u, 0x00000000u, 0x0000001du, 0x00030037u, 0x0000000du, 0x0000001eu, 0x00030037u, + 0x00000007u, 0x0000001fu, 0x000200f8u, 0x00000021u, 0x0004003bu, 0x00000007u, 0x00000140u, 0x00000007u, + 0x0004003bu, 0x0000000du, 0x00000143u, 0x00000007u, 0x0004003bu, 0x0000000du, 0x00000147u, 0x00000007u, + 0x0004003du, 0x00000006u, 0x0000013eu, 0x0000001fu, 0x000500aau, 0x0000002cu, 0x0000013fu, 0x0000013eu, + 0x00000046u, 0x000300f7u, 0x00000142u, 0x00000000u, 0x000400fau, 0x0000013fu, 0x00000141u, 0x00000146u, + 0x000200f8u, 0x00000141u, 0x0004003du, 0x00000008u, 0x00000144u, 0x0000001eu, 0x0003003eu, 0x00000143u, + 0x00000144u, 0x00050039u, 0x00000006u, 0x00000145u, 0x00000016u, 0x00000143u, 0x0003003eu, 0x00000140u, + 0x00000145u, 0x000200f9u, 0x00000142u, 0x000200f8u, 0x00000146u, 0x0004003du, 0x00000008u, 0x00000148u, + 0x0000001eu, 0x0003003eu, 0x00000147u, 0x00000148u, 0x00050039u, 0x00000006u, 0x00000149u, 0x00000010u, + 0x00000147u, 0x0003003eu, 0x00000140u, 0x00000149u, 0x000200f9u, 0x00000142u, 0x000200f8u, 0x00000142u, + 0x0004003du, 0x00000006u, 0x0000014au, 0x00000140u, 0x000200feu, 0x0000014au, 0x00010038u, +}; + constexpr uint32_t kSpv_vt_causal_conv1d_update[] = { 0x07230203u, 0x00010300u, 0x0008000bu, 0x00000391u, 0x00000000u, 0x00020011u, 0x00000001u, 0x00020011u, 0x00001151u, 0x0006000bu, 0x00000001u, 0x4c534c47u, 0x6474732eu, 0x3035342eu, 0x00000000u, 0x0003000eu, @@ -24116,6 +24777,10 @@ constexpr uint32_t kSpecIds_vt_cast[] = { 0u, 1u, }; +constexpr uint32_t kSpecIds_vt_causal_conv1d_fwd[] = { + 0u, +}; + constexpr uint32_t kSpecIds_vt_embedding[] = { 0u, 1u, 2u, }; @@ -24234,6 +24899,7 @@ const SpirvModule kSpirvModules[] = { {"vt_add", kSpv_vt_add, sizeof(kSpv_vt_add) / sizeof(uint32_t), nullptr, 0, 6u, 0x00000030u}, {"vt_attn_qk_norm_rope_gate", kSpv_vt_attn_qk_norm_rope_gate, sizeof(kSpv_vt_attn_qk_norm_rope_gate) / sizeof(uint32_t), kSpecIds_vt_attn_qk_norm_rope_gate, 3, 13u, 0x000003f0u}, {"vt_cast", kSpv_vt_cast, sizeof(kSpv_vt_cast) / sizeof(uint32_t), kSpecIds_vt_cast, 2, 4u, 0x0000000cu}, + {"vt_causal_conv1d_fwd", kSpv_vt_causal_conv1d_fwd, sizeof(kSpv_vt_causal_conv1d_fwd) / sizeof(uint32_t), kSpecIds_vt_causal_conv1d_fwd, 1, 12u, 0x00000303u}, {"vt_causal_conv1d_update", kSpv_vt_causal_conv1d_update, sizeof(kSpv_vt_causal_conv1d_update) / sizeof(uint32_t), nullptr, 0, 11u, 0x00000303u}, {"vt_embedding", kSpv_vt_embedding, sizeof(kSpv_vt_embedding) / sizeof(uint32_t), kSpecIds_vt_embedding, 3, 6u, 0x00000030u}, {"vt_exl3_gemm", kSpv_vt_exl3_gemm, sizeof(kSpv_vt_exl3_gemm) / sizeof(uint32_t), nullptr, 0, 3u, 0x00000001u}, diff --git a/src/vt/vulkan/vulkan_spirv.h b/src/vt/vulkan/vulkan_spirv.h index d7369eaedf..e084815580 100644 --- a/src/vt/vulkan/vulkan_spirv.h +++ b/src/vt/vulkan/vulkan_spirv.h @@ -18,7 +18,7 @@ // and extern declarations; vulkan_spirv.cpp carries the data, so adding shaders // costs one TU's compile time rather than all of them. // -// Produced by: Glslang Version: 11:16.4.0 +// Produced by: Glslang Version: 11:16.5.0 // Target environment: vulkan1.1 #ifndef VT_VULKAN_VULKAN_SPIRV_H_ #define VT_VULKAN_VULKAN_SPIRV_H_ diff --git a/tests/vt/test_vulkan_backend.cpp b/tests/vt/test_vulkan_backend.cpp index 2c19f594a2..5c7b34b25e 100644 --- a/tests/vt/test_vulkan_backend.cpp +++ b/tests/vt/test_vulkan_backend.cpp @@ -33,6 +33,7 @@ #include "vt/quant.h" #include "vt/vulkan/vulkan_context.h" #include "vt/vulkan/vulkan_spirv.h" +#include "support/test_env.h" // SetEnv/UnsetEnv -- MSVC has no setenv (#603) using vt::Backend; using vt::Device; @@ -60,7 +61,8 @@ TEST_CASE("the committed SPIR-V table is present and well-formed") { // point of the split: at the target shader surface the words must not be // re-parsed by every TU that merely needs the table. const size_t n = vt::vulkan::kSpirvModuleCount; - CHECK(n == 43); // +2: BACKEND-VULKAN-EXL3 (#2530); +15: BACKEND-VULKAN-TQ1_0 + CHECK(n == 44); // +1: BACKEND-VULKAN-GDN (vt_causal_conv1d_fwd); + // +2: BACKEND-VULKAN-EXL3 (#2530); +15: BACKEND-VULKAN-TQ1_0 // (vt_matmul_bt_tq2, vt_matmul_bt_tq2_grouped, vt_matmul_bt_tq2_dev, // vt_moe_gate_up_swiglu_grouped_tq2, vt_matmul_bt_tq2_grouped_dev, // vt_matmul_bt_tq2_dev; VK4 rope/moe) @@ -81,7 +83,8 @@ TEST_CASE("the committed SPIR-V table is present and well-formed") { "vt_rms_norm_wide", "vt_rope_from_cache", "vt_silu_and_mul", // BACKEND-VULKAN-GDN: the GDN / conv1d glue family. - "vt_causal_conv1d_update", "vt_gdn_post_conv", + "vt_causal_conv1d_update", "vt_causal_conv1d_fwd", + "vt_gdn_post_conv", "vt_gdn_state_gather", "vt_gdn_state_scatter", "vt_rms_norm_gated", "vt_sigmoid_gate_bf16", // BACKEND-VULKAN-GDN-CORE: the two recurrences. @@ -285,6 +288,13 @@ TEST_CASE("the committed SPIR-V table records each module's specialization const // q/k dtype, position dtype, llama3 flag, q/k head-dim overrides. REQUIRE(m.spec_id_count == 6); for (uint32_t want = 0; want < 6; ++want) CHECK(m.spec_ids[want] == want); + } else if (std::strcmp(m.name, "vt_causal_conv1d_fwd") == 0) { + // ONE axis, and it is not a dtype: VT_CONV_BLOCKS, the token blocks per + // (sequence, channel). Every dtype is a push-constant code, because the + // five tensors vary independently and a spec axis per tensor would be a + // module explosion for a kernel that runs once per GDN layer. + REQUIRE(m.spec_id_count == 1); + CHECK(m.spec_ids[0] == 0u); } else if (std::strcmp(m.name, "vt_moe_router_topk") == 0) { // E, K, renormalize flag, logits dtype. REQUIRE(m.spec_id_count == 4); @@ -590,6 +600,8 @@ TEST_CASE("Vulkan registers the W0 op set and NOT the unimplemented rest") { vt::OpId::kSigmoidGateBf16, vt::OpId::kRmsNormGated, vt::OpId::kGdnStateGather, vt::OpId::kGdnStateScatter, vt::OpId::kCausalConv1dUpdate, vt::OpId::kGdnPostConv, + // ...and the PREFILL conv, which closes the family. + vt::OpId::kCausalConv1dFwd, // BACKEND-VULKAN-GDN-CORE: the two gated-delta recurrences // themselves, which are where a GDN hybrid's prefill time // actually was. @@ -606,10 +618,9 @@ TEST_CASE("Vulkan registers the W0 op set and NOT the unimplemented rest") { // the two MoE router/combine ops (rope_cos_sin_cache, rope_neox, // moe_router_topk, moe_combine); the reference tier likewise serves // MatmulBTQuant's keep-quant path. Still genuinely unimplemented: the sampler - // beyond greedy argmax (kApplyTemperature) and the PRE-FILL conv - // (kCausalConv1dFwd -- its state write-back needs a different dispatch shape - // than the decode update, see src/vt/vulkan/vulkan_ops.cpp). - for (vt::OpId op : {vt::OpId::kApplyTemperature, vt::OpId::kCausalConv1dFwd}) { + // beyond greedy argmax (kApplyTemperature). The PRE-FILL conv used to be listed + // here and is native now (BACKEND-VULKAN-GDN, vt_causal_conv1d_fwd). + for (vt::OpId op : {vt::OpId::kApplyTemperature}) { CHECK_FALSE(vt::OpRegistered(op, DeviceType::kVULKAN)); } // ...but they no longer THROW, and this assertion used to say they did. @@ -2396,6 +2407,464 @@ TEST_CASE("the decode causal conv1d update runs NATIVELY on Vulkan, state roll i cpu.DestroyQueue(cq); } +TEST_CASE("the prefill causal conv1d runs NATIVELY on Vulkan, in both token mappings") { + if (!VulkanPresent()) return; + auto& ctx = vt::vulkan::VulkanContext::Get(); + Backend& vk = vt::GetBackend(DeviceType::kVULKAN); + Backend& cpu = vt::GetBackend(DeviceType::kCPU); + Queue vq = vk.CreateQueue(); + Queue cq = cpu.CreateQueue(); + const Device vd{DeviceType::kVULKAN, 0}; + const Device cd{DeviceType::kCPU, 0}; + + // A varlen batch of three sequences, lengths 5, 1, 9, one of them with no + // initial state. The shape is chosen for the SPLIT arm below: with three + // sequences of 24 channels the default grid is a single workgroup, a target + // of 5 groups therefore asks for 5 token blocks, and sequence 0 (length 5) + // then gets one token per block -- so t = 1 and t = 2, both < width and both + // reading the carried state, land OUTSIDE block 0 unless the shader gives + // them to it. That is the race the block-0 rule exists for. + // + // FIVE, NOT FOUR, deliberately. The blocks of one (sequence, channel) are + // adjacent invocations; with 4 they sat inside one of llvmpipe's 8-lane + // subgroups in this layout and the pre-fix shader PASSED 5 runs of 5. With 5 + // they straddle a subgroup and it fails 5 of 5. + constexpr int64_t kN = 3, kT = 15, kC = 24, kK = 4, kWidth = kK - 1; + const std::vector qsl = {0, 5, 6, 15}; + const std::vector has = {1, 0, 1}; + const std::vector x = Spread(kT * kC, 2.0f, 71u); + const std::vector w = Spread(kC * kK, 0.5f, 73u); + const std::vector bias = Spread(kC, 0.2f, 79u); + const std::vector state0 = Spread(kN * kC * kWidth, 0.5f, 83u); + + // CPU oracle, once. + Buf cx(cpu, kT * kC, 4), cw(cpu, kC * kK, 4), cb(cpu, kC, 4), co(cpu, kT * kC, 4), + cs(cpu, kN * kC * kWidth, 4), cqsl(cpu, kN + 1, 4), chas(cpu, kN, 4); + std::memcpy(cx.p(), x.data(), x.size() * 4); + std::memcpy(cw.p(), w.data(), w.size() * 4); + std::memcpy(cb.p(), bias.data(), bias.size() * 4); + std::memcpy(cs.p(), state0.data(), state0.size() * 4); + std::memcpy(cqsl.p(), qsl.data(), qsl.size() * 4); + std::memcpy(chas.p(), has.data(), has.size() * 4); + std::memset(co.p(), 0, kT * kC * 4); + { + Tensor xt = Tensor::Contiguous(cx.p(), vt::DType::kF32, cd, {kT, kC}); + Tensor wt = Tensor::Contiguous(cw.p(), vt::DType::kF32, cd, {kC, kK}); + Tensor bt = Tensor::Contiguous(cb.p(), vt::DType::kF32, cd, {kC}); + Tensor ot = Tensor::Contiguous(co.p(), vt::DType::kF32, cd, {kT, kC}); + Tensor st = Tensor::Contiguous(cs.p(), vt::DType::kF32, cd, {kN, kC, kWidth}); + Tensor qt = Tensor::Contiguous(cqsl.p(), vt::DType::kI32, cd, {kN + 1}); + Tensor ht = Tensor::Contiguous(chas.p(), vt::DType::kI32, cd, {kN}); + vt::CausalConv1dFwd(cq, ot, xt, wt, &bt, st, qt, ht, vt::CausalConv1dArgs{}); + } + const std::vector out_ref(co.as(), co.as() + kT * kC); + const std::vector state_ref(cs.as(), cs.as() + kN * kC * kWidth); + + // Restores the knob even when a CHECK below throws out of the case. + struct EnvRestore { + ~EnvRestore() { vllm_test::UnsetEnv("VT_VULKAN_CONV_TARGET_GROUPS"); } + } restore; + + // Arm "" is the default mapping (one invocation per (sequence, channel)); arm + // "5" is the opt-in split. The split runs several times because a race is a + // scheduling outcome: one green dispatch would not show it is absent. + std::vector out_default; + for (const char* target : {"", "5", "5", "5", "5"}) { + CAPTURE(std::string(target)); + vllm_test::SetEnv("VT_VULKAN_CONV_TARGET_GROUPS", target); + + Buf vx(vk, kT * kC, 4), vw(vk, kC * kK, 4), vb(vk, kC, 4), vo(vk, kT * kC, 4), + vs(vk, kN * kC * kWidth, 4), vqsl(vk, kN + 1, 4), vhas(vk, kN, 4); + vk.Copy(vq, vx.p(), x.data(), x.size() * 4); + vk.Copy(vq, vw.p(), w.data(), w.size() * 4); + vk.Copy(vq, vb.p(), bias.data(), bias.size() * 4); + vk.Copy(vq, vs.p(), state0.data(), state0.size() * 4); + vk.Copy(vq, vqsl.p(), qsl.data(), qsl.size() * 4); + vk.Copy(vq, vhas.p(), has.data(), has.size() * 4); + vk.Synchronize(vq); + Tensor xt = Tensor::Contiguous(vx.p(), vt::DType::kF32, vd, {kT, kC}); + Tensor wt = Tensor::Contiguous(vw.p(), vt::DType::kF32, vd, {kC, kK}); + Tensor bt = Tensor::Contiguous(vb.p(), vt::DType::kF32, vd, {kC}); + Tensor ot = Tensor::Contiguous(vo.p(), vt::DType::kF32, vd, {kT, kC}); + Tensor st = Tensor::Contiguous(vs.p(), vt::DType::kF32, vd, {kN, kC, kWidth}); + Tensor qt = Tensor::Contiguous(vqsl.p(), vt::DType::kI32, vd, {kN + 1}); + Tensor ht = Tensor::Contiguous(vhas.p(), vt::DType::kI32, vd, {kN}); + vt::CausalConv1dFwd(vq, ot, xt, wt, &bt, st, qt, ht, vt::CausalConv1dArgs{}); + vk.Synchronize(vq); + + CHECK(RanNative(vt::OpId::kCausalConv1dFwd)); + // The block count is the mechanism, and it is invisible in the numbers by + // design, so the specialization VALUE is asserted, not just the module. + { + const std::string want = std::string("vt_causal_conv1d_fwd|") + + (target[0] == '\0' ? "1" : "5"); + const std::vector keys = ctx.PipelineKeysFor("vt_causal_conv1d_fwd"); + std::string joined; + for (const std::string& k : keys) joined += k + " "; + CAPTURE(joined); + CHECK(std::find(keys.begin(), keys.end(), want) != keys.end()); + } + + std::vector out_got(kT * kC), state_got(kN * kC * kWidth); + vk.Copy(vq, out_got.data(), vo.p(), out_got.size() * 4); + vk.Copy(vq, state_got.data(), vs.p(), state_got.size() * 4); + vk.Synchronize(vq); + + const double nmse = NmseOf(out_ref, out_got); + MESSAGE("causal_conv1d_fwd NMSE vs the CPU oracle: " << nmse); + CHECK(nmse <= kGdnNmseTol); + // The state write-back moves RAW x samples, so it is exact, not a tolerance. + CHECK(std::memcmp(state_got.data(), state_ref.data(), state_got.size() * 4) == 0); + // Spelled out independently of the oracle for the two sequences at least + // `width` long: their new state is their own last `width` input rows. + for (int64_t s : {int64_t{0}, int64_t{2}}) { + const int64_t end = qsl[static_cast(s) + 1]; + for (int64_t c = 0; c < kC; ++c) { + for (int64_t j = 0; j < kWidth; ++j) { + CAPTURE(s); + CAPTURE(c); + CAPTURE(j); + CHECK(state_got[static_cast((s * kC + c) * kWidth + j)] == + x[static_cast((end - kWidth + j) * kC + c)]); + } + } + } + // The split changes which invocation computes an element, never the order + // of its own k-tap sum, so both mappings must agree to the bit. + if (target[0] == '\0') { + out_default = out_got; + } else { + CHECK(std::memcmp(out_got.data(), out_default.data(), out_got.size() * 4) == 0); + } + } + + vk.DestroyQueue(vq); + cpu.DestroyQueue(cq); +} + +namespace { + +// One prefill-conv call on one backend, everything uploaded from host vectors +// and downloaded back. Shared by the edge-case and compressed-state cases below +// so the CPU oracle and the Vulkan arm run the SAME setup code. +struct ConvFwdCase { + std::vector qsl; + std::vector has; // one flag per sequence + bool his_i8 = false; + int64_t his_byte_off = 0; // i8 view start within its allocation + int64_t c = 8, k = 4; + int64_t x_pad = 0; // extra elements per x row (padded stride) + int64_t state_extra = 0; // conv_state.shape[2] = k - 1 + state_extra + bool with_bias = true; + bool silu = true; + vt::DType state_dt = vt::DType::kF32; + vt::DType op_dt = vt::DType::kF32; // x, weight and bias storage + vt::DType out_dt = vt::DType::kF32; + std::vector x, w, bias, state0; // state0 is [n, c, k-1+state_extra] +}; + +struct ConvFwdResult { + std::vector out, state; // state as f32 whatever its storage dtype +}; + +// f32 values to the bytes of `dt` (f32, f16 or bf16), and back. +std::vector ConvEncode(const std::vector& v, vt::DType dt) { + std::vector out(v.size() * (dt == vt::DType::kF32 ? 4 : 2)); + for (size_t i = 0; i < v.size(); ++i) { + if (dt == vt::DType::kF32) { + std::memcpy(out.data() + i * 4, &v[i], 4); + } else { + const uint16_t h = dt == vt::DType::kBF16 ? vt::F32ToBF16(v[i]) : vt::F32ToF16(v[i]); + std::memcpy(out.data() + i * 2, &h, 2); + } + } + return out; +} + +std::vector ConvDecode(const std::vector& b, vt::DType dt) { + const size_t es = dt == vt::DType::kF32 ? 4 : 2; + std::vector out(b.size() / es); + for (size_t i = 0; i < out.size(); ++i) { + if (dt == vt::DType::kF32) { + std::memcpy(&out[i], b.data() + i * 4, 4); + } else { + uint16_t h; + std::memcpy(&h, b.data() + i * 2, 2); + out[i] = dt == vt::DType::kBF16 ? vt::BF16ToF32(h) : vt::F16ToF32(h); + } + } + return out; +} + +ConvFwdResult RunConvFwd(Backend& b, Queue q, Device d, const ConvFwdCase& cs) { + const int64_t n = static_cast(cs.qsl.size()) - 1; + const int64_t t = cs.qsl.back(); + const int64_t rs = cs.c + cs.x_pad; + const int64_t sw = cs.k - 1 + cs.state_extra; + const int64_t sn = n * cs.c * sw; + const size_t st_bytes = cs.state_dt == vt::DType::kF32 ? 4 : 2; + const std::vector xe = ConvEncode(cs.x, cs.op_dt); + const std::vector we = ConvEncode(cs.w, cs.op_dt); + const std::vector be = ConvEncode(cs.bias, cs.op_dt); + const size_t oes = cs.out_dt == vt::DType::kF32 ? 4 : 2; + Buf bx(b, xe.size() / 4 + 1, 4), bw(b, we.size() / 4 + 1, 4), bb(b, be.size() / 4 + 1, 4), + bo(b, static_cast(std::max(1, t) * cs.c), oes), + bs(b, static_cast(sn), st_bytes), bq(b, static_cast(n + 1), 4), + bh(b, static_cast(n + 8), 4); + b.Copy(q, bx.p(), xe.data(), xe.size()); + b.Copy(q, bw.p(), we.data(), we.size()); + if (cs.with_bias) b.Copy(q, bb.p(), be.data(), be.size()); + if (cs.state_dt == vt::DType::kF32) { + b.Copy(q, bs.p(), cs.state0.data(), cs.state0.size() * 4); + } else { + std::vector s16(cs.state0.size()); + for (size_t i = 0; i < s16.size(); ++i) s16[i] = vt::F32ToBF16(cs.state0[i]); + b.Copy(q, bs.p(), s16.data(), s16.size() * 2); + } + b.Copy(q, bq.p(), cs.qsl.data(), cs.qsl.size() * 4); + // Flags: i32 at offset 0, or i8 bytes starting at his_byte_off of a buffer + // whose other bytes are a pattern that would flip the answer if read. + std::vector hb(static_cast((n + 8) * 4), 0xA5); + if (cs.his_i8) { + for (int64_t s = 0; s < n; ++s) hb[static_cast(cs.his_byte_off + s)] = cs.has[s]; + } else { + for (int64_t s = 0; s < n; ++s) { + const int32_t v = cs.has[static_cast(s)]; + std::memcpy(hb.data() + s * 4, &v, 4); + } + } + b.Copy(q, bh.p(), hb.data(), hb.size()); + const std::vector seed = ConvEncode( + std::vector(static_cast(std::max(1, t) * cs.c), -7.0f), cs.out_dt); + b.Copy(q, bo.p(), seed.data(), seed.size()); + b.Synchronize(q); + + Tensor xt = Tensor::Contiguous(bx.p(), cs.op_dt, d, {t, cs.c}); + xt.stride[0] = rs; + Tensor wt = Tensor::Contiguous(bw.p(), cs.op_dt, d, {cs.c, cs.k}); + Tensor bt = Tensor::Contiguous(bb.p(), cs.op_dt, d, {cs.c}); + Tensor ot = Tensor::Contiguous(bo.p(), cs.out_dt, d, {t, cs.c}); + Tensor st = Tensor::Contiguous(bs.p(), cs.state_dt, d, {n, cs.c, sw}); + Tensor qt = Tensor::Contiguous(bq.p(), vt::DType::kI32, d, {n + 1}); + Tensor ht = cs.his_i8 + ? Tensor::Contiguous(static_cast(bh.p()) + cs.his_byte_off, + vt::DType::kI8, d, {n}) + : Tensor::Contiguous(bh.p(), vt::DType::kI32, d, {n}); + vt::CausalConv1dArgs args; + args.silu_activation = cs.silu; + vt::CausalConv1dFwd(q, ot, xt, wt, cs.with_bias ? &bt : nullptr, st, qt, ht, args); + b.Synchronize(q); + + ConvFwdResult r; + std::vector ob(static_cast(t * cs.c) * oes); + b.Copy(q, ob.data(), bo.p(), ob.size()); + r.state.resize(static_cast(sn)); + if (cs.state_dt == vt::DType::kF32) { + b.Copy(q, r.state.data(), bs.p(), r.state.size() * 4); + } else { + std::vector s16(r.state.size()); + b.Copy(q, s16.data(), bs.p(), s16.size() * 2); + for (size_t i = 0; i < s16.size(); ++i) r.state[i] = vt::BF16ToF32(s16[i]); + } + b.Synchronize(q); + r.out = ConvDecode(ob, cs.out_dt); + return r; +} + +ConvFwdCase MakeConvFwdCase(std::vector qsl, std::vector has, int64_t c, + int64_t k, int64_t x_pad, uint32_t seed) { + ConvFwdCase cs; + cs.qsl = std::move(qsl); + cs.has = std::move(has); + cs.c = c; + cs.k = k; + cs.x_pad = x_pad; + const int64_t n = static_cast(cs.qsl.size()) - 1; + const int64_t t = cs.qsl.back(); + cs.x = Spread(static_cast(std::max(1, t) * (c + x_pad)), 2.0f, seed); + cs.w = Spread(static_cast(c * k), 0.5f, seed + 1); + cs.bias = Spread(static_cast(c), 0.2f, seed + 2); + cs.state0 = Spread(static_cast(n * c * (k - 1)), 0.5f, seed + 3); + return cs; +} + +} // namespace + +TEST_CASE("the prefill causal conv1d matches the CPU oracle on its edge shapes") { + if (!VulkanPresent()) return; + Backend& vk = vt::GetBackend(DeviceType::kVULKAN); + Backend& cpu = vt::GetBackend(DeviceType::kCPU); + Queue vq = vk.CreateQueue(); + Queue cq = cpu.CreateQueue(); + const Device vd{DeviceType::kVULKAN, 0}; + const Device cd{DeviceType::kCPU, 0}; + + // Lengths 5, 0, 1, 7: an EMPTY sequence, and a length-1 sequence WITH initial + // state, whose new state is two shifted old taps plus one x sample. Flags + // 1,1,1,0. x rows are padded by 3 elements, so the row stride is not c. + ConvFwdCase base = MakeConvFwdCase({0, 5, 5, 6, 13}, {1, 1, 1, 0}, 40, 4, 3, 211u); + + struct Variant { + const char* name; + bool his_i8; + int64_t his_byte_off; + bool with_bias, silu; + vt::DType op_dt = vt::DType::kF32, out_dt = vt::DType::kF32; + }; + // i8 flags at every byte phase of a 32-bit word, the read a 4-aligned + // assumption gets wrong; and bias/silu both ways. + for (const Variant& v : {Variant{"i32 flags, bias, silu", false, 0, true, true}, + Variant{"i8 flags @0, no bias, no silu", true, 0, false, false}, + Variant{"i8 flags @1", true, 1, true, true}, + Variant{"i8 flags @2, no bias", true, 2, false, true}, + Variant{"i8 flags @3, no silu", true, 3, true, false}, + // Reduced-precision operands and output: the shader + // reads them through the 16-bit views and its own + // dtype codes, the CPU reference through LoadF32. + Variant{"bf16 x/w/bias, bf16 out", false, 0, true, true, + vt::DType::kBF16, vt::DType::kBF16}, + Variant{"f16 x/w/bias, f32 out", true, 1, true, true, + vt::DType::kF16, vt::DType::kF32}}) { + CAPTURE(std::string(v.name)); + ConvFwdCase cs = base; + cs.his_i8 = v.his_i8; + cs.his_byte_off = v.his_byte_off; + cs.with_bias = v.with_bias; + cs.silu = v.silu; + cs.op_dt = v.op_dt; + cs.out_dt = v.out_dt; + const auto before = vt::GetOpProviderStats(vt::OpId::kCausalConv1dFwd, DeviceType::kVULKAN); + const ConvFwdResult ref = RunConvFwd(cpu, cq, cd, cs); + const ConvFwdResult got = RunConvFwd(vk, vq, vd, cs); + const auto after = vt::GetOpProviderStats(vt::OpId::kCausalConv1dFwd, DeviceType::kVULKAN); + CHECK(RanNative(vt::OpId::kCausalConv1dFwd)); + CHECK(after.declines == before.declines); + CHECK(NmseOf(ref.out, got.out) <= kGdnNmseTol); + CHECK(std::memcmp(got.state.data(), ref.state.data(), ref.state.size() * 4) == 0); + } + + vk.DestroyQueue(vq); + cpu.DestroyQueue(cq); +} + +TEST_CASE("the prefill causal conv1d keeps a bf16 conv_state IN PLACE, bit-exact vs the f32 arm") { + if (!VulkanPresent()) return; + REQUIRE(vt::GetBackend(DeviceType::kVULKAN).SupportsCompressedConvState()); + Backend& vk = vt::GetBackend(DeviceType::kVULKAN); + Queue vq = vk.CreateQueue(); + const Device vd{DeviceType::kVULKAN, 0}; + + // The CPU reference reads f32 state only, so the oracle here is the native f32 + // arm on bf16-REPRESENTABLE inputs -- the same shape as the compressed-state + // update case above. With every x and state value exactly a bf16, the + // compressed arm must read the same taps, produce the same outputs, and store + // the same raw samples back. + ConvFwdCase cs = MakeConvFwdCase({0, 5, 6, 15}, {1, 0, 1}, 24, 4, 0, 307u); + for (float& v : cs.x) v = vt::BF16ToF32(vt::F32ToBF16(v)); + for (float& v : cs.state0) v = vt::BF16ToF32(vt::F32ToBF16(v)); + const ConvFwdResult f32_arm = RunConvFwd(vk, vq, vd, cs); + cs.state_dt = vt::DType::kBF16; + const ConvFwdResult bf16_arm = RunConvFwd(vk, vq, vd, cs); + CHECK(RanNative(vt::OpId::kCausalConv1dFwd)); + CHECK(std::memcmp(bf16_arm.out.data(), f32_arm.out.data(), f32_arm.out.size() * 4) == 0); + CHECK(std::memcmp(bf16_arm.state.data(), f32_arm.state.data(), f32_arm.state.size() * 4) == 0); + + vk.DestroyQueue(vq); +} + +TEST_CASE("the prefill causal conv1d DECLINES a conv_state row wider than K-1") { + if (!VulkanPresent()) return; + REQUIRE(vt::ReferenceTierEligible(DeviceType::kVULKAN)); + Backend& vk = vt::GetBackend(DeviceType::kVULKAN); + Backend& cpu = vt::GetBackend(DeviceType::kCPU); + Queue vq = vk.CreateQueue(); + Queue cq = cpu.CreateQueue(); + const Device vd{DeviceType::kVULKAN, 0}; + const Device cd{DeviceType::kCPU, 0}; + + // A speculative-decode-widened row (K-1 + 2). The CPU reference addresses + // rows with stride K-1; the op layer admits the wider row; the native kernel + // declines rather than pick one, so the answer is the reference's, byte for + // byte, and the decline is counted. Whether the reference's stride is right + // for widened rows is a question about the CPU kernel, not tested here. + ConvFwdCase cs = MakeConvFwdCase({0, 4, 9}, {1, 1}, 8, 4, 0, 401u); + cs.state_extra = 2; + cs.state0 = Spread(static_cast(2 * 8 * (3 + 2)), 0.5f, 404u); + const ConvFwdResult ref = RunConvFwd(cpu, cq, cd, cs); + const auto before = vt::GetOpProviderStats(vt::OpId::kCausalConv1dFwd, DeviceType::kVULKAN); + const ConvFwdResult got = RunConvFwd(vk, vq, vd, cs); + const auto after = vt::GetOpProviderStats(vt::OpId::kCausalConv1dFwd, DeviceType::kVULKAN); + CHECK(after.declines == before.declines + 1); + CHECK(std::memcmp(got.out.data(), ref.out.data(), ref.out.size() * 4) == 0); + CHECK(std::memcmp(got.state.data(), ref.state.data(), ref.state.size() * 4) == 0); + + vk.DestroyQueue(vq); + cpu.DestroyQueue(cq); +} + +TEST_CASE("the prefill causal conv1d DECLINES a width past its window, and stays correct") { + if (!VulkanPresent()) return; + REQUIRE(vt::ReferenceTierEligible(DeviceType::kVULKAN)); + Backend& vk = vt::GetBackend(DeviceType::kVULKAN); + Backend& cpu = vt::GetBackend(DeviceType::kCPU); + Queue vq = vk.CreateQueue(); + Queue cq = cpu.CreateQueue(); + const Device vd{DeviceType::kVULKAN, 0}; + const Device cd{DeviceType::kCPU, 0}; + + // K = 10 -> width 9, one past the shader's private window. The native kernel + // must hand the call to the next provider, not throw and not read past the + // array -- and the answer must still be the CPU oracle's. + constexpr int64_t kN = 1, kT = 12, kC = 8, kK = 10, kWidth = kK - 1; + const std::vector qsl = {0, 12}; + const std::vector has = {1}; + const std::vector x = Spread(kT * kC, 2.0f, 89u); + const std::vector w = Spread(kC * kK, 0.5f, 97u); + const std::vector state0 = Spread(kN * kC * kWidth, 0.5f, 101u); + + struct Arm { + Backend& b; + Queue qu; + Device d; + }; + std::vector outs[2], states[2]; + for (int arm = 0; arm < 2; ++arm) { + Arm a = arm == 0 ? Arm{cpu, cq, cd} : Arm{vk, vq, vd}; + Buf bx(a.b, kT * kC, 4), bw(a.b, kC * kK, 4), bo(a.b, kT * kC, 4), + bs(a.b, kN * kC * kWidth, 4), bq(a.b, kN + 1, 4), bh(a.b, kN, 4); + a.b.Copy(a.qu, bx.p(), x.data(), x.size() * 4); + a.b.Copy(a.qu, bw.p(), w.data(), w.size() * 4); + a.b.Copy(a.qu, bs.p(), state0.data(), state0.size() * 4); + a.b.Copy(a.qu, bq.p(), qsl.data(), qsl.size() * 4); + a.b.Copy(a.qu, bh.p(), has.data(), has.size() * 4); + a.b.Synchronize(a.qu); + Tensor xt = Tensor::Contiguous(bx.p(), vt::DType::kF32, a.d, {kT, kC}); + Tensor wt = Tensor::Contiguous(bw.p(), vt::DType::kF32, a.d, {kC, kK}); + Tensor ot = Tensor::Contiguous(bo.p(), vt::DType::kF32, a.d, {kT, kC}); + Tensor st = Tensor::Contiguous(bs.p(), vt::DType::kF32, a.d, {kN, kC, kWidth}); + Tensor qt = Tensor::Contiguous(bq.p(), vt::DType::kI32, a.d, {kN + 1}); + Tensor ht = Tensor::Contiguous(bh.p(), vt::DType::kI32, a.d, {kN}); + const auto before = vt::GetOpProviderStats(vt::OpId::kCausalConv1dFwd, DeviceType::kVULKAN); + vt::CausalConv1dFwd(a.qu, ot, xt, wt, nullptr, st, qt, ht, vt::CausalConv1dArgs{}); + a.b.Synchronize(a.qu); + if (arm == 1) { + const auto after = vt::GetOpProviderStats(vt::OpId::kCausalConv1dFwd, DeviceType::kVULKAN); + CHECK(after.declines == before.declines + 1); + } + outs[arm].resize(kT * kC); + states[arm].resize(kN * kC * kWidth); + a.b.Copy(a.qu, outs[arm].data(), bo.p(), outs[arm].size() * 4); + a.b.Copy(a.qu, states[arm].data(), bs.p(), states[arm].size() * 4); + a.b.Synchronize(a.qu); + } + // The fallback IS the CPU kernel, so this is exact, not a tolerance. + CHECK(std::memcmp(outs[1].data(), outs[0].data(), outs[0].size() * 4) == 0); + CHECK(std::memcmp(states[1].data(), states[0].data(), states[0].size() * 4) == 0); + + vk.DestroyQueue(vq); + cpu.DestroyQueue(cq); +} + // BACKEND-VULKAN-DEVICE-RESIDENT. The capability the model's indexed state-I/O // path needs: a COMPRESSED (bf16) conv_state addressed IN PLACE, which is what // Backend::SupportsCompressedConvState() advertises. The oracle is the arm this