Skip to content

Add Jinja2 backward kernel and PT2 wrapper templates for nobag embedding#83

Draft
aagalleg wants to merge 36 commits into
intel:mainfrom
aagalleg:feat/lookup_backward_pass_templates
Draft

Add Jinja2 backward kernel and PT2 wrapper templates for nobag embedding#83
aagalleg wants to merge 36 commits into
intel:mainfrom
aagalleg:feat/lookup_backward_pass_templates

Conversation

@aagalleg

@aagalleg aagalleg commented Jul 3, 2026

Copy link
Copy Markdown
Contributor

This PR adds the Jinja2 templates that generate the SYCL backward pass
kernels and PyTorch 2.0 compilation wrappers for no-bag (sequence)
unweighted embedding lookups on Intel XPU devices. Together with the
forward templates, this completes the full training pass template set.

Depends on #82

Changes

Backward Kernel Templates (src/codegen/training/backward/)

  • embedding_backward_split_kernel_templates.h: Jinja2 template
    generating both backward kernel variants for dense and split
    (rowwise Adagrad) paths:

    • SplitEmbeddingNobagBackwardCodegen{Dense|RowwiseAdagrad}UnweightedKernelWarpPerRow:
      warp-per-row kernel for short segments (SL < max_segment_length_per_warp),
      uses compute_grad_sum_unweighted_nobag for sub-group cooperative
      gradient accumulation
    • SplitEmbeddingNobagBackwardCodegen{Dense|RowwiseAdagrad}UnweightedKernelCtaPerRow:
      CTA-per-row kernel for long segments, processes all gradient
      contributions for a single embedding row
    • store_grad_sum helper (dense only): writes the accumulated
      gradient vector to grad_dev_weights with compile-time unrolling
      when kUseVecBlocking = false
    • split_rowwise_adagrad_table_update_kernel (split only): applies
      rowwise Adagrad weight updates with optional stochastic rounding,
      weight decay, and max-norm gradient clipping
  • embedding_backward_nobag_unweighted_host_template.cpp: Jinja2
    template generating the host-side backward dispatch function
    split_embedding_nobag_backward_codegen_{dense|rowwise_adagrad}_unweighted_exact_xpu.
    Three-step dispatch pipeline:

    1. SplitEmbeddingBackwardFindLongSegments kernel: classifies each
      run-length encoded segment as short or long
    2. CTA-per-row kernel for long segments (submitted first for
      pipelining)
    3. Warp-per-row kernel for short segments
      — Calls transpose_embedding_input for CSR→CSC index sorting before
      kernel dispatch. The split variant additionally handles momentum
      tensors, LXU cache lookups, and deterministic algorithm selection

PT2 Wrapper Template (src/codegen/training/pt2/)

  • embedding_forward_nobag_unweighted_pt2_wrapper_template.cpp:
    Single Jinja2 template (controlled by is_forward) generating both
    the forward and backward PT2 compilation wrappers for the split path:
    • Forward wrapper: delegates to
      split_embedding_nobag_forward_unweighted_xpu via
      torch::Dispatcher schema lookup
    • Backward wrapper: delegates to
      split_embedding_nobag_backward_codegen_rowwise_adagrad_unweighted_exact_xpu
      with full rowwise Adagrad parameter passthrough
    • Both wrappers registered via TORCH_LIBRARY_IMPL under the XPU
      dispatch key

cc: @flezaalv

aagalleg and others added 30 commits June 18, 2026 22:37
- Add invert_permute kernel to CMake build
- Implement invert_permute Python wrapper in ops.py
- Register invert_permute operator with schema existence check
- Add torch_library.h utility for schema validation
Add SYCL/XPU kernel implementation for invert_permute operation.
Add complete test coverage for invert_permute operator on XPU
devices, covering correctness, validation, parity, and performance.

Test coverage includes:
- Correctness tests for int32/int64 with edge cases (empty, single
  element, identity, reverse, random permutations)
- Input validation tests for invalid dimensions and dtypes
- Meta function tests for torch.compile compatibility
- PyTorch opcheck validation for operator conventions
- Parametric tests with varying sizes (1 to 1M elements)
- CPU-XPU parity tests to ensure consistent results
- Performance benchmarks measuring execution time and bandwidth
- CMakeLists: add permute_1d_sparse_data.cpp to build sources
- ops.py: add Python wrapper with type hints
- ops_registry.cpp: register operator schema in fbgemm namespace
Implement SYCL/XPU kernel implementation of permute_1D_sparse_data
operator for sparse jagged/1D format data permutation.
Add SYCL port of FBGEMM's asynchronous_complete_cumsum operator for
Intel XPU devices. The operator computes a complete cumulative sum
with a leading zero (e.g., [a, b, c] → [0, a, a+b, a+b+c]).
Integrate asynchronous_complete_cumsum operator into fbgemm-xpu:
- Add Python wrapper with complete cumsum documentation
- Register operator schema in torch library
- Include implementation in CMake build
Add comprehensive test suite for asynchronous_complete_cumsum
operator covering:
- Basic functionality with int32 and int64 dtypes
- Empty tensor handling
- Random input validation with numpy reference
Add SYCL infrastructure headers from intel/torch-xpu-ops/
to support advanced kernel implementations:
- DeviceProperties.h: Device capability queries and work group sizing
- SYCLContext.h: SYCL context management and namespace aliases
- SYCLHelpers.h: SYCL kernel submission and utility functions
- TensorInfo.h: Tensor metadata and dimension handling structures
- TensorOptions.h: Tensor configuration and options management
- Runtime.h: SYCL runtime utilities
- Macros.h: Common macro definitions
- Scalar.h: Scalar type conversion utilities

These headers provide the foundation for implementing 2D sparse data
permutation and other complex SYCL operations on XPU devices.
Add foundational utility headers and implementations to support
complex SYCL kernel operations:
- utils.h/cpp: Core constants, type definitions, kernel launch
  helpers, and device property queries
- dispatch_macros.h: Type dispatch macros for handling multiple
  data types (int32, int64, float, etc.)
- tensor_utils.h: Tensor manipulation and metadata utilities
- function_types.h: Symbol visibility definitions for shared
  library exports

These utilities provide essential infrastructure for implementing
2D sparse data permutation and other advanced operators on XPU
devices, including work group sizing, kernel launch helpers, and
type-safe dispatching mechanisms.
Add SYCL port of FBGEMM's permute_2D_sparse_data operator for
Intel XPU devices. This operator permutes 2D sparse data including
lengths [T, B], indices, and optional weights according to a
permutation vector, commonly used for reordering embedding table
features.

Implementation includes:
- SYCL kernels: permute_2D_lengths_kernel and permute_2D_data_kernel
- Host function: permute_2D_sparse_data_xpu
Integrate permute_2D_sparse_data operator into fbgemm-xpu:
- Add Python wrapper with type hints and documentation
- Register operator schema in torch library
- Include implementation files in CMake build (utils.cpp, SYCL
  kernels, and operator implementation)
Add comprehensive test suite for permute_2D_sparse_data operator
covering:
- Basic functionality with int32 and int64 data types
- Sparse data with and without weights
- Permutations with repeated indices
- Exact value validation
- CPU-XPU consistency verification
Fixes CMake configuration and import ordering to properly build and load
the block_bucketize_sparse_features XPU operator.

- Configure CMake for XPU-only PyTorch builds
- Import torch before _C extension to load libtorch.so dependencies
- Adjust test imports for consistency

All 18 tests passing.
… generation

Add jinja_environment.py module to support template-based code
generation for FBGEMM-XPU kernels.
Add common.py module with CodeTemplate class that provides
functionality for loading Jinja2 templates, rendering them with
context variables, and writing generated files with appropriate
headers.
Add torch_type_utils.py module with utilities for handling PyTorch
data types in template-based code generation.
Add generate_forward_split.py and generate_backward_split.py scripts
for generating SYCL embedding kernels from Jinja2 templates.
Add backward_utils.cpp and backward_utils.h with SYCL ports of
FBGEMM backward pass utilities for embedding operations.
…mization

Add vec4.h header implementing 4-element vectorized data structures
for efficient memory access in embedding operations.
Add feature_gates module to enable/disable features at runtime via environment variables.
Add pt2_arg_utils.h header defining argument index enumerations for
PyTorch 2 compiled embedding operations.
Add split_embeddings_cache_xpu.h header defining indices for UVM
(Unified Virtual Memory) cache performance statistics.
…version

Add stochastic_rounding.h header implementing stochastic rounding
algorithms for float-to-half precision conversions in embedding
operations.
Add weight_row.h header implementing abstractions for efficient
access to embedding table weight rows with support for both direct
table access and cache-resident data.
aagalleg added 6 commits July 3, 2026 00:04
Add Jinja2 template for generating optimized SYCL forward kernels
for small embedding dimensions (D <= 32).
Add embedding_forward_split_kernel_template.h as a Jinja2 template
for generating the main SYCL forward kernel for no-bag (sequence)
embeddings.
Add embedding_forward_nobag_unweighted_host_template.cpp as a
Jinja2 template for generating the host-side dispatch function for
no-bag (sequence) embedding forward passes.
…plit

Add embedding_backward_split_kernel_templates.h as a Jinja2 template
for generating SYCL backward kernels for no-bag unweighted embedding
lookups. Generates both dense (gradient-only) and split (rowwise
Adagrad optimizer) variants.
…ghted

Add embedding_backward_nobag_unweighted_host_template.cpp as a
Jinja2 template for generating the host-side backward dispatch
function for no-bag (sequence) embedding passes. Generates both
dense (gradient accumulation) and split (rowwise Adagrad) variants.
…ding

Add embedding_forward_nobag_unweighted_pt2_wrapper_template.cpp as a
Jinja2 template for generating PyTorch 2.0 compilation wrapper
functions for split embedding nobag unweighted operations. The
single template generates both forward and backward wrappers
(controlled by the is_forward flag).
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants