Skip to content

Fix loading HF tokenizers with no-op normalizer and non-tiling Split regexes - #225

Merged
larryliu0820 merged 1 commit into
meta-pytorch:mainfrom
dongjiang1989:fix-hf-split-and-noop-normalizer
Oct 19, 2026
Merged

Fix loading HF tokenizers with no-op normalizer and non-tiling Split regexes#225
larryliu0820 merged 1 commit into
meta-pytorch:mainfrom
dongjiang1989:fix-hf-split-and-noop-normalizer

Conversation

@dongjiang1989

@dongjiang1989 dongjiang1989 commented Oct 15, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Problem

tokenizer.json files that are valid per the HuggingFace tokenizers spec can fail to load or encode incorrectly. Found while enabling CI for XHToken/Spark-X2.5 models in executorch (pytorch/executorch#23525).

Bug 1: empty Sequence normalizer rejected

Spark-X2.5 ships "normalizer": {"type": "Sequence", "normalizers": []}. An empty sequence is valid in the Rust implementation (it normalizes nothing), but NormalizerConfig::create() threw Missing normalizers for Normalizer of type Sequence, so HFTokenizer::load() failed with LoadFailure.

Fix: allow an empty normalizers list; SequenceNormalizer with no children is already the identity.

Bug 2: Regex Split dropped non-matching text

RegexPreTokenizer treated every Regex-pattern Split as "return the matches themselves". That is only correct for invert=true, or for regexes that tile the input (GPT-2 / llama3 style), which is why it went unnoticed.

Per the HF spec, Split with invert=false uses matches as delimiters: text between matches must be kept, and the behavior field (Removed / Isolated / MergedWithPrevious) applies to the delimiters.

Spark-X2.5 isolates digits with {"pattern": {"Regex": "\\p{N}{1,3}"}, "behavior": "Isolated", "invert": false}. For a prompt with no digits there are no matches, so the old code returned zero pieces and encode() produced zero tokens for ordinary text.

Fix: a Regex pattern with an explicit behavior and invert=false now uses delimiter semantics (the same code path String patterns already used). invert=true and behavior-less configs keep the previous behavior, so existing configs (GPT-2, llama3, tekken, tiktoken) are unaffected — full ctest suite passes (15/15, including test_tekken, test_hf_tokenizer, test_pre_tokenizer).

Tests

  • NormalizerTest.EmptySequenceNormalizerFromConfig — empty Sequence is a no-op
  • PreTokenizerConfigTest.SplitWithRegexIsolatedKeepsNonMatchingText — non-tiling regex keeps text between matches; no-match input passes through whole
  • PreTokenizerConfigTest.SplitWithRegexInvertTrueReturnsMatches — invert=true unchanged

End-to-end verification

Built against the real XHToken/Spark-X2.5-1.7B/tokenizer.json (10MB BPE, Sequence pre-tokenizer with digit/CJK isolation, lookahead regex via PCRE2 fallback):

load: OK
encode("Once upon a time,") -> [19327 6420 259 1279 25]
decode -> "Once upon a time,"  (exact round-trip)

@dongjiang1989
Fix loading of HF tokenizers with no-op normalizer and non-tiling Spl…
4421403 
…it regexes

Two fixes for tokenizer.json files that are valid per the HuggingFace
tokenizers spec but failed to load or encode incorrectly:

1. NormalizerConfig::create() rejected an empty Sequence normalizer
   ({"type": "Sequence", "normalizers": []}). An empty list is valid
   and acts as a no-op, matching the Rust implementation. Spark-X2.5
   tokenizer.json ships this form and failed to load.

2. RegexPreTokenizer treated every Regex-pattern Split as "return the
   matches", which is only correct for invert=true or regexes that tile
   the input (GPT-2 / llama3 style). Per the HF spec, a Split with
   invert=false uses the matches as delimiters: non-matching text must
   be kept and the behavior field applies. Without this, patterns that
   do not tile the input (e.g. Spark-X2.5 digit isolation \p{N}{1,3}
   with behavior=Isolated) silently dropped all non-matching text, so
   encode() returned zero tokens for ordinary prompts. Now a Regex
   pattern with an explicit behavior and invert=false uses delimiter
   semantics; invert=true and behavior-less configs keep the previous
   behavior.

Verified end-to-end with the real XHToken/Spark-X2.5-1.7B tokenizer.json:
load OK, encode("Once upon a time,") -> [19327 6420 259 1279 25],
decode round-trips exactly.

This PR was authored with AI assistance using Claude Code.
@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Oct 15, 2026
@dongjiang1989 dongjiang1989 mentioned this pull request Oct 15, 2026
Open
dongjiang1989 added a commit to XHToken/executorch that referenced this pull request Oct 15, 2026
@dongjiang1989
Add CI integration for Spark-X2.5 models
c47aed0 
This commit adds CI test support for Spark-X2.5-1.7B and Spark-X2.5-4B
models to the ExecuTorch test infrastructure.

Changes:
- Added spark_x2_5_1_7b and spark_x2_5_4b to test_torchao_huggingface_checkpoints.sh
- Added both models to CI test matrix in trunk.yml
- Configured XNNPACK (x86_64) and TorchAO (ARM64) backend testing
- Use --local-dir for hf download to avoid read-only /mnt/hf_cache issues
- convert_weights now rewrites the no-op normalizer in Spark-X2.5
  tokenizer.json (empty Sequence, rejected by older revisions of the C++
  HF tokenizer) to null, which is semantically identical, so llama_main
  can load it. The root cause is also fixed upstream in
  meta-pytorch/tokenizers#225; the submodule pin will be bumped once
  that lands.

The Spark-X2.5 model infrastructure already exists in the codebase.
This commit completes the integration by adding automated CI testing.

Parts of this commit were authored with AI assistance using Claude Code.

Signed-off-by: dongjiang <dongjiang1989@126.com>
@dongjiang1989

dongjiang1989 commented Oct 19, 2026

Copy link
Copy Markdown
Contributor Author

cc @larryliu0820 Can you review this PR? Thx

larryliu0820
larryliu0820 approved these changes Oct 19, 2026
View reviewed changes

@larryliu0820 larryliu0820 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me

@larryliu0820
larryliu0820 merged commit 0d88e86 into meta-pytorch:main Oct 19, 2026
5 checks passed
@dongjiang1989
dongjiang1989 deleted the fix-hf-split-and-noop-normalizer branch October 20, 2026 00:37
@dongjiang1989

dongjiang1989 commented Oct 20, 2026

Copy link
Copy Markdown
Contributor Author

Thanks @larryliu0820

dongjiang1989 added a commit to XHToken/executorch that referenced this pull request Oct 23, 2026
@dongjiang1989
Add CI integration for Spark-X2.5 models
dee122e 
This commit adds CI test support for Spark-X2.5-1.7B and Spark-X2.5-4B
models to the ExecuTorch test infrastructure.

Changes:
- Added spark_x2_5_1_7b and spark_x2_5_4b to test_torchao_huggingface_checkpoints.sh
- Added both models to CI test matrix in trunk.yml
- Configured XNNPACK (x86_64) and TorchAO (ARM64) backend testing
- Use --local-dir for hf download to avoid read-only /mnt/hf_cache issues
- convert_weights now rewrites the no-op normalizer in Spark-X2.5
  tokenizer.json (empty Sequence, rejected by older revisions of the C++
  HF tokenizer) to null, which is semantically identical, so llama_main
  can load it. The root cause is also fixed upstream in
  meta-pytorch/tokenizers#225; the submodule pin will be bumped once
  that lands.

The Spark-X2.5 model infrastructure already exists in the codebase.
This commit completes the integration by adding automated CI testing.

Parts of this commit were authored with AI assistance using Claude Code.

Signed-off-by: dongjiang <dongjiang1989@126.com>
dongjiang1989 added a commit to XHToken/executorch that referenced this pull request Oct 23, 2026
@dongjiang1989
Add CI integration for Spark-X2.5 models
c99c501 
This commit adds CI test support for Spark-X2.5-1.7B and Spark-X2.5-4B
models to the ExecuTorch test infrastructure.

Changes:
- Added spark_x2_5_1_7b and spark_x2_5_4b to test_torchao_huggingface_checkpoints.sh
- Added both models to CI test matrix in trunk.yml
- Configured XNNPACK (x86_64) and TorchAO (ARM64) backend testing
- Use --local-dir for hf download to avoid read-only /mnt/hf_cache issues
- convert_weights now rewrites the no-op normalizer in Spark-X2.5
  tokenizer.json (empty Sequence, rejected by older revisions of the C++
  HF tokenizer) to null, which is semantically identical, so llama_main
  can load it. The root cause is also fixed upstream in
  meta-pytorch/tokenizers#225; the submodule pin will be bumped once
  that lands.

The Spark-X2.5 model infrastructure already exists in the codebase.
This commit completes the integration by adding automated CI testing.

Parts of this commit were authored with AI assistance using Claude Code.

Signed-off-by: dongjiang <dongjiang1989@126.com>
dongjiang1989 added a commit to XHToken/executorch that referenced this pull request Oct 28, 2026
@dongjiang1989
Add CI integration for Spark-X2.5 models
0551746 
This commit adds CI test support for Spark-X2.5-1.7B and Spark-X2.5-4B
models to the ExecuTorch test infrastructure.

Changes:
- Added spark_x2_5_1_7b and spark_x2_5_4b to test_torchao_huggingface_checkpoints.sh
- Added both models to CI test matrix in trunk.yml
- Configured XNNPACK (x86_64) and TorchAO (ARM64) backend testing
- Use --local-dir for hf download to avoid read-only /mnt/hf_cache issues
- convert_weights now rewrites the no-op normalizer in Spark-X2.5
  tokenizer.json (empty Sequence, rejected by older revisions of the C++
  HF tokenizer) to null, which is semantically identical, so llama_main
  can load it. The root cause is also fixed upstream in
  meta-pytorch/tokenizers#225; the submodule pin will be bumped once
  that lands.

The Spark-X2.5 model infrastructure already exists in the codebase.
This commit completes the integration by adding automated CI testing.

Parts of this commit were authored with AI assistance using Claude Code.

Signed-off-by: dongjiang <dongjiang1989@126.com>
dongjiang1989 added a commit to XHToken/executorch that referenced this pull request Nov 2, 2026
@dongjiang1989
Add CI integration for Spark-X2.5 models
fbe8e50 
This commit adds CI test support for Spark-X2.5-1.7B and Spark-X2.5-4B
models to the ExecuTorch test infrastructure.

Changes:
- Added spark_x2_5_1_7b and spark_x2_5_4b to test_torchao_huggingface_checkpoints.sh
- Added both models to CI test matrix in trunk.yml
- Configured XNNPACK (x86_64) and TorchAO (ARM64) backend testing
- Use --local-dir for hf download to avoid read-only /mnt/hf_cache issues
- convert_weights now rewrites the no-op normalizer in Spark-X2.5
  tokenizer.json (empty Sequence, rejected by older revisions of the C++
  HF tokenizer) to null, which is semantically identical, so llama_main
  can load it. The root cause is also fixed upstream in
  meta-pytorch/tokenizers#225; the submodule pin will be bumped once
  that lands.

The Spark-X2.5 model infrastructure already exists in the codebase.
This commit completes the integration by adding automated CI testing.

Parts of this commit were authored with AI assistance using Claude Code.

Signed-off-by: dongjiang <dongjiang1989@126.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@larryliu0820 larryliu0820 larryliu0820 approved these changes

Assignees

No one assigned

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@dongjiang1989 @larryliu0820