[kiymetakdemir]

Problem

HFTokenizer cannot load the CLIP tokenizer used by Stable Diffusion, SDXL and
FLUX.1 (openai/clip-vit-large-patch14). CLIP's tokenizer.json uses three
features that were missing:

• Lowercase normalizer: loading failed with Unsupported Normalizer type: Lowercase.
• RobertaProcessing post-processor: loading failed with
  Unsupported PostProcessor type: RobertaProcessing (a TODO in post_processor.h).
• BPE end_of_word_suffix ("</w>"): silently ignored, so every word-final token
  got the wrong id (a=64 instead of a</w>=320, cat=1481 instead of cat</w>=2368).

Changes

• LowercaseNormalizer: Unicode lowercasing per codepoint via unicode_tolower.
  U+0130 maps to i + U+0307 to match Rust's char::to_lowercase.
• RobertaProcessing: cls A sep for a single sequence, cls A sep sep B sep for
  a pair, ids passed through unchanged when special tokens are off.
  trim_offsets and add_prefix_space only affect offsets, which are not tracked.
• end_of_word_suffix: parsed from the BPE model and appended to the last symbol
  of each pre-tokenized word. With no suffix configured, encoding takes the
  existing code paths.

Testing

• test_hf_tokenizer.py::test_clip: token ids match transformers exactly for
  openai/clip-vit-large-patch14 on mixed case, non-ASCII case (İ, Greek),
  contractions, digits, whitespace runs, an inline <|endoftext|>, and a prompt
  longer than 77 tokens.
• C++ unit tests for each feature in test_normalizer.cpp, test_post_processor.cpp
  and test_hf_tokenizer.cpp; the end_of_word_suffix case was cross-checked
  against the HuggingFace tokenizers library.
• All Python tests (23) and C++ test suites pass locally.

This PR was authored with the help of Claude Code.

[shoumikhin]

@claude review effort=max