Conversation
tokenizer.json's ByteLevel post_processor carries no TemplateProcessing template, so ExtractBosEos left eos_id_/-bos_id_ unset on every Qwen-family checkpoint. When the model's config.json and generation_config.json also decline to name an eos_token_id, the request goes out with no eos at all and the model's <|im_end|> is generated then ignored: a Qwen3.5-9B-EXL3 chat reply emitted id 248046 at token 10 and ran to finish_reason=length, hallucinating further turns (measured live, ISSUE-LOCAL-01M3RY20H90NMK37V74EH34Z77). HF resolves the same id from tokenizer_config.json's eos_token name (string or AddedToken object; AutoTokenizer on the shipped config answers 248046, transformers 5.17.0). FromHfJson now reads the sibling tokenizer_config.json and special_tokens_map.json and fills bos_id_/eos_id_ from the declared names -- only where the post_processor left them unset, so a TemplateProcessing id keeps precedence (the pinned OPT case). The add_bos_token/add_eos_token keys stay unhonoured: HF's loaded class ignores them (measured) and EncodeWithSpecialTokens keeps reading only template_bos_/template_eos_. The DeepSeek-V2 pin's BosId()==-1 assertion was updated: HF resolves bos_token_id from the config name even when add_bos_token resolves False (measured, same transformers), so the pinned behaviour is the encode, not the getter. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: OMP:devin/swe-2 [omp]
…p.json Adversarial review of the eos/bos name fallback found the merge precedence inverted: the first-nonempty guard let special_tokens_map.json lock in a name before tokenizer_config.json was read, so the older file won. HF's from_pretrained applies the config's init kwargs after consuming the map file, so a declared key in tokenizer_config.json must overwrite the map's value. Unconditional overwrite regressed differently -- a config that does not declare the key would erase the map's name -- so contains() gates it: value() cannot distinguish absent from an explicit null. Verified empirically: the new 'tokenizer_config.json overrides special_tokens_map.json' subcase fails on the committed code (EosId -1 vs expected 19) and passes here; test_bpe is 30/30 green, 1020 assertions. FOLLOWING_AGENTS_PROTOCOL Following-Agents-Protocol: true AI-Assisted: true Assisted-by: OMP:devin/swe-2 [omp]
86a6d22 to
d5e6535
Compare
localai-org-maint-bot
left a comment
localai-org-maint-bot
left a comment
There was a problem hiding this comment.
The new sibling-file lookup is skipped for a valid bare relative filename. In FromHfJson, std::filesystem::path("tokenizer.json").parent_path() is empty, so if (!dir.empty()) bypasses the entire EOS/BOS fix even when tokenizer_config.json is in the current directory. Loading the same bytes as "./tokenizer.json" takes the new path instead. For the ByteLevel fixture with eos_token: "<|end|>", the first spelling therefore leaves EosId() at -1 while the second resolves 19.
Please resolve an empty parent as . (or allow the empty directory to participate in the sibling join) and add a regression that loads both path spellings from the fixture directory. This is a public Tokenizer::FromHfJson path, and callers such as the tokenizer example pass the supplied filename directly.
I read the complete diff at d5e6535 against current main fce3673 and checked the callers. This is a static finding: the C++ regression, oracle comparison, and repository gate could not run here because Python/CMake/the C++ compiler are absent. I cannot push a verified repair under the repository's gate-before-push requirement from this environment. @mudler @richiejp
This branch has not been deployed
3 participants
Checkpoint tokenizer.json files that ship a bare ByteLevel post_processor (every Qwen-family checkpoint) leave eos_id_/bos_id_ unset: ExtractBosEos reads only the TemplateProcessing template. When config.json and generation_config.json also decline to name eos, requests carry no eos id at all — the model emits <|im_end|> and the engine ignores it, generating fake turns to max_tokens. Measured live on Qwen3.5-9B-EXL3-4.00bpw (ROCm gfx1101): <|im_end|> (id 248046) emitted at token 10, request ran to finish_reason=length. Issue: #3364 (local ISSUE-LOCAL-01M3RY20H90NMK37V74EH34Z77, mirrored and synced).
HF resolves tokenizer.eos_token_id from the eos_token/bos_token NAMES in tokenizer_config.json (string or AddedToken object), optionally supplied by special_tokens_map.json — AutoTokenizer on this checkpoint answers 248046 (measured, transformers 5.17.0). FromHfJson now reads both sibling files and fills any id the post_processor left at -1. Template ids keep precedence; add_bos_token/add_eos_token stay unhonoured (HF ignores them on load — pinned); tokenizer_config.json wins over special_tokens_map.json via contains()-gated overwrite, matching from_pretrained kwarg ordering.
Verify: test_bpe 30/30, 1020 assertions (run in the ROCm toolchain image; the host lacks cmake so the checker is not host-reproducible). Mutation check: reverting the precedence fix fails the new override subcase with EosId -1 vs 19. Live verification: serving container rebuilt on this change returns finish_reason=stop with clean reasoning/content split, where the unfixed build ran to length.
Out of scope: add_bos/add_eos encode-side application (HF does not apply them either); GGUF path untouched (own kv EOS).
FOLLOWING_AGENTS_PROTOCOL
Following-Agents-Protocol: true
AI-Assisted: true
Assisted-by: OMP:devin/swe-2 [omp]