Honor null tokenizer_file in tokenizer_config.json

[rabeyanoor]

What does this PR do?

Fixes #48967

Problem

After the v5 update, AutoTokenizer.from_pretrained("AMR-KELEG/Sentence-ALDi") returns wrong token ids. The model repo ships both a vocab.txt (100K vocab) and a stale tokenizer.json (32K vocab), and its tokenizer_config.json contains "tokenizer_file": null.

In v4 this null meant "do not use tokenizer.json, build the tokenizer from the vocab files". Since #46279 (path traversal fix), the file resolved from the repo always overrides the value from tokenizer_config.json, so the null is ignored and the stale tokenizer.json gets loaded.

	vocab size	"هذه جملة" input_ids
before (v5 main)	32000	[2, 27966, 42, 1, 3]
after this PR	100000	[2, 2413, 9200, 3]

The "after" output matches the pre-5.0 behavior reported in the issue.

Fix

In PreTrainedTokenizerBase._from_pretrained, an explicit "tokenizer_file": null in the config is now kept when another vocab file (e.g. vocab.txt) is available to build the tokenizer.

• None is not a path, so honoring it does not reopen the path traversal issue fixed in Fix path traversal via vocab-file arguments in tokenizer_config.json #46279. Non-null values from the config are still ignored.
• If tokenizer.json is the only available file, it is still used, so repos that only ship tokenizer.json keep working.

Tests

Added test_null_tokenizer_file_in_config_ignores_tokenizer_json in tests/tokenization/test_tokenization_utils.py. It creates a local repo with a vocab.txt, a stale tokenizer.json and "tokenizer_file": null, and checks that the tokenizer is built from vocab.txt. The test fails without the fix and passes with it.
The existing test_vocab_file_in_config_does_not_escape_repo still passes.

I also ran the tokenization tests (tests/tokenization, bert, gpt2, t5, llama, albert, xlnet).

Before submitting

• Read the contributor guidelines
• Added a regression test
• Ran ruff format on the changed files

[github-actions]

---

CI recap

Dashboard: View test results in Grafana
Latest run: 35556173099
Result: success | Grafana metrics are not available yet.

[Rocketknight1]

cc @itazap