[dependabot]

Bumps tokenizers from 0.23.2 to 0.23.3.

Release notes

Sourced from tokenizers's releases.

Release v0.23.3 (python only)

Release v0.23.3

Python package release, other packages are unchanged

🚨 Fixs a breaking change introduced in v0.23.1

What's changed

Patch for the python package with three fixes:

• Loosen the huggingface_hub requirements to allow huggingface_hub v2 huggingface/tokenizers#2486 - Unblocks transformers installs
• Fix type hints not resolving for pyright: huggingface/tokenizers#2456
• Fix a breaking change from 0.23.1 regarding truncation and overflowing: huggingface/tokenizers#2091

Full Changelog: huggingface/tokenizers@v0.23.2...v0.23.3

Commits

• 3012770 comment out semver to run rust tests
• 62c96d3 fix: restore overflowing
• 580ffcb account for text_signature
• 14e6dce fix(python): ship submodule stubs as <sub>/init.pyi
• 935453f chore(python): delete the stale hand-written trainers stub
• 65903e3 ci: disable rust and node publishing on the 0.23.x branch
• 2b110f4 ci: remove CI.yml, it publishes to PyPI with a token on every tag
• ac1dfef ci(python-release): use main's Trusted Publishing release workflow
• 5ea330f release(python): v0.23.3, allow huggingface_hub <3
• c2b33f8 update release
• Additional commits viewable in compare view

[coderabbitai]

Important

Review skipped

Bot user detected.

To trigger a single review, invoke the @coderabbitai review command.

Run configuration

• Source mode: Auto
• Repository: AOSSIE-Org/OpenVerifiableLLM
• Branch: main
• File: .coderabbit.yaml
• Commit: 93211ec
• Layers: Repository YAML
• Review profile: ASSERTIVE
• Plan: Advanced
• Run ID: a2c9dad1-00cf-4c46-9d3b-716f3bd65e38

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• Trigger review

• Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

• X
• Mastodon
• Reddit
• LinkedIn

[socket-security]

Review the following changes in direct dependencies. Learn more about Socket for GitHub.

Diff	Package	Supply Chain Security	Vulnerability	Quality	Maintenance	License
	pypi/​tokenizers@​0.23.2 ⏵ 0.23.3

View full report

[ryoari]

Closing: the Python dependencies are pinned by hash in requirements/preparation.lock, which is evidence for the published run and is updated by hand. This changes the input file without the lock. Dependabot no longer opens pip updates (see the linked PR). (#284)

[dependabot]

OK, I won't notify you again about this release, but will get in touch when a new version is available. If you'd rather skip all updates until the next major or minor version, let me know by commenting @dependabot ignore this major version or @dependabot ignore this minor version. You can also ignore all major, minor, or patch releases for a dependency by adding an ignore condition with the desired update_types to your config file.

If you change your mind, just re-open this PR and I'll resolve any conflicts on it.