Skip to content

Fix Windows path separators breaking relative imports of Hub custom-generate modules - #48951

Open
dltsum wants to merge 1 commit into
huggingface:mainfrom
dltsum:fix-dynamic-module-posix
Open

dltsum wants to merge 1 commit into
huggingface:mainfrom
dltsum:fix-dynamic-module-posix

Conversation

@dltsum

@dltsum dltsum commented Sep 19, 2026

Copy link
Copy Markdown
Contributor

CPU CI GPU run-slow

What

When a Hub repo's custom_generate/generate.py relative-imports a sibling module (e.g. beam_search.py in transformers-community/group-beam-search and constrained-beam-search), get_cached_module_file recurses to fetch it and builds the child path like this:

f"{Path(module_file).parent / module_needed}.py"

On Windows, Path("custom_generate/generate.py").parent / "beam_search" renders as custom_generate\beam_search, and that backslash path is then looked up on the Hub, where file names always use forward slashes:

OSError: transformers-community/group-beam-search does not appear to have a file named
custom_generate\beam_search.py. Checkout
'https://huggingface.co/transformers-community/group-beam-search/tree/main' for available files.

So custom_generate strategies whose code spans several files cannot be loaded at all on Windows. (Sibling repos whose generate.py is self-contained, like dola and contrastive-search, are unaffected — this only triggers on relative imports.)

Fix

Normalize the Hub-relative path with as_posix() before the recursive lookup. Local filesystem uses of the same value are unaffected (Windows accepts forward slashes), and on POSIX systems the rendered path is unchanged.

Verification

Windows 11, Python 3.14:

  • GenerationIntegrationTests::test_hub_gen_strategies_2_transformers_community_group_beam_search and ..._3_transformers_community_constrained_beam_search: fail without the fix with the OSError above; pass with it.
  • tests/utils/test_dynamic_module_utils.py: 16 passed.

🤖 Generated with Claude Code

…om modules

get_cached_module_file recurses into relative imports of a remote
custom_generate/modeling module, building the child module path with
Path(module_file).parent / module_needed. On Windows that yields a
backslash path (e.g. 'custom_generate\beam_search.py') which the Hub
lookup then cannot find:

  OSError: transformers-community/group-beam-search does not appear to
  have a file named custom_generate\beam_search.py

Normalize with as_posix() so the Hub-relative path keeps forward slashes.
Local filesystem uses of the same value are unaffected (Windows accepts
forward slashes).

Co-Authored-By: Claude Code <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

CI recap

Dashboard: View test results in Grafana
Latest run: 35441462666:2
Result: success | Jobs: 16 | Tests: 190,608 | Failures: 0 | Duration: 17h 17m

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant