[mayuriphad]

Fixes explosion/spaCy#13730

When using fast tokenizers (like DeBERTa), the alignment IDs were being duplicated across adjacent spaCy tokens. This happened because char_to_sp_token mapped the trailing whitespace (which isn't part of the oken.text) to the preceding token.

Fast tokenizers often produce offset_mapping that includes the preceding space for wordpieces (e.g. _word). When that space character mapped to the previous token, the wordpiece was aligned to BOTH the previous token and the current token, resulting in duplicate alignments (�lign_lengths = 2).

This patch removes the + 1 from the range loop so that trailing spaces are not mapped to the preceding token. Wordpieces will only align to tokens whose non-whitespace characters they overlap with.

[Copilot]

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

[mayuriphad]

Hi maintainers, a gentle ping on this PR whenever you have a moment to review. I'm happy to make any changes or add tests if something needs adjusting. Thank you for your time!

[mayuriphad]

Hi, just a gentle ping on this PR. It's ready for review whenever you have time, and I'm happy to make any changes you'd like. Thanks!

[honnibal]

Thanks a lot for tracking this down — the diagnosis was spot on: the +1 maps each token's trailing space to it, so SentencePiece wordpieces (whose offsets include the leading space) get aligned to the previous token too.

Removing the +1 entirely does break one case though: whitespace-only wordpieces like GPT-2's Ġ end up aligned to nothing, which fails the existing check in test_model_wrapper.py that all fast-tokenizer wordpieces are aligned. (The file also got converted to CRLF line endings, which is why the diff showed every line changed.)

I've opened #429, which keeps the trailing-space mapping only as a fallback for wordpieces that don't overlap any token characters, with regression tests. You're credited as co-author on the commit. Closing this in favour of that one — thanks again!