Repository navigation
fix(paraformer): keep one keyed result per utterance when no tokens are predicted - #3797
Open
MohammadHijjawi97 wants to merge 1 commit into
Open
MohammadHijjawi97 wants to merge 1 commit into
MohammadHijjawi97 wants to merge 1 commit into
Conversation
…re predicted
When every utterance in a batch has a rounded predicted token length of
zero, Paraformer and ContextualParaformer returned a bare `[]` and
SeacoParaformer returned `([],)`. AutoModel then replaced the whole batch
with a single `{"text": ""}` without a key (Paraformer, Contextual) or
with nothing at all (SeACo). In the VAD pipeline the number of segment
results then no longer matches the number of segments, so the whole
recording is reported as an empty result.
Return one empty result per key instead, with the same fields the
regular decoding loop produces for an empty hypothesis, plus the
inference meta data.
Fixes modelscope#3782
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fixes #3782.
When the rounded predicted token length is zero for every utterance in a batch,
Paraformer.inferenceandContextualParaformer.inferencereturn a bare[], andSeacoParaformer.inferencereturns([],).AutoModel.inferencethen replaces the whole batch with a single{"text": ""}without a key (Paraformer, ContextualParaformer), or adds nothing for it (SeACo-Paraformer, the same bug with a worse outcome). Readinginference_with_vad, the number of segment results then no longer matches the number of VAD segments, so the whole recording would be returned as an empty result, and withspk_modelsetresults[_b]would raiseIndexError.The early return now produces one empty result per key, with the same fields the regular decoding loop produces for an empty hypothesis (
text: "", plustimestamp: []when timestamps are enabled, ortoken_int: []without a tokenizer), and returnsmeta_datalike the normal path. InParaformer, the existing key normalisation is moved above the check so the early return uses the same keys.Before (two utterances, batch_size=2, all token lengths 0):
After:
Type of change
Validation
python -m compileall funasr examples testsAdded
tests/test_paraformer_empty_token_batch.py. It uses a realWavFrontend(no cmvn) and model stand-ins whose predictor returns zero tokens, so no weights are downloaded. It covers Paraformer (with and without timestamps, without a tokenizer), ContextualParaformer, SeacoParaformer (CifPredictorV3 and another predictor), andAutoModel.inferenceend to end. All 6 tests fail onmainand pass with this change.User impact
Users decoding short, silent or noise-only inputs in a batch get one keyed empty result per input instead of a single unkeyed row, so a batch of segments with no predicted tokens no longer collapses the batch's results.
Notes for reviewers
The batch-level early return still skips the decoder, so its behaviour is unchanged apart from the returned structure. When only some utterances in a batch have zero tokens, the existing loop already handles them, and this change does not touch that path.