Skip to content

fix(paraformer): keep one keyed result per utterance when no tokens are predicted - #3797

Open
MohammadHijjawi97 wants to merge 1 commit into
modelscope:mainfrom
MohammadHijjawi97:fix/paraformer-empty-token-batch
Open

MohammadHijjawi97 wants to merge 1 commit into
modelscope:mainfrom
MohammadHijjawi97:fix/paraformer-empty-token-batch

Conversation

@MohammadHijjawi97

Copy link
Copy Markdown
Contributor

Summary

Fixes #3782.

When the rounded predicted token length is zero for every utterance in a batch, Paraformer.inference and ContextualParaformer.inference return a bare [], and SeacoParaformer.inference returns ([],). AutoModel.inference then replaces the whole batch with a single {"text": ""} without a key (Paraformer, ContextualParaformer), or adds nothing for it (SeACo-Paraformer, the same bug with a worse outcome). Reading inference_with_vad, the number of segment results then no longer matches the number of VAD segments, so the whole recording would be returned as an empty result, and with spk_model set results[_b] would raise IndexError.

The early return now produces one empty result per key, with the same fields the regular decoding loop produces for an empty hypothesis (text: "", plus timestamp: [] when timestamps are enabled, or token_int: [] without a tokenizer), and returns meta_data like the normal path. In Paraformer, the existing key normalisation is moved above the check so the early return uses the same keys.

Before (two utterances, batch_size=2, all token lengths 0):

[{'text': ''}]

After:

[{'key': 'a', 'text': ''}, {'key': 'b', 'text': ''}]

Type of change

  • Bug fix
  • Documentation
  • Example or demo
  • Runtime or deployment
  • Benchmark or evaluation
  • Model/training change

Validation

  • python -m compileall funasr examples tests
  • Docs or links checked
  • Runtime/deployment command tested

Added tests/test_paraformer_empty_token_batch.py. It uses a real WavFrontend (no cmvn) and model stand-ins whose predictor returns zero tokens, so no weights are downloaded. It covers Paraformer (with and without timestamps, without a tokenizer), ContextualParaformer, SeacoParaformer (CifPredictorV3 and another predictor), and AutoModel.inference end to end. All 6 tests fail on main and pass with this change.

python -m pytest tests/test_paraformer_empty_token_batch.py tests/test_paraformer_timestamp_padding.py tests/test_paraformer_timestamp_contract.py tests/test_paraformer_hotword_file.py tests/test_auto_model.py -k "not merge_thr"

User impact

Users decoding short, silent or noise-only inputs in a batch get one keyed empty result per input instead of a single unkeyed row, so a batch of segments with no predicted tokens no longer collapses the batch's results.

Notes for reviewers

The batch-level early return still skips the decoder, so its behaviour is unchanged apart from the returned structure. When only some utterances in a batch have zero tokens, the existing loop already handles them, and this change does not touch that path.

…re predicted

When every utterance in a batch has a rounded predicted token length of
zero, Paraformer and ContextualParaformer returned a bare `[]` and
SeacoParaformer returned `([],)`. AutoModel then replaced the whole batch
with a single `{"text": ""}` without a key (Paraformer, Contextual) or
with nothing at all (SeACo). In the VAD pipeline the number of segment
results then no longer matches the number of segments, so the whole
recording is reported as an empty result.

Return one empty result per key instead, with the same fields the
regular decoding loop produces for an empty hypothesis, plus the
inference meta data.

Fixes modelscope#3782

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

All-zero Paraformer token lengths collapse a batch to one unkeyed result

1 participant