[ANDROIDHASSAN]

Summary

sentence_postprocess removes <s>, </s>, <unk> and <OOV> from the tokens, then reads time_stamp by the position in the cleaned list. ts_prediction_lfr6_standard and time_stamp_lfr6_onnx give one timestamp per token, including <unk>, so every word after a filtered token gets the previous token's time. The last word's real end is lost as well.

Paraformer only strips eos, sos and blank from token_int, so <unk> reaches this function. Tokens 今 <unk> 天 好 through ts_prediction_lfr6_standard, sentence_postprocess and timestamp_sentence:

	今	天	好	sentence end
Timestamps for that token	210–809	1410–2009	2009–3105	3105
Before	210–809	809–1410	1410–2009	2009
After	210–809	1410–2009	2009–3105	3105

The fix keeps the timestamps of the tokens that survive the cleaning loop, in the three copies of the function (funasr, funasr_onnx, funasr_torch). If the list is too short to cover those tokens, it is left as it was before this change, because the branches below index it directly and would raise IndexError.

tests/test_sentence_postprocess_timestamps.py loads all three files by path, so it needs no model or torch. I added it to test-text-postprocessing.yml, which lists its tests by name and would otherwise skip it.

Type of change

• Bug fix
• Documentation
• Example or demo
• Runtime or deployment
• Benchmark or evaluation
• Model/training change

Validation

• python -m compileall funasr examples tests
• Docs or links checked
• Runtime/deployment command tested

With pytest 8.1.1, pypinyin 0.55.0 and rapidfuzz 3.14.5, the versions the workflow pins:

• New tests on main source: 3 failed, 2 passed. The Chinese, English and mixed cases fail, for example [[0, 100], [100, 200], [200, 300]] != [[0, 100], [200, 300], [300, 400]]. The other two are guards that pass before and after: a trailing </s> without a timestamp, and too few timestamps.
• With this change: tests/test_postprocess_hotwords.py tests/test_rich_transcription_postprocess.py tests/test_sentence_postprocess_timestamps.py gives 27 passed.
• Also with this change: tests/test_paraformer_timestamp_contract.py tests/test_paraformer_timestamp_padding.py plus the new file gives 13 passed.

These are unit tests plus the function chain above with synthetic CIF peaks. I did not run a real Paraformer model or the ONNX/libtorch runtime.

User impact

Anyone using Paraformer, BiCif Paraformer, SeACo Paraformer, E-Paraformer, the monotonic aligner, or the ONNX and libtorch runtimes gets correct word and sentence times when the output contains <unk>.

Notes for reviewers

• Output only changes for token lists that contain a filtered token before the last word.
• A short list keeps the old misaligned output, since it can't be matched to tokens reliably.
• Left out to keep this to one cause: in the core copy, the mixed branch's else appends a word with no timestamp, which shifts later words the same way (["你", "g5", "好", "吗"]). The runtime copies raise ValueError there. The runtime wash lists also lack <OOV>.

[LauraGPT]

感谢这个边界清楚的修复。我复核了 exact head 712e1d05f5c5a2e0035c07557536d44c74e62c99 相对当前 main@dd00b0e42261b628466037f3a522d713399360b6 的完整五文件 diff，并从 potential merge 403a04128026b121813a4e2481d267d65acae0b8 / tree 7b05ccfe70a3aff3149354e917c44fa9c036d1e1 重建了全部 4155 个 blob。

独立验证结果：

• 在 exact base 上保留 PR 新测试：3 failed, 2 passed，确实复现 <unk> 后时间戳左移；
• exact potential merge 上，新增测试及相关 hotword / rich-transcription / Paraformer timestamp 套件共 35 passed；
• 三份实现分别补测了特殊 token 位于首尾、英文 BPE 和中英混合的 9 个边界组合，全部通过；
• 变更 Python 文件 py_compile 通过。

目前没有新的静态代码修改要求。验证范围仍需准确限定为纯 CPU 文件级后处理：没有加载真实 Paraformer 权重，也没有执行 ONNX 引擎或 libtorch runtime。

当前阻塞是 exact-head CI，而不是独立 approval：该 head 仍为 0 个 check runs、0 个 commit statuses、statusCheckRollup=null，PR 的 mergeStateStatus=UNSTABLE；main 保护仍要求 status checks。因此本轮不会合并。需要让 test-text-postprocessing 在这个 exact head 上产生真实 hosted 结果后再做最终门控。

另外，#3801 报告的是 VAD 中文 chunk 拼接时文本 unit 融合导致的累计漂移，和本 PR 的“过滤 <unk> 后索引错位”是两个不同根因；本 PR 不应被当作 #3801 的修复。