Skip to content

[Bug] mmlu_gen_5_shot_chat_prompt测试无法使用 #628

Open
Open
[Bug] mmlu_gen_5_shot_chat_prompt测试无法使用#628
Labels
bugSomething isn't workingcontent_check_failedissue content check failed

Description

@Xiaoyue-X
Xiaoyue-X
opened on Oct 30, 2026

操作系统及版本

oe2403sp4

安装工具的python环境

docker容器中的python环境

python版本

其他

AISBench工具版本

3.1.20260609

AISBench执行命令

ais_bench --models vllm_api_general_chat --datasets gsm8k_gen_4_shot_cot_chat_prompt humaneval_gen_0_shot mmlu_gen_5_shot_chat_prompt

模型配置文件或自定义配置文件内容

models = [
dict(
attr="service",
type=VLLMCustomAPIChat,
abbr="vllm-api-general-chat",
path="",
model="qwen3.6",
stream=False,
request_rate=0,
use_timestamp=False,
retry=2,
api_key="",
host_ip="141.61.133.xxx",
host_port=8006,
url="",
max_out_len=131072,
batch_size=400,
trust_remote_code=False,
generation_kwargs=dict(
temperature=1.0,
top_p=0.95,
top_k=20,
min_p=0.0,
presence_penalty=0.0,
repetition_penalty=1.0,
ignore_eos=False,
),
pred_postprocessor=dict(type=extract_non_reasoning_content),
)
]

预期行为

No response

实际行为

/usr/local/python3.12.13/lib/python3.12/site-packages/triton/language/extra/cann/extension/math_ops.py:13: FutureWarning: triton.language.extra.cann.extension.atan2 is deprecated and will be removed in the next release; use triton.language.extra.cann.libdevice.atan2 instead.
@_deprecated(replacement="triton.language.extra.cann.libdevice.atan2")
/usr/local/python3.12.13/lib/python3.12/site-packages/triton/language/extra/cann/extension/math_ops.py:35: FutureWarning: triton.language.extra.cann.extension.isfinited is deprecated and will be removed in the next release; use triton.language.extra.cann.libdevice.isfinited instead.
@_deprecated(replacement="triton.language.extra.cann.libdevice.isfinited")
/usr/local/python3.12.13/lib/python3.12/site-packages/triton/language/extra/cann/extension/math_ops.py:49: FutureWarning: triton.language.extra.cann.extension.finitef is deprecated and will be removed in the next release; use triton.language.extra.cann.libdevice.finitef instead.
@_deprecated(replacement="triton.language.extra.cann.libdevice.finitef")
[2026-10-30 12:06:23,003] [ais_bench] [INFO] Loading vllm_api_general_chat: /vllm-workspace/vllm-ascend/benchmark/ais_bench/benchmark/configs/./models/vllm_api/vllm_api_general_chat.py
[2026-10-30 12:06:23,143] [ais_bench] [INFO] Loading gsm8k_gen_4_shot_cot_chat_prompt: /vllm-workspace/vllm-ascend/benchmark/ais_bench/benchmark/configs/./datasets/gsm8k/gsm8k_gen_4_shot_cot_chat_prompt.py
[2026-10-30 12:06:23,148] [ais_bench] [INFO] Loading humaneval_gen_0_shot: /vllm-workspace/vllm-ascend/benchmark/ais_bench/benchmark/configs/./datasets/humaneval/humaneval_gen_0_shot.py
[2026-10-30 12:06:23,151] [ais_bench] [INFO] Loading mmlu_gen_5_shot_chat_prompt: /vllm-workspace/vllm-ascend/benchmark/ais_bench/benchmark/configs/./datasets/mmlu/mmlu_gen_5_shot_chat_prompt.py
[2026-10-30 12:06:23,159] [ais_bench] [INFO] Loading example: /vllm-workspace/vllm-ascend/benchmark/ais_bench/benchmark/configs/./summarizers/example.py
[2026-10-30 12:06:23,184] [ais_bench] [INFO] Current exp folder: outputs/default/20261030_120615
Traceback (most recent call last):
File "/usr/local/python3.12.13/lib/python3.12/site-packages/mmengine/config/lazy.py", line 68, in build
module = importlib.import_module(self._module)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.12.13/lib/python3.12/importlib/init.py", line 90, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "", line 1387, in _gcd_import
File "", line 1360, in _find_and_load
File "", line 1324, in _find_and_load_unlocked
ModuleNotFoundError: No module named 'ais_bench.benchmark.utils.text_postprocessors'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/python3.12.13/bin/ais_bench", line 6, in
sys.exit(main())
^^^^^^
File "/vllm-workspace/vllm-ascend/benchmark/ais_bench/benchmark/cli/main.py", line 6, in main
task_manager.run()
File "/vllm-workspace/vllm-ascend/benchmark/ais_bench/benchmark/cli/task_manager.py", line 35, in run
cfg = self.config_manager.load_config(self.workflow)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm-ascend/benchmark/ais_bench/benchmark/cli/config_manager.py", line 104, in load_config
self._dump_and_reload_config()
File "/vllm-workspace/vllm-ascend/benchmark/ais_bench/benchmark/cli/config_manager.py", line 333, in _dump_and_reload_config
recur_convert_config_type(self.cfg)
File "/vllm-workspace/vllm-ascend/benchmark/ais_bench/benchmark/cli/utils.py", line 39, in recur_convert_config_type
cfg[key] = recur_convert_config_type(value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm-ascend/benchmark/ais_bench/benchmark/cli/utils.py", line 42, in recur_convert_config_type
cfg[i] = recur_convert_config_type(item) if isinstance(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm-ascend/benchmark/ais_bench/benchmark/cli/utils.py", line 39, in recur_convert_config_type
cfg[key] = recur_convert_config_type(value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm-ascend/benchmark/ais_bench/benchmark/cli/utils.py", line 39, in recur_convert_config_type
cfg[key] = recur_convert_config_type(value)
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/vllm-workspace/vllm-ascend/benchmark/ais_bench/benchmark/cli/utils.py", line 35, in recur_convert_config_type
for key, value in cfg.items():
^^^^^^^^^^^
File "/usr/local/python3.12.13/lib/python3.12/site-packages/mmengine/config/config.py", line 239, in items
items.append((key, self.build_lazy(value)))
^^^^^^^^^^^^^^^^^^^^^^
File "/usr/local/python3.12.13/lib/python3.12/site-packages/mmengine/config/config.py", line 217, in build_lazy
value = value.build()
^^^^^^^^^^^^^
File "/usr/local/python3.12.13/lib/python3.12/site-packages/mmengine/config/lazy.py", line 70, in build
raise type(e)(f'Failed to import {self._module} '
ModuleNotFoundError: Failed to import ais_bench.benchmark.utils.text_postprocessors in /vllm-workspace/vllm-ascend/benchmark/ais_bench/benchmark/configs/./datasets/mmlu/mmlu_gen_5_shot_chat_prompt.py, line 6 for No module named 'ais_bench.benchmark.utils.text_postprocessors'
[ERROR] 2026-10-30-12:06:23 (PID:1326756, Device:-1, RankID:-1) ERR99999 UNKNOWN applicaiton exception

前置检查

  • 我已读懂主页文档的快速入门，无法解决问题
  • 我已检索过FAQ，无重复问题
  • 我已搜索过现有Issue，无重复问题
  • 我已更新到最新版本，问题仍存在

Activity

  1. Xiaoyue-X
    added
    bugSomething isn't working
    on Oct 30, 2026

  2. github-actions commented on Oct 30, 2026

    @github-actions
    github-actionsBot
    on Oct 30, 2026 – with GitHub Actions

    🌟欢迎给AISBench benchmark测评工具提issue，您可以先尝试以下途径寻求issue的解决方法：

    1. 【强烈推荐❤️‍🔥】确保issue描述完整后，可以试着将issue交给Ask DeepWiki回答，deepwiki包含了和工具相关的所有知识库
    2. 在 工具历史issue 中通过问题的关键日志片段或关键词检索相似issue
    3. 在 官方文档 的搜索栏中通过问题的关键日志片段或关键词检索
    4. 寻求仓库维护者的帮助（issue评论区中 @SJTUyh）

  3. github-actions commented on Oct 30, 2026

    @github-actions
    github-actionsBot
    on Oct 30, 2026 – with GitHub Actions

    🤖 基于AI机器人的issue内容完整性检查结果:

    问题内容检查结果

    检查状态

    FAIL

    详细检查

    • 操作系统及版本: COMPLETE
    • 安装工具的python环境: COMPLETE
    • python版本: INCOMPLETE
    • AISBench工具版本: COMPLETE
    • AISBench执行命令: COMPLETE
    • 模型配置文件或自定义配置文件内容: COMPLETE
    • 实际行为: COMPLETE

    改进建议

    • python版本: 请提供具体的Python版本号，例如根据日志中的 /usr/local/python3.12.13/lib/python3.12/ 路径，可以填写 3.12.13，而不是 其他。

    补充说明

    无

    👉 如果想重新检查，在评论区@issue_checker即可。

    【强烈推荐❤️‍🔥】确保issue描述完整后，可以试着将issue交给Ask DeepWiki回答，deepwiki包含了和工具相关的所有知识库

  4. github-actions
    added
    content_check_failedissue content check failed
    on Oct 30, 2026

  5. aisbench-issue-master commented on Oct 30, 2026

    @aisbench-issue-master
    aisbench-issue-masterBot
    on Oct 30, 2026

    感谢反馈。从日志看，本 issue 的失败点并不在 MMLU 数据部署或模型配置，而是配置加载阶段的模块导入失败，属于运行环境问题。

    关键报错

    日志末尾的最终异常为：

    ModuleNotFoundError: Failed to import ais_bench.benchmark.utils.text_postprocessors
in /vllm-workspace/vllm-ascend/benchmark/ais_bench/benchmark/configs/./datasets/mmlu/mmlu_gen_5_shot_chat_prompt.py, line 6
for No module named 'ais_bench.benchmark.utils.text_postprocessors'

    调用栈为 cli/main.py → cli/task_manager.py:35 → cli/config_manager.py:104/333 → cli/utils.py:39/42/35 → mmengine/config/config.py:217 → mmengine/config/lazy.py:70。

    两点说明：

    1. 日志开头的 triton.language.extra.cann.extension.* is deprecated 只是 FutureWarning，与本问题无关，可直接忽略。
    2. 失败发生在配置解析阶段（Config.build_lazy 触发 lazy import），任务尚未开始跑数据，因此可以排除 MMLU 数据集文件缺失、5-shot dev split 不足、模型请求失败等运行期原因；也未进入 vllm_api_general_chat 的推理流程。

    环境层面的判断

    值得注意的是：ais_bench 命令入口在 /usr/local/python3.12.13/bin/ais_bench，但 traceback 中的 main.py、cli/config_manager.py、cli/utils.py 全部解析到源码目录 /vllm-workspace/vllm-ascend/benchmark。这说明当前环境的 ais_bench 是「源码目录 + 安装元数据」混合形态；而配置文件中引用的 ais_bench.benchmark.utils.text_postprocessors 在该解释器实际可导入的 ais_bench 包内不存在。这类现象通常来自以下几种环境不一致，而非配置写错：

    • 同一解释器下存在多份 ais_bench（site-packages 中的旧安装 + 源码目录），sys.path 优先级导致实际导入到版本不匹配的一份；
    • 源码目录与 pip 安装元数据（版本 3.1.20260609）不同步，源码中的 config 引用了当前安装里尚未包含的模块；
    • PYTHONPATH 等环境变量指向了另一份 ais_bench 源码，覆盖了正常解析顺序。

    建议的排查步骤

    请在同一个解释器（/usr/local/python3.12.13）下依次执行，并把输出贴回本 issue：

    # 1. 确认实际解析到的包路径（重点）
python -c "import ais_bench, ais_bench.benchmark.utils as u; print(ais_bench.__file__); print(u.__file__)"

# 2. 确认该 utils 目录下是否存在 text_postprocessors
ls $(python -c "import ais_bench.benchmark.utils as u, os; print(os.path.dirname(u.__file__))")

# 3. 确认安装形态与版本是否一致
pip show ais_bench 2>/dev/null || pip list | grep -i ais
pip check

# 4. 确认是否有路径覆盖
env | grep -i pythonpath

    处理方向：

    • 若第 1 步打印出的路径是 site-packages 而非 /vllm-workspace/vllm-ascend/benchmark，说明存在残留安装，请清理后统一安装形态（在源码根目录用 pip install -e . 安装到该解释器，或完全改用 pip 安装并删除源码路径依赖）。
    • 若第 2 步确认源码目录中也没有 text_postprocessors.py，则属于源码与配置版本不一致，请同步到与配置匹配的版本（issue 中「我已更新到最新版本，问题仍存在」未勾选，建议先更新到最新版本再复现）。
    • 若 PYTHONPATH 指向了另一份 ais_bench，请修正后重试原命令。

    其他

    • 机器人的完整性检查提示 python 版本填写为「其他」，从日志路径 /usr/local/python3.12.13/ 可确认实际为 3.12.13，建议更新 issue 描述。
    • 在上述信息补齐前，尚无法判定是「残留安装」还是「版本不匹配」，因此暂不关闭本 issue。

  6. Xiaoyue-X commented on Oct 31, 2026

    @Xiaoyue-X
    Xiaoyue-X
    on Oct 31, 2026
    Author

    [root@server133 home]# python -c "import ais_bench, ais_bench.benchmark.utils as u; print(ais_bench.file); print(u.file)"
    /vllm-workspace/vllm-ascend/benchmark/ais_bench/init.py
    /vllm-workspace/vllm-ascend/benchmark/ais_bench/benchmark/utils/init.py
    [root@server133 home]# ls $(python -c "import ais_bench.benchmark.utils as u, os; print(os.path.dirname(u.file))")
    init.py pycache config core file image_process.py logging model_postprocessors.py postprocess prompt results visualization
    源码目录中也text_postprocessors.py，该版本无法使用mmlu_gen_5_shot_chat_prompt

  7. Xiaoyue-X commented on Oct 31, 2026

    @Xiaoyue-X
    Xiaoyue-X
    on Oct 31, 2026
    Author

    ais_bench/benchmark/configs/./datasets/mmlu/mmlu_gen_5_shot_chat_prompt.py, line 6 'ais_bench.benchmark.utils.text_postprocessors' 改为ais_bench.benchmark.utils.postprocess.text_postprocessors

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't workingcontent_check_failedissue content check failed

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions