Skip to content

feat(tools): semantic tool-name suggestion for registry misses - #215

Open
raymondginger2018-sudo wants to merge 1 commit into
HKUDS:mainfrom
raymondginger2018-sudo:feat/tool-semantic-hint
Open

feat(tools): semantic tool-name suggestion for registry misses#215
raymondginger2018-sudo wants to merge 1 commit into
HKUDS:mainfrom
raymondginger2018-sudo:feat/tool-semantic-hint

Conversation

@raymondginger2018-sudo

Copy link
Copy Markdown
Contributor

Summary

When the LLM hallucinates or misremembers a tool name, the registry returns "not found". This PR adds a cheap, deterministic first mitigation: given the missed name and the available tool names, suggest the closest candidates by token-overlap + sequence similarity.

Design

  • Pure mechanism: no LLM calls, no embeddings, no network — uses difflib.SequenceMatcher + token-overlap scoring.
  • Safety guard: semantic discovery is only a hint fed back to the model as an error message. Execution still requires the exact registered name plus the permission engine — it never widens the callable surface.
  • Zero new dependencies (stdlib only).

Files

  • core/agent_runtime/tools/semantic_hint.py (new, 82 lines)

Testing

  • Module is side-effect free and importable standalone.
  • Callable surface unchanged; no changes to existing registry code.

Inspired by GenAI lesson 17 (P2-A7): Taskweaver-style semantic plugin discovery, adapted as a non-executable hint.

When the LLM hallucinates or misremembers a tool name, the registry
returns 'not found'. This module adds a cheap first mitigation:
given the missed name and available tool names, suggest the closest
candidates by token-overlap similarity (no LLM, no embeddings —
pure static scoring via difflib.SequenceMatcher).

Design guard: semantic discovery is only a hint fed back to the model
as an error message; execution still requires the exact registered
name plus the permission engine. It never widens the callable surface.
@raymondginger2018-sudo

Copy link
Copy Markdown
Contributor Author

设计说明

问题:当 agent 调用工具名拼写有微小偏差(如 execute_bash 写成 exec_bashread_file 写成 readfile),注册表查找直接返回空,agent 进入"工具不存在"的死胡同。

解法:用 difflib.SequenceMatcher 计算名字相似度 + token 交集双重判定,返回 Top-3 候选提示。pure mechanism,无 I/O,无 LLM 调用。

关键设计决策

  • 阈值 0.6 定得宽松(宁可误报也不漏报),因为结果是 suggestion 而非自动 fallback
  • _tokens() 按 snake_case 和 camelCase 分词,但对短名退化到字符级
  • 所有函数都没有副作用,纯计算,适合缓存在 runtime 启动时预热

测试建议:提供几个已知 miss case 作为回归测试即可(exec_bashexecute_bashfile_readread_file

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant