feat(providers): opt-in tiered loop routing to cheap models - #224
Open
raymondginger2018-sudo wants to merge 2 commits into
Open
feat(providers): opt-in tiered loop routing to cheap models#224raymondginger2018-sudo wants to merge 2 commits into
raymondginger2018-sudo wants to merge 2 commits into
Conversation
Contributor
Author
关键设计决策补充
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
设计说明
问题:agent 循环里大多数轮次是短小的 routine 工具步骤(读文件、跑命令、查状态),用旗舰模型处理成本和延迟都浪费。
解法:
TieredLoopProvider——一个 proxy wrapper,包住 runner 正在使用的 provider。当(且仅当)请求满足"轻量"条件时,转发给便宜的 light provider;其余全部走 primary。关键设计决策:
chat_with_retry/chat两个方法,其他属性通过__getattr__透传给 primary——对 harness 其他部分完全透明边界情况:
enabled=False时永远走 primarychat_with_retry→ 回退到chat测试建议:构造 4 种消息组合(短新对话/长新对话/短多轮/长多轮)验证路由选择