feat(subagent): adaptive rate-limit governor - #43
Conversation
Provider 429 storms from a large subagent fleet now feed a shared RateLimitGovernor: a sliding window drives AIMD shrink of the launch gate capacity (halve on throttling, pause at sustained 429s, additive recovery on consecutive successes). 429 retries honor Retry-After with full-jitter exponential backoff (cap 120s). The root runtime is stamped with the fleet governor so descendant attempts report into the window. Signed-off-by: asto <asto18089@126.com>
|
Thanks @asto18089 for taking the time to contribute. This repository is observing a maintainer-managed PR intake gate in dry-run mode, so this pull request is staying open. This note helps maintainers prepare the allowlist before any enforcement is considered. Please read |
Review follow-up to the adaptive rate-limit governor: - Stamp the fleet governor on every manager-spawned runtime at the `spawn_background_with_assignment_options` chokepoint. Only the direct workflow root was stamped before, so the interactive `agent` tool chain (the primary swarm path) never reported 429s and the AIMD scheduler stayed inert there. - Time-driven recovery: queued launches periodically probe the governor so a pause lifts once the window drains even when the in-flight fleet finished without a final success. Without it the queue froze until each queued child hit its wall-time deadline. - `set_max_capacity` now applies the configured launch capacity to the live gate directly (raising or lowering) but keeps capacity 0 while paused, so runtime limit changes cannot silently lift a pause. `with_launch_concurrency`/`update_runtime_limits` route through it. - `DynamicGate` grants queued waiters an already-counted permit through a oneshot channel: a waiter cancelled after dispatch drops the permit, whose `Drop` re-dispatches the slot instead of losing the wakeup. - Drop the stale `#[allow(dead_code)]` on `rate_limit_governor` and fix the spawn-stamping doc comments. - Add `forkguard_rate_limit_governor_*` and `forkguard_dynamic_gate_*` behavior tests (window-drain recovery, pause survives a limit change, cancelled-waiter grant re-dispatch). Signed-off-by: asto <asto18089@126.com>
审阅结论(父仓 PR 配对审计)结论:问题真实(蜂群 128 并发下 429 锁步重试 + t0 洪峰,现有机制防不住)、方案对症(准入层 AIMD 是正确 scope,
另含: 实测: 遗留披露:仓库退避实现至此有 4 套( |
摘要
蜂群模式(父仓 PR 配对)下子智能体数量可达底座设计上限(128 并发),会大量触发模型提供商的 429 并行限流。本 PR 为子智能体 launch gate 增加限流自适应调度:
launch_gate从固定容量tokio::sync::Semaphore改为自定义DynamicGate(可收缩容量;Semaphore收缩只能换Arc,有 permit 在手时静默失效)。RateLimitGovernor(引擎级共享,经SubAgentRuntime派生树继承):60s 滑动窗口统计 429 率,AIMD 策略——窗口内限流 ≥2 次或比例 >30% 容量减半,≥4 次暂停新 launch(排队原因queued: waiting for provider rate-limit recovery);限流事件老化后按「连续成功」或时间驱动探针(排队子智能体周期调用recover_if_window_drained,避免在飞集群清空后队列冻结到 wall-time 超时)以 1/4 容量恢复、加性回升。运行时改 launch 并发经 governor 生效,不会静默解除暂停。Retry-After;无则指数退避 + full jitter(250ms 起、cap 120s)错峰防 thundering herd;QuotaExhausted不算临时限流,走既有失败路径。spawn_background_with_assignment_options汇聚点为全部 spawn 路径统一 stamp(交互会话agent工具链、Op::SpawnSubAgent、direct workflow 根 runtime),整棵派生树经SubAgentRuntime继承;不 stamp 的裸 runtime(测试、tool-only)才保持None。验证
cargo fmt --check、cargo clippy -p codewhale-tui --all-targets新代码 0 警告cargo test -p codewhale-tui --lib subagent::491 通过 0 失败(含 governor 12 个单测:窗口统计、AIMD 升降、暂停/恢复/时间自愈边界、gate 取消重派发);cargo test --lib forkguard_60 通过 0 失败配套
父仓 PR(gitlink 已指向本 head
6d81cfa83,评审修复见下方「评审跟进」);合并后需发布 r14(tagpinvou-v0.9.5-r14)并由父仓收尾更新docs/fork-modifications.md与scripts/fork-guard.sh的PUBLISHED_HEAD/PUBLISHED_COMMITS(37→39)。父仓已在同 PR 内完成docs/fork-modifications.md(zh/en)与 T5 指纹登记。评审跟进(2026-09-06)
审计发现三处实质问题,已随 head
6d81cfa83修复:agent工具链(core/engine.rs根 runtime →spawn_background_*)整棵树 governor 恒为None,AIMD 在主用例上是死代码。现改为 manager 在 spawn 汇聚点统一 stamp,mod.rs失实注释同步修正。另含:
set_max_capacity改为「未暂停时即应用配置容量(可升可降)、暂停时保持 0」,运行时改限不再静默解除暂停;摘除过时#[allow(dead_code)];新增 3 条forkguard_*行为测试钉住自愈/暂停保持/取消重派发。Signed-off-by: asto asto18089@126.com