diff --git a/CHANGELOG.md b/CHANGELOG.md index 8a43abdc3d..c9310c5436 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -18,6 +18,25 @@ All notable changes to Raven are documented here. query's length. The research product, the deck launcher, the sub-agent fetch lane and the onboarding wizard resolve the same borrowed key. +- The reasoning effort can be set **per conversation**: the *Reasoning + effort* row of the WebUI's model picker, beside the sub-agent tier, and + `/effort ` in the TUI (`/effort` alone + reports, `default` returns to the global setting). The pick outranks + `agents.defaults.reasoningEffort` and the conversation's mode, is kept on the + conversation's own record so it survives a gateway restart and comes back on + resume, lands on the next turn, and moves Raven's own model calls only. + New RPC `session.set_effort` (report, set or clear; every answer names what + the default is on this install and the words), and `session.info` carries + `reasoning_effort` for the status bar. The using-raven guide has a section on + it. Clearing `agents.defaults.reasoningEffort` (*Provider default*) reaches a + running gateway's next turn as well: the turn sends the effort in force on + every call, the provider's own default included, instead of leaving a + cleared key to the copy the provider took at boot. A fork keeps its + parent's pick with its model; the agent's own `raven_config` door accepts + null for `agents.defaults.reasoningEffort` as the settings page does; and + inside a TUI direct chat with an instance `/effort` is refused rather than + moving the main conversation while saying "this conversation". + - The settings dialog's **Skills** and **Plugins** pages have an **Add** button. Neither page can install anything -- a skill is fetched from Skill Hub by `use_skill` and an MCP server is connected from the PlugHub catalogue @@ -29,7 +48,6 @@ All notable changes to Raven are documented here. with the lead-in after it. The Plugins page says under the button that a server the catalogue does not list goes into `tools.mcpServers` in the config file by hand, naming the file the gateway reported. - - The history slot is filled by a **history strategy**, `context.history.strategy`: `compacting` (the new default) or `curator` (the previous behaviour, one setting away). Compacting keeps the conversation verbatim until the window @@ -285,6 +303,18 @@ All notable changes to Raven are documented here. ### Changed +- A conversation's reasoning-effort choices now come from the model it runs on: + LiteLLM's declaration first, a provider-scoped + `model_overlay.reasoningEfforts` that outranks the table when stated, the + Codex account catalogue's own `supported_reasoning_levels` for its models + (read from the cache by the turn and reports, refreshed by an explicit set + off the event loop, with an expired answer kept as last-good), and no control + at all for a model that declares none. The turn and the status bar clamp a + stored pick to the current model, so switching models cannot send a level the + new model does not accept; the pick itself is kept and applies again if a + later model declares it. The global `agents.defaults.reasoningEffort` is + unchanged. + - Empty and abnormal model replies are handled the way Hermes Agent handles them. A reply that is only the silence marker `[SILENT]` ends the turn with nothing sent; a scheduled run, a sub-agent's report or a Sentinel notice @@ -555,6 +585,11 @@ All notable changes to Raven are documented here. before it, the turn sent no closing text, and the answer was recorded but never shown. +- The settings page showed the reasoning effort as **Low** when the key was + not set at all, while the provider's own default was what ran. The unset + state is now its own rung, *Provider default*, which also clears the key + (`agents.defaults.reasoningEffort` accepts null) rather than leaving a word + the install never chose. - In the TUI on native Windows, typing with a Chinese IME no longer duplicates characters. The console was told to re-encode keys through Kitty keyboard / modifyOtherKeys, which doubles IME commits before they reach Raven's input diff --git a/CONTEXT.md b/CONTEXT.md index cc3f4a45af..7996835018 100644 --- a/CONTEXT.md +++ b/CONTEXT.md @@ -274,6 +274,38 @@ _Avoid_: putting the scope of the control on the rungs. Each row says only what distinguishes it; that raven's own effort is unchanged is stated once by whichever surface draws the control. +**Session Reasoning Effort** (`agent/session_effort.py`; `session.set_effort` over RPC, `/effort` in +the TUI, the effort row of the WebUI's model picker): +The reasoning effort one session picked for its own model calls, kept on the session's +metadata beside `model` (`reasoning_effort`; null once cleared, since a metadata patch sets +keys and never removes one) rather than on the loop's in-memory policy, so it outlives the +gateway and a resumed conversation runs at what it was left at, and a fork carries it with +the model (`FORK_CARRIES` in `session/manager.py`). `_run_agent_loop` is handed +it by the turn entrance that holds the record, the way the model is, and ranks it: the +configured `agents.defaults.reasoningEffort` underneath, a Session Mode's own +`reasoningEffort` over that, this pick over both, a hook's rollback override on top. The +ranking is `session_effort.in_force`, one rule with one home: the turn runs at it +(`session_reasoning_effort` on the loop) and `session.info` / `session.set_effort` report +it, and the turn sends the word in force on every call, None included, so a default cleared +on a running gateway lands on its next turn -- the provider's own copy of the key, frozen at +boot, is never consulted. The +pick's words are the levels the conversation's model declares +(`providers/effort_levels.py`: LiteLLM's static table first, a provider's +`model_overlay.reasoningEfforts` over it); `session.set_effort` offers and validates exactly +those, a model that declares none has no row, and a stored pick the current model does not +declare is dropped to None at the turn and in `session.info` (`session_effort.clamp`). The +install-wide `agents.defaults.reasoningEffort` keeps `REASONING_EFFORTS` +(`config/schema.py`), shared with the console's checker and the agent's own `raven_config` +door -- a key that is clearable on both (null is the provider's own default) and that the +settings page draws, when unset, as a rung of its own rather than as `low`. Lands on the +next turn, like a tier; +reaches no sub-agent, unlike one -- a sub-agent is still told the configured default +(`RAVEN_PARENT_REASONING_EFFORT`), and the TUI's `/effort` is refused inside a direct chat +with an instance, since the pick is the main conversation's and an instance has none. +_Avoid_: reading the pick off `SessionPolicy`. A mode switch replaces the whole policy and a +restart empties it; the record is the pick's one home, and the policy carries only the +mode's own effort. + **Subagent** (`agent/subagent/`): A background agent task spawned by `SubagentManager`. Runs with its own tool set; its result re-enters the session as a `SUBAGENT`-origin `TurnRequest` via Spine submit. Bounded by diff --git a/docs-site/docs/commands.md b/docs-site/docs/commands.md index 7dcd58c760..ac2813fe2b 100644 --- a/docs-site/docs/commands.md +++ b/docs-site/docs/commands.md @@ -48,6 +48,7 @@ on a specific command. From a source checkout, prefix commands with `uv run`. | Task | Read | | --- | --- | | Choose a conversation or one-shot workflow | [Using Raven](using-raven.md) | +| Set how much the model thinks, globally or per conversation | [Using Raven](using-raven.md#set-the-reasoning-effort) | | Configure shipped or external agents | [Agent Integrations](agent-integrations.md) | | Continue or steer a specialist task | [Working with sub-agents](agent-collaboration.md) | | Share browser work with the agent | [Browser collaboration](browser-collaboration.md) | diff --git a/docs-site/docs/commands.zh.md b/docs-site/docs/commands.zh.md index 617490cf51..508fe8058c 100644 --- a/docs-site/docs/commands.zh.md +++ b/docs-site/docs/commands.zh.md @@ -47,6 +47,7 @@ | 任务 | 阅读 | | --- | --- | | 选择对话或单次工作流 | [使用 Raven](using-raven.md) | +| 设置模型的思考深度（全局或单个对话） | [使用 Raven](using-raven.md#set-the-reasoning-effort) | | 配置随 Raven 发布或外部 Agent | [Agent 集成](agent-integrations.md) | | 继续或引导专用任务 | [与子 Agent 协作](agent-collaboration.md) | | 与 Agent 共享浏览器工作 | [浏览器协作](browser-collaboration.md) | diff --git a/docs-site/docs/using-raven.md b/docs-site/docs/using-raven.md index 290461ab26..556993d3b9 100644 --- a/docs-site/docs/using-raven.md +++ b/docs-site/docs/using-raven.md @@ -57,6 +57,41 @@ Use the UI's conversation picker for interactive sessions. A sub-agent `instance` continues that child's conversation; it is not the host session id or a DAG node id. See [DAG Orchestration](orchestration.md). +## Set the reasoning effort + +The reasoning effort is how much the model thinks before each answer. The words +a conversation may pick are the ones its model declares: `minimal`, `low`, +`medium` and `high` on most models, and `xhigh` or `max` where the provider +publishes them. A model that declares none offers no row at all. Each provider +maps the words to its own request (Anthropic's adaptive effort and thinking +budgets, the `reasoning_effort` field of OpenAI-style APIs), and a model with no +thinking to move ignores them. Left unset, the provider's own default applies. + +Two places set it: + +- **For every conversation**: in the WebUI, open Settings, then *Parameters* on + the *Chat* row, then *Reasoning effort*; or set + `agents.defaults.reasoningEffort` in `~/.raven/config.json`. The first rung, + *Provider default*, clears the key. +- **For one conversation**: the *Reasoning effort* row of the model picker in + the WebUI (open it from the model chip beside the composer), or `/effort + ` in the TUI. `/effort` alone reports the effort in force, and + `/effort default` returns the conversation to the global setting. Inside a + direct chat with a sub-agent instance the command is refused: the pick is + the main conversation's, and an instance has none of its own. + +A model's words can also be declared in the provider's +`model_overlay.reasoningEfforts` (Settings, then *Advanced* on the provider, +*Add* under display names). A non-empty declaration outranks LiteLLM's table +for that model, including a route the table marks as not reasoning; leave it +blank to fall back to the table. + +A pick made for one conversation outranks the global setting and the +conversation's mode. It is kept on the conversation's own record, so it survives +a gateway restart and comes back when the conversation is resumed, and it lands +on the next turn. It moves Raven's own model calls only; sub-agents keep the +configured default. + ## Add capability deliberately Use a skill for reusable instructions, MCP for external tools, and a plugin for diff --git a/docs-site/docs/using-raven.zh.md b/docs-site/docs/using-raven.zh.md index fe613c1421..34883bf2bb 100644 --- a/docs-site/docs/using-raven.zh.md +++ b/docs-site/docs/using-raven.zh.md @@ -49,6 +49,32 @@ raven agent --continue -m "Expand the previous summary; do not modify files" 交互式会话通过 UI 的对话选择器切换。子 Agent 的 `instance` 续接子 Agent 对话， 不是宿主 session id 或 DAG node id，详见[DAG 编排](orchestration.md)。 +## 设置思考深度 { #set-the-reasoning-effort } + +思考深度是模型每次回答前思考多少。可选的词由当前模型声明：多数模型是 `minimal`、 +`low`、`medium`、`high`，厂商提供时还会有 `xhigh`、`max`；没有声明任何档位的模型 +不会出现这一行。各厂商把这些词映射成自己的请求参数（Anthropic 的自适应思考强度与 +思考预算，OpenAI 风格接口的 `reasoning_effort` 字段），没有思考能力的模型会忽略它。 +不设置时使用厂商自己的默认值。 + +两处可以设置： + +- **对所有对话**：在 WebUI 打开设置页，在「主对话」一行点「参数」，再选「思考深度」； + 或在 `~/.raven/config.json` 里设置 `agents.defaults.reasoningEffort`。第一档 + 「厂商默认」会清掉这个键。 +- **只对当前对话**：WebUI 模型选择器（点输入框旁的模型标签打开）里的「思考深度」行， + 或 TUI 里的 `/effort <词>`。单独输入 `/effort` 查看当前生效的档位，`/effort default` + 让这个对话重新跟随全局设置。在与某个子智能体实例直聊时这条命令会被拒绝：这个选择 + 属于主对话，实例自己没有思考深度可设。 + +也可以在厂商的 `model_overlay.reasoningEfforts` 里声明某个模型的可用档位 +（设置页该厂商的「高级」→「显示名」下的「添加」）。非空声明优先于 LiteLLM 的表， +包括表里标为不支持 reasoning 的路由；留空则回退到表。 + +单个对话的选择优先于全局设置和该对话的模式。它记录在对话自己的档案里，网关重启后仍在， +恢复对话时一并带回，从下一轮开始生效。它只影响 Raven 自己的模型调用；子智能体仍按全局 +默认值运行。 + ## 有目的地增加能力 { #add-capability-deliberately } 可复用指令用 skill，外部工具用 MCP，运行时贡献用 plugin。需要独立身份和执行配置 diff --git a/i18n/messages.json b/i18n/messages.json index cd942f4b6e..48a1418962 100644 --- a/i18n/messages.json +++ b/i18n/messages.json @@ -241,6 +241,16 @@ "help": "查看或设置推理强度（对运行中的 agent 生效）" } }, + "effort": { + "en": { + "name": "effort", + "help": "show or set how much the model thinks for this conversation [word] - the words are the ones the model declares" + }, + "zh": { + "name": "思考深度", + "help": "查看或设置本会话的思考深度 [word] - 可选词由当前模型声明" + } + }, "fast": { "zh": { "name": "快速模式", @@ -6170,8 +6180,8 @@ "zh": "模式" }, "gui.tier.scope": { - "en": "Reasoning depth, sub-agent mode and the rest follow this setting. This conversation only.", - "zh": "模型思考深度、子智能体运行强度等都跟着这一档走。只作用于本会话。" + "en": "The tier the sub-agents this conversation dispatches run at. This conversation only.", + "zh": "派出的子智能体按这一档运行。只作用于本会话。" }, "gui.tier.failed": { "en": "Could not switch to {name}", @@ -6185,6 +6195,54 @@ "en": "This is a deployment's own mode, not a built-in tier, so it is not passed on to sub-agents.", "zh": "这是部署自带的模式，不是内置档位，因此不会下发给子智能体。" }, + "gui.effort.title": { + "en": "Reasoning effort", + "zh": "思考深度" + }, + "gui.effort.default": { + "en": "Default", + "zh": "默认" + }, + "gui.effort.minimal": { + "en": "Minimal", + "zh": "最少" + }, + "gui.effort.low": { + "en": "Low", + "zh": "低" + }, + "gui.effort.medium": { + "en": "Medium", + "zh": "中" + }, + "gui.effort.high": { + "en": "High", + "zh": "高" + }, + "gui.effort.xhigh": { + "en": "Extra high", + "zh": "极高" + }, + "gui.effort.max": { + "en": "Max", + "zh": "最高" + }, + "gui.effort.scope": { + "en": "How much the model thinks before each answer, for this conversation only. Sub-agents are not affected.", + "zh": "模型每次回答前思考多少，只作用于本会话，不影响子智能体。" + }, + "gui.effort.now": { + "en": "Default follows the global setting, which is {name} right now.", + "zh": "「默认」跟随设置页里的全局值，当前是「{name}」。" + }, + "gui.effort.provider": { + "en": "the provider's own default", + "zh": "厂商默认" + }, + "gui.effort.failed": { + "en": "Could not set the reasoning effort to {name}", + "zh": "思考深度切到「{name}」没有成功" + }, "gui.imodel.title": { "en": "Model", "zh": "模型" @@ -8441,6 +8499,10 @@ "en": "Shown as", "zh": "显示成什么" }, + "gui.settings.providers.effort_ph": { + "en": "Effort levels (comma-separated)", + "zh": "思考深度档位（逗号分隔）" + }, "gui.settings.providers.description_ph": { "en": "Description (optional)", "zh": "说明（可留空）" @@ -8597,6 +8659,10 @@ "en": "Reasoning effort", "zh": "思考深度" }, + "gui.settings.roles.effort_default": { + "en": "Provider default", + "zh": "厂商默认" + }, "gui.settings.roles.effort_minimal": { "en": "Minimal", "zh": "最少" diff --git a/raven/agent/loop/organ_glue.py b/raven/agent/loop/organ_glue.py index 735c8f937c..33f4ff0647 100644 --- a/raven/agent/loop/organ_glue.py +++ b/raven/agent/loop/organ_glue.py @@ -549,7 +549,7 @@ async def _report_skill_read(self, session_key: str, tool_name: str, args: dict[ # CancelledError is not an Exception and still propagates. _WATCH_JUDGEMENT_TIMEOUT_S: float = 60.0 - async def _judge_watch_work(self, message: str, effort_kwargs: dict[str, str]) -> Any: + async def _judge_watch_work(self, message: str, effort_kwargs: dict[str, str | None]) -> Any: """The turn's watch-work verdict, a quiet no when the model does not answer.""" import asyncio @@ -583,10 +583,11 @@ async def _note_watch_work( ) -> str: """The steering line this look earned, or "". The CALLER places it. - ``reasoning_effort`` is the turn's pinned effort: the judgement this + ``reasoning_effort`` is the effort the turn runs at: the judgement this helper may pay for is a model call of the turn like any other, so it - asks for the session's effort too. ``None`` passes nothing and the - provider's configured default stands. + asks for the same one, sent the same way -- None included, which is + the provider's own default rather than the copy the provider took at + boot (``_run_agent_loop``'s ``gen_overrides`` says why). Returned separately rather than appended to ``result`` because of where the result goes next: every tool result is fenced as untrusted data @@ -612,7 +613,7 @@ async def _note_watch_work( reasoning, and ignored, while a fact arriving in a tool result was acted on. """ - effort_kwargs: dict[str, str] = {} if reasoning_effort is None else {"reasoning_effort": reasoning_effort} + effort_kwargs: dict[str, str | None] = {"reasoning_effort": reasoning_effort} # Only a real hand-off answers the question this line asks for the rest # of the turn. The first cut treated every spawn as one and silenced the # nudges on refusal too -- at exactly the moment the model chose to run diff --git a/raven/agent/loop/turn_path.py b/raven/agent/loop/turn_path.py index 3f1b8e5bca..51011bc5d7 100644 --- a/raven/agent/loop/turn_path.py +++ b/raven/agent/loop/turn_path.py @@ -119,6 +119,7 @@ reply_kind, resend_budget, ) +from raven.agent.session_effort import of as session_effort_of from raven.agent.tools.registry import call_failed from raven.agent.tools.removals import RemovalWatch from raven.agent.window import shrink @@ -779,8 +780,10 @@ async def _synthesize_final_on_exhaustion( # the turn's last word. synth_messages = messages + [{"role": "user", "content": prompt}] # The wrap-up is a model call of the same turn, so it pays the turn's - # effort; absent, the provider's configured default stands. - effort_kwargs: dict[str, Any] = {} if reasoning_effort is None else {"reasoning_effort": reasoning_effort} + # effort, sent the way the turn's other calls send it: None included, + # which is the provider's own default rather than the copy it took at + # boot (``gen_overrides`` in ``_run_agent_loop`` says why). + effort_kwargs: dict[str, Any] = {"reasoning_effort": reasoning_effort} # The wrap-up is asked for at the cliff edge, where the transcript is # longest, so it is sized to the room beside it like any other call. if reply_room := self._reply_room(synth_messages, None, model): @@ -933,6 +936,7 @@ async def _run_agent_loop( # noqa: C901 (cc 100: pre-existing, above the ceilin rungs_used: set[str] | None = None, reply_awaited: bool = False, window_reading: dict[str, int] | None = None, + session_effort: str | None = None, ) -> tuple[str | None, list[str], list[dict], LoopOutcome]: """Run the agent iteration loop. @@ -997,6 +1001,11 @@ async def _run_agent_loop( # noqa: C901 (cc 100: pre-existing, above the ceilin newest window a call of this loop measured. It outlives the loop the way ``spend`` does: a turn that fails or is stopped files it on its marker, which can be the only row the turn adds. + + ``session_effort`` is the reasoning effort the session picked for its + own model calls (``raven.agent.session_effort``), read off its record + by the caller the way the model is. It outranks a mode's effort and + the configured default; a hook's rollback override still outranks it. """ spend = spend if spend is not None else TurnSpend(session_key) messages = initial_messages @@ -1128,6 +1137,15 @@ async def _shrink(pressure: WindowPressure, at_model: str | None) -> ShrinkResul # Read once, here: a mode switched mid-turn lands on the next turn. policy = self.session_policy(session_key or "") iteration_cap = policy.max_iterations or self.max_iterations + # The effort this turn's own calls run at, short of a hook's rollback + # override: the session's own pick over its mode's word over the + # configured default as the file has it now -- ``session_effort.in_force``, + # the one rule ``session.info`` reports from too. The pick wins because + # a person set it on this conversation, the mode's is the catalogue's + # word for every conversation in that mode, and all three land on the + # next turn. None is an answer, the provider's own default, and every + # call below sends it as one. + turn_effort = self.session_reasoning_effort(session_key, session_effort) # The other bound on this turn, and the only one a product supplies rather # than the loop: whatever a hook left under ``TURN_BUDGETS_KEY``. Read here # for the same reason the cap is, and defaulting to unbounded, so an agent @@ -1451,16 +1469,19 @@ async def _end_on_the_partial(last: LLMResponse) -> str: # TokenWise before-hook: strategies may rewrite messages, tools, # or model (e.g. CacheOptimizer marks cache_control blocks). - # The session's pinned effort first, a hook's rollback override on - # top: a mode that asks for more thinking sets the turn's default, - # and a gate re-sampling one call may still move that one call. + # The session's effort first, a hook's rollback override on top: a + # mode that asks for more thinking, or the session's own pick, sets + # the turn's default, and a gate re-sampling one call may still + # move that one call. gen_overrides = { - # The configured default underneath, the session's pin over it, - # a hook's rollback override on top. The default is here rather - # than left to the provider because the provider's copy is - # frozen at construction (see ``default_reasoning_effort``). - **({"reasoning_effort": self.default_reasoning_effort} if self.default_reasoning_effort else {}), - **({"reasoning_effort": policy.reasoning_effort} if policy.reasoning_effort else {}), + # The effort in force (``turn_effort``), a hook's rollback + # override on top. Always sent, None included: the provider's + # own copy of the key is frozen at construction (see + # ``default_reasoning_effort``), so a key left out of the call + # let a default cleared on a running gateway keep running at + # the word the gateway booted with. An explicit None is the + # provider's own default, nothing on the wire. + "reasoning_effort": turn_effort, **(pending_gen_overrides or {}), } # Kept for an unchanged resend: it asks again with whatever rode this call. @@ -1860,7 +1881,7 @@ async def _end_on_the_partial(last: LLMResponse) -> str: tool_call.arguments, str(result), agent_output=getattr(result, "agent_output", False) ), watch_request, - reasoning_effort=policy.reasoning_effort, + reasoning_effort=turn_effort, ) # A call that bound a Worker Table (load_playbook) moved the # dispatch enums under an identity assembled before it. @@ -2642,7 +2663,7 @@ async def _end_on_the_partial(last: LLMResponse) -> str: list(fallbacks_left), on_token_delta=wrap_up_gate or on_token_delta, on_reasoning_delta=on_reasoning_delta, - reasoning_effort=policy.reasoning_effort, + reasoning_effort=turn_effort, prompt=_MAX_ITER_SYNTHESIS_PROMPT if stalled_tool is None else _STALLED_SYNTHESIS_PROMPT, fallback=( _STALLED_STATIC_FALLBACK.format(tool=stalled_tool) @@ -3236,6 +3257,9 @@ async def _attempt(seed: list[dict], attempt: int): rungs_used=rungs_used, reply_awaited=reply_awaited, window_reading=reading, + # Read here, beside the model, because this is where the + # session is: the loop below is handed a key, not a record. + session_effort=session_effort_of(session.metadata), ) # Taken BEFORE the first attempt, because the loop appends to the list it is diff --git a/raven/agent/loop/wiring.py b/raven/agent/loop/wiring.py index 9b81c37a0d..9a07592970 100644 --- a/raven/agent/loop/wiring.py +++ b/raven/agent/loop/wiring.py @@ -44,6 +44,7 @@ from raven.agent.subagent.role import is_subagent_process from raven.agent.tools.ask_user import DEFAULT_TIMEOUT_S from raven.contracts.token_strategy import UsageSnapshot +from raven.providers import effort_levels if TYPE_CHECKING: from collections.abc import Sequence @@ -512,13 +513,66 @@ def default_reasoning_effort(self) -> str | None: it only at a restart. Sent as an explicit argument instead, which is the path a session's pinned effort already takes. - None passes nothing, which leaves the provider's own default standing - -- an explicit None would switch it off instead. + None is an answer, not an absence: the turn sends it as an explicit + None, which the provider takes as its own default (nothing on the + wire). Leaving the key out instead let the provider fill it from that + frozen copy, so a key cleared on a running gateway kept every call at + the word it booted with (``session_effort.in_force`` says the rule). """ from raven.config.live import held, reasoning_effort return held(_EFFORT_KEY, lambda: reasoning_effort(self._live_config)) + def session_reasoning_effort(self, session_key: str | None, pick: str | None = None) -> str | None: + """The reasoning effort this session's own calls run at, short of a hook's override. + + ``session_effort.in_force`` over this loop's inputs: the pick the caller + read off the session's record, the session's mode's word, and the + configured default as the file has it now. The session's model then + bounds the answer (``session_effort.clamp``): a level it does not + declare, by LiteLLM's table or its provider overlay, is dropped to None + rather than sent. Beside ``session_tier`` for the reason it gives -- the + turn and the RPC need the one rule, and the policy it reads lives here + -- and reached on ``self`` by the turn, so nothing imports anything new. + """ + from raven.agent import session_effort + from raven.providers.overlay import declared_efforts, overlay_for + from raven.providers.registry import canonical_provider_name, find_by_model, split_model_id + + model_id = ( + self.session_model(session_key) if session_key and self.has_session_binding(session_key) else self.model + ) + value = session_effort.in_force( + pick, self.session_policy(session_key or "").reasoning_effort, self.default_reasoning_effort + ) + provider, _rest = split_model_id(model_id) + if not provider: + spec = find_by_model(model_id) + provider = getattr(spec, "name", "") if spec else "" + try: + from raven.config.schema import _provider_section + + section = ( + _provider_section( + self._live_config.get("providers") or {}, + canonical_provider_name(provider), + ) + if provider + else None + ) + except Exception: # noqa: BLE001 - a bad edit must not fail the turn over an overlay + section = None + overlay = overlay_for(section, provider, model_id) if section is not None else None + clamped = session_effort.clamp(value, effort_levels.levels_for(model_id, declared_efforts(overlay))) + if value is not None and clamped is None: + logger.debug( + "session {}: reasoning effort {} is not declared by {}; using the provider default", + session_key, + value, + model_id, + ) + return clamped + @property def personalization_enabled(self) -> bool: """Whether the personalization flow runs, as the file has it now. diff --git a/raven/agent/session_effort.py b/raven/agent/session_effort.py new file mode 100644 index 0000000000..d859485fdb --- /dev/null +++ b/raven/agent/session_effort.py @@ -0,0 +1,118 @@ +"""The reasoning effort one session picked for its own model calls. + +A conversation can ask for more or less thinking than the configured default +(``agents.defaults.reasoningEffort``) without moving that default: a quick +question is not worth the time a deep one is, and the person asking is the one +who knows which is which. The pick rides the session's metadata beside +``model``, and for the same reason -- it is this conversation's own, it has to +outlive the gateway (a policy held only in memory goes back to the default on +every restart, which is what the tier does), and the turn reads it where it +reads the model. + +Where it ranks (``in_force``, the one rule the turn runs at and the RPC +reports): the configured default underneath, a Session Mode's own +``reasoningEffort`` over that, this pick over both, and a hook's rollback +override over the lot (``_run_agent_loop``'s ``gen_overrides``). The model then +bounds the result (``clamp``): a level the current model does not declare is +dropped to None, the provider's own default, and the stored pick is left on the +record so it applies again if a later model declares it. Raven's own calls +only: a sub-agent is told the configured default, as before. + +The words are ``REASONING_EFFORT_LADDER`` (``contracts/llm_provider.py``); the +provider maps each to its own wire, and a model with no thinking to move +ignores it. The install-wide ``agents.defaults.reasoningEffort`` keeps its +four-word ``REASONING_EFFORTS`` list; this module accepts the six-word ladder +because a session pick may now be ``xhigh``/``max``. +""" + +from __future__ import annotations + +from typing import Any + +from loguru import logger + +from raven.contracts.llm_provider import REASONING_EFFORT_LADDER + +#: Where the pick is kept on a session: the metadata dict ``model`` lives in. +#: Null, not absent, once cleared -- a metadata patch can set a key and not +#: remove one, and both spellings read as "follows the default" here. +METADATA_KEY = "reasoning_effort" + + +def of(metadata: Any) -> str | None: + """The pick a session record carries, or None while it follows the default. + + A word outside ``REASONING_EFFORT_LADDER`` reads as none: a record written + by a later raven is not a request this one can send, and the turn is better + off at the default than refused by the provider. + """ + if not isinstance(metadata, dict): + return None + stored = metadata.get(METADATA_KEY) + return stored if isinstance(stored, str) and stored in REASONING_EFFORT_LADDER else None + + +def in_force(pick: str | None, mode_effort: str | None, configured: str | None) -> str | None: + """The effort a session's own calls run at, short of a hook's rollback override. + + One rule with one home. The turn freezes it at its start (``_run_agent_loop``) + and ``session.info`` / ``session.set_effort`` report it, and the two used to + spell the ranking out apart -- and to disagree on what no word meant. The + turn left the key out of the call, which let the provider fill it from its + own copy, frozen at boot from the same config key; so a default cleared on + a running gateway kept every call at the word it booted with, while the RPC + reported the clear as in force. Here None is one answer with one meaning, + the provider's own default, and the turn sends it as such: an explicit None, + nothing on the wire. The provider's boot copy is never consulted. + + The pick is the record's (``of`` / ``read``), the mode's word is the + session's policy's, and the configured default is the file's as it is now + (the loop's live reader, the RPC's fresh load): each caller reads its own + inputs and this is the one place they are ranked. + """ + return pick or mode_effort or configured or None + + +def clamp(value: str | None, levels: tuple[str, ...] | None) -> str | None: + """The value if this model declares it; None otherwise. + + ``None`` levels mean the table could not be read, which is not a model + answer: the caller's value is left alone (fail open). An empty tuple means + the model declares no reasoning at all, so even a valid word is dropped. + """ + if levels is None: + return value + return value if value in levels else None + + +def read(sessions: Any, session_key: str | None) -> str | None: + """What one session picked, read off its record; None for one following the default.""" + if sessions is None or not session_key: + return None + try: + # `peek`, not `get_or_create`, for the reason knowledge_scope.read + # gives: a read must not cache a fresh empty session under a key the + # manager has never seen. + held = sessions.peek(session_key) + except Exception: # noqa: BLE001 - a session that cannot be read has no pick + return None + return of(getattr(held, "metadata", None)) if held is not None else None + + +def write(sessions: Any, session_key: str, effort: str | None) -> bool: + """Record the pick (None clears it), and answer whether it reached the disk. + + A session that has no record yet -- a draft the first message has not + filed -- holds it on its in-memory copy, which the first turn's save then + writes. The answer tells the two apart for a caller that cares; + ``session.set_effort`` does not pass it on, since the next turn runs at + the pick either way and the control is drawn from the reply's ``effort``. + """ + persisted = False + try: + persisted = bool(sessions.append_metadata_patch(session_key, {METADATA_KEY: effort})) + except Exception: # noqa: BLE001 - the in-memory copy below still carries it + logger.warning("session {}: failed to persist the reasoning effort", session_key) + if not persisted: + sessions.get_or_create(session_key).metadata[METADATA_KEY] = effort + return persisted diff --git a/raven/config/schema.py b/raven/config/schema.py index 836861299a..39f5d8fad5 100644 --- a/raven/config/schema.py +++ b/raven/config/schema.py @@ -353,6 +353,12 @@ class ModelOverlay(Base): request may actually carry is still decided by `capabilities.supports_vision` and `ProviderSpec`, which is why a wrong tag here costs a wrong picture of a model and never a wrong call. + + ``reasoning_efforts`` is the exception to the display-only rule: it states + the words this model accepts for a per-conversation pick, and + ``providers/effort_levels.py`` prefers it over the LiteLLM table. A word + outside the effort ladder is refused at the RPC door; an empty list means + nothing was stated and falls back to the table. """ label: str = "" @@ -360,6 +366,7 @@ class ModelOverlay(Base): capabilities: list[str] = Field(default_factory=list) input_modalities: list[str] = Field(default_factory=list) output_modalities: list[str] = Field(default_factory=list) + reasoning_efforts: list[str] = Field(default_factory=list) class ProviderEndpoint(Base): @@ -797,6 +804,16 @@ class GatewayPageConfig(Base): """The built-in tiers, cheapest first. The order is what makes a clamp possible, so it is a constant here rather than something read back off the catalogue.""" +REASONING_EFFORTS: tuple[str, ...] = ("minimal", "low", "medium", "high") +"""The words ``agents.defaults.reasoningEffort`` accepts, least first: the +install-wide default. A session's own pick is checked against the levels its +model declares (``providers/effort_levels.py``), which may also carry +``xhigh``/``max``; this list is that picker's fallback only when the LiteLLM +table cannot be read. Each provider maps a word to its own wire -- the adaptive +efforts and thinking budgets in +``providers/anthropic_messages_provider.py``, ``reasoning_effort`` on the +OpenAI-shaped ones -- and a model with no thinking to move ignores it.""" + ROUTE_REQUIREMENTS: tuple[str, ...] = ("image_generation", "image_search") """What a route may declare its target's own pipeline spends. diff --git a/raven/config/self_surface.py b/raven/config/self_surface.py index 82a2b87d56..8394a201f8 100644 --- a/raven/config/self_surface.py +++ b/raven/config/self_surface.py @@ -45,6 +45,7 @@ from raven.config.agent_names import GENERIC_AGENT from raven.config.keys import key_as_written as _spelled from raven.config.loader import EXTENSION_KEYS, get_config_path, read_raw_or_raise +from raven.config.schema import REASONING_EFFORTS from raven.providers.registry import PROVIDERS from raven.utils.atomic_io import atomic_update @@ -152,7 +153,6 @@ class Section: settings: tuple[Setting, ...] = field(default_factory=tuple) -_REASONING = ("minimal", "low", "medium", "high") _WEB_SEARCH = ( "serper", "anysearch", @@ -212,13 +212,17 @@ def _skill_hub_in_effect(raw: dict[str, Any]) -> bool: note='value is {"provider": "", "model": ""}; takes over from this ' "conversation's next turn", ), + # The words are ``REASONING_EFFORTS``, the one list the console's + # checker and the session's pick read too; null hands the choice + # back to the provider's own default, as the settings page offers. Setting( "agents.defaults.reasoningEffort", - "Reasoning effort sent with each model call", + "Reasoning effort sent with each model call; null is the provider's own default", "enum", _E.NEXT_TURN, writer="settings", - choices=_REASONING, + choices=REASONING_EFFORTS, + nullable=True, ), Setting( "agents.defaults.maxToolIterations", diff --git a/raven/providers/catalog.py b/raven/providers/catalog.py index dbaf50f184..bf2c4f2c76 100644 --- a/raven/providers/catalog.py +++ b/raven/providers/catalog.py @@ -69,6 +69,7 @@ class ModelRow: capabilities: tuple[str, ...] = () input_modalities: tuple[str, ...] = () output_modalities: tuple[str, ...] = () + reasoning_efforts: tuple[str, ...] = () @property def described(self) -> bool: @@ -215,6 +216,7 @@ def _with_overlay(row: ModelRow, overlay: "ModelOverlay | None") -> ModelRow: "capabilities": clean_tags(overlay.capabilities, CAPABILITIES) or row.capabilities, "input_modalities": clean_tags(overlay.input_modalities, MODALITIES) or row.input_modalities, "output_modalities": clean_tags(overlay.output_modalities, MODALITIES) or row.output_modalities, + "reasoning_efforts": tuple(overlay.reasoning_efforts) if overlay.reasoning_efforts else row.reasoning_efforts, } described = row.described or bool(overlay.label or overlay.description) return replace(row, **changed, source=SOURCE_OVERLAY if described else row.source) diff --git a/raven/providers/codex_catalog.py b/raven/providers/codex_catalog.py index c5cfb84fd1..b170d493f5 100644 --- a/raven/providers/codex_catalog.py +++ b/raven/providers/codex_catalog.py @@ -9,6 +9,11 @@ Hidden entries are dropped. The catalogue marks some (`codex-auto-review`, a watermarked variant) `visibility: "hide"` -- they are reachable but not meant to be offered. + +The cache also keeps each model's ``supported_reasoning_levels``, the account's +own declaration of the effort words it takes. ``account_efforts`` answers from +that cache for the turn, or fetches when the RPC says it may; a level outside +this build's ladder (``ultra``) is dropped there. """ from __future__ import annotations @@ -16,6 +21,8 @@ import time from typing import Any +from raven.contracts.llm_provider import REASONING_EFFORT_LADDER + CATALOG_URL = "https://chatgpt.com/backend-api/codex/models" #: A client newer than any release, so the catalogue answers with every model the @@ -39,8 +46,8 @@ #: offline, are states that change -- an account's entitlements are not. _FAILURE_TTL_SECONDS = 30 -#: (credential fingerprint, monotonic stamp, slugs) -_cache: tuple[object, float, tuple[str, ...]] | None = None +#: (credential fingerprint, monotonic stamp, slug -> the levels it states) +_cache: tuple[object, float, dict[str, tuple[str, ...]]] | None = None #: (credential fingerprint, monotonic stamp) _failure: tuple[object, float] | None = None @@ -84,22 +91,57 @@ def account_models(*, timeout: float = 5.0, strict: bool = False) -> tuple[str, now = time.monotonic() fingerprint = _credential_fingerprint() if _cache is not None and _cache[0] == fingerprint and now - _cache[1] < _CACHE_TTL_SECONDS: - return _cache[2] + return tuple(_cache[2]) if not strict and _failure is not None and _failure[0] == fingerprint and now - _failure[1] < _FAILURE_TTL_SECONDS: return () try: - slugs = _fetch(timeout=timeout) + models = _fetch(timeout=timeout) except Exception: if strict: raise _failure = (fingerprint, now) return () - _cache = (fingerprint, now, slugs) + _cache = (fingerprint, now, models) _failure = None - return slugs + return tuple(models) + + +def account_efforts(slug: str, *, refresh: bool = False, timeout: float = 5.0) -> tuple[str, ...]: + """The reasoning levels this account's catalogue states for ``slug``, least first. + + Never ``None``: an answer that exists is kept as last-good past its refresh + time, because a turn may not wait on the network and dropping a stored pick + to the provider default is safer than sending an unvalidated word. ``()`` + means the catalogue has no answer to offer -- it never was read, the fetch + failed with nothing cached, or it does not offer this model (or states no + levels for it). The turn and reports read cache-only; an explicit effort + set passes ``refresh=True`` to fetch, and a failed refresh falls back to + the last-good answer. + """ + global _cache, _failure + + now = time.monotonic() + fingerprint = _credential_fingerprint() + cached = _cache if _cache is not None and _cache[0] == fingerprint else None + if cached is not None and now - cached[1] < _CACHE_TTL_SECONDS: + return cached[2].get(slug, ()) + if not refresh: + return cached[2].get(slug, ()) if cached is not None else () + if _failure is not None and _failure[0] == fingerprint and now - _failure[1] < _FAILURE_TTL_SECONDS: + return cached[2].get(slug, ()) if cached is not None else () + + try: + models = _fetch(timeout=timeout) + except Exception: + _failure = (fingerprint, now) + return cached[2].get(slug, ()) if cached is not None else () + + _cache = (fingerprint, now, models) + _failure = None + return models.get(slug, ()) def reset_cache() -> None: @@ -110,7 +152,7 @@ def reset_cache() -> None: _failure = None -def _fetch(*, timeout: float) -> tuple[str, ...]: +def _fetch(*, timeout: float) -> dict[str, tuple[str, ...]]: import httpx from raven.providers.chatgpt_token import access_token_and_account @@ -128,19 +170,29 @@ def _fetch(*, timeout: float) -> tuple[str, ...]: ) response.raise_for_status() - return _visible_slugs(response.json()) + return _visible_models(response.json()) -def _visible_slugs(payload: Any) -> tuple[str, ...]: +def _visible_models(payload: Any) -> dict[str, tuple[str, ...]]: + """Every offered slug, in catalogue order, with the levels it states.""" if not isinstance(payload, dict): - return () + return {} - out: list[str] = [] + out: dict[str, tuple[str, ...]] = {} for entry in payload.get("models") or (): if not isinstance(entry, dict) or entry.get("visibility") == "hide": continue slug = entry.get("slug") if isinstance(slug, str) and slug: - out.append(slug) + out[slug] = _levels(entry) + + return out + - return tuple(out) +def _levels(entry: dict[str, Any]) -> tuple[str, ...]: + """The ladder words this entry offers, least first; an unstated set is empty.""" + raw = entry.get("supported_reasoning_levels") + offered = { + str(item.get("effort")) for item in raw or () if isinstance(item, dict) and isinstance(item.get("effort"), str) + } + return tuple(name for name in reversed(REASONING_EFFORT_LADDER) if name in offered) diff --git a/raven/providers/effort_levels.py b/raven/providers/effort_levels.py new file mode 100644 index 0000000000..936408dde2 --- /dev/null +++ b/raven/providers/effort_levels.py @@ -0,0 +1,255 @@ +"""Which reasoning-effort levels a model declares, from LiteLLM's static table +or the Codex account catalogue. + +LiteLLM's table is read offline from ``import_litellm().model_cost``; +``get_model_info`` is deliberately not used because it may query a provider's +endpoint. The Codex account catalogue is the one provider-owned source: its +``supported_reasoning_levels`` are the account's own declaration for models no +static table carries. It is network data, so the turn reads only the cache +``codex_catalog`` keeps; ``refresh`` lets the RPC warm that cache. A level the +account states outside this build's ladder (``ultra``) is not offered. + +The answer is tri-state for the static table: a tuple of levels when the model +or the user declares them, ``()`` when the model declares none, and ``None`` +when the table itself cannot be read (an internal failure, not a model answer). +The Codex account source never answers ``None``: an expired answer is kept as +last-good, and a cold cache has no levels to offer (``()``), so a stored pick is +never sent against an unreadable account. Level semantics mirror LiteLLM's own +validator: medium/high are the baseline, minimal/low are opt-out, xhigh/max are +opt-in. ``none`` is ignored: Raven's None means "provider default", not +"reasoning off". +""" + +from __future__ import annotations + +from collections import defaultdict +from functools import lru_cache +from typing import Any, Sequence + +#: The per-level flags, in the order the derivation asks about them. +EFFORT_FLAGS = ( + "supports_minimal_reasoning_effort", + "supports_low_reasoning_effort", + "supports_xhigh_reasoning_effort", + "supports_max_reasoning_effort", +) + +#: The public order, least first. A declared overlay is normalized to it. +LEVEL_ORDER = ("minimal", "low", "medium", "high", "xhigh", "max") + + +def levels_for(model: str, declared: Sequence[str] = (), *, refresh: bool = False) -> tuple[str, ...] | None: + """The levels this model declares, or ``()``/``None`` per the module docstring. + + ``refresh`` lets an explicit effort set ask the Codex account catalogue, + off the event loop; the turn and reports leave it false and read the cache + only, where an expired answer is still its last-good one. + """ + if declared: + known = set(declared) + return tuple(name for name in LEVEL_ORDER if name in known) + from raven.providers.registry import canonical_provider_name, split_model_id + + provider, rest = split_model_id(model) + if canonical_provider_name(provider) == "openai_codex": + from raven.providers import codex_catalog + + return codex_catalog.account_efforts(rest, refresh=refresh) + index = _index() + if index is None: + return None + by_key, by_name = index + route = _route_row(model, by_key) + if route is not None and route.get("supports_reasoning") is False: + return () + authority = _authority_row(model, by_name) + if route is not None: + merged = _merged(route, authority) + return _derive(merged) if merged.get("supports_reasoning") is True else () + if authority is not None and authority.get("supports_reasoning") is True: + return _derive(authority) + return () + + +def reset() -> None: + """Drop the cached table/index; tests and a LiteLLM reload use it.""" + _index.cache_clear() + + +def _table() -> dict[str, dict[str, Any]] | None: + try: + from raven.providers.litellm_setup import import_litellm + + table = getattr(import_litellm(), "model_cost", None) + except Exception: + return None + if not isinstance(table, dict) or not table: + return None + return {key: row for key, row in table.items() if isinstance(row, dict)} + + +@lru_cache(maxsize=1) +def _index() -> tuple[dict[str, dict[str, Any]], dict[str, tuple[tuple[str, dict[str, Any]], ...]]] | None: + table = _table() + if not table: + return None + by_name: dict[str, list[tuple[str, dict[str, Any]]]] = defaultdict(list) + for key, row in table.items(): + by_name[_normalized_name(key)].append((key, row)) + return table, {key: tuple(rows) for key, rows in by_name.items()} + + +def _candidates(model: str) -> list[tuple[str, bool]]: + """Table keys to try, each with whether a bare key needs provider agreement.""" + from raven.providers.registry import metadata_model_id, split_model_id + + provider, rest = split_model_id(model) + out: list[tuple[str, bool]] = [(model, False)] + metadata = metadata_model_id(model) + if metadata: + out.append((metadata, False)) + raw_head = (model or "").partition("/")[0] + if provider and rest and raw_head and raw_head != provider: + out.append((f"{provider}/{rest}", False)) + from raven.providers.wire import metadata_candidates + + for candidate in metadata_candidates(model): + out.append((candidate, "/" not in candidate)) + try: + from litellm.utils import _get_potential_model_names + + names = _get_potential_model_names(model=model, custom_llm_provider=None) + for name in ( + "combined_model_name", + "stripped_model_name", + "combined_stripped_model_name", + "split_model", + ): + value = names.get(name) + if isinstance(value, str) and value: + out.append((value, "/" not in value)) + except Exception: + pass + seen: set[str] = set() + unique: list[tuple[str, bool]] = [] + for candidate, needs_agreement in out: + if candidate in seen: + continue + seen.add(candidate) + unique.append((candidate, needs_agreement)) + return unique + + +def _provider_names(provider: str) -> set[str]: + from raven.providers.registry import canonical_provider_name, find_by_name + + spec = find_by_name(provider) if provider else None + names = set(spec.route_names) if spec is not None else set() + if spec is not None: + metadata_prefix = getattr(spec, "metadata_prefix", None) + if metadata_prefix: + names.add(metadata_prefix) + if provider: + names.add(canonical_provider_name(provider)) + return {canonical_provider_name(name) for name in names if name} + + +def _provider_agrees(row_provider: Any, model: str) -> bool: + from raven.providers.registry import canonical_provider_name, find_by_model, split_model_id + + if not isinstance(row_provider, str) or not row_provider: + return False + provider, _rest = split_model_id(model) + if not provider: + spec = find_by_model(model) + provider = getattr(spec, "name", "") if spec else "" + return canonical_provider_name(row_provider) in _provider_names(provider) + + +def _route_row(model: str, by_key: dict[str, dict[str, Any]]) -> dict[str, Any] | None: + for candidate, needs_agreement in _candidates(model): + row = by_key.get(candidate) + if row is None: + continue + if needs_agreement and not _provider_agrees(row.get("litellm_provider"), model): + continue + return row + return None + + +def _derive(row: dict[str, Any]) -> tuple[str, ...]: + levels: list[str] = [] + if row.get("supports_minimal_reasoning_effort") is not False: + levels.append("minimal") + if row.get("supports_low_reasoning_effort") is not False: + levels.append("low") + levels += ["medium", "high"] + if row.get("supports_xhigh_reasoning_effort") is True: + levels.append("xhigh") + if row.get("supports_max_reasoning_effort") is True: + levels.append("max") + return tuple(levels) + + +def _canonical_vendor(model: str) -> str: + from raven.providers.catalog import _vendor_id + from raven.providers.registry import canonical_provider_name, find_by_model, split_model_id + from raven.providers.registry_data import row_for + + provider, _rest = split_model_id(model) + if not provider: + spec = find_by_model(model) + provider = getattr(spec, "name", "") if spec else "" + if not provider: + return "" + row = row_for(canonical_provider_name(provider), _vendor_id(provider, model)) + return split_model_id(row.model_id)[0] if row is not None else "" + + +def _row_provider_names(row: dict[str, Any]) -> set[str]: + name = row.get("litellm_provider") + return _provider_names(name) if isinstance(name, str) and name else set() + + +def _authority_row( + model: str, + by_name: dict[str, tuple[tuple[str, dict[str, Any]], ...]], +) -> dict[str, Any] | None: + from raven.providers.registry import split_model_id + + route, rest = split_model_id(model) + rows = by_name.get(_normalized_name(rest or model), ()) + declaring = [(key, row) for key, row in rows if row.get("supports_reasoning") is True] + if not declaring: + return None + vendor = _canonical_vendor(model) + vendor_names = _provider_names(vendor) if vendor else set() + accepted = vendor_names | _provider_names(route) + matching = [(key, row) for key, row in declaring if _row_provider_names(row) & accepted] + if not matching: + return None + vendor_rows = [pair for pair in matching if _row_provider_names(pair[1]) & vendor_names] + pool = vendor_rows or matching + return max( + pool, + key=lambda pair: ( + sum(1 for flag in EFFORT_FLAGS if pair[1].get(flag) is not None), + pair[0], + ), + )[1] + + +def _merged(route: dict[str, Any], authority: dict[str, Any] | None) -> dict[str, Any]: + merged = dict(route) + if authority is None: + return merged + for flag in ("supports_reasoning", *EFFORT_FLAGS): + if merged.get(flag) is None: + merged[flag] = authority.get(flag) + return merged + + +def _normalized_name(model_id: str) -> str: + from raven.providers.registry_data import _normalized_name as normalize + + return normalize(model_id) diff --git a/raven/providers/overlay.py b/raven/providers/overlay.py new file mode 100644 index 0000000000..888ddbc0df --- /dev/null +++ b/raven/providers/overlay.py @@ -0,0 +1,38 @@ +"""The user's own description of one model, read off a provider section.""" + +from __future__ import annotations + +from typing import Any + + +def overlay_for(section: Any, provider: str, model: str) -> Any | None: + """The overlay entry for (provider, model), keyed by identity, or None. + + ``section`` is either a parsed provider config (``model_overlay``) or the + live config's raw section (``modelOverlay``); the two readers must not + carry separate spellings of the same lookup. + """ + from raven.providers.wire import merge_key + + if section is None: + return None + if isinstance(section, dict): + overlay = section.get("modelOverlay") or section.get("model_overlay") or {} + else: + overlay = getattr(section, "model_overlay", None) or {} + target = merge_key(provider, model) + for stored, value in overlay.items(): + if merge_key(provider, stored) == target: + return value + return None + + +def declared_efforts(overlay: Any) -> tuple[str, ...]: + """The reasoning-effort words an overlay states, from a model or a raw dict.""" + if overlay is None: + return () + if isinstance(overlay, dict): + values = overlay.get("reasoningEfforts") or overlay.get("reasoning_efforts") or () + else: + values = getattr(overlay, "reasoning_efforts", ()) or () + return tuple(value for value in values if isinstance(value, str)) diff --git a/raven/rpc/methods/console.py b/raven/rpc/methods/console.py index ee47300d9f..2269d380d9 100644 --- a/raven/rpc/methods/console.py +++ b/raven/rpc/methods/console.py @@ -25,7 +25,7 @@ from raven.config.env_file import MIRRORED_KEYS, refresh_env_file from raven.config.keys import key_as_written as _as_written -from raven.config.schema import WEB_VENDOR_ENV_VARS, WebFetchProvider, WebSearchProvider +from raven.config.schema import REASONING_EFFORTS, WEB_VENDOR_ENV_VARS, WebFetchProvider, WebSearchProvider from raven.rpc import LOCAL_CHANNEL from raven.rpc.errors import ConfigValidationError from raven.rpc.files import UPLOAD_DIR as _UPLOAD_DIR @@ -1137,6 +1137,16 @@ def chk(v: Any) -> int | None: return chk +def _chk_enum_or_null(key: str, *allowed: Any): + """An enum a surface may also clear: null takes the key back to its default.""" + inner = _chk_enum(key, *allowed) + + def chk(v: Any) -> Any: + return None if v is None else inner(v) + + return chk + + _SETTINGS_SIMPLE_KEYS: dict[str, Any] = { "tools.exec.timeout": _chk_int("tools.exec.timeout", 5, 3600), "tools.web.search.apiKey": _chk_str("tools.web.search.apiKey", 200), @@ -1178,7 +1188,9 @@ def chk(v: Any) -> int | None: "embedding": _chk_pin_pair("embedding", "model", "provider"), "vision": _chk_pin_pair("vision", "model", "provider"), "agents.defaults.enablePersonalization": _chk_bool("agents.defaults.enablePersonalization"), - "agents.defaults.reasoningEffort": _chk_enum("agents.defaults.reasoningEffort", "minimal", "low", "medium", "high"), + # Clearable: null hands the choice back to the provider's own default, + # which is what an unset key means and what the settings page offers. + "agents.defaults.reasoningEffort": _chk_enum_or_null("agents.defaults.reasoningEffort", *REASONING_EFFORTS), "permissions.mode": _chk_enum("permissions.mode", "ask", "smart", "full"), "agents.defaults.maxToolIterations": _chk_int("agents.defaults.maxToolIterations", 1, 200), "agents.defaults.contextWindowTokens": _chk_int_or_null("agents.defaults.contextWindowTokens", 1024, 100_000_000), diff --git a/raven/rpc/methods/model.py b/raven/rpc/methods/model.py index af0d0f0697..28aea3b392 100644 --- a/raven/rpc/methods/model.py +++ b/raven/rpc/methods/model.py @@ -263,7 +263,7 @@ def _model_labels(slug: str, models: "list[str]", *, section: Any = _UNLOADED) - for model in models: row = describe(slug, model, overlay=_overlay_for(overlays, slug, model)) window = resolve_context_window(model, allow_fetch=False) - if not (row.described or row.tagged or window): + if not (row.described or row.tagged or window or row.reasoning_efforts): # A model only the vendor's own list names still has the kind that # list proved, which is what keeps an embedding model out of the # chat slot's column. @@ -275,6 +275,8 @@ def _model_labels(slug: str, models: "list[str]", *, section: Any = _UNLOADED) - entry["description"] = row.description if row.capabilities: entry["capabilities"] = list(row.capabilities) + if row.reasoning_efforts: + entry["reasoning_efforts"] = list(row.reasoning_efforts) if row.input_modalities: entry["input_modalities"] = list(row.input_modalities) if row.output_modalities: @@ -738,6 +740,18 @@ def _stated_overlay(parsed: "ModelAddModelParams") -> dict[str, object]: raise ConfigValidationError(f"unknown {field}: {', '.join(unknown)}", data={"field": field}) if cleaned: stated[field] = list(cleaned) + if parsed.reasoning_efforts is not None: + from raven.contracts.llm_provider import REASONING_EFFORT_LADDER + + allowed = tuple(reversed(REASONING_EFFORT_LADDER)) + cleaned = clean_tags(parsed.reasoning_efforts, allowed) + if len(cleaned) != len(set(parsed.reasoning_efforts)): + unknown = sorted(set(parsed.reasoning_efforts) - set(cleaned)) + raise ConfigValidationError( + f"unknown reasoning effort: {', '.join(unknown)}", + data={"field": "reasoning_efforts"}, + ) + stated["reasoningEfforts"] = list(cleaned) return stated diff --git a/raven/rpc/methods/session.py b/raven/rpc/methods/session.py index 40ec25f8a7..9f771816c2 100644 --- a/raven/rpc/methods/session.py +++ b/raven/rpc/methods/session.py @@ -36,8 +36,12 @@ from loguru import logger +from raven.agent import session_effort from raven.config.loader import drain_migration_notices, load_config, raven_home from raven.config.raven import load_raven_config +from raven.config.schema import REASONING_EFFORTS +from raven.providers import effort_levels +from raven.providers.overlay import declared_efforts, overlay_for from raven.providers.rates import resolve_context_window from raven.providers.usage import reported_cost, token_count from raven.rpc.errors import ( @@ -274,6 +278,9 @@ async def _default_session_info( # through turn.send's bookkeeping, and it is as much in flight as a # typed one. "running": bool(session_key) and turn_module.is_session_answering(session_key), + # What this session's own calls think at, for the status bar beside the + # model: its own pick, else what it would run at without one. + "reasoning_effort": _effort_in_force(agent_loop, config, session_key, model_id), } # Nudge the status bar to run `raven upgrade` when the cached latest release @@ -544,6 +551,60 @@ async def session_create( } +def _mode_effort(agent_loop: object, session_key: str | None) -> str | None: + """The session's mode's own effort, where the catalogue sets one (``raven-code`` does).""" + policy = getattr(agent_loop, "session_policy", None) + held = policy(session_key or "") if callable(policy) and session_key else None + mode_effort = getattr(held, "reasoning_effort", None) + return mode_effort if isinstance(mode_effort, str) and mode_effort else None + + +def _configured_effort(config: Any) -> str | None: + """``agents.defaults.reasoningEffort`` as the file has it now, or None for an unset key. + + Read fresh rather than off the loop, the way ``default_reasoning_effort`` + reads the file, so a settings edit shows here without a restart. + """ + configured = getattr(getattr(getattr(config, "agents", None), "defaults", None), "reasoning_effort", None) + return configured if isinstance(configured, str) and configured else None + + +def _default_effort(agent_loop: object, config: Any, session_key: str | None) -> str | None: + """What a session runs at with no pick of its own: the one rule, with no pick. + + ``session_effort.in_force`` ranks it, the same rule the turn runs at, so + what this reports and what the next turn sends cannot drift apart: None is + the provider's own default on both sides. + """ + return session_effort.in_force(None, _mode_effort(agent_loop, session_key), _configured_effort(config)) + + +def _effort_levels(model_id: str, config: Any, *, refresh: bool = False) -> tuple[str, ...] | None: + """What this model declares, with the provider's overlay as the authority.""" + from raven.providers.registry import canonical_provider_name, find_by_model, split_model_id + + provider, _rest = split_model_id(model_id) + if not provider: + spec = find_by_model(model_id) + provider = getattr(spec, "name", "") if spec else "" + section = getattr(config, "providers", {}).get(canonical_provider_name(provider)) if provider else None + overlay = overlay_for(section, provider, model_id) if section is not None else None + return effort_levels.levels_for(model_id, declared_efforts(overlay), refresh=refresh) + + +def _effort_in_force( + agent_loop: object, + config: Any, + session_key: str | None, + model_id: str | None = None, +) -> str | None: + """The effort this session's own calls run at: its pick over the default, clamped to its model.""" + model_id = model_id or _session_model(agent_loop, config, session_key) + picked = session_effort.read(manager_for(agent_loop, config), session_key) if session_key else None + value = session_effort.in_force(picked, _mode_effort(agent_loop, session_key), _configured_effort(config)) + return session_effort.clamp(value, _effort_levels(model_id, config)) + + def _read_knowledge(agent_loop: object, config: object, session_key: str | None) -> tuple[str, ...]: """What one session may search, for the bundle that reports it.""" from raven.agent import knowledge_scope @@ -1594,6 +1655,83 @@ def _current() -> str | None: return {"mode": tier, "availableModes": menu} +async def session_set_effort( + params: dict, + *, + agent_loop_factory: "AgentLoopFactory | None" = None, +) -> dict: + """``session.set_effort`` -- report, set or clear this session's reasoning effort. + + The effort this session's own model calls run at, as a person picked it + for this conversation: the shape ``session.set_mode`` has -- three calls + told apart by which fields are present -- over a different knob. A tier + moves what raven asks of its sub-agents; this moves how much raven's own + model thinks before each answer, and reaches no sub-agent. + + Kept on the session's record (``raven.agent.session_effort``) rather than + on the loop's in-memory policy, so it outlives the gateway and a resumed + conversation runs at what it was left at. A set lands on the next turn. + + Every answer carries ``default`` -- what the session runs at without a + pick of its own -- so a control can say what "default" means on this + install, and ``options``, the levels this model declares, so it need not + spell the words itself. A draft names its model, since it has no session + record to read one from. + + A report reads the account catalogue's cache; only an explicit set or + clear refreshes it, and that refresh runs off the event loop, so a UI read + never blocks on the network. + """ + session_key = str(params.get("session_key") or "") + raw = params.get("effort") + wanted = str(raw) if isinstance(raw, str) and raw else None + requested_model = params.get("model") + loop = _safe_invoke_factory(agent_loop_factory) + config = load_config() + mgr = manager_for(loop, config) + model_id = ( + str(requested_model) + if not session_key and isinstance(requested_model, str) and requested_model + else _session_model(loop, config, session_key) + ) + refresh_catalogue = wanted is not None + model_levels = ( + await asyncio.to_thread(_effort_levels, model_id, config, refresh=True) + if refresh_catalogue + else _effort_levels(model_id, config) + ) + + def _reply(stored: str | None) -> dict: + return { + "session_key": session_key, + "effort": session_effort.clamp(stored, model_levels), + "default": session_effort.clamp(_default_effort(loop, config, session_key or None), model_levels), + "options": list(model_levels) if model_levels is not None else list(REASONING_EFFORTS), + } + + if wanted is None and not params.get("clear"): + # A draft asks too, with no key: the model, the default and the words + # are what its first turn will run at, and the row is drawn before it + # is sent. + return _reply(session_effort.read(mgr, session_key) if session_key else None) + if not session_key: + raise ConfigValidationError("session_key is required to set or clear a reasoning effort") + effort = None if params.get("clear") else wanted + if effort is not None: + allowed = model_levels if model_levels is not None else REASONING_EFFORTS + if not allowed: + raise ConfigValidationError(f"{model_id} declares no reasoning effort") + if effort not in allowed: + raise ConfigValidationError( + f"no reasoning effort {effort!r} for {model_id}; the words are {', '.join(allowed)}" + ) + previous = session_effort.read(mgr, session_key) + session_effort.write(mgr, session_key, effort) + if previous != effort: + logger.info("session {} reasoning effort {} -> {}", session_key, previous or "", effort or "") + return _reply(effort) + + async def session_set_harness( params: dict, *, @@ -1691,6 +1829,9 @@ async def _set_mode(params: dict) -> dict: async def _set_harness(params: dict) -> dict: return await session_set_harness(params, agent_loop_factory=agent_loop_factory) + async def _set_effort(params: dict) -> dict: + return await session_set_effort(params, agent_loop_factory=agent_loop_factory) + dispatcher.register("session.create", _create) dispatcher.register("session.close", _close) dispatcher.register("session.resume", _resume) @@ -1707,6 +1848,7 @@ async def _set_harness(params: dict) -> dict: dispatcher.register("session.branch", _branch) dispatcher.register("session.export", _export) dispatcher.register("session.set_mode", _set_mode) + dispatcher.register("session.set_effort", _set_effort) dispatcher.register("session.set_harness", _set_harness) async def _set_knowledge(params: dict) -> dict: diff --git a/raven/rpc/models.py b/raven/rpc/models.py index fbf14a5f6f..72905be81d 100644 --- a/raven/rpc/models.py +++ b/raven/rpc/models.py @@ -2278,6 +2278,11 @@ class ModelAddModelParams(_Strict): carries the icon row every other model has. Names outside the published vocabulary are refused rather than stored -- a surface draws one icon per name and has nothing to draw for a name it has never heard of. + + ``reasoning_efforts`` is not display data: it is this model's declaration + for the per-conversation effort picker, and the server validates it against + the effort ladder. ``None`` leaves a stored declaration alone; ``[]`` clears + it. """ slug: str @@ -2289,6 +2294,7 @@ class ModelAddModelParams(_Strict): capabilities: list[str] | None = None input_modalities: list[str] | None = None output_modalities: list[str] | None = None + reasoning_efforts: list[str] | None = None session_id: str | None = None @@ -3484,6 +3490,35 @@ class SessionSetModeResult(_Strict): ) +class SessionSetEffortParams(_Strict): + session_key: str + effort: str | None = Field( + None, + description="The reasoning effort this session's own model calls run at, one of the words in options. " + "Omit it to report without changing.", + ) + clear: bool = Field( + False, + description="Drop the session's own pick and follow the default again. Wins over effort when both are given.", + ) + model: str | None = Field( + None, + description="The model to answer for when the caller has no session yet (a draft). " + "Omit for a session's own model.", + ) + + +class SessionSetEffortResult(_Strict): + session_key: str + effort: str | None = Field(None, description="The session's own pick, or null while it follows the default.") + default: str | None = Field( + None, + description="What the session runs at with no pick of its own: its mode's effort where the catalogue sets " + "one, else agents.defaults.reasoningEffort, else null for the provider's own default.", + ) + options: list[str] = Field(default_factory=list, description="The words this model accepts, least first.") + + # --------------------------------------------------------------------------- # Method registry — used by tests/test_rpc_schema_match.py to walk every # method and compare its Pydantic Params/Result models against the OpenRPC @@ -3814,6 +3849,13 @@ class SessionInitInfo(_Strict): ), ) running: bool = Field(default=False, description="A turn is in flight on this session right now.") + reasoning_effort: str | None = Field( + default=None, + description=( + "The reasoning effort this session's own model calls run at: its own pick (session.set_effort), " + "else its mode's, else agents.defaults.reasoningEffort; null when the provider's own default stands." + ), + ) read_only: bool | None = Field( default=None, description="True for an Imported Conversation, which nothing can change; absent on every other session.", @@ -6994,6 +7036,7 @@ class ImportedUploadResult(_Strict): "session.set_knowledge": (SessionSetKnowledgeParams, SessionSetKnowledgeResult), "session.set_harness": (SessionSetHarnessParams, SessionSetHarnessResult), "session.set_mode": (SessionSetModeParams, SessionSetModeResult), + "session.set_effort": (SessionSetEffortParams, SessionSetEffortResult), # ext.list / cron.* / settings.* / channels.status / fs.* -- the console "ext.list": (ExtListParams, ExtListResult), "cron.list": (CronListParams, CronListResult), diff --git a/raven/session/manager.py b/raven/session/manager.py index ceea47263f..11846a5a08 100644 --- a/raven/session/manager.py +++ b/raven/session/manager.py @@ -37,6 +37,16 @@ IMPORTED_CHANNEL = "imported" +#: The metadata a fork carries over from its parent: the model and provider it +#: was switched onto, and the reasoning effort picked for it. The effort key is +#: ``raven.agent.session_effort.METADATA_KEY``, spelt here rather than imported +#: because the session layer does not import the agent's; a test holds the two +#: spellings together. A value that is null -- a pick cleared, which a metadata +#: patch leaves as null rather than removing -- is not carried: the fork follows +#: the default, as its parent does. +FORK_CARRIES: tuple[str, ...] = ("model", "provider", "reasoning_effort") + + def _refuse_imported(key: str) -> None: if key.partition(":")[0] == IMPORTED_CHANNEL: raise ValueError(f"{key!r} is an imported conversation; the session store does not keep it") @@ -1306,10 +1316,11 @@ def fork(self, source_key: str, *, title: str | None = None) -> "Session | None" # every inherited message to the annotator a second time. child.metadata["annotated_through"] = len(child.messages) # A fork continues its parent's conversation, so it continues on its - # parent's model. The caller re-points the live binding, but that lives - # in memory only -- without carrying the record too, the fork drops to - # the default the first time it is resumed in a new process. - for slot in ("model", "provider"): + # parent's model, and at the reasoning effort picked for it. The caller + # re-points the live binding, but that lives in memory only -- without + # carrying the record too, the fork drops to the default the first time + # it is resumed in a new process. + for slot in FORK_CARRIES: inherited = (source.metadata or {}).get(slot) if inherited: child.metadata[slot] = inherited diff --git a/rpc-schema/openrpc.json b/rpc-schema/openrpc.json index 0396114886..47fef28806 100644 --- a/rpc-schema/openrpc.json +++ b/rpc-schema/openrpc.json @@ -1392,6 +1392,17 @@ } } }, + { + "name": "reasoning_efforts", + "required": false, + "schema": { + "type": "array", + "items": { + "type": "string" + }, + "description": "The reasoning-effort words this model accepts, least first. Omit to leave a stored declaration alone; [] clears it." + } + }, { "name": "session_id", "required": false, @@ -3533,6 +3544,80 @@ }, "errors": [] }, + { + "name": "session.set_effort", + "summary": "Report, set or clear the reasoning effort this session's own model calls run at. Neither field reports; clear=true drops the pick so the session follows the default again; effort= runs from the next turn on. Kept on the session's record, so it outlives the gateway. Raven's own calls only: sub-agents keep the configured default.", + "params": [ + { + "name": "session_key", + "required": true, + "schema": { + "type": "string" + } + }, + { + "name": "effort", + "required": false, + "schema": { + "type": "string", + "description": "One of the words in options. Omit it to report without changing." + } + }, + { + "name": "clear", + "required": false, + "schema": { + "type": "boolean", + "description": "Drop the session's own pick and follow the default again. Wins over effort when both are given." + } + }, + { + "name": "model", + "required": false, + "schema": { + "type": "string", + "description": "The model to answer for when the caller has no session yet (a draft). Omit for a session's own model." + } + } + ], + "result": { + "name": "SessionSetEffortResult", + "schema": { + "type": "object", + "additionalProperties": false, + "required": [ + "session_key" + ], + "properties": { + "session_key": { + "type": "string" + }, + "effort": { + "type": [ + "string", + "null" + ], + "description": "The session's own pick, or null while it follows the default." + }, + "default": { + "type": [ + "string", + "null" + ], + "description": "What the session runs at with no pick of its own: its mode's effort where the catalogue sets one, else agents.defaults.reasoningEffort, else null for the provider's own default." + }, + "options": { + "type": "array", + "items": { + "type": "string" + }, + "description": "The words this model accepts, least first." + } + } + } + }, + "errors": [] + }, { "name": "session.set_knowledge", "summary": "Choose which knowledge bases this session's turns may search.", @@ -12334,6 +12419,13 @@ "type": "boolean", "description": "A turn is in flight on this session right now." }, + "reasoning_effort": { + "type": [ + "string", + "null" + ], + "description": "The reasoning effort this session's own model calls run at: its own pick (session.set_effort), else its mode's, else agents.defaults.reasoningEffort; null when the provider's own default stands." + }, "read_only": { "type": "boolean", "description": "True for an Imported Conversation, which nothing can change; absent on every other session." diff --git a/tests/test_agent_loop_context_overflow.py b/tests/test_agent_loop_context_overflow.py index 20a6679ce8..088a41bc84 100644 --- a/tests/test_agent_loop_context_overflow.py +++ b/tests/test_agent_loop_context_overflow.py @@ -837,3 +837,15 @@ async def test_summary_always_uses_the_session_model(workspace): assert final == "answer after compaction" assert provider.summary_models and all(m == "stub" for m in provider.summary_models) + + +@pytest.fixture(autouse=True) +def _fake_model_declares_every_level(monkeypatch): + """The turn clamps to the model's levels; the scripted model declares them all.""" + from raven.agent.loop import wiring + + monkeypatch.setattr( + wiring.effort_levels, + "levels_for", + lambda model, declared=(), refresh=False: ("minimal", "low", "medium", "high", "xhigh", "max"), + ) diff --git a/tests/test_agent_loop_empty_recovery.py b/tests/test_agent_loop_empty_recovery.py index c86b64b4c5..32fe37ac64 100644 --- a/tests/test_agent_loop_empty_recovery.py +++ b/tests/test_agent_loop_empty_recovery.py @@ -5213,3 +5213,15 @@ async def after_turn(key, record): assert [record["final_content"] for record in seen] == ["[SILENT]"] assert seen[0]["messages"][-1]["content"] == "[SILENT]" + + +@pytest.fixture(autouse=True) +def _fake_model_declares_every_level(monkeypatch): + """The turn clamps to the model's levels; the scripted model declares them all.""" + from raven.agent.loop import wiring + + monkeypatch.setattr( + wiring.effort_levels, + "levels_for", + lambda model, declared=(), refresh=False: ("minimal", "low", "medium", "high", "xhigh", "max"), + ) diff --git a/tests/test_agent_loop_iteration_hooks.py b/tests/test_agent_loop_iteration_hooks.py index 67c202dabc..49ec6ca756 100644 --- a/tests/test_agent_loop_iteration_hooks.py +++ b/tests/test_agent_loop_iteration_hooks.py @@ -290,7 +290,9 @@ async def test_a_session_policy_pins_the_reasoning_effort_of_its_calls(tmp_path) other = _ScriptedProvider([_text("plain")]) quiet = _loop(tmp_path, other, []) await quiet._process_message(_req()) - assert "reasoning_effort" not in other.calls[0] + # Sent as None, the provider's own default, rather than left out for the + # provider to fill from the copy it took at boot. + assert other.calls[0]["reasoning_effort"] is None @pytest.mark.asyncio @@ -577,3 +579,15 @@ def test_the_loop_wires_this_sessions_tier_into_the_subagent_manager(tmp_path): loop.set_session_policy("cli:c", mode="max") assert loop.subagents._session_tier("cli:c") == "max", "a session's own tier reaches the manager" assert loop.subagents._session_tier("cli:other") == "high", "and only that session's" + + +@pytest.fixture(autouse=True) +def _fake_model_declares_every_level(monkeypatch): + """The turn clamps to the model's levels; fake/model declares them all.""" + from raven.agent.loop import wiring + + monkeypatch.setattr( + wiring.effort_levels, + "levels_for", + lambda model, declared=(), refresh=False: ("minimal", "low", "medium", "high", "xhigh", "max"), + ) diff --git a/tests/test_agent_loop_session_effort.py b/tests/test_agent_loop_session_effort.py index 77304a186f..a6d2f8e2a8 100644 --- a/tests/test_agent_loop_session_effort.py +++ b/tests/test_agent_loop_session_effort.py @@ -4,20 +4,27 @@ names a reasoning effort, every model call of that turn -- the ReAct steps, the wrap-up after an exhausted budget and the head summary a compaction pays for -- carries it as an explicit argument. -A policy that names none passes nothing, so the provider's configured default -applies exactly as before, and another session's turns are untouched. +A policy that names none sends the configured default as the file has it now, +None included: an unset key is the provider's own default, and the provider's +own copy of the key -- frozen at boot -- is never what a call falls back to. +Another session's turns are untouched. """ from __future__ import annotations import pytest +from raven.agent import session_effort +from raven.providers import effort_levels as effort_levels_module + +_REAL_LEVELS_FOR = effort_levels_module.levels_for from raven.agent.loop import AgentLoop from raven.agent.loop.bundles import EngineWiring, ToolWiring, TurnPolicy from raven.agent.window import compaction +from raven.config.live import LiveConfig from raven.config.raven import CheckpointConfig, RuntimeConfig from raven.config.schema import CompactionConfig -from raven.contracts.llm_provider import LLMResponse, ToolCallRequest +from raven.contracts.llm_provider import GenerationSettings, LLMResponse, ToolCallRequest from raven.providers.base import LLMProvider from raven.spine import ChatType, Origin, Source, TurnRequest @@ -87,14 +94,15 @@ async def test_a_session_policys_effort_reaches_every_model_call(tmp_path): @pytest.mark.asyncio -async def test_a_policy_naming_no_effort_passes_none_so_the_configured_default_stands(tmp_path): +async def test_a_policy_naming_no_effort_sends_the_configured_default_none_included(tmp_path): provider = _ScriptedProvider([_text("done")]) loop = _loop(tmp_path, provider) loop.set_session_policy("cli:c", mode="high") await loop._process_message(_req()) - assert "reasoning_effort" not in provider.calls[0], "an explicit None would override the provider's sentinel" + assert "reasoning_effort" in provider.calls[0], "the key is never left to the provider's own boot copy" + assert provider.calls[0]["reasoning_effort"] is None, "an unset key is the provider's own default" @pytest.mark.asyncio @@ -105,7 +113,144 @@ async def test_another_session_runs_on_the_defaults(tmp_path): await loop._process_message(_req(chat_id="d")) - assert "reasoning_effort" not in provider.calls[0] + assert provider.calls[0]["reasoning_effort"] is None + + +class _BootedProvider(LLMProvider): + """A real provider booted with a configured effort, which ``chat_with_retry`` + and ``chat_stream`` resolve through their sentinel when a call leaves the + argument out. Answers from ``responses`` in order, the last one repeating.""" + + def __init__(self, booted: str, responses: list[LLMResponse] | None = None) -> None: + super().__init__(api_key="test") + self.generation = GenerationSettings(reasoning_effort=booted) + self.efforts: list[str | None] = [] + self._responses = list(responses or [_text("done")]) + + async def chat( + self, + messages, + tools=None, + model=None, + max_tokens=4096, + temperature=0.7, + reasoning_effort=None, + tool_choice=None, + ): + self.efforts.append(reasoning_effort) + return self._responses[min(len(self.efforts) - 1, len(self._responses) - 1)] + + def get_default_model(self) -> str: + return "fake/model" + + +def _live_default(tmp_path, effort: str | None) -> LiveConfig: + """The config file as a settings edit leaves it: the key held to ``effort``.""" + import json + + path = tmp_path / "config.json" + path.write_text(json.dumps({"agents": {"defaults": {"reasoningEffort": effort}}}), encoding="utf-8") + return LiveConfig(path) + + +@pytest.mark.asyncio +@pytest.mark.parametrize( + ("live", "sent"), + [ + pytest.param(None, None, id="cleared-to-provider-default"), + pytest.param("low", "low", id="control-a-word-reaches-the-call"), + ], +) +async def test_a_default_cleared_on_a_running_gateway_reaches_the_next_turn(tmp_path, live, sent): + """The provider's copy of ``agents.defaults.reasoningEffort`` is frozen at + boot. A key cleared afterwards (the picker's *Provider default*) used to + leave the call's argument out, which the provider filled from that copy: + the gateway kept calling at the word it booted with while ``session.info`` + reported the clear. The turn now sends the effort in force, None included, + so the clear is on the wire from the next turn.""" + provider = _BootedProvider(booted="high") + loop = _loop(tmp_path, provider) + loop._live_config = _live_default(tmp_path, live) + + await loop._process_message(_req()) + + assert provider.efforts == [sent] + # The one rule the RPC reports from answers the same. + assert loop.session_reasoning_effort("cli:c", None) == sent + + +# The effort in force, under three readings the turn's side calls must agree +# with: the configured default where the policy names no word, the session's +# own pick over it, and a cleared key -- which is sent as an explicit None, not +# left out. The provider is booted at ``high`` in every case, so a call that +# leaves the argument out is caught: the provider fills it from the copy it +# took at boot, and ``high`` is a word no case asks for. +_SIDE_CALL_CASES = [ + pytest.param("low", None, "low", id="configured-default-where-the-policy-names-none"), + pytest.param("low", "high", "high", id="the-sessions-own-pick"), + pytest.param(None, None, None, id="cleared-is-sent-as-none-not-left-out"), +] + + +def _picked(loop: AgentLoop, pick: str | None) -> None: + """The session's own pick, on the record the turn entrance reads it off.""" + if pick is not None: + session_effort.write(loop.sessions, "cli:c", pick) + + +@pytest.mark.asyncio +@pytest.mark.parametrize(("live", "pick", "sent"), _SIDE_CALL_CASES) +async def test_the_wrap_up_runs_at_the_effort_in_force_not_the_policys_word(tmp_path, live, pick, sent): + """The exhaustion wrap-up asks for the effort the turn froze -- the pick, + else the configured default as the file has it -- and sends it the way the + turn's other calls do, None included. The policy names no word here, so a + wrap-up handed the policy's effort instead of the turn's sends None where a + word is due; and one that leaves the key out on None lets the provider + fill it from the copy it took at boot, which is the clear never reaching + the wire.""" + provider = _BootedProvider("high", [_tool_call("list_dir", {"path": "."})] * 3 + [_text("summary")]) + loop = _loop(tmp_path, provider, max_iterations=1) + loop._live_config = _live_default(tmp_path, live) + _picked(loop, pick) + + out = await loop._process_message(_req()) + + assert out is not None + # The one ReAct step, then the wrap-up: both at the effort in force. + assert provider.efforts == [sent, sent] + + +def _with_oncall_roster(monkeypatch) -> None: + """The ``spawn`` tool's roster names an on-call agent, so a look at a path + earns the judgement. On the class: the tool keeps slots.""" + from raven.agent.subagent.backends import AgentMeta + from raven.agent.subagent.spawn_tool import SpawnTool + + roster = [AgentMeta("Raven-Oncall", "", False, False, owns_watched_work=True)] + monkeypatch.setattr(SpawnTool, "_agents", lambda self: roster) + + +@pytest.mark.asyncio +@pytest.mark.parametrize(("live", "pick", "sent"), _SIDE_CALL_CASES) +async def test_the_watch_work_judgement_is_paid_at_the_effort_in_force(tmp_path, monkeypatch, live, pick, sent): + """The judgement a look at a path earns is a model call of the turn, made + through ``chat_stream``: the turn hands the helper the effort it froze and + the provider sees that word -- or an explicit None -- never the copy it + booted with. Pinned at the provider rather than at the helper's door, so + the call site handing the policy's word, or the helper dropping the key, + both show up here.""" + verdict = _text('{"watched": true, "paths": ["/tmp/arena"]}') + provider = _BootedProvider("high", [_tool_call("list_dir", {"path": "."}), verdict, _text("done")]) + loop = _loop(tmp_path, provider) + loop._live_config = _live_default(tmp_path, live) + _with_oncall_roster(monkeypatch) + _picked(loop, pick) + + out = await loop._process_message(_req()) + + assert out is not None + # The ReAct step, the judgement the look earned, and the step after it. + assert provider.efforts == [sent, sent, sent] @pytest.mark.asyncio @@ -264,3 +409,110 @@ async def spy(state, name, args, result, message, reasoning_effort=None): assert out is not None assert handed == ["max"], handed + + +@pytest.fixture(autouse=True) +def _fake_model_declares_every_level(monkeypatch): + from raven.agent.loop import wiring + + monkeypatch.setattr( + wiring.effort_levels, + "levels_for", + lambda model, declared=(), refresh=False: ("minimal", "low", "medium", "high", "xhigh", "max"), + ) + + +async def test_a_pick_the_model_does_not_declare_is_dropped(tmp_path, monkeypatch): + from raven.agent.loop import wiring + + monkeypatch.setattr( + wiring.effort_levels, + "levels_for", + lambda model, declared=(), refresh=False: ("medium", "high"), + ) + loop = _loop(tmp_path, _ScriptedProvider([_text("ok")])) + assert loop.session_reasoning_effort("cli:c", "minimal") is None + assert loop.session_reasoning_effort("cli:c", "high") == "high" + + +async def test_a_model_that_declares_no_reasoning_drops_the_pick(tmp_path, monkeypatch): + from raven.agent.loop import wiring + + monkeypatch.setattr(wiring.effort_levels, "levels_for", lambda model, declared=(), refresh=False: ()) + loop = _loop(tmp_path, _ScriptedProvider([_text("ok")])) + assert loop.session_reasoning_effort("cli:c", "high") is None + + +async def test_an_unreadable_table_keeps_the_pick(tmp_path, monkeypatch): + from raven.agent.loop import wiring + + monkeypatch.setattr(wiring.effort_levels, "levels_for", lambda model, declared=(), refresh=False: None) + loop = _loop(tmp_path, _ScriptedProvider([_text("ok")])) + assert loop.session_reasoning_effort("cli:c", "high") == "high" + + +def _live_with_legacy_overlay(tmp_path) -> LiveConfig: + import json + + path = tmp_path / "config.json" + path.write_text( + json.dumps( + { + "providers": { + "zhipu": { + "modelOverlay": { + "zai/glm-5": {"reasoningEfforts": ["high"]}, + } + } + } + } + ), + encoding="utf-8", + ) + return LiveConfig(path) + + +async def test_a_legacy_provider_spelling_reaches_the_overlay(tmp_path, monkeypatch): + """The file kept its former provider name; the turn must read the overlay + the same way the RPC's parsed config does, or the status bar and the turn + disagree.""" + from raven.agent.loop import wiring + + monkeypatch.setattr( + wiring.effort_levels, + "levels_for", + lambda model, declared=(), refresh=False: tuple(declared) if declared else ("minimal", "low"), + ) + loop = AgentLoop( + provider=_ScriptedProvider([_text("ok")]), + workspace=tmp_path, + model="zai/glm-5", + policy=TurnPolicy(max_iterations=6), + tools=ToolWiring(restrict_to_workspace=True), + ) + loop._live_config = _live_with_legacy_overlay(tmp_path) + assert loop.session_reasoning_effort("cli:c", "high") == "high" + + +async def test_the_turn_reads_a_codex_accounts_levels_from_the_cache(tmp_path, monkeypatch): + """The turn must never fetch the account catalogue; the RPC warms it.""" + from raven.agent.loop import wiring + from raven.providers import codex_catalog + + monkeypatch.setattr(wiring.effort_levels, "levels_for", _REAL_LEVELS_FOR) + calls = [] + + def efforts(slug, *, refresh=False): + calls.append((slug, refresh)) + return ("high",) + + monkeypatch.setattr(codex_catalog, "account_efforts", efforts) + loop = AgentLoop( + provider=_ScriptedProvider([_text("ok")]), + workspace=tmp_path, + model="openai-codex/gpt-5.6-sol", + policy=TurnPolicy(max_iterations=6), + tools=ToolWiring(restrict_to_workspace=True), + ) + assert loop.session_reasoning_effort("cli:c", "high") == "high" + assert calls == [("gpt-5.6-sol", False)] diff --git a/tests/test_agent_loop_usage_sink.py b/tests/test_agent_loop_usage_sink.py index 2632df0d2d..c3d28761ef 100644 --- a/tests/test_agent_loop_usage_sink.py +++ b/tests/test_agent_loop_usage_sink.py @@ -481,6 +481,76 @@ async def test_no_configured_effort_sends_none(workspace, monkeypatch): assert provider.efforts == [None] +async def _one_turn(agent: AgentLoop, session_key: str = "s1") -> None: + await agent._process_message( + TurnRequest( + origin=Origin.USER, + source=Source(channel="test", chat_id="c1", sender_id="user", chat_type=ChatType.DM), + text="hi", + ), + session_key=session_key, + ) + + +@pytest.mark.asyncio +async def test_the_sessions_own_pick_outranks_the_configured_effort(workspace, monkeypatch): + """`session.set_effort` keeps the pick on the session's record, and the + turn reads it there, where it reads the model: over the file's default.""" + _point_config(workspace, monkeypatch, {"agents": {"defaults": {"reasoningEffort": "high"}}}) + provider = EffortProvider("stub") + agent = _make_agent(workspace, provider, model="stub", window=None) + agent.sessions.get_or_create("s1").metadata["reasoning_effort"] = "low" + + await _one_turn(agent) + + assert provider.efforts == ["low"] + + +@pytest.mark.asyncio +async def test_the_sessions_own_pick_outranks_its_modes_effort(workspace, monkeypatch): + """A mode's effort is the catalogue's word for every conversation in that + mode; the pick is a person's word for this one, so it wins -- and without a + pick the mode's effort stands, as before.""" + _point_config(workspace, monkeypatch, {}) + provider = EffortProvider("stub") + agent = _make_agent(workspace, provider, model="stub", window=None) + agent.set_session_policy("s1", mode="deep", reasoning_effort="medium") + + await _one_turn(agent) + agent.sessions.get_or_create("s1").metadata["reasoning_effort"] = "minimal" + await _one_turn(agent) + + assert provider.efforts == ["medium", "minimal"] + + +@pytest.mark.asyncio +async def test_a_pick_this_raven_does_not_know_is_ignored(workspace, monkeypatch): + """A record written by a later raven may carry a word this one cannot + send; the turn runs at the default rather than being refused.""" + _point_config(workspace, monkeypatch, {"agents": {"defaults": {"reasoningEffort": "high"}}}) + provider = EffortProvider("stub") + agent = _make_agent(workspace, provider, model="stub", window=None) + agent.sessions.get_or_create("s1").metadata["reasoning_effort"] = "turbo" + + await _one_turn(agent) + + assert provider.efforts == ["high"] + + +@pytest.mark.asyncio +async def test_a_cleared_pick_is_the_default_again(workspace, monkeypatch): + """Cleared is null on the record, not absent (a metadata patch sets keys); + both read as no pick.""" + _point_config(workspace, monkeypatch, {"agents": {"defaults": {"reasoningEffort": "medium"}}}) + provider = EffortProvider("stub") + agent = _make_agent(workspace, provider, model="stub", window=None) + agent.sessions.get_or_create("s1").metadata["reasoning_effort"] = None + + await _one_turn(agent) + + assert provider.efforts == ["medium"] + + # --------------------------------------------------------------------------- # # the cost is the turn's: every iteration, and every call it delegated # # --------------------------------------------------------------------------- # @@ -1262,3 +1332,15 @@ async def test_a_turn_whose_calls_read_no_window_still_reports_a_whole_usage(wor assert TurnUsage.model_validate(sink).context_used == 0 assert reported_usage(sink) is None + + +@pytest.fixture(autouse=True) +def _fake_model_declares_every_level(monkeypatch): + """The turn clamps to the model's levels; the scripted model declares them all.""" + from raven.agent.loop import wiring + + monkeypatch.setattr( + wiring.effort_levels, + "levels_for", + lambda model, declared=(), refresh=False: ("minimal", "low", "medium", "high", "xhigh", "max"), + ) diff --git a/tests/test_config_self_surface.py b/tests/test_config_self_surface.py index 7ce3ffb60b..4766460df8 100644 --- a/tests/test_config_self_surface.py +++ b/tests/test_config_self_surface.py @@ -76,7 +76,7 @@ def test_every_concrete_path_is_a_schema_field(): for s in surface.all_settings(): if "*" in s.path or s.kind == "pin" or s.session or s.stored_at: continue - if surface.default_of(s.path) is None and not s.nullable and s.path != "agents.defaults.reasoningEffort": + if surface.default_of(s.path) is None and not s.nullable: missing.append(s.path) assert missing == [] @@ -122,6 +122,9 @@ def test_find_prefers_an_exact_entry_and_binds_wildcards(): ("tools.disabledTools", ["exec"], True), ("tools.disabledTools", "exec", False), ("agents.defaults.contextWindowTokens", None, True), + ("agents.defaults.reasoningEffort", None, True), + ("agents.defaults.reasoningEffort", "high", True), + ("agents.defaults.reasoningEffort", "ultra", False), ("tools.exec.timeout", None, False), ("agents.defaults.model", {"provider": "openrouter", "model": "x/y"}, True), ("agents.defaults.model", {"provider": "openrouter"}, False), @@ -136,6 +139,24 @@ def test_check_value(path, value, ok): surface.check_value(setting, value) +def test_the_reasoning_effort_words_have_one_owner(): + """The agent's door offers the words ``REASONING_EFFORTS`` spells and none + of its own, and clears the key the way the settings page does: null is + the provider's own default on both doors.""" + from raven.config.schema import REASONING_EFFORTS + from raven.rpc.methods import console + + setting, _ = surface.find("agents.defaults.reasoningEffort") + assert setting.choices == REASONING_EFFORTS + assert setting.nullable + page = console._SETTINGS_SIMPLE_KEYS["agents.defaults.reasoningEffort"] + for word in REASONING_EFFORTS: + assert surface.check_value(setting, word) == word == page(word) + assert surface.check_value(setting, None) is None + assert page(None) is None + assert setting.describe()["nullable"] is True + + def test_change_line_states_the_effect_and_the_risk(): line = surface.change_line({"action": "set", "path": "permissions.mode", "value": "full"}) assert "permissions.mode" in line and "full" in line diff --git a/tests/test_provider_catalog.py b/tests/test_provider_catalog.py index d2fff982b9..80e36360af 100644 --- a/tests/test_provider_catalog.py +++ b/tests/test_provider_catalog.py @@ -769,3 +769,11 @@ def test_the_qwen_vendor_is_shelved_under_the_cloud_that_runs_it() -> None: from raven.providers.registry import find_by_model assert find_by_model("dashscope/qwen-plus") is spec + + +def test_describe_carries_the_overlay_effort_levels() -> None: + from raven.config.schema import ModelOverlay + from raven.providers.catalog import describe + + row = describe("hosted_vllm", "mine", overlay=ModelOverlay(reasoning_efforts=["low", "high"])) + assert row.reasoning_efforts == ("low", "high") diff --git a/tests/test_provider_effort_levels.py b/tests/test_provider_effort_levels.py new file mode 100644 index 0000000000..b744c1b07c --- /dev/null +++ b/tests/test_provider_effort_levels.py @@ -0,0 +1,206 @@ +from __future__ import annotations + +import pytest + +from raven.providers import codex_catalog, effort_levels + + +def _row(**flags): + return {"supports_reasoning": True, "litellm_provider": "openai", **flags} + + +@pytest.fixture +def table(monkeypatch): + def use(rows): + monkeypatch.setattr(effort_levels, "_table", lambda: rows) + effort_levels.reset() + + yield use + effort_levels.reset() + + +def test_a_bare_model_key_is_found_through_its_provider(table): + table({"gpt-5": _row()}) + assert effort_levels.levels_for("openai/gpt-5") == ("minimal", "low", "medium", "high") + + +def test_a_model_absent_from_the_table_has_no_levels(table): + table({"gpt-5": _row()}) + assert effort_levels.levels_for("anthropic/claude-opus-4-5") == () + + +def test_the_table_being_unreadable_is_not_a_model_answer(table): + table(None) + assert effort_levels.levels_for("anthropic/claude-opus-4-5") is None + + +def test_a_declared_list_wins_without_reading_the_table(table): + table(None) + assert effort_levels.levels_for("anything", ("high",)) == ("high",) + + +def test_minimal_and_low_are_allowed_unless_declared_false(table): + table({"gpt-5": _row(supports_minimal_reasoning_effort=True)}) + assert effort_levels.levels_for("openai/gpt-5") == ("minimal", "low", "medium", "high") + + +def test_xhigh_and_max_are_opt_in(table): + table( + { + "gpt-5.5-pro": _row( + supports_minimal_reasoning_effort=False, + supports_low_reasoning_effort=False, + supports_xhigh_reasoning_effort=True, + ) + } + ) + assert effort_levels.levels_for("openai/gpt-5.5-pro") == ("medium", "high", "xhigh") + + +def test_a_route_that_declares_no_reasoning_has_no_levels(table): + table({"openrouter/anthropic/claude-opus-4-5": {"supports_reasoning": False}}) + assert effort_levels.levels_for("openrouter/anthropic/claude-opus-4-5") == () + + +def test_the_pinned_table_declares_the_models_the_spec_names(): + """The real 1.85.0 table, not a fixture: these are the values the design + measured. A LiteLLM upgrade that moves them must fail here and be re-read.""" + effort_levels.reset() + assert effort_levels.levels_for("openai/gpt-5") == ("minimal", "low", "medium", "high") + assert effort_levels.levels_for("openai/gpt-5.5-pro") == ("medium", "high", "xhigh") + assert effort_levels.levels_for("anthropic/claude-opus-4-5") == ("minimal", "low", "medium", "high") + assert effort_levels.levels_for("anthropic/claude-opus-4-7") == ( + "minimal", + "low", + "medium", + "high", + "xhigh", + "max", + ) + assert effort_levels.levels_for("zai/glm-5") == ("minimal", "low", "medium", "high") + assert effort_levels.levels_for("zhipu/glm-5.3-flash") == () + + +def test_a_route_row_inherits_the_vendors_missing_flags(table): + table( + { + "claude-opus-4-6": _row(supports_max_reasoning_effort=True, litellm_provider="anthropic"), + "openrouter/anthropic/claude-opus-4.6": { + "supports_reasoning": True, + "supports_max_reasoning_effort": None, + }, + } + ) + assert "max" in effort_levels.levels_for("openrouter/anthropic/claude-opus-4.6") + + +def test_a_route_that_says_false_is_vetoed_even_when_the_vendor_says_true(table): + table( + { + "claude-opus-4-5": _row(litellm_provider="anthropic"), + "perplexity/anthropic/claude-opus-4-5": {"supports_reasoning": False}, + } + ) + assert effort_levels.levels_for("perplexity/anthropic/claude-opus-4-5") == () + assert "high" in effort_levels.levels_for("anthropic/claude-opus-4-5") + + +def test_a_bare_row_from_an_unrelated_vendor_is_not_borrowed(table): + table({"some-model": {**_row(), "litellm_provider": "somevendor"}}) + assert effort_levels.levels_for("bedrock/some-model") == () + + +def test_an_unknown_provider_spelling_still_finds_its_row(table): + table({"github_copilot/claude-opus-4.5": _row()}) + assert effort_levels.levels_for("github-copilot/claude-opus-4.5") == ( + "minimal", + "low", + "medium", + "high", + ) + + +def test_a_model_with_no_registry_row_uses_its_route_vendor(table): + table({"zai/glm-5": {**_row(), "litellm_provider": "zai"}}) + assert effort_levels.levels_for("bigmodel/glm-5") == ("minimal", "low", "medium", "high") + + +def test_an_empty_table_is_unreadable_not_a_model_answer(table): + table({}) + assert effort_levels.levels_for("openai/gpt-5") is None + + +def test_a_route_silent_on_reasoning_keeps_its_own_level_flags(table): + table( + { + "claude-opus-4-6": _row(supports_xhigh_reasoning_effort=True, litellm_provider="anthropic"), + "openrouter/anthropic/claude-opus-4.6": { + "supports_reasoning": None, + "supports_xhigh_reasoning_effort": False, + }, + } + ) + assert "xhigh" not in effort_levels.levels_for("openrouter/anthropic/claude-opus-4.6") + + +def test_an_openrouter_alias_is_consulted(table): + table({"openrouter/vendor/some-model": _row()}) + assert effort_levels.levels_for("vendor/some-model") == ("minimal", "low", "medium", "high") + + +def test_an_openai_codex_model_reads_the_account_catalogue(monkeypatch): + calls = [] + + def efforts(slug, *, refresh=False): + calls.append((slug, refresh)) + return ("low", "medium", "high", "xhigh", "max") + + monkeypatch.setattr(codex_catalog, "account_efforts", efforts) + assert effort_levels.levels_for("openai-codex/gpt-5.6-sol") == ( + "low", + "medium", + "high", + "xhigh", + "max", + ) + assert calls == [("gpt-5.6-sol", False)] + + +def test_the_account_catalogue_is_refreshed_only_when_asked(monkeypatch): + calls = [] + + def efforts(slug, *, refresh=False): + calls.append(refresh) + return ("high",) + + monkeypatch.setattr(codex_catalog, "account_efforts", efforts) + assert effort_levels.levels_for("openai-codex/gpt-5.6-sol", refresh=True) == ("high",) + assert calls == [True] + + +def test_a_cold_account_catalogue_has_no_levels(monkeypatch): + monkeypatch.setattr(codex_catalog, "account_efforts", lambda slug, *, refresh=False: ()) + assert effort_levels.levels_for("openai-codex/gpt-5.6-sol") == () + + +def test_an_unlisted_account_model_has_no_levels(monkeypatch): + monkeypatch.setattr(codex_catalog, "account_efforts", lambda slug, *, refresh=False: ()) + assert effort_levels.levels_for("openai-codex/gpt-5.6-sol") == () + + +def test_an_overlay_beats_the_account_catalogue(monkeypatch): + def boom(slug, *, refresh=False): + raise AssertionError("the overlay must short-circuit the catalogue") + + monkeypatch.setattr(codex_catalog, "account_efforts", boom) + assert effort_levels.levels_for("openai-codex/gpt-5.6-sol", ("high",)) == ("high",) + + +def test_a_non_codex_model_does_not_read_the_catalogue(monkeypatch, table): + table({"gpt-5": _row()}) + + def boom(slug, *, refresh=False): + raise AssertionError("only openai_codex models read the account catalogue") + + monkeypatch.setattr(codex_catalog, "account_efforts", boom) + assert effort_levels.levels_for("openai/gpt-5") == ("minimal", "low", "medium", "high") diff --git a/tests/test_providers_codex_catalog.py b/tests/test_providers_codex_catalog.py index 977b2cc92f..f86eba21b5 100644 --- a/tests/test_providers_codex_catalog.py +++ b/tests/test_providers_codex_catalog.py @@ -220,3 +220,177 @@ def boom(url, **kwargs): _serve(monkeypatch, {"models": [{"slug": "gpt-5.6-sol", "visibility": "list"}]}) assert codex_catalog.account_models() == ("gpt-5.6-sol",), "the picker inherited a report's failure" + + +def test_account_efforts_returns_the_supported_levels_least_first( + monkeypatch: pytest.MonkeyPatch, + signed_in, +) -> None: + """The account catalogue lists the levels each model takes; `ultra` is not a + word this build's ladder carries, so it is dropped.""" + _serve( + monkeypatch, + { + "models": [ + { + "slug": "gpt-5.6-sol", + "visibility": "list", + "supported_reasoning_levels": [ + {"effort": effort} for effort in ("low", "medium", "high", "xhigh", "max", "ultra") + ], + } + ] + }, + ) + + assert codex_catalog.account_efforts("gpt-5.6-sol", refresh=True) == ( + "low", + "medium", + "high", + "xhigh", + "max", + ) + + +def test_account_efforts_is_empty_until_the_catalogue_is_read( + monkeypatch: pytest.MonkeyPatch, + signed_in, +) -> None: + assert codex_catalog.account_efforts("gpt-5.6-sol") == () + + +def test_account_efforts_is_empty_for_a_model_it_does_not_list( + monkeypatch: pytest.MonkeyPatch, + signed_in, +) -> None: + _serve(monkeypatch, {"models": [{"slug": "gpt-5.6-sol", "visibility": "list"}]}) + + assert codex_catalog.account_efforts("gpt-5.4", refresh=True) == () + + +def test_account_efforts_uses_the_cached_catalogue_without_a_refresh( + monkeypatch: pytest.MonkeyPatch, + signed_in, +) -> None: + seen = _serve( + monkeypatch, + {"models": [{"slug": "gpt-5.6-sol", "visibility": "list", "supported_reasoning_levels": [{"effort": "high"}]}]}, + ) + + assert codex_catalog.account_efforts("gpt-5.6-sol", refresh=True) == ("high",) + assert codex_catalog.account_efforts("gpt-5.6-sol") == ("high",) + + assert len(seen) == 1 + + +def test_account_efforts_is_empty_when_the_catalogue_cannot_be_reached( + monkeypatch: pytest.MonkeyPatch, + signed_in, +) -> None: + _serve(monkeypatch, {"models": []}, status=500) + + assert codex_catalog.account_efforts("gpt-5.6-sol", refresh=True) == () + + +def test_a_hidden_model_has_no_levels(monkeypatch: pytest.MonkeyPatch, signed_in) -> None: + _serve( + monkeypatch, + { + "models": [ + { + "slug": "codex-auto-review", + "visibility": "hide", + "supported_reasoning_levels": [{"effort": "high"}], + } + ] + }, + ) + + assert codex_catalog.account_efforts("codex-auto-review", refresh=True) == () + + +def test_a_visible_model_without_levels_declares_none(monkeypatch: pytest.MonkeyPatch, signed_in) -> None: + _serve(monkeypatch, {"models": [{"slug": "gpt-5.6-sol", "visibility": "list"}]}) + + assert codex_catalog.account_efforts("gpt-5.6-sol", refresh=True) == () + + +def test_an_expired_answer_still_serves_the_cache_only_read( + monkeypatch: pytest.MonkeyPatch, + signed_in, +) -> None: + seen = _serve( + monkeypatch, + { + "models": [ + { + "slug": "gpt-5.6-sol", + "visibility": "list", + "supported_reasoning_levels": [{"effort": effort} for effort in ("low", "medium", "high")], + } + ] + }, + ) + assert codex_catalog.account_efforts("gpt-5.6-sol", refresh=True) == ("low", "medium", "high") + real = codex_catalog.time.monotonic + monkeypatch.setattr(codex_catalog.time, "monotonic", lambda: real() + 301) + assert codex_catalog.account_efforts("gpt-5.6-sol") == ("low", "medium", "high") + assert len(seen) == 1 + + +def test_a_failed_refresh_keeps_the_last_good_levels( + monkeypatch: pytest.MonkeyPatch, + signed_in, +) -> None: + _serve( + monkeypatch, + { + "models": [ + { + "slug": "gpt-5.6-sol", + "visibility": "list", + "supported_reasoning_levels": [{"effort": "high"}], + } + ] + }, + ) + assert codex_catalog.account_efforts("gpt-5.6-sol", refresh=True) == ("high",) + real = codex_catalog.time.monotonic + monkeypatch.setattr(codex_catalog.time, "monotonic", lambda: real() + 301) + + def offline(*, timeout): + raise RuntimeError("offline") + + monkeypatch.setattr(codex_catalog, "_fetch", offline) + assert codex_catalog.account_efforts("gpt-5.6-sol", refresh=True) == ("high",) + + +def test_an_expired_answer_clamps_a_stale_pick(monkeypatch: pytest.MonkeyPatch, signed_in) -> None: + from raven.agent import session_effort + + _serve( + monkeypatch, + { + "models": [ + { + "slug": "gpt-5.6-sol", + "visibility": "list", + "supported_reasoning_levels": [ + {"effort": effort} for effort in ("low", "medium", "high", "xhigh", "max") + ], + } + ] + }, + ) + assert codex_catalog.account_efforts("gpt-5.6-sol", refresh=True) == ( + "low", + "medium", + "high", + "xhigh", + "max", + ) + real = codex_catalog.time.monotonic + monkeypatch.setattr(codex_catalog.time, "monotonic", lambda: real() + 301) + levels = codex_catalog.account_efforts("gpt-5.6-sol") + assert levels == ("low", "medium", "high", "xhigh", "max") + assert session_effort.clamp("minimal", levels) is None diff --git a/tests/test_raven_config_tool.py b/tests/test_raven_config_tool.py index b29314cd55..ef885a6645 100644 --- a/tests/test_raven_config_tool.py +++ b/tests/test_raven_config_tool.py @@ -218,6 +218,39 @@ async def test_a_settings_page_setting_goes_through_the_lent_caller(config_file) assert "60 -> 120" in reply and "next turn" in reply +@pytest.mark.asyncio +async def test_the_reasoning_effort_may_be_cleared_through_the_lent_caller(config_file): + """``null`` hands the choice back to the provider's own default, as the + settings page's *Provider default* rung does; the door used to refuse it + while the page accepted it.""" + calls = Calls({"settings.set": {"applied": True, "previous": "high"}}) + tool = RavenConfigTool() + tool.set_rpc_caller(calls) + reply = await _run(tool, action="set", path="agents.defaults.reasoningEffort", value="null") + assert calls.calls == [("settings.set", {"key": "agents.defaults.reasoningEffort", "value": None})] + assert "next turn" in reply + + +@pytest.mark.asyncio +async def test_a_cleared_reasoning_effort_lands_as_null_in_the_file(config_file): + """Through the settings service itself: the key is written as null, which + is what the live reader takes as the provider's own default.""" + from raven.rpc.methods import console as rpc_console + + raw = json.loads(config_file.read_text()) + raw["agents"] = {"defaults": {"reasoningEffort": "high"}} + config_file.write_text(json.dumps(raw), encoding="utf-8") + + async def writer(method, params): + assert method == "settings.set" + return await rpc_console.settings_set(params) + + tool = RavenConfigTool() + tool.set_rpc_caller(writer) + await _run(tool, action="set", path="agents.defaults.reasoningEffort", value="null") + assert json.loads(config_file.read_text())["agents"]["defaults"]["reasoningEffort"] is None + + @pytest.mark.asyncio async def test_the_image_provider_may_be_set_to_volcengine(config_file): """The choices are the registry's, so a provider the roles card offers the image diff --git a/tests/test_rpc_model.py b/tests/test_rpc_model.py index 9b2f57a27e..5bca21d1db 100644 --- a/tests/test_rpc_model.py +++ b/tests/test_rpc_model.py @@ -2243,3 +2243,172 @@ async def test_add_model_preserves_metadata_under_a_bare_id(fake_home: Path) -> assert row["label"] == "Team model" assert row["description"] == "Keep this description" assert row["capabilities"] == ["reasoning"] + + +async def test_add_model_stores_declared_effort_levels(fake_home: Path) -> None: + await model_add_model( + { + "slug": "anthropic", + "model": "mine", + "reasoning_efforts": ["low", "high", "xhigh"], + } + ) + raw = json.loads((fake_home / ".raven" / "config.json").read_text()) + overlay = raw["providers"]["anthropic"]["modelOverlay"]["anthropic/mine"] + assert overlay["reasoningEfforts"] == ["low", "high", "xhigh"] + + +async def test_add_model_refuses_an_unknown_effort(fake_home: Path) -> None: + with pytest.raises(ConfigValidationError): + await model_add_model( + { + "slug": "anthropic", + "model": "mine", + "reasoning_efforts": ["turbo"], + } + ) + + +async def test_add_model_with_an_empty_list_clears_the_declaration(fake_home: Path) -> None: + await model_add_model( + { + "slug": "anthropic", + "model": "mine", + "reasoning_efforts": ["high"], + } + ) + await model_add_model({"slug": "anthropic", "model": "mine", "reasoning_efforts": []}) + raw = json.loads((fake_home / ".raven" / "config.json").read_text()) + assert "anthropic/mine" not in (raw["providers"]["anthropic"].get("modelOverlay") or {}) + + +async def test_model_options_label_carries_declared_efforts(fake_home: Path) -> None: + await model_add_model( + { + "slug": "anthropic", + "model": "mine", + "reasoning_efforts": ["low", "high"], + } + ) + result = await model_options({"include_providers": True}) + provider = next(p for p in result["providers"] if p["slug"] == "anthropic") + assert provider["model_labels"]["anthropic/mine"]["reasoning_efforts"] == ["low", "high"] + + +async def test_a_stored_overlay_reaches_the_effort_read(fake_home: Path) -> None: + from raven.rpc.methods.session import session_set_effort + from raven.session.manager import SessionManager + + await model_add_model( + { + "slug": "anthropic", + "model": "mine", + "reasoning_efforts": ["low", "high"], + } + ) + + class _Loop: + def __init__(self, path: Path) -> None: + self.sessions = SessionManager(path) + + def session_policy(self, key): + from raven.agent.loop._shared import SessionPolicy + + return SessionPolicy() + + result = await session_set_effort( + {"session_key": "", "model": "anthropic/mine"}, + agent_loop_factory=lambda: _Loop(fake_home), + ) + assert result["options"] == ["low", "high"] + + +async def test_a_codex_model_gets_its_options_from_the_account_catalogue(fake_home: Path, monkeypatch) -> None: + from raven.providers import codex_catalog + from raven.rpc.methods.session import session_set_effort + from raven.session.manager import SessionManager + + calls = [] + + def efforts(slug, *, refresh=False): + calls.append((slug, refresh)) + return ("low", "medium", "high", "xhigh", "max") + + monkeypatch.setattr(codex_catalog, "account_efforts", efforts) + + class _Loop: + def __init__(self, path: Path) -> None: + self.sessions = SessionManager(path) + + def session_policy(self, key): + from raven.agent.loop._shared import SessionPolicy + + return SessionPolicy() + + result = await session_set_effort( + {"session_key": "", "model": "openai-codex/gpt-5.6-sol"}, + agent_loop_factory=lambda: _Loop(fake_home), + ) + assert result["options"] == ["low", "medium", "high", "xhigh", "max"] + assert calls == [("gpt-5.6-sol", False)] + + +async def test_the_bundle_reads_the_codex_catalogue_without_refreshing(tmp_path, monkeypatch) -> None: + from types import SimpleNamespace + from unittest.mock import MagicMock + + from raven.providers import codex_catalog + from raven.rpc.methods import session as session_module + from raven.session.manager import SessionManager + + calls = [] + + def efforts(slug, *, refresh=False): + calls.append((slug, refresh)) + return ("high",) + + monkeypatch.setattr(codex_catalog, "account_efforts", efforts) + loop = MagicMock() + loop.model = "openai-codex/gpt-5.6-sol" + loop.session_model = lambda key: "openai-codex/gpt-5.6-sol" + loop.sessions = SessionManager(tmp_path) + loop.session_policy = lambda key: SimpleNamespace(reasoning_effort=None) + held = loop.sessions.get_or_create("tui:1") + held.metadata["reasoning_effort"] = "high" + loop.sessions.save(held) + info = await session_module._default_session_info(loop, session_module.load_config(), "tui:1") + assert info["reasoning_effort"] == "high" + assert calls == [("gpt-5.6-sol", False)] + + +async def test_an_effort_set_refreshes_the_codex_catalogue(tmp_path, monkeypatch) -> None: + from raven.providers import codex_catalog + from raven.rpc.methods.session import session_set_effort + from raven.session.manager import SessionManager + + calls = [] + + def efforts(slug, *, refresh=False): + calls.append((slug, refresh)) + return ("low", "medium", "high", "xhigh", "max") + + monkeypatch.setattr(codex_catalog, "account_efforts", efforts) + + class _Loop: + def __init__(self, path: Path) -> None: + self.sessions = SessionManager(path) + + def session_model(self, key): + return "openai-codex/gpt-5.6-sol" + + def session_policy(self, key): + from raven.agent.loop._shared import SessionPolicy + + return SessionPolicy() + + result = await session_set_effort( + {"session_key": "tui:1", "effort": "high"}, + agent_loop_factory=lambda: _Loop(tmp_path), + ) + assert result["effort"] == "high" + assert calls == [("gpt-5.6-sol", True)] diff --git a/tests/test_rpc_session.py b/tests/test_rpc_session.py index e982a0edec..99d58e34bd 100644 --- a/tests/test_rpc_session.py +++ b/tests/test_rpc_session.py @@ -3483,6 +3483,199 @@ async def test_a_clear_wins_over_a_mode_given_alongside_it(): assert loop.policies["tui:1"] == "high" +# --------------------------------------------------------------------------- +# session.set_effort: this session's own reasoning effort +# --------------------------------------------------------------------------- + + +class _EffortLoop: + """A loop over a real session manager, which is where the pick is kept.""" + + def __init__(self, tmp_path: Path, mode_effort: str | None = None) -> None: + self.sessions = SessionManager(tmp_path) + self._mode_effort = mode_effort + + def session_policy(self, key): + from raven.agent.loop._shared import SessionPolicy + + return SessionPolicy(reasoning_effort=self._mode_effort) + + +def _configured_effort(monkeypatch: pytest.MonkeyPatch, effort: str | None) -> None: + cfg = load_config() + cfg.agents.defaults.reasoning_effort = effort + monkeypatch.setattr(session_module, "load_config", lambda: cfg) + + +def _filed(mgr: SessionManager, key: str) -> None: + """A session with a record on disk, which is what a metadata patch lands on.""" + mgr.save(mgr.get_or_create(key)) + + +@pytest.fixture(autouse=True) +def levels(monkeypatch): + """Keep the effort tests off the real LiteLLM table; a test may narrow it.""" + + def use(words): + monkeypatch.setattr( + session_module.effort_levels, + "levels_for", + lambda model, declared=(), refresh=False: None if words is None else tuple(words), + ) + + use(["minimal", "low", "medium", "high"]) + return use + + +async def test_an_effort_read_reports_no_pick_the_default_and_the_words(tmp_path, monkeypatch) -> None: + _configured_effort(monkeypatch, "low") + loop = _EffortLoop(tmp_path) + result = await session_module.session_set_effort({"session_key": "tui:1"}, agent_loop_factory=lambda: loop) + assert result == { + "session_key": "tui:1", + "effort": None, + "default": "low", + "options": ["minimal", "low", "medium", "high"], + } + + +async def test_an_unset_default_reports_null_not_low(tmp_path, monkeypatch) -> None: + """No configured effort means the provider's own default, which has no + word: reporting a rung for it is the mistake the settings page made.""" + _configured_effort(monkeypatch, None) + loop = _EffortLoop(tmp_path) + result = await session_module.session_set_effort({"session_key": "tui:1"}, agent_loop_factory=lambda: loop) + assert result["default"] is None + + +async def test_an_effort_set_lands_on_the_record_and_outlives_the_manager(tmp_path, monkeypatch) -> None: + _configured_effort(monkeypatch, "low") + loop = _EffortLoop(tmp_path) + _filed(loop.sessions, "tui:1") + result = await session_module.session_set_effort( + {"session_key": "tui:1", "effort": "high"}, agent_loop_factory=lambda: loop + ) + assert result["effort"] == "high" and result["default"] == "low" + assert loop.sessions.peek("tui:1").metadata["reasoning_effort"] == "high" + # On disk, not only on the cached copy: a fresh manager reads it back. + assert SessionManager(tmp_path).peek("tui:1").metadata["reasoning_effort"] == "high" + again = await session_module.session_set_effort({"session_key": "tui:1"}, agent_loop_factory=lambda: loop) + assert again["effort"] == "high" + + +async def test_an_effort_clear_returns_the_session_to_the_default(tmp_path, monkeypatch) -> None: + _configured_effort(monkeypatch, "low") + loop = _EffortLoop(tmp_path) + _filed(loop.sessions, "tui:1") + await session_module.session_set_effort({"session_key": "tui:1", "effort": "high"}, agent_loop_factory=lambda: loop) + result = await session_module.session_set_effort( + {"session_key": "tui:1", "clear": True}, agent_loop_factory=lambda: loop + ) + assert result["effort"] is None and result["default"] == "low" + # Null rather than absent: a patch sets keys, and both read as "no pick". + assert loop.sessions.peek("tui:1").metadata["reasoning_effort"] is None + assert SessionManager(tmp_path).peek("tui:1").metadata["reasoning_effort"] is None + + +async def test_an_effort_clear_wins_over_an_effort_given_alongside_it(tmp_path, monkeypatch) -> None: + _configured_effort(monkeypatch, None) + loop = _EffortLoop(tmp_path) + _filed(loop.sessions, "tui:1") + result = await session_module.session_set_effort( + {"session_key": "tui:1", "effort": "high", "clear": True}, agent_loop_factory=lambda: loop + ) + assert result["effort"] is None + assert loop.sessions.peek("tui:1").metadata["reasoning_effort"] is None + + +async def test_the_modes_effort_is_the_default_where_the_catalogue_sets_one(tmp_path, monkeypatch) -> None: + """raven-code's modes each set an effort; that, not the file's, is what a + session in such a mode runs at without a pick of its own.""" + _configured_effort(monkeypatch, "low") + loop = _EffortLoop(tmp_path, mode_effort="medium") + result = await session_module.session_set_effort({"session_key": "tui:1"}, agent_loop_factory=lambda: loop) + assert result["default"] == "medium" + + +async def test_an_unknown_effort_is_refused_naming_the_words(tmp_path, monkeypatch) -> None: + from raven.rpc.errors import ConfigValidationError + + _configured_effort(monkeypatch, None) + loop = _EffortLoop(tmp_path) + _filed(loop.sessions, "tui:1") + with pytest.raises(ConfigValidationError) as exc: + await session_module.session_set_effort( + {"session_key": "tui:1", "effort": "turbo"}, agent_loop_factory=lambda: loop + ) + assert "minimal, low, medium, high" in str(exc.value) + assert "reasoning_effort" not in loop.sessions.peek("tui:1").metadata, "a refused set must not half-land" + + +async def test_a_draft_reads_the_default_but_cannot_set_without_a_key(tmp_path, monkeypatch) -> None: + """The page's draft has no session yet; it still draws the row from the + default and the words, and stages a pick until the first message mints one.""" + from raven.rpc.errors import ConfigValidationError + + _configured_effort(monkeypatch, "medium") + loop = _EffortLoop(tmp_path) + result = await session_module.session_set_effort({"session_key": ""}, agent_loop_factory=lambda: loop) + assert result == { + "session_key": "", + "effort": None, + "default": "medium", + "options": ["minimal", "low", "medium", "high"], + } + with pytest.raises(ConfigValidationError): + await session_module.session_set_effort({"effort": "high"}, agent_loop_factory=lambda: loop) + with pytest.raises(ConfigValidationError): + await session_module.session_set_effort({"clear": True}, agent_loop_factory=lambda: loop) + + +async def test_an_effort_set_on_a_draft_is_held_until_its_first_save(tmp_path, monkeypatch) -> None: + """A conversation the first message has not filed yet has no record to + patch; the pick waits on the in-memory copy, which that first save writes.""" + _configured_effort(monkeypatch, None) + loop = _EffortLoop(tmp_path) + result = await session_module.session_set_effort( + {"session_key": "tui:draft", "effort": "minimal"}, agent_loop_factory=lambda: loop + ) + assert result["effort"] == "minimal" + held = loop.sessions.get_or_create("tui:draft") + assert held.metadata["reasoning_effort"] == "minimal" + loop.sessions.save(held) + assert SessionManager(tmp_path).peek("tui:draft").metadata["reasoning_effort"] == "minimal" + + +async def test_the_bundle_reports_the_effort_in_force(tmp_path, monkeypatch) -> None: + """The status bar sits beside the model; it shows the pick where there is + one and the default otherwise, the same answer the next turn runs at.""" + from unittest.mock import MagicMock + + _configured_effort(monkeypatch, "low") + # MagicMock for the unrelated skills/tools enumeration, as the session-info + # tests above do; the manager and the policy are real. + loop = MagicMock() + loop.model = "vendor-a/model" + loop.session_model = lambda key: "vendor-a/model" + loop.sessions = SessionManager(tmp_path) + loop.session_policy = _EffortLoop(tmp_path).session_policy + _filed(loop.sessions, "tui:1") + config = session_module.load_config() + before = await session_module._default_session_info(loop, config, "tui:1") + assert before["reasoning_effort"] == "low" + await session_module.session_set_effort({"session_key": "tui:1", "effort": "high"}, agent_loop_factory=lambda: loop) + after = await session_module._default_session_info(loop, config, "tui:1") + assert after["reasoning_effort"] == "high" + fresh = await session_module._default_session_info(loop, config, None) + assert fresh["reasoning_effort"] == "low" + + +async def test_set_effort_is_registered_beside_set_mode() -> None: + d = Dispatcher() + register_session_methods(d, agent_loop_factory=lambda: None) + assert "session.set_effort" in d.methods() and "session.set_mode" in d.methods() + + # --------------------------------------------------------------------------- # session.list {archived} and the lazy auto-archive pass # --------------------------------------------------------------------------- @@ -4302,3 +4495,107 @@ async def test_delete_of_an_imported_conversation(tmp_path: Path) -> None: assert not state.exists() assert state.resolve() not in tidx._INDEXERS assert await session_delete({"session_id": key}) == {"deleted": None, "still_on_disk": False} + + +async def test_effort_options_come_from_the_models_declared_levels(tmp_path, monkeypatch, levels): + _configured_effort(monkeypatch, "low") + levels(["medium", "high", "xhigh"]) + loop = _EffortLoop(tmp_path) + result = await session_module.session_set_effort({"session_key": "tui:1"}, agent_loop_factory=lambda: loop) + assert result["options"] == ["medium", "high", "xhigh"] + + +async def test_a_declared_level_is_refused_when_the_model_does_not_carry_it(tmp_path, monkeypatch, levels): + from raven.rpc.errors import ConfigValidationError + + _configured_effort(monkeypatch, "low") + levels(["medium", "high", "xhigh"]) + loop = _EffortLoop(tmp_path) + with pytest.raises(ConfigValidationError): + await session_module.session_set_effort( + {"session_key": "tui:1", "effort": "minimal"}, agent_loop_factory=lambda: loop + ) + + +async def test_a_draft_read_names_the_model_it_asks_about(tmp_path, monkeypatch): + _configured_effort(monkeypatch, "low") + seen = {} + + def resolve(model, declared=(), refresh=False): + seen["model"] = model + return ("low", "high") + + monkeypatch.setattr(session_module.effort_levels, "levels_for", resolve) + loop = _EffortLoop(tmp_path) + result = await session_module.session_set_effort( + {"session_key": "", "model": "openai/gpt-5.5-pro"}, agent_loop_factory=lambda: loop + ) + assert seen["model"] == "openai/gpt-5.5-pro" + assert result["options"] == ["low", "high"] + + +async def test_a_model_that_declares_no_reasoning_offers_no_words(tmp_path, monkeypatch, levels): + _configured_effort(monkeypatch, "low") + levels([]) + loop = _EffortLoop(tmp_path) + result = await session_module.session_set_effort({"session_key": "tui:1"}, agent_loop_factory=lambda: loop) + assert result["options"] == [] + + +async def test_the_reported_pick_is_clamped_to_the_model(tmp_path, monkeypatch, levels): + _configured_effort(monkeypatch, "low") + levels(["medium", "high", "xhigh"]) + loop = _EffortLoop(tmp_path) + _filed(loop.sessions, "tui:1") + await session_module.session_set_effort({"session_key": "tui:1", "effort": "high"}, agent_loop_factory=lambda: loop) + levels(["medium"]) + result = await session_module.session_set_effort({"session_key": "tui:1"}, agent_loop_factory=lambda: loop) + assert result["effort"] is None and result["default"] is None + + +async def test_the_bundle_reports_the_clamped_effort(tmp_path, monkeypatch, levels): + from unittest.mock import MagicMock + + _configured_effort(monkeypatch, "low") + loop = MagicMock() + loop.model = "vendor-a/model" + loop.session_model = lambda key: "vendor-a/model" + loop.sessions = SessionManager(tmp_path) + loop.session_policy = _EffortLoop(tmp_path).session_policy + _filed(loop.sessions, "tui:1") + config = session_module.load_config() + await session_module.session_set_effort({"session_key": "tui:1", "effort": "high"}, agent_loop_factory=lambda: loop) + levels(["low", "medium"]) + info = await session_module._default_session_info(loop, config, "tui:1") + assert info["reasoning_effort"] is None + + +async def test_a_pick_comes_back_when_a_later_model_declares_it(tmp_path, monkeypatch, levels): + _configured_effort(monkeypatch, "low") + levels(["medium", "high"]) + loop = _EffortLoop(tmp_path) + _filed(loop.sessions, "tui:1") + await session_module.session_set_effort({"session_key": "tui:1", "effort": "high"}, agent_loop_factory=lambda: loop) + levels(["medium"]) + dropped = await session_module.session_set_effort({"session_key": "tui:1"}, agent_loop_factory=lambda: loop) + assert dropped["effort"] is None + levels(["medium", "high"]) + back = await session_module.session_set_effort({"session_key": "tui:1"}, agent_loop_factory=lambda: loop) + assert back["effort"] == "high" + assert loop.sessions.peek("tui:1").metadata["reasoning_effort"] == "high" + + +async def test_a_model_param_does_not_override_a_sessions_model(tmp_path, monkeypatch): + _configured_effort(monkeypatch, "low") + seen = {} + + def resolve(model, declared=(), refresh=False): + seen["model"] = model + return ("low", "high") + + monkeypatch.setattr(session_module.effort_levels, "levels_for", resolve) + loop = _EffortLoop(tmp_path) + await session_module.session_set_effort( + {"session_key": "tui:1", "model": "other/model"}, agent_loop_factory=lambda: loop + ) + assert seen["model"] == session_module.load_config().agents.defaults.model diff --git a/tests/test_rpc_session_init_bundle.py b/tests/test_rpc_session_init_bundle.py index c88954223e..226613668f 100644 --- a/tests/test_rpc_session_init_bundle.py +++ b/tests/test_rpc_session_init_bundle.py @@ -312,6 +312,9 @@ async def test_default_session_info_key_set_matches_expected_v030(fake_agent_loo "usage", "endpoint", "running", + # What the session's own calls think at, for the status bar beside the + # model: its pick (session.set_effort), else the default in force. + "reasoning_effort", } assert set(info) == expected_keys, ( f"init bundle key set drift: unexpected={set(info) - expected_keys}, missing={expected_keys - set(info)}" diff --git a/tests/test_rpc_settings.py b/tests/test_rpc_settings.py index 854c66f73a..7c4d3b9ea6 100644 --- a/tests/test_rpc_settings.py +++ b/tests/test_rpc_settings.py @@ -1202,6 +1202,7 @@ async def test_settings_set_new_scalar_keys_write_and_refuse(cfg, key, good, bad "key", [ "agents.defaults.contextWindowTokens", + "agents.defaults.reasoningEffort", "context.curatorModel", "sessionTitle.provider", "sessions.autoArchiveAfterDays", diff --git a/tests/test_session_effort.py b/tests/test_session_effort.py new file mode 100644 index 0000000000..944954f604 --- /dev/null +++ b/tests/test_session_effort.py @@ -0,0 +1,95 @@ +"""raven.agent.session_effort: the pick on a session's record and how it is read.""" + +from __future__ import annotations + +from pathlib import Path + +from raven.agent import session_effort +from raven.session.manager import SessionManager + + +def test_a_record_with_no_pick_reads_as_none() -> None: + assert session_effort.of({}) is None + assert session_effort.of({"reasoning_effort": None}) is None + assert session_effort.of(None) is None + assert session_effort.of("high") is None + + +def test_only_a_word_this_raven_knows_reads_as_a_pick() -> None: + assert session_effort.of({"reasoning_effort": "high"}) == "high" + assert session_effort.of({"reasoning_effort": "turbo"}) is None + assert session_effort.of({"reasoning_effort": 3}) is None + + +def test_a_read_needs_a_manager_and_a_key_and_caches_nothing(tmp_path: Path) -> None: + mgr = SessionManager(tmp_path) + assert session_effort.read(None, "tui:1") is None + assert session_effort.read(mgr, "") is None + assert session_effort.read(mgr, "tui:never") is None + # peek, not get_or_create: the miss left no ghost behind. + assert mgr.peek("tui:never") is None + + +def test_a_write_reaches_the_disk_for_a_filed_session(tmp_path: Path) -> None: + mgr = SessionManager(tmp_path) + mgr.save(mgr.get_or_create("tui:1")) + assert session_effort.write(mgr, "tui:1", "low") is True + assert session_effort.read(mgr, "tui:1") == "low" + assert session_effort.read(SessionManager(tmp_path), "tui:1") == "low" + assert session_effort.write(mgr, "tui:1", None) is True + assert session_effort.read(SessionManager(tmp_path), "tui:1") is None + + +def test_a_write_on_an_unfiled_session_waits_on_the_copy_in_memory(tmp_path: Path) -> None: + mgr = SessionManager(tmp_path) + assert session_effort.write(mgr, "tui:draft", "high") is False + assert session_effort.read(mgr, "tui:draft") == "high" + mgr.save(mgr.get_or_create("tui:draft")) + assert session_effort.read(SessionManager(tmp_path), "tui:draft") == "high" + + +def test_a_manager_that_cannot_patch_still_holds_the_pick(tmp_path: Path) -> None: + class _Broken(SessionManager): + def append_metadata_patch(self, key, patch): + raise OSError("disk went away") + + mgr = _Broken(tmp_path) + assert session_effort.write(mgr, "tui:1", "medium") is False + assert session_effort.read(mgr, "tui:1") == "medium" + + +def test_in_force_ranks_the_pick_over_the_mode_over_the_configured_default() -> None: + assert session_effort.in_force("minimal", "medium", "high") == "minimal" + assert session_effort.in_force(None, "medium", "high") == "medium" + assert session_effort.in_force(None, None, "high") == "high" + + +def test_in_force_answers_none_for_the_providers_own_default() -> None: + """None, not absence: the turn sends it and the RPC reports it as one answer.""" + assert session_effort.in_force(None, None, None) is None + assert session_effort.in_force("", "", "") is None + + +def test_of_accepts_a_word_from_the_extended_ladder() -> None: + assert session_effort.of({"reasoning_effort": "xhigh"}) == "xhigh" + assert session_effort.of({"reasoning_effort": "max"}) == "max" + + +def test_of_still_rejects_a_word_outside_the_ladder() -> None: + assert session_effort.of({"reasoning_effort": "turbo"}) is None + + +def test_clamp_keeps_a_declared_value() -> None: + assert session_effort.clamp("high", ("minimal", "low", "medium", "high")) == "high" + + +def test_clamp_drops_an_undeclared_value() -> None: + assert session_effort.clamp("max", ("minimal", "low", "medium", "high")) is None + + +def test_clamp_drops_everything_when_the_model_declares_none() -> None: + assert session_effort.clamp("high", ()) is None + + +def test_clamp_is_a_no_op_when_the_table_could_not_be_read() -> None: + assert session_effort.clamp("high", None) == "high" diff --git a/tests/test_session_manager.py b/tests/test_session_manager.py index 4d10931ea6..f722a240ca 100644 --- a/tests/test_session_manager.py +++ b/tests/test_session_manager.py @@ -1345,6 +1345,47 @@ def test_fork_of_an_unswitched_parent_carries_no_model(tmp_path: Path): assert "provider" not in child.metadata +def test_fork_inherits_the_reasoning_effort_the_parent_picked(tmp_path: Path): + """The pick rides the record beside the model, and a fork continues the + conversation it was picked for -- so it continues at it, restart or not.""" + from raven.agent import session_effort + + mgr = SessionManager(tmp_path) + src = _seed(mgr, "cli:src10", ("user", "a")) + src.metadata["model"] = "anthropic/claude-opus-4-8" + src.metadata[session_effort.METADATA_KEY] = "high" + mgr.save(src) + + child = mgr.fork("cli:src10") + + reloaded = SessionManager(tmp_path).get_or_create(child.key) + assert session_effort.of(reloaded.metadata) == "high" + assert reloaded.metadata["model"] == "anthropic/claude-opus-4-8" + + +def test_fork_of_a_parent_following_the_default_carries_no_effort(tmp_path: Path): + """A cleared pick is null on the record, not absent -- a metadata patch sets + keys and never removes one -- and the fork starts without the key either way.""" + from raven.agent import session_effort + + mgr = SessionManager(tmp_path) + src = _seed(mgr, "cli:src11", ("user", "a")) + src.metadata[session_effort.METADATA_KEY] = None + mgr.save(src) + + child = mgr.fork("cli:src11") + + assert session_effort.METADATA_KEY not in child.metadata + + +def test_the_fork_carries_the_key_the_effort_pick_is_kept_under(): + """The session layer spells the key rather than importing the agent's; + this is what keeps the two spellings one.""" + from raven.agent import session_effort + + assert session_effort.METADATA_KEY in manager_mod.FORK_CARRIES + + def test_fork_resets_pending_clarification(tmp_path: Path): """The child does not carry the source's clarification wait-state.""" mgr = SessionManager(tmp_path) diff --git a/tests/test_watch_work.py b/tests/test_watch_work.py index 550e2098f3..0eeb681e0a 100644 --- a/tests/test_watch_work.py +++ b/tests/test_watch_work.py @@ -796,7 +796,10 @@ async def test_the_hand_off_judgement_asks_for_the_turns_effort_too(): @pytest.mark.asyncio -async def test_no_pinned_effort_passes_nothing_so_the_provider_default_stands(): +async def test_no_pinned_effort_sends_the_providers_own_default_as_none(): + """None is sent, not left out: an argument left out is filled by the + provider from the copy of the key it took at boot, which is how a default + cleared on a running gateway kept running at the old word.""" loop, state = _loop() await loop._note_watch_work(state, "read_file", {"path": "/tmp/arena/run.sh"}, "r", "run /tmp/arena") - assert loop._llm_kwargs == [{}], "an explicit None would override the provider's sentinel" + assert loop._llm_kwargs == [{"reasoning_effort": None}] diff --git a/ui-tui/src/__tests__/effortCommand.test.ts b/ui-tui/src/__tests__/effortCommand.test.ts new file mode 100644 index 0000000000..7147a0dee1 --- /dev/null +++ b/ui-tui/src/__tests__/effortCommand.test.ts @@ -0,0 +1,152 @@ +// SPDX-License-Identifier: MIT +// Copyright (c) 2026 EverMind. +// See NOTICES.md. +// +// `/effort`: showing and changing how much the model thinks for this +// conversation. The pick is the session's own and lives on the server, so the +// command only ever paints from the reply. + +import { beforeEach, describe, expect, it, vi } from 'vitest' + +import { enterDirect, resetDirectChat } from '../app/directChatStore.js' +import { RESET_WORDS } from '../app/slash/commands/core.js' +import { sessionCommands } from '../app/slash/commands/session.js' +import { getUiState, patchUiState, resetUiState } from '../app/uiStore.js' +import { rpcErrorMessage } from '../lib/rpc.js' + +const cmd = sessionCommands.find(c => c.name === 'effort')! + +const WORDS = ['minimal', 'low', 'medium', 'high'] + +const run = ( + arg: string, + rpc = vi.fn(() => Promise.resolve({ session_key: 's1', effort: null, default: 'low', options: WORDS })), + sid: null | string = 's1' +) => { + const lines: string[] = [] + cmd.run( + arg, + { + gateway: { rpc }, + guarded: + (fn: (r: T) => void) => + (r: null | T) => { + if (r !== null) { + fn(r) + } + }, + guardedErr: (e: unknown) => { + lines.push(`error: ${rpcErrorMessage(e)}`) + }, + sid, + stale: () => false, + transcript: { sys: (text: string) => lines.push(text) } + } as never, + 'effort' + ) + + return { lines, rpc } +} + +const flush = () => new Promise(resolve => setTimeout(resolve, 0)) + +beforeEach(() => { + resetUiState() + resetDirectChat() + patchUiState({ info: { model: 'vendor/model', skills: {}, tools: {}, reasoning_effort: 'low' } }) +}) + +describe('/effort', () => { + it('refuses without a session and asks the gateway nothing', () => { + const { lines, rpc } = run('high', undefined, null) + + expect(lines).toEqual(['no active session']) + expect(rpc).not.toHaveBeenCalled() + }) + + it('reports the default when nothing is picked, naming what the default is', async () => { + const { lines, rpc } = run('') + await flush() + + expect(rpc).toHaveBeenCalledWith('session.set_effort', { session_key: 's1' }, { quiet: true }) + expect(lines).toEqual(['reasoning effort: default (low; words: minimal, low, medium, high)']) + }) + + it('names the provider when no effort is configured anywhere', async () => { + const rpc = vi.fn(() => Promise.resolve({ session_key: 's1', effort: null, default: null, options: WORDS })) + const { lines } = run('', rpc) + await flush() + + expect(lines).toEqual(["reasoning effort: default (the provider's own; words: minimal, low, medium, high)"]) + expect(getUiState().info?.reasoning_effort).toBe('') + }) + + it('sets a word from the reply and moves the status bar with it', async () => { + const rpc = vi.fn(() => Promise.resolve({ session_key: 's1', effort: 'high', default: 'low', options: WORDS })) + const { lines } = run('High', rpc) + await flush() + + expect(rpc).toHaveBeenCalledWith('session.set_effort', { session_key: 's1', effort: 'high' }, { quiet: true }) + expect(lines).toEqual(['reasoning effort: high (this conversation; default is low; words: minimal, low, medium, high)']) + expect(getUiState().info?.reasoning_effort).toBe('high') + }) + + // The words are `/mode`'s: one set, so the two commands cannot drift apart. + it.each([...RESET_WORDS])('clears the pick on `%s` and shows the default again', async word => { + const rpc = vi.fn(() => Promise.resolve({ session_key: 's1', effort: null, default: 'medium', options: WORDS })) + patchUiState(state => ({ ...state, info: { ...state.info!, reasoning_effort: 'high' } })) + const { lines } = run(word, rpc) + await flush() + + expect(rpc).toHaveBeenCalledWith('session.set_effort', { session_key: 's1', clear: true }, { quiet: true }) + expect(lines).toEqual(['reasoning effort: default (medium; words: minimal, low, medium, high)']) + expect(getUiState().info?.reasoning_effort).toBe('medium') + }) + + it.each(['', 'high', 'default'])( + 'refuses `/effort %s` inside a direct chat, naming the instance, and asks the gateway nothing', + arg => { + // The pick is the main conversation's; sent from an instance's pane it + // moved the main record while the reply there said "this conversation". + enterDirect('Raven-Code', 'refactor-auth') + const { lines, rpc } = run(arg) + + expect(lines).toEqual([ + "/effort is the main conversation's and does not reach Raven-Code/refactor-auth -- Esc returns to Raven" + ]) + expect(rpc).not.toHaveBeenCalled() + expect(getUiState().info?.reasoning_effort).toBe('low') + } + ) + + it('lands the gateway refusal on the transcript and leaves the status bar alone', async () => { + const rpc = vi.fn(() => + Promise.reject(new Error("no reasoning effort 'turbo'; the words are minimal, low, medium, high")) + ) + const { lines } = run('turbo', rpc) + await flush() + + expect(lines).toEqual(["error: no reasoning effort 'turbo'; the words are minimal, low, medium, high"]) + expect(getUiState().info?.reasoning_effort).toBe('low') + }) +}) + + +it('reports the words the model declares', async () => { + const rpc = vi.fn(() => Promise.resolve({ + session_key: 's1', effort: 'high', default: 'medium', + options: ['medium', 'high', 'xhigh'], + })) + const { lines } = run('', rpc) + await flush() + expect(lines.join('\n')).toContain('medium, high, xhigh') +}) + +it('says so when the model declares no reasoning effort', async () => { + const rpc = vi.fn(() => Promise.resolve({ + session_key: 's1', effort: null, default: null, options: [], + })) + const { lines } = run('', rpc) + await flush() + expect(lines.join('\n')).toContain('declares no reasoning effort') +}) diff --git a/ui-tui/src/__tests__/slashParity.test.ts b/ui-tui/src/__tests__/slashParity.test.ts index 58b5f2a2e3..13890ae285 100644 --- a/ui-tui/src/__tests__/slashParity.test.ts +++ b/ui-tui/src/__tests__/slashParity.test.ts @@ -18,6 +18,7 @@ const MUTATING_COMMANDS = [ 'busy', 'clear', 'compress', + 'effort', 'fast', 'model', 'new', diff --git a/ui-tui/src/app/slash/commands/core.ts b/ui-tui/src/app/slash/commands/core.ts index 9200878076..ba453bdf52 100644 --- a/ui-tui/src/app/slash/commands/core.ts +++ b/ui-tui/src/app/slash/commands/core.ts @@ -58,7 +58,7 @@ const flagFromArg = (arg: string, current: boolean): boolean | null => { return null } -const RESET_WORDS = new Set(['reset', 'clear', 'default']) +export const RESET_WORDS = new Set(['reset', 'clear', 'default']) const CYCLE_WORDS = new Set(['cycle', 'toggle']) const DETAILS_USAGE = diff --git a/ui-tui/src/app/slash/commands/session.ts b/ui-tui/src/app/slash/commands/session.ts index 4e1cf893d2..92d867cc83 100644 --- a/ui-tui/src/app/slash/commands/session.ts +++ b/ui-tui/src/app/slash/commands/session.ts @@ -14,6 +14,7 @@ import type { SessionUsageResponse, VoiceToggleResponse } from '../../../gatewayTypes.js' +import type { SessionSetEffortResult } from '../../../rpc/generated.js' import type { PanelSection } from '../../../types.js' import type { SlashCommand } from '../types.js' @@ -26,9 +27,11 @@ import { } from '../../../domain/messages.js' import { formatVoiceRecordKey, parseVoiceRecordKey } from '../../../lib/platform.js' import { fmtK } from '../../../lib/text.js' +import { getDirectChat } from '../../directChatStore.js' import { DEFAULT_INDICATOR_STYLE, INDICATOR_STYLES, type IndicatorStyle } from '../../interfaces.js' import { patchOverlayState } from '../../overlayStore.js' import { patchUiState } from '../../uiStore.js' +import { RESET_WORDS } from './core.js' // A model switch is per conversation; `--default` also changes what new ones start on. The picker passes ` --provider // `; a bare `/model ` carries no provider. Parse both into the @@ -630,6 +633,71 @@ export const sessionCommands: SlashCommand[] = [ } }, + { + help: 'show or set how much the model thinks for this conversation [word]', + name: 'effort', + usage: '/effort [word] -- default/reset/clear follow the global setting again; the words are the ones the model declares', + run: (arg, ctx) => { + if (!ctx.sid) { + return ctx.transcript.sys('no active session') + } + + // Refused inside a direct chat rather than redirected: the pick is the + // main conversation's, and the instance on screen has no reasoning + // effort of its own to set from here (`/mode` has one per instance; + // this does not). Sent anyway, it moved the main conversation's record + // while the reply, in the instance's pane, said "this conversation". + const { active } = getDirectChat() + + if (active) { + return ctx.transcript.sys( + `/effort is the main conversation's and does not reach ${active.agent}/${active.handle} -- Esc returns to Raven` + ) + } + + // The session's own reasoning effort, kept on its record by the server: + // a bare `/effort` reports, a word sets it from the next turn on, and + // the reset words (`/mode`'s) drop it so the conversation follows the + // global setting (or its mode's) again. The words are the server's to + // refuse, and its refusal names them, so none are spelt here. + const want = arg.trim().toLowerCase() + const params: Record = { session_key: ctx.sid } + + if (RESET_WORDS.has(want)) { + params.clear = true + } else if (want) { + params.effort = want + } + + ctx.gateway + .rpc('session.set_effort', params, { quiet: true }) + .then( + ctx.guarded(r => { + const fallback = r.default ?? null + const fallbackLabel = fallback ?? "the provider's own" + const options = r.options ?? [] + if (options.length === 0) { + ctx.transcript.sys('reasoning effort: this model declares no reasoning effort') + } else { + const words = options.join(', ') + ctx.transcript.sys( + r.effort + ? `reasoning effort: ${r.effort} (this conversation; default is ${fallbackLabel}; words: ${words})` + : `reasoning effort: default (${fallbackLabel}; words: ${words})` + ) + } + // The status bar shows what the next turn runs at, as session.info + // reports it: the pick, else the default. + patchUiState(state => ({ + ...state, + info: state.info ? { ...state.info, reasoning_effort: r.effort ?? fallback ?? '' } : state.info + })) + }) + ) + .catch(ctx.guardedErr) + } + }, + { help: 'toggle fast mode [normal|fast|status|on|off|toggle]', name: 'fast', diff --git a/ui-tui/src/i18n/messages.generated.ts b/ui-tui/src/i18n/messages.generated.ts index 257f531a8c..cf3ea8fd98 100644 --- a/ui-tui/src/i18n/messages.generated.ts +++ b/ui-tui/src/i18n/messages.generated.ts @@ -154,6 +154,10 @@ export const SLASH_TEXT: Record> = { name: '推理强度', help: '查看或设置推理强度（对运行中的 agent 生效）' }, + effort: { + name: '思考深度', + help: '查看或设置本会话的思考深度 [word] - 可选词由当前模型声明' + }, fast: { name: '快速模式', help: '开关快速模式 [normal|fast|status|on|off|toggle]' @@ -260,6 +264,10 @@ export const SLASH_TEXT: Record> = { name: 'export', help: 'export the transcript to Markdown' }, + effort: { + name: 'effort', + help: 'show or set how much the model thinks for this conversation [word] - the words are the ones the model declares' + }, lang: { name: 'lang', help: 'switch UI language [en|zh]' @@ -1892,11 +1900,24 @@ export const UI_TEXT: Record> = { 'gui.pb.required': 'required', 'gui.pb.no_default': '—', 'gui.tier.title': 'Mode', - 'gui.tier.scope': 'Reasoning depth, sub-agent mode and the rest follow this setting. This conversation only.', + 'gui.tier.scope': 'The tier the sub-agents this conversation dispatches run at. This conversation only.', 'gui.tier.failed': 'Could not switch to {name}', 'gui.tier.mode': 'Mode', 'gui.tier.mode_scope': "This is a deployment's own mode, not a built-in tier, so it is not passed on to sub-agents.", + 'gui.effort.title': 'Reasoning effort', + 'gui.effort.default': 'Default', + 'gui.effort.minimal': 'Minimal', + 'gui.effort.low': 'Low', + 'gui.effort.medium': 'Medium', + 'gui.effort.high': 'High', + 'gui.effort.xhigh': 'Extra high', + 'gui.effort.max': 'Max', + 'gui.effort.scope': + 'How much the model thinks before each answer, for this conversation only. Sub-agents are not affected.', + 'gui.effort.now': 'Default follows the global setting, which is {name} right now.', + 'gui.effort.provider': "the provider's own default", + 'gui.effort.failed': 'Could not set the reasoning effort to {name}', 'gui.imodel.title': 'Model', 'gui.imodel.agent': "Agent's own", 'gui.imodel.agent_tip': 'Answering with whatever this agent chose. Click to give this instance its own model.', @@ -2515,6 +2536,7 @@ export const UI_TEXT: Record> = { 'gui.settings.providers.remove_label': 'Remove the name for {model}', 'gui.settings.providers.model_id': 'Model ID', 'gui.settings.providers.label_ph': 'Shown as', + 'gui.settings.providers.effort_ph': 'Effort levels (comma-separated)', 'gui.settings.providers.description_ph': 'Description (optional)', 'gui.settings.providers.model_first': 'Pick a model first', 'gui.settings.providers.label_first': 'Enter a display name', @@ -2556,6 +2578,7 @@ export const UI_TEXT: Record> = { 'gui.settings.roles.video': 'Video', 'gui.settings.roles.video_use': 'Used by video_generate; without it the tool is not registered', 'gui.settings.roles.effort': 'Reasoning effort', + 'gui.settings.roles.effort_default': 'Provider default', 'gui.settings.roles.effort_minimal': 'Minimal', 'gui.settings.roles.effort_low': 'Low', 'gui.settings.roles.effort_medium': 'Medium', @@ -5357,10 +5380,22 @@ export const UI_TEXT: Record> = { 'gui.pb.required': '必填', 'gui.pb.no_default': '—', 'gui.tier.title': '模式', - 'gui.tier.scope': '模型思考深度、子智能体运行强度等都跟着这一档走。只作用于本会话。', + 'gui.tier.scope': '派出的子智能体按这一档运行。只作用于本会话。', 'gui.tier.failed': '切换到 {name} 没有成功', 'gui.tier.mode': '模式', 'gui.tier.mode_scope': '这是部署自带的模式，不是内置档位，因此不会下发给子智能体。', + 'gui.effort.title': '思考深度', + 'gui.effort.default': '默认', + 'gui.effort.minimal': '最少', + 'gui.effort.low': '低', + 'gui.effort.medium': '中', + 'gui.effort.high': '高', + 'gui.effort.xhigh': '极高', + 'gui.effort.max': '最高', + 'gui.effort.scope': '模型每次回答前思考多少，只作用于本会话，不影响子智能体。', + 'gui.effort.now': '「默认」跟随设置页里的全局值，当前是「{name}」。', + 'gui.effort.provider': '厂商默认', + 'gui.effort.failed': '思考深度切到「{name}」没有成功', 'gui.imodel.title': '模型', 'gui.imodel.agent': '跟随子智能体', 'gui.imodel.agent_tip': '用的是子智能体自己选的模型。点击可为这个实例单独指定模型。', @@ -5926,6 +5961,7 @@ export const UI_TEXT: Record> = { 'gui.settings.providers.remove_label': '移除 {model} 的显示名', 'gui.settings.providers.model_id': '模型 ID', 'gui.settings.providers.label_ph': '显示成什么', + 'gui.settings.providers.effort_ph': '思考深度档位（逗号分隔）', 'gui.settings.providers.description_ph': '说明（可留空）', 'gui.settings.providers.model_first': '先选一个模型', 'gui.settings.providers.label_first': '填个显示名', @@ -5965,6 +6001,7 @@ export const UI_TEXT: Record> = { 'gui.settings.roles.video': '视频', 'gui.settings.roles.video_use': 'video_generate 工具使用；不配则该工具不注册', 'gui.settings.roles.effort': '思考深度', + 'gui.settings.roles.effort_default': '厂商默认', 'gui.settings.roles.effort_minimal': '最少', 'gui.settings.roles.effort_low': '低', 'gui.settings.roles.effort_medium': '中', diff --git a/ui-tui/src/rpc/generated.ts b/ui-tui/src/rpc/generated.ts index 5bf5cf1096..4fc911e629 100644 --- a/ui-tui/src/rpc/generated.ts +++ b/ui-tui/src/rpc/generated.ts @@ -213,6 +213,10 @@ export interface SessionInitInfo { * A turn is in flight on this session right now. */ running?: boolean; + /** + * The reasoning effort this session's own model calls run at: its own pick (session.set_effort), else its mode's, else agents.defaults.reasoningEffort; null when the provider's own default stands. + */ + reasoning_effort?: string | null; /** * True for an Imported Conversation, which nothing can change; absent on every other session. */ @@ -3988,6 +3992,10 @@ export interface ModelAddModelParams { capabilities?: string[]; input_modalities?: string[]; output_modalities?: string[]; + /** + * The reasoning-effort words this model accepts, least first. Omit to leave a stored declaration alone; [] clears it. + */ + reasoning_efforts?: string[]; session_id?: string; } /** @@ -4771,6 +4779,44 @@ export interface SessionSetModeResult { description?: string; }[]; } +/** + * This interface was referenced by `RavenRpcRoot`'s JSON-Schema + * via the `definition` "SessionSetEffortParams". + */ +export interface SessionSetEffortParams { + session_key: string; + /** + * One of the words in options. Omit it to report without changing. + */ + effort?: string; + /** + * Drop the session's own pick and follow the default again. Wins over effort when both are given. + */ + clear?: boolean; + /** + * The model to answer for when the caller has no session yet (a draft). Omit for a session's own model. + */ + model?: string; +} +/** + * This interface was referenced by `RavenRpcRoot`'s JSON-Schema + * via the `definition` "SessionSetEffortResult". + */ +export interface SessionSetEffortResult { + session_key: string; + /** + * The session's own pick, or null while it follows the default. + */ + effort?: string | null; + /** + * What the session runs at with no pick of its own: its mode's effort where the catalogue sets one, else agents.defaults.reasoningEffort, else null for the provider's own default. + */ + default?: string | null; + /** + * The words this model accepts, least first. + */ + options?: string[]; +} /** * This interface was referenced by `RavenRpcRoot`'s JSON-Schema * via the `definition` "SessionSetKnowledgeParams". diff --git a/ui-web/scripts/fixture-params.mjs b/ui-web/scripts/fixture-params.mjs index 74b359cef4..912e78b567 100644 --- a/ui-web/scripts/fixture-params.mjs +++ b/ui-web/scripts/fixture-params.mjs @@ -27,6 +27,7 @@ export const PARAMS = { 'session.compress': { session_id: 'b' }, 'session.branch': { session_id: 'b' }, 'session.set_mode': { session_key: 'b', mode: 'max' }, + 'session.set_effort': { session_key: 'b', effort: 'high' }, 'turn.subscribe': { session_key: 'a' }, 'turn.unsubscribe': { subscription_id: 'sub-1' }, 'turn.send': { session_key: 'a', content: 'look at the market' }, diff --git a/ui-web/scripts/gates/boot-order.test.mjs b/ui-web/scripts/gates/boot-order.test.mjs index bfe709d77e..7b2bbff162 100644 --- a/ui-web/scripts/gates/boot-order.test.mjs +++ b/ui-web/scripts/gates/boot-order.test.mjs @@ -26,7 +26,7 @@ const installText = moduleText('app/install.ts') rest of the draw shells. */ const STEPS = [ 'lookLoad', 'paneLoad', 'setRail', 'sessionDraw', 'sessionOpen', - 'drawPerm', 'loadTier', 'drawCtx', 'bumpWs', 'drawSettings', + 'drawPerm', 'loadTier', 'loadEffort', 'drawCtx', 'bumpWs', 'drawSettings', 'setRuntime', 'goState', ] @@ -51,6 +51,7 @@ async function harness(rows = [], { connected = false, selectionOnly = true } = 'src/chrome/behaviour/panes': { load: step('paneLoad') }, 'src/state/perm': { draw: step('drawPerm') }, 'src/state/tier': { load: step('loadTier') }, + 'src/state/effort': { load: step('loadEffort') }, 'src/state/ctxChip': { draw: step('drawCtx') }, 'src/state/failureBar': { bootError: (where, error) => { throw new Error(`${where}: ${error}`) }, diff --git a/ui-web/src/app/boot.ts b/ui-web/src/app/boot.ts index 91640a845e..f349c60146 100644 --- a/ui-web/src/app/boot.ts +++ b/ui-web/src/app/boot.ts @@ -30,6 +30,7 @@ import { setCurrent as sessionSet } from '../lib/session' import { hasUpdateFlag } from '../rpc/capabilities' import { gateway } from '../rpc/gateway' import { draw as drawCtx } from '../state/ctxChip' +import { load as loadEffort } from '../state/effort' import { setRuntime } from '../state/envChip' import { bootError } from '../state/failureBar' import { load as loadLang } from '../state/lang/pick' @@ -175,6 +176,8 @@ async function sequence(): Promise { restored never changes session, so the chip would stay hidden on the one screen where the reader is about to start a conversation. */ loadTier() + /* The effort row's first read, for the same reason. */ + loadEffort() /* The import row's first read. Here rather than with the seam installers: those run before the socket opens, and a read there is refused as "not connected" -- which left a run the gateway lost to a restart, still in @@ -217,6 +220,7 @@ export function bootPage(): void { ['sessionOpen', (): void => { const first = sessionRows()[0]; if (first) void sessionOpen(first) }], ['drawPerm', (): void => drawPerm()], ['loadTier', (): void => { void loadTier() }], + ['loadEffort', (): void => { void loadEffort() }], ['drawCtx', (): void => drawCtx()], ['bumpWs', (): void => bumpWs()], ['drawSettings', (): void => redrawSettings()], diff --git a/ui-web/src/app/install.ts b/ui-web/src/app/install.ts index 7f9c2b5132..ca229919ba 100644 --- a/ui-web/src/app/install.ts +++ b/ui-web/src/app/install.ts @@ -30,7 +30,7 @@ import { capabilitiesSource, loadExt } from '../features/installed/source' import { knowledgeSource } from '../features/knowledge/source' import { reveal as revealDocument } from '../features/knowledge/store' import { memorySource } from '../features/memory/source' -import { blockSendForModel, modelSource, openModelsForMissingProvider, tierSource } from '../features/model/source' +import { blockSendForModel, effortSource, modelSource, openModelsForMissingProvider, tierSource } from '../features/model/source' import { onboardSource } from '../features/onboard/source' import { isOpen as onboardOpen, setBodies as setOnboardBodies, subscribe as onOnboard } from '../features/onboard/store' import { markNew } from '../features/rail/store' @@ -157,6 +157,7 @@ export function installSources(): void { sources.banner = bannerSource sources.settings = settingsSource sources.tier = tierSource + sources.effort = effortSource sources.model = modelSource composer.beforeSend = blockSendForModel diff --git a/ui-web/src/components/ModelTags.tsx b/ui-web/src/components/ModelTags.tsx index 6a883ce2c0..cf368197e2 100644 --- a/ui-web/src/components/ModelTags.tsx +++ b/ui-web/src/components/ModelTags.tsx @@ -192,6 +192,7 @@ export interface ModelTagFacts { capabilities?: string[] input_modalities?: string[] output_modalities?: string[] + reasoning_efforts?: string[] context_window?: number /* Which bucket a model list files this under, straight off the wire (`registry_data.kind_of`). A string rather than the model domain's `Kind` diff --git a/ui-web/src/features/model/ModelPicker.test.tsx b/ui-web/src/features/model/ModelPicker.test.tsx index 588e5fc216..68a6d11898 100644 --- a/ui-web/src/features/model/ModelPicker.test.tsx +++ b/ui-web/src/features/model/ModelPicker.test.tsx @@ -4,6 +4,7 @@ import { afterEach, describe, expect, it, vi } from 'vitest' import { resetTranslator, setTranslator } from '../../i18n/t' import * as confirmStore from '../../state/confirm' +import * as effort from '../../state/effort' import * as pageStore from '../../state/page' import { resetSources, setSources } from '../../state/sources' import * as tier from '../../state/tier' @@ -1141,3 +1142,64 @@ describe('the model picker, to a screen reader', () => { expect(recents().map((b) => b.getAttribute('aria-checked'))).toEqual(['true']) }) }) + +/* The reasoning effort rides the picker as a second row under the tier, for + the composer's opening only, drawn once the server has answered + (state/effort.ts). */ +describe('the effort row', () => { + const WORDS = ['minimal', 'low', 'medium', 'high'] + const seg = (): HTMLElement[] => [...document.querySelectorAll('.mpick .model-effort .model-seg button')] + + afterEach(() => { effort._resetForTests() }) + + it('offers the default rung and every word as one segmented control, and asks for the one clicked', async () => { + install() + const asked: Array = [] + setSources({ + model: (await import('./store')).source(), + effort: { + read: async () => { asked.push(undefined); return { effort: null, default: 'low', options: WORDS } }, + set: async (id) => { asked.push(id); return { effort: id, default: 'low', options: WORDS } }, + }, + }) + await act(async () => { await effort.load() }) + mount() + openIt() + expect(seg().map((b) => b.textContent)).toEqual(['gui.effort.default', 'gui.effort.minimal', 'gui.effort.low', 'gui.effort.medium', 'gui.effort.high']) + expect(seg().map((b) => b.getAttribute('aria-checked'))).toEqual(['true', 'false', 'false', 'false', 'false']) + /* One explanation behind the question mark, naming what default means. */ + const help = document.querySelector('.mpick .model-effort .model-tier-help')! + expect(help.getAttribute('data-tip')).toBe('gui.effort.scope gui.effort.now {"name":"gui.effort.low"}') + expect(help.hasAttribute('data-tip-wrap')).toBe(true) + await act(async () => { fireEvent.click(seg()[4]!) }) + expect(asked).toEqual([undefined, 'high']) + expect(seg().map((b) => b.getAttribute('aria-checked'))).toEqual(['false', 'false', 'false', 'false', 'true']) + /* Back to the default rung: a clear, not a word. */ + await act(async () => { fireEvent.click(seg()[0]!) }) + expect(asked).toEqual([undefined, 'high', null]) + expect(seg()[0]!.getAttribute('aria-checked')).toBe('true') + /* The picker stays up: an effort is not a pick of a model. */ + expect(pick()).not.toBeNull() + }) + + it('draws no row before the server answers, and none for a settings slot', async () => { + install() + mount() + openIt() + expect(document.querySelector('.mpick .model-effort')).toBeNull() + act(() => store.close()) + setSources({ + model: (await import('./store')).source(), + effort: { + read: async () => ({ effort: null, default: 'low', options: WORDS }), + set: async (id) => ({ effort: id, default: 'low', options: WORDS }), + }, + }) + await act(async () => { await effort.load() }) + act(() => { + store.open(document.getElementById('modelChip'), undefined, undefined, + { kind: 'text', title: 'Chat', pick: async () => {} }) + }) + expect(document.querySelector('.mpick .model-effort')).toBeNull() + }) +}) diff --git a/ui-web/src/features/model/ModelPicker.tsx b/ui-web/src/features/model/ModelPicker.tsx index 605b234f1b..265ea4608d 100644 --- a/ui-web/src/features/model/ModelPicker.tsx +++ b/ui-web/src/features/model/ModelPicker.tsx @@ -14,10 +14,10 @@ * and the window its vendor publishes (components/ModelTags). The id is on * the title, the provider is the head above, and a pick is a click. * - * Two rows that are not models ride along when the composer opened it: the - * sub-agent tier (state/tier.ts), a setting of the same conversation drawn - * where the model is chosen rather than as a chip of its own, and the door to - * the settings page. + * Three rows that are not models ride along when the composer opened it: the + * sub-agent tier (state/tier.ts) and the reasoning effort (state/effort.ts), + * two settings of the same conversation drawn where the model is chosen rather + * than as chips of their own, and the door to the settings page. */ import { useCallback, useEffect, useLayoutEffect, useRef, useState, useSyncExternalStore } from 'react' @@ -25,6 +25,7 @@ import { useCallback, useEffect, useLayoutEffect, useRef, useState, useSyncExter import { ModelTagDefs, ModelTags } from '../../components/ModelTags' import { ProviderIcon } from '../../components/ProviderMark' import { t } from '../../i18n/t' +import * as effort from '../../state/effort' import * as lang from '../../state/lang' import * as tier from '../../state/tier' import { recent } from './recent' @@ -94,6 +95,50 @@ function TierRow(): JSX.Element | null { ) } +/** A bulb: how much the model thinks before it answers. */ +const THINK = 'M12 3.5c-3.3 0-5.8 2.5-5.8 5.6 0 1.9.9 3.2 1.9 4.2.7.7 1.1 1.4 1.2 2.3h5.4c.1-.9.5-1.6 1.2-2.3 1-1 1.9-2.3 1.9-4.2 0-3.1-2.5-5.6-5.8-5.6ZM9.8 19h4.4' + +/* The effort row: the default rung and every word the server offers as one + segmented control, the one in force pressed. Drawn under the tier row for + the same opening and under the same rule -- a conversation's setting, so + not for a settings slot -- and only once the server has answered, for the + reason state/effort.ts's load gives (an older gateway has no row to draw). + + The same shape as the tier row on purpose: a reader meets the two as one + block of per-conversation settings. The explanation sits behind the question + mark and says what "default" means on this install, because that is the one + rung whose meaning is not in its name. */ +function EffortRow(): JSX.Element | null { + useSyncExternalStore(effort.subscribe, effort.get) + if (!effort.offered()) return null + const heading = effort.heading() + const note = effort.scopeNote() + return ( +
+ + + {heading} + + + + + + {effort.options().map((row) => ( + + ))} + +
+ ) +} + /* The word for a model: the registry's label where the provider carries one, the id without its vendor half otherwise -- the rule the chip follows too, so the chip and the row it was picked from say the same thing. */ @@ -391,6 +436,7 @@ function Pick(): JSX.Element { ) : null} {at.scope !== 'default' && !at.offer.title ? : null} + {at.scope !== 'default' && !at.offer.title ? : null} {at.footer ? (