diff --git a/prd-spec/bugfixes/adk-evaluator/2026-10-27-evaluator-model-calls.md b/prd-spec/bugfixes/adk-evaluator/2026-10-27-evaluator-model-calls.md new file mode 100644 index 000000000..a1584b070 --- /dev/null +++ b/prd-spec/bugfixes/adk-evaluator/2026-10-27-evaluator-model-calls.md @@ -0,0 +1,58 @@ +# Run each evaluation pass once, not once per case + +[中文版](2026-10-27-evaluator-model-calls.zh.md) + +## Metadata + +- Change ID: `adk-evaluator` +- Created/revised: 2026-10-27 +- Status: implemented +- Component: no maintained component spec covers evaluation; see the no-impact conclusion below + +## Background, goals, and scope + +`ADKEvaluator.evaluate()` loops over the eval cases and, for each case and each run, calls `generate_actual_outputs()`. That method already runs the agent on **every** case of the eval set. Evaluating `C` cases with `R` runs therefore makes `C x R x C` agent runs instead of `C x R`. With a real model each run is a paid call. + +Measured on 1.1.19 and confirmed unchanged on `main` (`2301477`) by reading the loop: with a scripted model that counts calls, 3 cases x 1 run made 9 model calls, 3 x 2 made 18, and 6 x 2 made 72. A 27-case set with 2 runs would make 1458 calls instead of 54. The scores themselves are not wrong: each case still reads its own result, so the defect is cost and latency, not correctness. The `veadk eval --evaluator adk` command uses this evaluator, so it is affected too. The `DeepevalEvaluator` was not examined. + +The fix runs the agent once per run and scores every case from those runs. It does not change thresholds, metrics, the result or failure structures, the public signatures, the eval-set formats, or the `DeepevalEvaluator`. + +## Scenarios and requirements + +- FR-1: Given `C` cases and `num_runs = R`, `evaluate()` calls `generate_actual_outputs()` exactly `R` times, so the agent runs `C x R` times in total. +- FR-2: Given the same actual outputs as before, each case is scored on the same invocations in the same run order, and returns the same results and failure messages as before. +- FR-3: Given a failing case, only that case is reported as failed, and the other cases are unaffected. +- FR-4: Each run's actual values are captured as ADK invocations right after its pass, because the next pass overwrites them in place. + +## Design and contract impact + +Hoist the run loop above the case loop. After each pass, convert the actual values of all cases into ADK `Invocation` objects (the existing conversion, moved into `_to_actual_invocation`) and keep them per run. Inside the case loop, the case's actual invocations are those of every run, in run order, which is what the loop produced before. Alternatives rejected: caching `generate_actual_outputs()` results per call index couples to the loop order; changing `generate_actual_outputs()` to take a case would alter a shared `BaseEvaluator` contract used by `DeepevalEvaluator`. + +Public imports, signatures, defaults, return values, eval-set formats, configuration, dependencies, concurrency, and security are unchanged. The only observable change is that the agent is called fewer times. Session ids are still regenerated before each run. No documentation page describes the call pattern. + +No component-spec impact: `specs/` has no evaluation component, and this changes neither an interface nor state, only how many times an existing call is repeated. No spec is created for a single bugfix. + +## Tasks and acceptance + +| Requirement | Task | Acceptance | Verification | +| --- | --- | --- | --- | +| FR-1 | T-1: add a failing regression test, then hoist the run loop | AC-1: 3 cases x 2 runs makes 2 passes, not 6 | `uv run --extra dev --extra eval pytest tests/test_adk_evaluator.py -q` | +| FR-2, FR-3 | T-2: keep the conversion identical, add a failing-case test | AC-2: a wrong tool call fails only its own case and the failure names the trajectory metric | same command | +| FR-4 | T-3: convert after each pass | AC-3: results across runs stay in run order | covered by AC-1 and AC-2 | +| All | T-4: run Ruff and the focused tests, reconcile both languages | AC-4: checks pass and limits are recorded | see delivery record | + +## Risks and review + +With a stochastic model, a case now uses results from the same `C x R` runs rather than from `C` repeated sweeps, so its distribution is unchanged and only the extra, discarded runs disappear. Nothing in the loop depended on those extra runs. Callers that counted on the extra runs for rate-limit warm-up are not known and are not expected. + +Design reviewed directly: scope, compatibility, tests, and bilingual equivalence have no open blockers. User approval: request on 2026-10-27 to submit this fix upstream. + +## Delivery record + +- Baseline: `2301477`; verification date: 2026-10-27 +- Scope: `adk_evaluator.py` loop, one regression test file, and this bilingual PRD; T-1 through T-4 complete +- Regression before the fix: fail as expected, `assert 6 == 2` for the pass count; the failing-case test passed on both versions +- Focused checks after the fix: pass, 4 tests via `uv run --extra dev --extra eval pytest tests/test_adk_evaluator.py tests/test_evaluator.py -q` +- Real-model evaluation: not_run, a scripted model measured the call counts and no provider was called +- `DeepevalEvaluator`: not_run, not examined +- UI validation: not_applicable, no frontend change diff --git a/prd-spec/bugfixes/adk-evaluator/2026-10-27-evaluator-model-calls.zh.md b/prd-spec/bugfixes/adk-evaluator/2026-10-27-evaluator-model-calls.zh.md new file mode 100644 index 000000000..4eb464c91 --- /dev/null +++ b/prd-spec/bugfixes/adk-evaluator/2026-10-27-evaluator-model-calls.zh.md @@ -0,0 +1,58 @@ +# 每轮评测只运行一次,而不是每个用例各运行一次 + +[English](2026-10-27-evaluator-model-calls.md) + +## 元数据 + +- 变更 ID:`adk-evaluator` +- 创建/修订日期:2026-10-27 +- 状态:implemented +- 组件:目前没有维护中的评测组件规格;见下文的“无影响”结论 + +## 背景、目标与范围 + +`ADKEvaluator.evaluate()` 遍历评测用例,并对每个用例的每一轮调用 `generate_actual_outputs()`。该方法本身已经会在评测集的**所有**用例上运行智能体。因此评测 `C` 个用例、`R` 轮,会运行 `C x R x C` 次智能体,而不是 `C x R` 次。使用真实模型时,每次运行都是一次付费调用。 + +在 1.1.19 上实测,并通过阅读代码确认 `main`(`2301477`)上的循环没有变化:用会计数的脚本化模型,3 个用例 x 1 轮产生 9 次模型调用,3 x 2 产生 18 次,6 x 2 产生 72 次。27 个用例、2 轮会产生 1458 次调用,而不是 54 次。评分本身没有错误:每个用例仍读取自己的结果,所以缺陷是成本和耗时,而不是正确性。`veadk eval --evaluator adk` 命令使用该评测器,因此同样受影响。`DeepevalEvaluator` 未检查。 + +修复方式是每轮只运行一次智能体,并用这些轮次的结果给每个用例评分。不改变阈值、指标、结果与失败信息的结构、公开签名、评测集格式,也不改变 `DeepevalEvaluator`。 + +## 场景与需求 + +- FR-1:给定 `C` 个用例和 `num_runs = R`,`evaluate()` 恰好调用 `generate_actual_outputs()` `R` 次,因此智能体总共运行 `C x R` 次。 +- FR-2:给定与之前相同的实际输出,每个用例在相同的调用上、按相同的轮次顺序评分,并返回与之前相同的结果和失败信息。 +- FR-3:给定一个失败的用例,只有该用例被报告为失败,其他用例不受影响。 +- FR-4:每一轮的实际值在该轮运行后立即转换为 ADK 调用对象,因为下一轮会原地覆盖它们。 + +## 设计与契约影响 + +把轮次循环提到用例循环之上。每轮运行后,把所有用例的实际值转换为 ADK `Invocation` 对象(沿用现有转换逻辑,移入 `_to_actual_invocation`),并按轮次保存。在用例循环内,该用例的实际调用就是每一轮中它自己的调用,按轮次顺序排列,与之前循环产生的结果一致。否决的方案:按调用序号缓存 `generate_actual_outputs()` 的结果,会与循环顺序耦合;让 `generate_actual_outputs()` 接收一个用例参数,会改变 `DeepevalEvaluator` 共用的 `BaseEvaluator` 契约。 + +公开导入、签名、默认值、返回值、评测集格式、配置、依赖、并发与安全均不变。唯一可观察的变化是智能体被调用的次数减少。每轮运行前仍会重新生成会话 ID。没有文档页面描述这一调用模式。 + +对组件规格无影响:`specs/` 中没有评测组件,本变更既不改变接口也不改变状态,只改变一个已有调用的重复次数。不为单个缺陷修复新建规格。 + +## 任务与验收 + +| 需求 | 任务 | 验收 | 验证 | +| --- | --- | --- | --- | +| FR-1 | T-1:先添加失败的回归测试,再提升轮次循环 | AC-1:3 个用例 x 2 轮只运行 2 次,而不是 6 次 | `uv run --extra dev --extra eval pytest tests/test_adk_evaluator.py -q` | +| FR-2, FR-3 | T-2:保持转换逻辑不变,添加失败用例测试 | AC-2:错误的工具调用只让它自己的用例失败,且失败信息包含轨迹指标名称 | 同上 | +| FR-4 | T-3:每轮运行后立即转换 | AC-3:各轮结果保持轮次顺序 | 由 AC-1 和 AC-2 覆盖 | +| 全部 | T-4:运行 Ruff 和聚焦测试,核对两种语言 | AC-4:检查通过且记录了局限 | 见交付记录 | + +## 风险与评审 + +使用带随机性的模型时,每个用例现在使用同一批 `C x R` 次运行的结果,而不是 `C` 次重复遍历的结果,因此分布不变,只是多余且被丢弃的运行消失了。循环中没有任何逻辑依赖这些多余的运行。未发现也不预期有调用方依赖额外运行来做限流预热。 + +设计已直接评审:范围、兼容性、测试与双语一致性均无未解决的阻塞项。用户批准:2026-10-27 请求将此修复提交到上游。 + +## 交付记录 + +- 基线:`2301477`;验证日期:2026-10-27 +- 范围:`adk_evaluator.py` 的循环、一个回归测试文件,以及本双语 PRD;T-1 至 T-4 已完成 +- 修复前的回归:按预期失败,运行次数断言为 `assert 6 == 2`;失败用例测试在两个版本上都通过 +- 修复后的聚焦检查:通过,通过 `uv run --extra dev --extra eval pytest tests/test_adk_evaluator.py tests/test_evaluator.py -q` 运行 4 个测试 +- 真实模型评测:not_run,用脚本化模型测得调用次数,没有调用任何提供方 +- `DeepevalEvaluator`:not_run,未检查 +- 界面验证:not_applicable,没有前端改动 diff --git a/tests/test_adk_evaluator.py b/tests/test_adk_evaluator.py new file mode 100644 index 000000000..2d7ed92ca --- /dev/null +++ b/tests/test_adk_evaluator.py @@ -0,0 +1,110 @@ +# Copyright (c) 2025 Beijing Volcano Engine Technology Co., Ltd. and/or its affiliates. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + + +from types import SimpleNamespace + +import pytest +from google.adk.evaluation.eval_case import ( + EvalCase, + IntermediateData, + Invocation, + SessionInput, +) +from google.adk.evaluation.eval_set import EvalSet +from google.genai import types + +from veadk.evaluation.adk_evaluator import ADKEvaluator + + +def _eval_set(case_count: int) -> EvalSet: + """`case_count` single-turn cases, each expecting one distinct tool call.""" + cases = [] + for i in range(case_count): + cases.append( + EvalCase( + eval_id=f"case-{i}", + conversation=[ + Invocation( + invocation_id=f"case-{i}-inv", + user_content=types.Content( + role="user", parts=[types.Part(text=f"question {i}")] + ), + final_response=types.Content( + role="model", parts=[types.Part(text="ok")] + ), + intermediate_data=IntermediateData( + tool_uses=[ + types.FunctionCall(name=f"tool_{i}", args={"n": i}) + ] + ), + ) + ], + session_input=SessionInput( + app_name="test_app", user_id="test_user", state={} + ), + ) + ) + return EvalSet(eval_set_id="test_set", eval_cases=cases) + + +class _FakeRunEvaluator(ADKEvaluator): + """Replaces the model run: every pass reports each case's expected values as actual. + + `generate_actual_outputs()` runs the agent on ALL cases, so each call is one real model + call per case in production. `wrong_tool` makes one case call the wrong tool. + """ + + def __init__(self, wrong_tool: str | None = None) -> None: + super().__init__(agent=SimpleNamespace(name="fake_agent")) + self.passes = 0 + self._wrong_tool = wrong_tool + + async def generate_actual_outputs(self): + self.passes += 1 + for case in self.invocation_list: + for inv in case.invocations: + inv.actual_output = inv.expected_output + inv.actual_tool = list(inv.expected_tool) + if self._wrong_tool and self.invocation_list: + self.invocation_list[1].invocations[0].actual_tool = [ + {"name": self._wrong_tool, "args": {}} + ] + + +@pytest.mark.asyncio +async def test_each_run_runs_the_cases_once(): + # 3 cases x 2 runs must cost 2 passes over the cases, not 3 x 2 = 6. + evaluator = _FakeRunEvaluator() + + _, failures = await evaluator.evaluate( + eval_set=_eval_set(3), num_runs=2, print_detailed_results=False + ) + + assert failures == [] + assert evaluator.passes == 2 + + +@pytest.mark.asyncio +async def test_a_failing_case_is_still_reported_for_that_case_only(): + evaluator = _FakeRunEvaluator(wrong_tool="some_other_tool") + + results, failures = await evaluator.evaluate( + eval_set=_eval_set(3), num_runs=2, print_detailed_results=False + ) + + assert len(results) == 1 # one eval file + assert len(results[0]) == 6 # 3 cases x 2 metrics + assert len(failures) == 1 + assert "tool_trajectory_avg_score" in failures[0] diff --git a/veadk/evaluation/adk_evaluator/adk_evaluator.py b/veadk/evaluation/adk_evaluator/adk_evaluator.py index 2c922e01c..63da7eceb 100644 --- a/veadk/evaluation/adk_evaluator/adk_evaluator.py +++ b/veadk/evaluation/adk_evaluator/adk_evaluator.py @@ -90,6 +90,30 @@ def __init__( """ super().__init__(agent=agent, name=name) + @staticmethod + def _to_actual_invocation(inv) -> Invocation: + """Packs BaseEvaluator's actual data of one invocation into an ADK Invocation.""" + user_content = genai_types.Content( + role="user", + parts=[genai_types.Part(text=inv.input or "")], + ) + actual_final = genai_types.Content( + role=None, + parts=[genai_types.Part(text=inv.actual_output or "")], + ) + # Collect the tool calls observed during actual execution + actual_tool_calls = [ + SimpleNamespace(name=t.get("name"), args=t.get("args", {})) + for t in (inv.actual_tool or []) + ] + # Pack a full actual Invocation for ADK metrics + return Invocation( + invocation_id=inv.invocation_id, + user_content=user_content, + final_response=actual_final, + intermediate_data=IntermediateData(tool_uses=actual_tool_calls), + ) + @override async def evaluate( self, @@ -171,7 +195,28 @@ async def evaluate( evaluation_result_list = [] - # For each eval case, generate actual outputs num_runs times using BaseEvaluator + # Run the agent num_runs times. generate_actual_outputs() already runs EVERY eval + # case, so it must be called once per run, not once per case and run (that would + # multiply the model calls by the number of cases). Each pass overwrites the actual + # values in place, so convert them to ADK Invocations right after it. + actual_invocations_by_run: list[list[list[Invocation]]] = [] + for _ in range(num_runs): + for agent_information in self.agent_information_list: + agent_information["session_id"] = str(uuid.uuid4()) + + await self.generate_actual_outputs() + + actual_invocations_by_run.append( + [ + [ + self._to_actual_invocation(inv) + for inv in eval_case_data.invocations + ] + for eval_case_data in self.invocation_list + ] + ) + + # For each eval case, score the actual invocations of all runs per metric for case_idx, eval_case_data in enumerate(self.invocation_list): # Convert BaseEvaluator's expected data into ADK Invocation list expected_invocations: list[Invocation] = [] @@ -200,41 +245,12 @@ async def evaluate( ) ) - # Collect actual invocations across runs - actual_invocations_all_runs: list[Invocation] = [] - for _ in range(num_runs): - for agent_information in self.agent_information_list: - agent_information["session_id"] = str(uuid.uuid4()) - - # Generate actual outputs for all cases in this run via BaseEvaluator - await self.generate_actual_outputs() - - # Convert BaseEvaluator's actual data into ADK Invocation list - for inv in eval_case_data.invocations: - user_content = genai_types.Content( - role="user", - parts=[genai_types.Part(text=inv.input or "")], - ) - actual_final = genai_types.Content( - role=None, - parts=[genai_types.Part(text=inv.actual_output or "")], - ) - # Collect the tool calls observed during actual execution - actual_tool_calls = [ - SimpleNamespace(name=t.get("name"), args=t.get("args", {})) - for t in (inv.actual_tool or []) - ] - # Pack a full actual Invocation for ADK metrics - actual_invocations_all_runs.append( - Invocation( - invocation_id=inv.invocation_id, - user_content=user_content, - final_response=actual_final, - intermediate_data=IntermediateData( - tool_uses=actual_tool_calls - ), - ) - ) + # This case's actual invocations from every run, in run order + actual_invocations_all_runs: list[Invocation] = [ + invocation + for run_invocations in actual_invocations_by_run + for invocation in run_invocations[case_idx] + ] # Repeat expected invocations to align with num_runs expected_invocations_repeated = expected_invocations * num_runs