Skip to content

feat(observability): LLMOps five-dimension metric aggregation - #221

Open
raymondginger2018-sudo wants to merge 1 commit into
HKUDS:mainfrom
raymondginger2018-sudo:feat/llmops-metrics
Open

feat(observability): LLMOps five-dimension metric aggregation#221
raymondginger2018-sudo wants to merge 1 commit into
HKUDS:mainfrom
raymondginger2018-sudo:feat/llmops-metrics

Conversation

@raymondginger2018-sudo

Copy link
Copy Markdown
Contributor

Summary

Adds aggregation for the five LLMOps metrics: Quality / Harm / Honesty / Cost / Latency.

This module consumes recorded per-call records and aggregates them into the five dimensions:

  • Cost — from token usage × a pluggable per-model price table (USD)
  • Latency — from recorded durations (ms), p50/p95/max
  • Quality / Harm / Honesty — machine-observable proxies + optional LLM-as-judge hook (judge_fn) for sampled labels; without a judge these stay None (unmeasured) rather than fabricated

Design

  • Pure mechanism: consumes a list of record dicts and returns a summary dict
  • No I/O, no network, no model calls by default
  • Pluggable price table: DEEPCODE_LLM_PRICES env or caller-supplied table
  • Honest about unmeasured dimensions (no fabricated scores)

File

  • core/observability/llmops.py (new, 158 lines)

Part of GenAI lesson 14 LLMOps family.

@raymondginger2018-sudo

Copy link
Copy Markdown
Contributor Author

设计说明

问题:agent 运行的质量、安全、诚实、成本、延迟五个维度分散在各处日志里,没有统一聚合视图。管理者难以回答"这周 agent 系统整体健康吗"。

解法:五维聚合器——每个维度定义评分函数(0-100),最终输出综合报告。价格表通过 DEEPCODE_LLM_PRICES 环境变量注入(默认内置常见模型价格)。

关键设计决策

  • 五维正交:Quality(任务完成率)、Harm(安全事件)、Honesty(幻觉/误导)、Cost(token 费用)、Latency(响应延迟)——避免一维分数掩盖问题
  • 价格表可注入,不硬编码——环境变量 DEEPCODE_LLM_PRICES 传 JSON
  • 输入是结构化事件列表,输出是聚合摘要,无副作用

测试建议:构造各维度的事件样本,验证评分单调性(更多坏事件 → 更低分)

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant