Skip to content

feat: 列级分析查询面与造数入口 — columns/predicates/transform/跨表键 (#165-#169) - #170

Merged
c2j merged 14 commits into
mainfrom
feat/issue-165_to_168
Sep 8, 2026
Merged

c2j merged 14 commits into
mainfrom
feat/issue-165_to_168

Conversation

@c2j

@c2j c2j commented Sep 8, 2026

Copy link
Copy Markdown
Owner

概述

打通「列级分析 → 造数/mock 机器可读入口」主线,一次性落地 #165#169 五个 issue。实施计划见 docs/plans/2026-09-08-issue-165-169-column-analysis-surface.md(Momus 两轮审核通过,决策 D1–D6 全部锁定)。

Closes #165, closes #166, closes #167, closes #168, closes #169

主要变更

#169 函数包裹列的字面量过滤 → HardFilter transform

  • HardFilter.transform: Option<FilterTransform>(serde "fn"),白名单 {substr, substring→substr, nvl, trim, upper, lower}FunctionCall + SpecialFunction 双变体(关键字语法 SUBSTR(x FROM 1 FOR 2) 不再漏)
  • 六个比较操作符统一支持;substr(col,1,v_len) 等含变量实参自动排除
  • ⚠️ 序列化采用 is_human_readable() 分支手写实现——skip_serializing_if 会破坏 bincode 固定布局(已被 regress_mark 8/20 失败暴露后修复),后续 bincode 持久化结构沿用此模式

#168 %ROWTYPE 记录字段 → 跨表等值键

  • JoinConditionSource 新增 RecordField 变体;WHERE/JOIN ON 中一侧为已解析记录字段时产出跨表 JoinCondition(如 par_sys_purchase.security_id ↔ mid_yjqs_detail.security_id
  • column_source 的三段解析规则抽取为 resolve_record_field 复用;plain join / 记录-vs-参数 / 未注册记录三组负例锁死

#167 PL IF/CASE 条件 → 表列谓词

  • 新增 branch-aware PredicateExtractor(走完整 PlBlock,分支结构在 extract_body_sql 摊平前收集)
  • 置信度规则落地(记录字段/裸列→high;SELECT INTO 主表变量→medium;维表变量→low + param_table_hint;函数/动态 SQL→low 保留 origin,绝不静默 high)
  • PredicateClause.transform 携带 feat: 函数包裹列的字面量过滤纳入 HardFilter(substr/nvl/trim) #169 同款变换描述;游标 WHERE HardFilter 不混入谓词列表
  • store 新增 procedure_predicates 侧表;CLI codeweb predicates

#165 按过程导出 ColumnAnalysis(CLI + MCP + HTTP 三面同 schema)

  • 方案A(review 发现的 store 期数据丢失修复)merge_table_access_edges 对全部诊断字段做保序并集(此前「保留第一条」导致同过程多语句同表时 hard_filters/join_conditions 丢失)
  • 新增 codeweb columns --procedure|--package [--table] --format json;后端 graph::columns 三面共享,字段与 ColumnAnalysis 1:1
  • MCP 新增 codeweb_column_analysis / codeweb_lineage(6→8 工具);HTTP 新增 GET /api/v1/columnsGET /api/v1/lineage
  • lineage 目标解析抽取为 graph::lineage::parse_lineage_target 纯函数(cmd_lineage 行为不变)

#166 文档

  • README(中英)CLI/HTTP/MCP 三表、user-guide §6 三个新小节、DeveloperGuide ColumnAnalysis 字段表(含 transform/RecordField)与 mock 造数场景、getting-started(_zh) 可照跑示例、serve-api-guide 双端点——全部示例由真实 --help/真实 JSON 输出生成,QA 清单 5/5

⚠️ 破坏性说明

  • STORE_VERSION 9 → 12(v10 诊断并集+transform、v11 谓词侧表、v12 PredicateClause.transform)。旧 store 会被版本门禁拒绝并提示 codeweb analyze 重建;增量 analyze 场景经 store_is_current() 自动重建
  • 谓词存储键按 RoutineId::normalized() 小写化(与 NodeKey::from_node 查找路径一致)——修复 review 发现的大写过程名查找 miss

验证

cargo build --features full                              ✅
cargo test --features full -- --skip test_path_mapping_applied --skip test_serve_
                                                         ✅ 39 测试二进制全 ok,754+ 单测 0 失败
cargo clippy --features full -- -D warnings              ✅
cargo fmt --all -- --check                               ✅

新增 ~45 个测试(extractor 单测 15、predicates 单测 11、store round-trip、lineage 目标解析 6、集成 regress_columns/regress_predicates/mcp/serve)。TDD Red→Green 全程留痕(含 kill-switch 式修复验证)。

已披露的合规偏差

  1. tests/regress_issue_159_sequence_informed.rs 版本戳字面量 9→12(跟踪 STORE_VERSION 的机械同步,语义断言不变;版本拒绝基线测试未动)
  2. tests/mcp_test.rs 工具清单断言 6→8(新增工具的必要同步,plan-sanctioned)
  3. builder 谓词键测试原始大小写断言 "proc:P""proc:p"(跟随正确的归一化修复)

已知限制

  • MCP/HTTP 的 lineage 使用默认 LineageConfig(不读取项目 codeweb.toml [lineage] 覆盖项);与 CLI 默认行为一致
  • predicates 暂无 MCP/HTTP 入口(决策 D3:YAGNI,issue 验收未强制)

🤖 Generated with Claude Code

c2j and others added 9 commits September 8, 2026 16:49
覆盖 #165 columns 查询面 / #166 文档 / #167 PL 谓词 / #168 记录字段跨表键 / #169 transform 白名单;决策 D1-D6 全部锁定(方案A 合并、RecordField 变体、独立 predicates 命令、合并版本 bump、双变体白名单、store 侧表方案)。

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
fix #169, fix #168)

- HardFilter 增加 transform(FilterTransform,serde "fn");白名单 {substr,substring,nvl,trim,upper,lower},FunctionCall+SpecialFunction 双变体;六个比较操作符统一支持
- transform 序列化采用 is_human_readable() 分支手写实现(bincode 固定布局 / JSON 省略 None)
- JoinConditionSource 新增 RecordField 变体;column_source 记录字段解析抽取为 record_field_source/resolve_record_field 复用;等值一侧为已解析记录字段时产出跨表 JoinCondition
- 诊断类型补 Hash derive(方案A 并集合并前置)

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
- PredicateExtractor 走整个 PlBlock 保留分支结构(extract_body_sql 摊平前收集)
- 置信度规则:记录字段/裸列→high;SELECT INTO 主表变量→medium;维表变量→low + param_table_hint;函数/动态 SQL→low 保留 origin,绝不静默 high
- 条件转换复用 #169 column_transform_of 与 #168 记录字段解析;PredicateClause.transform 沿用 HardFilter 的 is_human_readable 序列化模式
- 过程内 SELECT INTO 变量源追踪;SELECT INTO 目标/变量数不匹配时 parse_log 告警;游标 WHERE HardFilter 不混入谓词列表

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
- v10: merge_table_access_edges 诊断字段并集(#165 方案A)+ transform 预留(#169)
- v11: procedure_predicates 侧表(HashMap<NodeKey, Vec<PlPredicate>>,serde default,merge 时并集)
- v12: PredicateClause.transform(bincode 布局变更)
- 版本拒绝基线测试不变;版本戳字面量测试同步至 12

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
- merge_table_access_edges 对 join_conditions/hard_filters/enum_mappings/select_into/insert_columns/update_columns/column_refs 做保序 HashSet 并集,alias_map 首见优先;column_mappings/read_tables 语义不变(方案A,修复同过程多语句同表时诊断字段「保留第一条」丢失)
- 过程构建期走整个 PlBlock 收集谓词,procedure_predicates 经 RoutineId::normalized() 以与 NodeKey::from_node 一致的小写键写入(修复大小写不匹配查找 miss)
- project/mod.rs 接线 ctx.procedure_predicates → GraphStore

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
column_analysis_of_routine/column_analysis_of_package 扫入边+出边逐字段去重聚合,--table 过滤;AggregatedColumnAnalysis 字段与 ColumnAnalysis 1:1(schema_version 1),供 CLI/MCP/HTTP 三面共享,不另造 schema

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
- columns --procedure|--package [--table] --format json:旧 store(<v10) 软提示重跑 analyze;unknown 过程非零退出不静默
- predicates --procedure --format json:旧 store(<v12) 软提示;输出 {schema_version, procedure, predicates}
- regress_columns / regress_predicates 端到端回归(含大写过程名大小写无关解析、记录字段跨表键、param_table_hint)

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
…165)

- MCP 新增 codeweb_column_analysis / codeweb_lineage(与 CLI JSON 字段 1:1,空图守卫与错误措辞对齐既有工具);工具清单测试 6→8
- HTTP 新增 GET /api/v1/columns、GET /api/v1/lineage(400 非法输入 / 404 未命中,同 serde 结构直出)
- lineage 目标解析抽取为 graph::lineage::parse_lineage_target 纯函数(6 个单测),cmd_lineage 行为不变

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
- README 中英 CLI/HTTP/MCP 三表补齐(示例与 --help 逐字一致)
- user-guide §6 新增 lineage/columns/predicates 小节;DeveloperGuide 增加 ColumnAnalysis 字段表(含 transform/RecordField)与 mock 造数消费场景
- getting-started(_zh) 增加可照跑的列级血缘示例;serve-api-guide 补 /columns 与 /lineage 端点(真实 JSON 输出)

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
Comment thread src/parser/predicates.rs Outdated
impl Visitor for PredicateExtractor<'_> {
fn visit_pl_statement(&mut self, stmt: &PlStatement) -> VisitorResult {
match stmt {
PlStatement::If(spanned) => self.push_condition(

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[bug] PredicateExtractor only records PlIfStmt.condition and each searched-CASE when.condition. walk_pl_statement walks elsifs[].condition via walk_expr, but this visitor does not implement visit_expr, so ELSIF predicates are never collected. Simple CASE (CASE r.x WHEN '1', where PlCaseStmt.expression is Some and when.condition is just the WHEN value) is pushed as a bare literal and degrades to Low/table_predicate: null instead of r.x = '1'. The plan explicitly lists elsifs: Vec<PlElsif> and PlCaseStmt.expression; both are untested. Enterprise PL/SQL that branches on ELSIF or simple CASE will under-report constraints to mock/codegen consumers.

Suggestion: In the PlStatement::If arm, also push_condition every spanned.elsifs entry (same PredicateKind::If, elsif span/line). In the Case arm, when spanned.expression is Some, synthesize expression = when.condition (or a dedicated comparison Expr) before push_condition. Add fixtures for IF … ELSIF … and CASE rec.col WHEN '1'.

Comment thread src/parser/predicates.rs Outdated
return Some(ConditionResolution::Direct(vec![resolved]));
}
}
let name = expr_name(expr)?.to_lowercase();

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[bug] After the record-field resolved_clause miss, condition_operand does expr_name(expr)?. expr_name only matches ColumnRef/PlVariable, so any FunctionCall/SpecialFunction returns None and bails out before (a) the SELECT INTO var_sources lookup and (b) the sole-cursor-table naked-column fallback that would have attached transform. Consequences: IF substr(stock_kind,1,2) = '05' (naked column, one cursor table — the documented 裸列→high path) and IF substr(v_kind,1,2) = '05' (SELECT INTO var) both become Low with no table_predicate, even though the same substr on r.stock_kind is High. The Derived arm also hard-codes transform: None, so even a future inner-name lookup would drop the #169 descriptor. Only the record-field transform path is tested.

Suggestion: Do not ? on expr_name(expr). Prefer, in order: record-field resolve (already done); var_sources lookup using expr_name(expr) or the inner transform target name; then the names.len()==1 && tables.len()==1 fallback. Thread transform through the Derived PredicateClause as well. Add tests for naked-column substr(...) and substr(v_into_var, …).

Comment thread src/main.rs Outdated
.ok_or_else(|| error::CodeWebError::ExportError {
message: format!("No PL predicates found for '{}'", procedure),
})?;
let procedure_name = display

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[bug] cmd_predicates sets procedure from NodeKey::from_node(...).to_string() after split_once(':'). For a packaged routine that display key is proc:pkg.prc, so JSON procedure becomes pkg.prc. column_analysis_of_routine instead uses id.name (prc) plus a separate package field. Predicates JSON has no package field. Mock consumers joining columns and predicates on procedure will miss packaged routines. Tests only cover standalone procedures, so this never fails in CI.

Suggestion: Use the same identity as columns: id.name plus package: id.package (and schema if you want it). Do not parse it back out of the NodeKey display string. Add an end-to-end packaged-procedure predicates test that asserts procedure/package match codeweb columns.

Comment thread src/main.rs Outdated
&name,
crate::graph::search::MatchMode::Substring,
false,
false,

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] cmd_columns / cmd_predicates match ResolveResult::Ambiguous and return a non-zero error, but they call resolve_single_node(..., all_matches=false, fail_on_multiple=false). That helper returns Single (first hit) on multiple substring matches, so the Ambiguous arm is unreachable. MCP resolve_node and HTTP resolve_node use the same flags, so they also silently pick the first match (MCP then reports "No nodes matching" only for Empty). For a machine-readable mock entry point this can emit another routine's filters without failing.

Suggestion: Pass fail_on_multiple=true (4th argument) in CLI/MCP/HTTP, and make MCP/HTTP distinguish empty vs ambiguous. Add a two-proc substring fixture that expects a non-zero / 4xx response.

Comment thread src/main.rs Outdated
.cloned()
.filter(|predicates| !predicates.is_empty())
.ok_or_else(|| error::CodeWebError::ExportError {
message: format!("No PL predicates found for '{}'", procedure),

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] A procedure that exists but has no IF/CASE (or whose predicates were never stored because collect_block_predicates skips empty vecs) exits non-zero with No PL predicates found. codeweb columns returns empty arrays for a routine with no diagnostics. Empty branch constraints are a valid mock input (predicates: []); treating them as an error makes “unknown name” and “no branches” look similar to automations (they do differ in the message, but both are hard failures).

Suggestion: On a resolved procedure with no stored predicates, print {schema_version, procedure, predicates: []} and exit 0; reserve non-zero for unresolved/ambiguous names.

Comment thread src/parser/extractor.rs Outdated
}
}

/// #168: an equi-comparison where exactly one side is a `%ROWTYPE` record field

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[suggestion] Several new comments restate control flow or embed review/issue history instead of a short WHY. The extract_record_field_join block restates the match arms and the no-guess policy already visible in the code; parse_lineage_target (src/graph/lineage.rs:1916) narrates why cmd_lineage was left duplicated. Same pattern on collect_diagnostics and HardFilter's serialize impl (the bincode-vs-JSON constraint there is a useful WHY, but it is longer than needed).

Suggestion: Keep one sentence on the non-obvious constraint (bincode field count; RecordField vs alias lookup). Drop the issue-number / review-finding / “intentionally left untouched” narration; if CLI lineage parsing should stay in sync, share parse_lineage_target instead of documenting the fork.

c2j and others added 4 commits September 8, 2026 17:53
- If 臂遍历 elsifs 逐条产出谓词;简单 CASE(expression: Some)合成 expression = WHEN 值比较,不再把裸字面量降级为 Low
- condition_operand 移除 expr_name(expr)? 提前断链:记录字段 → var_sources(expr_name 或 transform 目标列名)→ sole-table fallback 顺序保持;裸列 substr 与 SELECT INTO 变量 substr 均正确解析
- Derived 臂携带 transform(删除硬编码 None);PL 变量的 transform 提取打通

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
…/F5)

- predicates JSON 的 procedure 取 RoutineId 裸名并新增 package 字段,与 columns 可 join(此前包内过程为 pkg.prc)
- columns/predicates/MCP/HTTP 四处新查询面 fail_on_multiple=true:多匹配非零/400/error JSON 显式失败,消灭静默取首个(MCP 区分 Empty 与 Ambiguous 文案;HTTP 歧义 400、未命中 404)
- resolved 无分支过程返回 predicates: [] exit 0,非零保留给未解析/歧义
- cmd_lineage 改用共享 parse_lineage_target(行为不变,regress_lineage 套件守护);文档同步

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
仅注释:保留非显性约束(bincode 固定字段数等)并压缩篇幅,删除控制流复述与评审/issue 编号叙事

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
六项评审发现的核实记录与 F1-F6 修复方案

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
@c2j

c2j commented Sep 8, 2026

Copy link
Copy Markdown
Owner Author

六条评审意见已全部修复(计划经 Momus 审核),提交 351736d / fafeaa2 / 8a3f0c1

# 修复 提交
① ELSIF + 简单 CASE 漏采 If 臂遍历 elsifs 逐条产出;简单 CASE 合成 expression = WHEN 值 比较(High + 列 clauses)。新增单测 elsif_conditions_collected_as_predicatessimple_case_synthesizes_expression_comparison + e2e 351736d
expr_name(expr)? 断链 移除提前 bail;fallback 顺序保持(记录字段 → var_sources(含 transform 目标列名)→ sole-table);Derived 臂携带 transform。新增 naked_column_substr_resolves_via_sole_tableselect_into_var_substr_resolves_via_var_source 351736d
③ predicates 身份不一致 procedure 取 RoutineId 裸名 + 新增 package 字段,与 columns 可 join;e2e predicates_identity_matches_columns_for_packaged_procedure fafeaa2
④ 歧义静默取首个 四处新查询面 fail_on_multiple=true;CLI 非零 + stderr;MCP 区分 Empty/Ambiguous 文案;HTTP 歧义 400、未命中 404;新增双过程 substring fixture ×4 fafeaa2
⑤ 空谓词非零退出 resolved 无分支过程 → predicates: [] exit 0;非零保留给未解析/歧义 fafeaa2
⑥ 注释叙事 复述性/issue 编号注释精简;cmd_lineage 改用共享 parse_lineage_target(regress_lineage 三套件未动全绿守护) 8a3f0c1 / fafeaa2

门禁:build/clippy/fmt 干净,39 测试二进制 0 失败(757 单测)。无既有测试断言改动;STORE_VERSION 保持 12。

Comment thread src/main.rs Outdated
&name,
crate::graph::search::MatchMode::Substring,
false,
false,

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[bug] Previous F4 asked every new query surface to pass fail_on_multiple=true so ResolveResult::Ambiguous is reachable. The follow-up flipped the procedure arm of cmd_columns, cmd_predicates, MCP codeweb_column_analysis, and HTTP GET /api/v1/columns, but the package arm of cmd_columns still calls resolve_single_node(..., false, false). With fail_on_multiple=false, multiple substring hits return Single(first) rather than Ambiguous, so the _ => "Ambiguous match" arm below it is dead and codeweb columns --package pkg_order silently succeeds when both pkg_order and pkg_order_header exist. MCP/HTTP package resolution already uses true, so the CLI is now the inconsistent surface. The new columns_ambiguous_substring_fails_explicitly test only covers --procedure.

Suggestion: Pass true on this call (same as the procedure arm), and add a --package sibling of columns_ambiguous_substring_fails_explicitly so the Ambiguous path cannot regress independently.

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

已修复(2028d0b):--packagefail_on_multiple 翻转为 true,与 procedure 臂及 MCP/HTTP 对齐;新增 columns_ambiguous_package_fails_explicitly(Red:双包 substring 静默 exit 0 → Green:非零 + stderr 歧义提示),防此路径独立回退。门禁全绿(39 测试二进制 0 失败)。

…llow-up)

package 臂 resolve_single_node 补上 fail_on_multiple=true(此前仍静默取首个匹配,Ambiguous 臂不可达);新增 columns_ambiguous_package_fails_explicitly 防回归

Ultraworked with [Sisyphus](https://github.com/code-yeongyu/oh-my-openagent)

Co-authored-by: Sisyphus <clio-agent@sisyphuslabs.ai>
@c2j
c2j merged commit b36ef62 into main Sep 8, 2026
2 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment