[syarihu]

What

Addresses an issue in auto-keyword extraction where multiple sub-compounds of the same compound phrase (e.g. 排他 / 制御 / 排他制御, 自動 / 自動更新 / 自動更新処理, トークン / アクセストークン) were redundantly selected, unnecessarily exhausting the keyword cap.

This PR introduces longest-match priority deduplication, suppresses over-splitting of kanji compound candidates, trims uninformative suffix stop words, expands Japanese general stop words, and tightens the automatic keyword extraction cap from 15 to 8.

Changes

• src/keywords.rs:
  • Suppressed indiscriminate splitting of 4-character kanji compounds into 2-character sub-compounds, preventing redundant candidate generation.
  • Added suffix handling for 〜時 (e.g. 発生時 -> skipped because 発生 is a stop word; 失効時 -> trimmed to stem 失効).
  • Added automatic trimming of trailing 2-character stop words (e.g. 自動更新処理 -> 自動更新, 排他制御手順 -> 排他制御).
  • Expanded JP_STOP_WORDS with common procedural and relational terms (自動, 手動, 検討, 概要, 目的, 方法, 方針, 結果, 注意, 内容, 前提, 実施, 導入, 参照, 共有, 反映, 保存, 調査, 動作, 連携, 提供, 定義, 考慮, 適用, 部分, 全体, 自身, 以降, 以前, 程度, 通常, 以下, 以上, 未満, 各種, 複数, 一覧).
  • Implemented is_subword and deduplicate_substrings to remove shorter sub-compounds when a longer compound exists (longest-match priority), with safeguards against false positives on ASCII terms (e.g. avoiding matching format inside information).
  • Reduced MAX_AUTO_KEYWORDS from 15 to 8.
  • Added unit tests: test_is_subword, test_deduplicate_substrings, and test_user_example_candidate_extraction.
• src/main.rs:
  • Updated the default --threshold for lk keywords --regen from 15 to 8 to align with the new cap.

Tests

• cargo test: 237 unit / 135 integration (372 passed, 1 pre-existing ignored).
• cargo clippy --all-targets: clean.
• cargo fmt --check: clean.

Summary by CodeRabbit

• Improvements
  • Japanese keyword extraction now produces cleaner, more relevant terms by removing redundant substring matches and expanded stopwords.
  • Keyword suggestions are capped at 8 results, with fallback suggestions limited to 5.
  • Regeneration commands now use the updated keyword threshold by default.
  • Japanese text handling better trims unwanted suffixes and avoids generating unnecessary compound terms.

[coderabbitai]

Navigate logical layers of code changes, visualize relationships, and explore their blast radius.

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Advanced

Run ID: d0caaa97-bae2-4722-bc1e-34df8d153661

📥 Commits

Reviewing files that changed from the base of the PR and between d92a9ff and 738e07b.

📒 Files selected for processing (2)

• src/keywords.rs
• src/main.rs

Included review availability: Your plan provides up to 10 included reviews per hour; 9 remain after this review.

---

📝 Walkthrough

Walkthrough

Changes

Keyword processing

Layer / File(s)	Summary
Keyword rules and extraction src/keywords.rs, src/main.rs	The automatic keyword limit and CLI regeneration threshold change from 15 to 8. Japanese stopwords and Kanji candidate trimming are expanded. Public substring matching and deduplication helpers are added.
Ranked result filtering and validation src/keywords.rs	Laya-ranked terms now require a 0.50 score, remove substring duplicates, and cap results at 8. Fallback terms are deduplicated and capped at 5. Tests cover Japanese, ASCII, stopword, and candidate extraction behavior.

Priority: ⬇️ Low

Estimated code review effort: 3 (Moderate) | ~20 minutes

Change: Bug fix

Sequence Diagram(s)

sequenceDiagram
  participant Laya
  participant KeywordFilter
  participant CandidateExtractor
  Laya->>KeywordFilter: provide ranked keyword terms
  KeywordFilter->>KeywordFilter: filter scores and deduplicate substrings
  KeywordFilter->>CandidateExtractor: use fallback candidates when needed
  CandidateExtractor-->>KeywordFilter: return trimmed and deduplicated terms

Loading

Merge Risk: ⚪ Minimal · up to 738e0

Keyword generation now returns fewer, deduplicated terms with stricter Japanese filtering; no concrete correctness, data-integrity, or availability issue is identified, so the change is merge-ready with normal checks.

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Docstring Coverage	✅ Passed	Docstring coverage is 88.89% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 9 functions across 2 files.
Linked Issues check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check	✅ Passed	Check skipped because no linked issues were found for this pull request.
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The title clearly summarizes the main changes: deduplication of sub-compound keywords and stricter auto-extraction limits.

✨ Finishing Touches

📝 Generate docstrings

• Commit to this branch
• Create a new PR

🧪 Generate unit tests (beta)

• Commit to this branch
• Create a new PR

---

_{Comment @coderabbitai help to get the list of available commands.}