[BUG]: Some non-ASCII characters are incorrectly flagged as spelling errors

[surfissh]

Issue Description

When writing comments or documentation, non-ASCII characters (such as certain characters from Chinese, Japanese, or other languages) are incorrectly identified and flagged as spelling errors.

Ignoring non-ASCII characters or single-character words in the spell checker may helpful to resolve these false positives.

Operating System

Linux (Arch)

Editor

Zed

Codebook Version

No response

Configuration

Steps to Reproduce

1. Write some non-ASCII characters (e.g. "简体中文").

Expected Behavior

These characters should be ignored by spell checker.

Actual Behavior

These characters incorrectly identified and flagged as spelling errors.

Code Sample

// 简体中文
// 繁體中文
// にほんご
// ภาษาไทย
// 한국어
// Tiếng Việt

Log Output

Additional Context

No response

[blopker]

Hey! Yes, this is an issue. I was hoping to eventually add support for more languages to fix this. However, when trying to add Korean, I learned that it's not as trivial as adding more dictionaries. I think the right way to go here is to just ignore words that contain Unicode that's out of range of the currently supported languages (for now).

Happy to take a PR that does this if you want to take a look.

[aizigao]

ignore_patterns = [
    "\\b[ATCG]+\\b",             # DNA sequences
    "\\d{3}-\\d{2}-\\d{4}",      # Social Security Number format
    "^[A-Z]{2,}$",               # All caps words like "HTML", "CSS"
    "https?://[^\\s]+",          # URLs
    "\\p{Han}",                  # 汉字
    "\\p{Hiragana}",             # 平假名
  # 如果你也想一并跳过日文 / 韩文，可以加：
    "\\p{Hangul}",               # 韩文
    "\\p{Katakana}"              # 片假名
]

add this

[m15a]

For Japanese ignore patterns, the chōonpu (ー) used in some Katakana/Hiragana words (e.g. ブイヤベース) is not captured by \p{Katakana} or \p{Hiragana}.

I fixed this by using script extensions (i.e. \p{scx:Katakana}) or by adding ー to the ignore_patterns list.