perf(rules): O(log N) line index + standard regex fast path + fancy_regex prefilter

[c2j]

Summary

Fixes 3 performance bottlenecks in rule execution:

1. O(log N) line index (was O(N²))

byte_index_to_line_col scanned from byte 0 for every match → O(M×N). Now pre-builds newline index once, binary search → O(log N) per match.
4MB + 10 literal rules: >30s → 404ms

2. Standard regex fast path (was always fancy_regex)

execute_regex_pattern used fancy_regex (backtracking) for ALL patterns. Now tries regex::Regex (DFA) first, falls back to fancy_regex only for backreferences/lookaround.

3. Fancy_regex prefilter for backreferences

Backreference patterns (\1\2) on large files (4MB) caused pathological fancy_regex backtracking. Added simplified pre-filter via standard regex + 1000-match cap.
4MB + 2 backref rules: >30s → 488ms

Final benchmark

4MB + 10 regex rules : 425ms
4MB +  2 backref    : 488ms
260KB + 21 rules    : 5.29s
annotation validate : 66 passed, 18 failed (18 pre-existing)

Closes #44

Summary by CodeRabbit

• Performance Improvements

  • Improved rule matching performance, especially for regular expressions and large source files.
  • Accelerated conversion of match locations into line and column positions.
  • Added optimized handling for complex regular expressions with backreferences.

• Bug Fixes

  • Improved consistency and accuracy of reported match locations.

• Release

  • Updated the application version to 0.4.14.

[coderabbitai]

Caution

Review failed

The pull request is closed.

ℹ️ Recent review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: b709954e-68c0-4191-b305-749602d97c70

📥 Commits

Reviewing files that changed from the base of the PR and between 25fcade and 731a19e.

📒 Files selected for processing (5)

• Cargo.toml
• crates/astgrep-rules/src/engine/traversal/execution.rs
• crates/astgrep-rules/src/engine/traversal/location.rs
• crates/astgrep-rules/src/engine/traversal/pattern.rs
• crates/astgrep-rules/src/engine/traversal/types.rs

---

📝 Walkthrough

Walkthrough

Rule execution now uses cached newline indexes for location conversion, optimized standard-regex matching, and prefiltered fallback matching for backreferences. Sequential literal matching also reuses precomputed line starts, and the crate version is bumped to 0.4.14.

Changes

Rule execution performance

Layer / File(s)	Summary
Cached byte-to-location conversion crates/astgrep-rules/src/engine/traversal/types.rs, crates/astgrep-rules/src/engine/traversal/location.rs, crates/astgrep-rules/src/engine/traversal/pattern.rs	RuleExecutionEngine caches line starts and performs binary-search byte-to-line/column conversion across location-producing paths.
Indexed literal finding conversion crates/astgrep-rules/src/engine/traversal/execution.rs	Sequential batched literal matching precomputes line starts and passes them into finding construction.
Regex execution fast paths crates/astgrep-rules/src/engine/traversal/pattern.rs	Regex matching prefers cached regex::Regex execution and uses a backreference-stripped prefilter before capped fancy_regex validation.
Crate release metadata Cargo.toml	The package version changes from 0.4.13 to 0.4.14.

Estimated code review effort: 4 (Complex) | ~45 minutes

Sequence Diagram(s)

sequenceDiagram
  participant RuleExecutionEngine
  participant regexRegex
  participant fancyRegex
  participant FindingBuilder
  RuleExecutionEngine->>regexRegex: compile and execute standard regex
  regexRegex-->>RuleExecutionEngine: matches or compilation failure
  RuleExecutionEngine->>fancyRegex: prefilter candidates and validate fallback matches
  fancyRegex-->>RuleExecutionEngine: validated spans
  RuleExecutionEngine->>FindingBuilder: convert offsets and construct findings

Loading

Possibly related PRs

• c2j/astgrep#34: Updates related pattern routing and conversion of match offsets into finding locations.
• c2j/astgrep#43: Modifies the related literal batching, regex caching, and traversal execution pipeline.

✨ Finishing Touches

📝 Generate docstrings

• Create stacked PR
• Commit on current branch

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch perf/regex-fast-path

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}