diff --git a/CHANGELOG.md b/CHANGELOG.md index a617680..519142b 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -54,6 +54,19 @@ below. the public API). - `EngineError` implements `Clone`; `StringGenerator` implements `Debug` and `FusedIterator`; `ExecutionProfileBuilder` implements `Debug` and `Clone`. +- `GenerationOrder`, the order `generate_strings`/`iter_strings` walk a + language in. `Exhaustive` is the previous behaviour: shortest strings + first, one path expanded in full before the next. `Sampled` covers every + shape the automaton holds before asking any of them for a second string, + and picks representative characters (`a`, `0`, `A`, ` `, ...) spread over + each range, so `.*abc.*` yields `abc`, `abc `, `aabc`, `0abc`, ... instead + of a million variations of `abc\u{0}`. It stays deterministic and pages + with `offset` the same way. +- `GenerationOptions`, what `generate_strings`/`iter_strings` may generate: + the order, plus an optional charset (`with_charset(CharRange)`) that keeps + generation to a set of characters. Only strings made entirely of them come + out — a path needing a ruled-out character is dropped whole, never + shortened — so `.*abc.*` over `[ -~]` yields `abc`, `abc `, `abc!`, ... ### Changed - `Term::to_regex`/`to_pattern` and `FastAutomaton::to_regex` are now fallible @@ -70,8 +83,10 @@ below. `&[a, b]`, `[&a, &b]`, and `Vec` all work without cloning. - `repeat` now takes `impl RangeBounds` (e.g. `3..6`, `..=2`) instead of explicit min/max parameters. -- `generate_strings` now takes `(limit, offset)` for pagination instead of a - single `count`. +- `generate_strings` now takes `(limit, offset, options)`: pagination instead + of a single `count`, plus the `GenerationOptions` to generate under (an + order, and optionally a charset). A `GenerationOrder` converts into options, + so it can be passed on its own. `iter_strings` takes the same `options`. - `is_empty`, `is_total`, and `is_empty_string` now return `Result` instead of `bool`. - `are_equivalent`/`is_subset_of` renamed to `equivalent`/`subset`. diff --git a/README.md b/README.md index 60d6480..a26235d 100644 --- a/README.md +++ b/README.md @@ -8,7 +8,7 @@ The `regex` crate tells you whether a *string* matches a pattern. **RegexSolver treats patterns as the sets of strings they match** — so you can intersect, subtract, compare, complement, and enumerate them, and get the result back as a regex. ```rust -use regexsolver::Term; +use regexsolver::{Term, fast_automaton::GenerationOrder}; let a: Term = "(ab|xy){2}".parse()?; let b: Term = ".*xy".parse()?; @@ -21,13 +21,13 @@ assert_eq!(both.to_pattern()?, "(ab|xy)xy"); assert!(both.matches("abxy")?); // ...and sample them: -assert_eq!(both.generate_strings(2, 0)?, ["xyxy", "abxy"]); +assert_eq!(both.generate_strings(2, 0, GenerationOrder::Exhaustive)?, ["xyxy", "abxy"]); ``` ## What would you use this for? - **Safe migrations** - `old_rule.subset(&new_rule)?`: does the new validation pattern accept *everything* the old one did? -- **Test-data generation** - `term.generate_strings(100, 0)?`: produce strings matching any pattern, with pagination. +- **Test-data generation** - `term.generate_strings(100, 0, GenerationOrder::Sampled)?`: produce strings matching any pattern, spread over the cases the pattern allows, restricted to the characters you can use, with pagination. - **Rule analysis**: find shadowed or overlapping routes, firewall rules, and validators with `intersection` / `difference`. - **Equivalence proofs** - `a.equivalent(&b)?`: show that two differently-written patterns match exactly the same strings. - **Pattern simplification**: every operation returns a `Term` you can turn back into a regex pattern with `to_pattern()`. @@ -90,8 +90,8 @@ RegexSolver is based on the [regex-syntax](https://docs.rs/regex-syntax/0.8.5/re | `concat(&self, terms)` / `repeat(&self, range)` | Sequence and repeat languages; `range` is any Rust range expression (`2..=5`, `1..`, `..3`, ...). | | `equivalent(&self, other)` / `subset(&self, other)` | Compare languages. | | `is_empty()` / `is_total()` / `length()` / `cardinality()` | Analyze a language: matches nothing? everything? string lengths? how many strings? | -| `generate_strings(limit, offset)` | Enumerate matching strings eagerly (call `minimize()` once first when paginating). | -| `iter_strings()` | Lazy iterator equivalent; computes the automaton once and yields strings in batches. | +| `generate_strings(limit, offset, options)` | Enumerate matching strings eagerly (call `determinize()` or `minimize()` once first when paginating). | +| `iter_strings(options)` | Lazy iterator equivalent; computes the deterministic automaton once and yields strings in batches. | | `to_pattern()` / `to_automaton()` / `to_regex()` | Convert back out. | All fallible operations return `Result<_, EngineError>`. @@ -166,7 +166,7 @@ Automaton operations can blow up on adversarial inputs, so the engine is built t ### Time-Bounded Execution ```rust -use regexsolver::{Term, execution_profile::{ExecutionProfile, ExecutionProfileBuilder}, error::EngineError}; +use regexsolver::{Term, execution_profile::{ExecutionProfile, ExecutionProfileBuilder}, error::EngineError, fast_automaton::GenerationOrder}; let term = Term::from_pattern(".*abc.*cdef.*sqdsqf.*")?; @@ -176,7 +176,7 @@ let execution_profile = ExecutionProfileBuilder::new() // We run the operation with the defined limitation execution_profile.run(|| { - assert_eq!(EngineError::OperationTimeOutError, term.generate_strings(1000, 1_000_000).unwrap_err()); + assert_eq!(EngineError::OperationTimeOutError, term.generate_strings(1000, 1_000_000, GenerationOrder::Exhaustive).unwrap_err()); }); ``` diff --git a/benches/operations.rs b/benches/operations.rs index 46ebc36..cb5e2bc 100644 --- a/benches/operations.rs +++ b/benches/operations.rs @@ -13,7 +13,7 @@ use criterion::{BatchSize, BenchmarkId, Criterion, criterion_group, criterion_main}; use regex_charclass::char::Char; -use regexsolver::fast_automaton::FastAutomaton; +use regexsolver::fast_automaton::{FastAutomaton, GenerationOptions, GenerationOrder}; use regexsolver::regex::RegularExpression; use regexsolver::{CharRange, Term}; use std::hint::black_box; @@ -215,13 +215,42 @@ fn bench_generate_strings(c: &mut Criterion) { let automaton = dfa("[a-z]{1,4}"); group.bench_function("first_2000", |b| { - b.iter(|| black_box(&automaton).generate_strings(2000, 0).unwrap()) + b.iter(|| { + black_box(&automaton) + .generate_strings(2000, 0, GenerationOrder::Exhaustive) + .unwrap() + }) }); // The offset fast-skips whole subtrees by counting paths. let deep = dfa("[a-z]{1,10}"); group.bench_function("deep_offset", |b| { - b.iter(|| black_box(&deep).generate_strings(100, 1_000_000).unwrap()) + b.iter(|| { + black_box(&deep) + .generate_strings(100, 1_000_000, GenerationOrder::Exhaustive) + .unwrap() + }) + }); + + // Sampling walks the automaton once per pass instead of settling on one + // path, so it pays for the paths it spreads over. + group.bench_function("sampled_2000", |b| { + b.iter(|| { + black_box(&automaton) + .generate_strings(2000, 0, GenerationOrder::Sampled) + .unwrap() + }) + }); + + // A charset costs one intersection per transition condition, up front. + let printable = CharRange::new_from_range(Char::new(' ')..=Char::new('~')); + let options = GenerationOptions::from(GenerationOrder::Exhaustive).with_charset(printable); + group.bench_function("charset_2000", |b| { + b.iter(|| { + black_box(&automaton) + .generate_strings(2000, 0, options.clone()) + .unwrap() + }) }); group.finish(); diff --git a/examples/generate.rs b/examples/generate.rs index 9c9fa6e..d80d09a 100644 --- a/examples/generate.rs +++ b/examples/generate.rs @@ -2,22 +2,55 @@ //! //! ```text //! cargo run --example generate -- "[a-z]{2}[0-9]" 20 +//! cargo run --example generate -- "[a-z]{2}[0-9]" 20 sampled +//! cargo run --example generate -- ".{4}" 20 sampled "[ -~]" //! ``` -use regexsolver::Term; +use regexsolver::regex::RegularExpression; +use regexsolver::{ + Term, + fast_automaton::{GenerationOptions, GenerationOrder}, +}; fn main() -> Result<(), Box> { let mut args = std::env::args().skip(1); let Some(pattern) = args.next() else { - eprintln!("Usage: cargo run --example generate -- [count]"); + eprintln!( + "Usage: cargo run --example generate -- [count] [exhaustive|sampled] [charset]" + ); std::process::exit(2); }; let count: usize = args.next().map(|c| c.parse()).transpose()?.unwrap_or(10); + // `exhaustive` sweeps the language in order, `sampled` spreads the strings + // over the shapes the pattern allows (see `GenerationOrder`). + let order = match args.next().as_deref() { + None | Some("exhaustive") => GenerationOrder::Exhaustive, + Some("sampled") => GenerationOrder::Sampled, + Some(other) => { + eprintln!("Unknown order {other:?}, expected `exhaustive` or `sampled`"); + std::process::exit(2); + } + }; + + let mut options = GenerationOptions::from(order); + + // A charset is a plain character class, e.g. "[ -~]" for printable ASCII: + // only the strings made entirely of its characters are generated. + if let Some(charset) = args.next() { + match RegularExpression::new(&charset)? { + RegularExpression::Character(charset) => options = options.with_charset(charset), + _ => { + eprintln!("The charset has to be a single character class, e.g. \"[ -~]\""); + std::process::exit(2); + } + } + } + // Minimize once: pagination over the same minimized term yields // disjoint, consistent pages (see `Term::generate_strings`). let term = Term::from_pattern(&pattern)?.minimize()?; - for string in term.generate_strings(count, 0)? { + for string in term.generate_strings(count, 0, options)? { println!("{string:?}"); } diff --git a/examples/relate.rs b/examples/relate.rs index 12eefb6..c5d601d 100644 --- a/examples/relate.rs +++ b/examples/relate.rs @@ -4,7 +4,7 @@ //! cargo run --example relate -- "(abc|de){2}" ".*xy" //! ``` -use regexsolver::Term; +use regexsolver::{Term, fast_automaton::GenerationOrder}; fn main() -> Result<(), Box> { let mut args = std::env::args().skip(1); @@ -34,7 +34,10 @@ fn main() -> Result<(), Box> { println!("a ∩ b = [] (no string matches both)"); } else { println!("a ∩ b = {}", intersection.to_pattern()?); - println!(" e.g. {:?}", intersection.generate_strings(5, 0)?); + println!( + " e.g. {:?}", + intersection.generate_strings(5, 0, GenerationOrder::Sampled)? + ); } let pattern_or_empty = |term: Term| -> Result> { diff --git a/proptest-regressions/regex/analyze/number_of_states.txt b/proptest-regressions/regex/analyze/number_of_states.txt new file mode 100644 index 0000000..205171e --- /dev/null +++ b/proptest-regressions/regex/analyze/number_of_states.txt @@ -0,0 +1,7 @@ +# Seeds for failure cases proptest has generated in the past. It is +# automatically read and these particular cases re-run before any +# novel cases are generated. +# +# It is recommended to check this file in to source control so that +# everyone who runs the test benefits from these saved cases. +cc c974b0759317710e3901bbc0b13a84ff1cd218b3abbaae98f4fe0ab7bc03505a # shrinks to regex = Alternation([Character(RangeSet([Char('a'), Char('a')])), Concat([Repetition(Character(RangeSet([Char('a'), Char('a')])), 0, None), Character(RangeSet([Char('a'), Char('a')]))])]) diff --git a/src/execution_profile.rs b/src/execution_profile.rs index f0e0593..fcb7bf0 100644 --- a/src/execution_profile.rs +++ b/src/execution_profile.rs @@ -11,7 +11,7 @@ use crate::error::EngineError; /// /// ## Limiting the number of states /// ``` -/// use regexsolver::{Term, execution_profile::{ExecutionProfile, ExecutionProfileBuilder}, error::EngineError}; +/// use regexsolver::{Term, execution_profile::{ExecutionProfile, ExecutionProfileBuilder}, error::EngineError, fast_automaton::GenerationOrder}; /// /// let term1 = Term::from_pattern(".*abcdef.*").unwrap(); /// let term2 = Term::from_pattern(".*defabc.*").unwrap(); @@ -27,7 +27,7 @@ use crate::error::EngineError; /// /// ## Limiting the execution time /// ``` -/// use regexsolver::{Term, execution_profile::{ExecutionProfile, ExecutionProfileBuilder}, error::EngineError}; +/// use regexsolver::{Term, execution_profile::{ExecutionProfile, ExecutionProfileBuilder}, error::EngineError, fast_automaton::GenerationOrder}; /// /// let term = Term::from_pattern(".*abc.*cdef.*sqdsqf.*").unwrap(); /// @@ -36,7 +36,7 @@ use crate::error::EngineError; /// .build(); /// /// execution_profile.run(|| { -/// assert_eq!(EngineError::OperationTimeOutError, term.generate_strings(1000, 1_000_000).unwrap_err()); +/// assert_eq!(EngineError::OperationTimeOutError, term.generate_strings(100_000_000, 0, GenerationOrder::Exhaustive).unwrap_err()); /// }); /// ``` /// @@ -119,6 +119,15 @@ impl ExecutionProfile { ThreadLocalParams::get_execution_profile() } + /// Whether a execution deadline is configured. When it is not, + /// [`assert_not_timed_out`](Self::assert_not_timed_out) can be + /// skipped entirely instead of being computed for a check that + /// cannot fail. + #[inline] + pub fn limits_execution_time(&self) -> bool { + self.execution_deadline.is_some() + } + /// Assert that `execution_timeout` is not exceeded. /// /// Return empty if `execution_timeout` is not exceeded. @@ -136,6 +145,16 @@ impl ExecutionProfile { } } + /// Whether a maximum number of states is configured. When it is not, the + /// state-count heuristics feeding + /// [`assert_max_number_of_states`](Self::assert_max_number_of_states) can + /// be skipped entirely instead of being computed for a check that cannot + /// fail. + #[inline] + pub fn limits_number_of_states(&self) -> bool { + self.max_number_of_states.is_some() + } + /// Assert that `max_number_of_states` is not exceeded. /// /// `max_number_of_states` is the largest number of states an automaton may @@ -372,7 +391,7 @@ impl ThreadLocalParams { #[cfg(test)] mod tests { - use crate::{Term, regex::RegularExpression}; + use crate::{Term, fast_automaton::GenerationOrder, regex::RegularExpression}; use super::*; @@ -531,7 +550,10 @@ mod tests { assert!(term.is_total().is_ok()); assert!(term.cardinality().is_ok()); assert!(term.minimize().is_ok()); - assert!(term.generate_strings(5, 0).is_ok()); + assert!( + term.generate_strings(5, 0, GenerationOrder::Exhaustive) + .is_ok() + ); // ...and the rest of the API never needed one. assert!(term.concat(std::slice::from_ref(&other)).is_ok()); @@ -577,7 +599,8 @@ mod tests { .run(|| { assert_eq!( EngineError::OperationTimeOutError, - term.generate_strings(100, 1_000_000).unwrap_err() + term.generate_strings(100_000_000, 1_000_000, GenerationOrder::Exhaustive) + .unwrap_err() ); let run_duration = Instant::now().duration_since(start_time).as_millis(); diff --git a/src/fast_automaton/generate.rs b/src/fast_automaton/generate.rs index 2d7c85a..ac42751 100644 --- a/src/fast_automaton/generate.rs +++ b/src/fast_automaton/generate.rs @@ -1,18 +1,85 @@ use crate::{EngineError, execution_profile::ExecutionProfile}; -use ahash::{AHashSet, RandomState}; +use ahash::RandomState; use indexmap::IndexSet; use super::*; use std::cmp::Ordering; use std::collections::BinaryHeap; +use std::ops::Range; + +/// Each transition condition's index into the range pool the generation +/// resolved, the charset already taken out; `None` for the conditions the +/// charset leaves nothing of. +type RangeIds<'a> = AHashMap<&'a Condition, Option>; + +/// The order in which [`FastAutomaton::generate_strings`] walks a language. +#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Hash)] +pub enum GenerationOrder { + /// Shortest strings first, each path expanded from the low end of its + /// character ranges before the next path is visited: `.*abc.*` yields + /// `abc`, `abc\u{0}`, `abc\u{1}`, ... This sweeps the language in a stable + /// order, and is the cheapest way to page through it with `offset`. + #[default] + Exhaustive, + /// Samples the language instead of sweeping it: a few strings per path + /// before moving to the next one, with representative characters (`a`, + /// `0`, `A`, ` `, ...) spread over each range rather than always its first + /// one. `.*abc.*` yields `abc`, `0abc`, `aabc`, `abc0`, ... — the shapes + /// the pattern allows, instead of a million variations of one of them, + /// which is what makes it usable to derive test cases. + /// + /// Shape comes first: the strings cover every path the automaton holds + /// before any path is asked for a second one, so a `limit` smaller than + /// the number of shapes is spent entirely on distinct shapes, and only a + /// larger one starts varying the characters within them. + /// + /// Deterministic, and pages with `offset` like [`Exhaustive`](Self::Exhaustive). + /// Each pass takes twice as many strings per path as the previous one, so + /// a finite language is still enumerated in full given a large enough + /// `limit`; those repeated passes make it slower than `Exhaustive`. + Sampled, +} + +/// The characters a sampled string reaches for first, in order of preference: +/// one per kind of character a range is usually built from, so that an early +/// sample lands on a letter, a digit or a space rather than on `\u{0}`. The +/// list is rotated by position, so neighbouring characters of a sample differ. +const SAMPLE_CHARS: [char; 10] = [ + 'a', + '0', + 'A', + ' ', + '_', + '~', + '\n', + '\u{e9}', + '\u{4e2d}', + '\u{1f600}', +]; + +/// The block of code points `char` cannot hold: [`Char`] values skip it, so +/// scalar values have to be shifted down past it to be counted. +const SURROGATES: Range = 0xD800..0xE000; + +/// The most strings to reserve room for up front. `limit` is caller-controlled +/// and huge values (up to `usize::MAX`) are legitimate ways to ask for +/// everything, so it cannot size the allocation on its own; past this hint the +/// set grows as it fills. +const STRINGS_CAPACITY_LIMIT: usize = 1 << 12; + +/// How much a [`PathCache`] may hold — a finite language can still have far +/// more paths than fit in memory. Past these, recording gives up and the later +/// sampled passes search the automaton again: time spent instead of memory. +const CACHE_IDS_LIMIT: usize = 1 << 20; +const CACHE_PATHS_LIMIT: usize = 1 << 17; #[derive(Clone, Eq, PartialEq)] struct QueueItem { score: usize, depth: usize, state: usize, - ranges: Vec, - hash: u64, + /// The path's transitions as indices into [`Generation::range_pool`] + ranges: Vec, } impl Ord for QueueItem { @@ -22,7 +89,7 @@ impl Ord for QueueItem { .cmp(&self.score) .then_with(|| self.depth.cmp(&other.depth)) .then_with(|| self.state.cmp(&other.state)) - .then_with(|| self.hash.cmp(&other.hash)) + .then_with(|| self.ranges.cmp(&other.ranges)) } } @@ -32,182 +99,516 @@ impl PartialOrd for QueueItem { } } +/// What [`FastAutomaton::generate_strings`] is allowed to generate: the order +/// to walk the language in, and the characters it may use. +/// +/// [`GenerationOrder`] converts into it, so an order can be passed on its own +/// wherever options are expected. +/// +/// # Examples +/// +/// ``` +/// use regexsolver::{CharRange, Term, fast_automaton::{GenerationOptions, GenerationOrder}}; +/// use regexsolver::regex_charclass::char::Char; +/// +/// let term = Term::from_pattern(".{2}").unwrap(); +/// +/// // An order on its own. +/// let strings = term.generate_strings(3, 0, GenerationOrder::Sampled).unwrap(); +/// +/// // Sampled, and restricted to lowercase letters. +/// let lowercase = CharRange::new_from_range(Char::new('a')..=Char::new('z')); +/// let options = GenerationOptions::from(GenerationOrder::Sampled).with_charset(lowercase); +/// +/// let strings = term.generate_strings(3, 0, options).unwrap(); +/// assert!(strings.iter().all(|s| s.chars().all(|c| c.is_ascii_lowercase()))); +/// ``` +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub struct GenerationOptions { + order: GenerationOrder, + charset: Option, +} + +impl GenerationOptions { + /// Default options: [`GenerationOrder::Exhaustive`], over every character + /// the automaton allows. + pub fn new() -> Self { + Self::default() + } + + /// Returns a copy of these options walking the language in `order`. + pub fn with_order(mut self, order: GenerationOrder) -> Self { + self.order = order; + self + } + + /// Returns a copy of these options restricted to `charset`. + /// + /// Only strings made entirely of those characters are generated: a path + /// through a transition that `charset` rules out is dropped whole, never + /// shortened. Restricting to characters the automaton never matches + /// generates nothing. + /// + /// A [`CharRange`] is built from bounds, or out of a character class + /// pattern through [`RegularExpression`](crate::regex::RegularExpression): + /// + /// ``` + /// use regexsolver::{CharRange, regex::RegularExpression}; + /// use regexsolver::regex_charclass::char::Char; + /// + /// let printable = CharRange::new_from_range(Char::new(' ')..=Char::new('~')); + /// + /// let no_controls = match RegularExpression::new("\\P{C}").unwrap() { + /// RegularExpression::Character(charset) => charset, + /// other => panic!("not a character class: {other}"), + /// }; + /// ``` + pub fn with_charset(mut self, charset: CharRange) -> Self { + self.charset = Some(charset); + self + } + + /// The order the language is walked in. + pub fn order(&self) -> GenerationOrder { + self.order + } + + /// The characters generation is restricted to, `None` when it is not. + pub fn charset(&self) -> Option<&CharRange> { + self.charset.as_ref() + } +} + +impl From for GenerationOptions { + fn from(order: GenerationOrder) -> Self { + GenerationOptions { + order, + charset: None, + } + } +} + impl FastAutomaton { - /// Generates up to `limit` distinct strings matched by the automaton, skipping the first `offset` strings. + /// Generates up to `limit` distinct strings matched by the automaton under + /// the given [`GenerationOptions`], skipping the first `offset` strings. + /// + /// `options` is a [`GenerationOrder`] on its own, or a full + /// [`GenerationOptions`] to also restrict the characters used. /// /// Strings are only guaranteed to be distinct **within a single call**: /// the offset fast-skips by counting paths, and in a non-deterministic /// automaton the same string can be reached through several paths, so /// calls with different offsets may repeat strings (or skip some). /// [`determinize`](Self::determinize) (and ideally - /// [`minimize`](Self::minimize)) first to make pages disjoint. - #[tracing::instrument(level = "debug", skip(self), fields(states = self.number_of_states(), deterministic=self.is_deterministic(), limit=limit, offset=offset))] + /// [`minimize`](Self::minimize)) first to make pages disjoint. Offsets are + /// also only consistent between calls made with the same options. + #[tracing::instrument(level = "debug", skip(self, options), fields(states = self.number_of_states(), deterministic=self.is_deterministic(), limit=limit, offset=offset, order=tracing::field::Empty, charset=tracing::field::Empty))] pub fn generate_strings( &self, limit: usize, - mut offset: usize, + offset: usize, + options: impl Into, + ) -> Result, EngineError> { + let options = options.into(); + + // Serializing the charset is not free: only when the span is recorded. + let span = tracing::Span::current(); + if !span.is_disabled() { + span.record("order", tracing::field::debug(options.order)); + span.record( + "charset", + tracing::field::debug(options.charset.as_ref().map(|charset| charset.to_regex())), + ); + } + + self.generate(limit, offset, &options) + } + + /// [`generate_strings`](Self::generate_strings) over borrowed options, for + /// the callers holding them across several batches. + pub(crate) fn generate( + &self, + limit: usize, + offset: usize, + options: &GenerationOptions, ) -> Result, EngineError> { if self.is_empty() || limit == 0 { return Ok(vec![]); } - let (_, max) = self.length(); - let max_len = max.unwrap_or(u32::MAX) as usize; + let mut generation = Generation::new(self, limit, offset, options.charset())?; - let execution_profile = ExecutionProfile::get(); - let num_states = self.transitions.len(); + match options.order { + GenerationOrder::Exhaustive => { + generation.walk(self, None, None)?; + } + GenerationOrder::Sampled => { + // A pass takes at most `window` combinations per path, so no + // single path can spend the whole `limit` on itself. The + // windows double and pick up where the previous one stopped: + // a pass that exhausts the automaton without filling `limit` + // is followed by one digging deeper into the same paths, until + // a pass finds nothing left to cover. Exhausting the automaton + // also proves the paths finite and leaves them in `cache`, so + // the passes after it replay them instead of searching again. + let mut window = 0..1; + let mut cache = PathCache::new(); + loop { + let covered = if cache.complete { + generation.replay(&cache, &window)? + } else { + generation.walk(self, Some(&window), Some(&mut cache))? + }; + if covered == 0 || generation.emitter.is_full() { + break; + } + window = window.end..window.end.saturating_mul(2).saturating_add(1); + } + } + } + + Ok(generation.emitter.strings.into_iter().collect()) + } +} - // ----------------------------------------------------------------- - // 1. REVERSE BFS: Precalculate exact distances to Accept State - // ----------------------------------------------------------------- - let mut incoming = vec![vec![]; num_states]; - let mut dist_q = std::collections::VecDeque::new(); - let mut dist = vec![usize::MAX; num_states]; +/// The state of a single [`FastAutomaton::generate_strings`] call, shared by +/// every pass a [`GenerationOrder::Sampled`] generation makes over the +/// automaton. +struct Generation<'a> { + /// Number of transitions from each state to the nearest accept state; + /// `usize::MAX` for the states that cannot reach one. + distances: Vec, + /// Length of the longest string the automaton matches. + max_len: usize, + /// The characters each transition stands for, resolved once: the paths + /// refer to them by index (see [`QueueItem::ranges`]). + range_pool: Vec, + /// Each transition condition's index into + /// [`range_pool`](Self::range_pool), the charset already taken out. A + /// condition the charset leaves nothing of holds `None`, which is what + /// makes its transition impassable. + range_ids: RangeIds<'a>, + emitter: Emitter, +} - for state in self.states() { - if self.is_accepted(state as _) { - dist[state] = 0; - dist_q.push_back(state); +/// Turns the paths the search pops into strings: the half of a [`Generation`] +/// the emission mutates, kept apart from the search data so that a borrow of +/// the range pool can live alongside it. +struct Emitter { + limit: usize, + offset: usize, + strings: IndexSet, + execution_profile: ExecutionProfile, +} + +/// The accepting paths a sampled pass popped, in pop order — flat, path `i` +/// being `ids[starts[i]..starts[i + 1]]`. A pass that runs out of paths has +/// recorded all of them, and the passes after it replay the cache instead of +/// searching the automaton again. +struct PathCache { + ids: Vec, + starts: Vec, + /// The cache holds every path of the automaton and can stand in for it. + complete: bool, + /// Recording outgrew [`CACHE_IDS_LIMIT`]/[`CACHE_PATHS_LIMIT`] and gave + /// up; the cache stays empty and every pass searches. + overflowed: bool, +} + +impl PathCache { + fn new() -> Self { + PathCache { + ids: vec![], + starts: vec![0], + complete: false, + overflowed: false, + } + } + + fn record(&mut self, path: &[u32]) { + if self.overflowed { + return; + } + if self.ids.len().saturating_add(path.len()) > CACHE_IDS_LIMIT + || self.starts.len() > CACHE_PATHS_LIMIT + { + self.overflowed = true; + self.ids = vec![]; + self.starts = vec![0]; + return; + } + + self.ids.extend_from_slice(path); + self.starts.push(self.ids.len()); + } + + fn paths(&self) -> impl Iterator { + self.starts + .windows(2) + .map(|window| &self.ids[window[0]..window[1]]) + } +} + +/// What every transition condition leaves once the charset is taken out. +/// Resolved up front rather than as the walk reaches them, so that the search +/// already knows which transitions are impassable: a pool of the non-empty +/// ranges, and each condition's index into it — `None` for the conditions the +/// charset leaves nothing of. +fn resolve_ranges<'a>( + automaton: &'a FastAutomaton, + charset: Option<&CharRange>, +) -> Result<(Vec, RangeIds<'a>), EngineError> { + let mut range_pool: Vec = Vec::new(); + let mut range_ids: RangeIds = AHashMap::with_capacity(automaton.transitions.len()); + + for state in automaton.states() { + for (cond, _) in automaton.transitions_from(state) { + if range_ids.contains_key(cond) { + continue; } - for (_cond, &to_state) in self.transitions_from(state as _) { + let range = cond.to_range(&automaton.spanning_set)?; + let range = match charset { + Some(charset) => range.intersection(charset), + None => range, + }; + let id = if range.is_empty() { + None + } else { + range_pool.push(range); + Some((range_pool.len() - 1) as u32) + }; + range_ids.insert(cond, id); + } + } + + Ok((range_pool, range_ids)) +} + +/// REVERSE BFS: the exact distance from every state to an accept state, which +/// drives the A* search and prunes the states that never accept; `usize::MAX` +/// for the states that cannot reach one. A state the charset leaves no way out +/// of (no id in `range_ids`) is one of those dead ends. +fn distances_to_accept(automaton: &FastAutomaton, range_ids: &RangeIds) -> Vec { + let num_states = automaton.transitions.len(); + let mut incoming = vec![vec![]; num_states]; + let mut dist_q = VecDeque::new(); + let mut distances = vec![usize::MAX; num_states]; + + for state in automaton.states() { + if automaton.is_accepted(state) { + distances[state] = 0; + dist_q.push_back(state); + } + for (cond, &to_state) in automaton.transitions_from(state) { + if range_ids[cond].is_some() { incoming[to_state].push(state); } } + } - while let Some(state) = dist_q.pop_front() { - let d = dist[state]; - for &prev in &incoming[state] { - if dist[prev] == usize::MAX { - dist[prev] = d + 1; - dist_q.push_back(prev); - } + while let Some(state) = dist_q.pop_front() { + let d = distances[state]; + for &prev in &incoming[state] { + if distances[prev] == usize::MAX { + distances[prev] = d + 1; + dist_q.push_back(prev); } } + } - // ----------------------------------------------------------------- - // 2. A* SEARCH: Find matching strings instantly - // ----------------------------------------------------------------- - let mut ranges_cache: AHashMap<&Condition, CharRange> = AHashMap::with_capacity(num_states); - let mut strings = IndexSet::with_capacity_and_hasher(limit, RandomState::default()); - let mut visited = AHashSet::with_capacity(num_states); + distances +} - let mut q = BinaryHeap::new(); - let start_state = self.start_state(); +/// The ranges of a path's transitions, looked up from the pool. +fn resolve<'p>(pool: &'p [CharRange], path: &[u32]) -> Vec<&'p CharRange> { + path.iter().map(|&id| &pool[id as usize]).collect() +} + +impl<'a> Generation<'a> { + fn new( + automaton: &'a FastAutomaton, + limit: usize, + offset: usize, + charset: Option<&CharRange>, + ) -> Result { + let (range_pool, range_ids) = resolve_ranges(automaton, charset)?; + let distances = distances_to_accept(automaton, &range_ids); + let (_, max) = automaton.length(); + + Ok(Generation { + distances, + max_len: max.unwrap_or(u32::MAX) as usize, + range_pool, + range_ids, + emitter: Emitter { + limit, + offset, + strings: IndexSet::with_capacity_and_hasher( + limit.min(STRINGS_CAPACITY_LIMIT), + RandomState::default(), + ), + execution_profile: ExecutionProfile::get(), + }, + }) + } + + /// A* SEARCH: walks the automaton once, shortest path first, emitting the + /// strings of every accepting path it pops until `limit` strings are + /// collected or the automaton runs out of paths. + /// + /// `window` restricts each path to the combinations whose index falls + /// inside it; `None` takes them all. `cache`, when given, records the + /// accepting paths in pop order, and running out of paths marks it + /// complete: [`replay`](Self::replay) then stands in for the next passes. + /// Returns how many combinations the pass covered, the ones `offset` + /// skipped included. + fn walk( + &mut self, + automaton: &'a FastAutomaton, + window: Option<&Range>, + mut cache: Option<&mut PathCache>, + ) -> Result { + let start_state = automaton.start_state(); // If the start state can't reach an accept state, exit immediately - if dist[start_state] != usize::MAX { - q.push(QueueItem { - score: dist[start_state], - depth: 0, - state: start_state, - ranges: vec![], - hash: 0u64, - }); + if self.distances[start_state] == usize::MAX { + return Ok(0); } + let mut covered = 0usize; + + let mut q = BinaryHeap::new(); + q.push(QueueItem { + score: self.distances[start_state], + depth: 0, + state: start_state, + ranges: vec![], + }); + while let Some(QueueItem { score: _, depth: current_depth, state, - mut ranges, - hash: h, + ranges, }) = q.pop() { - execution_profile.assert_not_timed_out()?; + self.emitter.execution_profile.assert_not_timed_out()?; - if self.is_accepted(state) { - if current_depth == 0 { - if offset > 0 { - offset -= 1; - } else { - strings.insert(String::new()); - } - } else { - Self::ranges_to_strings( - &mut strings, - &ranges, - limit, - &mut offset, - &execution_profile, - )?; + if automaton.is_accepted(state) { + if let Some(cache) = cache.as_deref_mut() { + cache.record(&ranges); } - if strings.len() >= limit { + let resolved = resolve(&self.range_pool, &ranges); + covered = covered.saturating_add(match window { + Some(window) => self.emitter.emit_sampled(&resolved, window)?, + None => self.emitter.emit_all(&resolved)?, + }); + + if self.emitter.is_full() { break; } } - if current_depth >= max_len { + if current_depth >= self.max_len { continue; } - let next_depth = current_depth + 1; - let mut valid_transitions = Vec::new(); - - for (cond, &to_state) in self.transitions_from(state) { - let to_state_usize = to_state; + self.expand(automaton, &mut q, current_depth + 1, state, ranges); + } - // DEAD-END PRUNING: Instantly kill paths that cannot accept - if dist[to_state_usize] == usize::MAX { - continue; - } + // An empty queue means every path was popped, so a recording cache + // now holds them all. + if q.is_empty() + && let Some(cache) = cache + && !cache.overflowed + { + cache.complete = true; + } - let hash = - Self::path_mix(h, Self::mix64(state as u64 ^ Self::mix64(to_state as u64))); - - if visited.insert((to_state, next_depth, hash)) { - let range = match ranges_cache.get(cond) { - Some(range) => range.clone(), - None => { - let range = cond.to_range(&self.spanning_set)?; - ranges_cache.insert(cond, range.clone()); - range - } - }; + Ok(covered) + } - valid_transitions.push((to_state_usize, range, hash)); - } + /// Queues every passable one-transition extension of a popped path, the + /// last one taking over the path's own vector instead of cloning it. + fn expand( + &self, + automaton: &'a FastAutomaton, + q: &mut BinaryHeap, + next_depth: usize, + state: State, + mut ranges: Vec, + ) { + let mut valid_transitions = Vec::new(); + + for (cond, &to_state) in automaton.transitions_from(state) { + // DEAD-END PRUNING: Instantly kill paths that cannot accept + if self.distances[to_state] == usize::MAX { + continue; } - // Vector Reuse Optimization - if let Some((last_state, last_range, last_hash)) = valid_transitions.pop() { - for (to_state, range, hash) in valid_transitions { - let mut new_ranges = ranges.clone(); - new_ranges.push(range); - q.push(QueueItem { - score: next_depth + dist[to_state], // A* Score Formula - depth: next_depth, - state: to_state, - ranges: new_ranges, - hash, - }); - } + // ...and the transitions the charset closed off. + let Some(range_id) = self.range_ids[cond] else { + continue; + }; - ranges.push(last_range); + valid_transitions.push((to_state, range_id)); + } + + // Vector Reuse Optimization + if let Some((last_state, last_id)) = valid_transitions.pop() { + for (to_state, range_id) in valid_transitions { + let mut new_ranges = ranges.clone(); + new_ranges.push(range_id); q.push(QueueItem { - score: next_depth + dist[last_state], // A* Score Formula + score: next_depth + self.distances[to_state], // A* Score Formula depth: next_depth, - state: last_state, - ranges, - hash: last_hash, + state: to_state, + ranges: new_ranges, }); } - } - Ok(strings.into_iter().collect()) + ranges.push(last_id); + q.push(QueueItem { + score: next_depth + self.distances[last_state], // A* Score Formula + depth: next_depth, + state: last_state, + ranges, + }); + } } - fn ranges_to_strings( - strings: &mut IndexSet, - ranges: &Vec, - count: usize, - offset: &mut usize, - execution_profile: &ExecutionProfile, - ) -> Result<(), EngineError> { - if strings.len() >= count { - return Ok(()); + /// Emits `window` from every path of a complete [`PathCache`], in the + /// order the search popped them: what a [`walk`](Self::walk) pass would + /// do, minus the search. + fn replay(&mut self, cache: &PathCache, window: &Range) -> Result { + let mut covered = 0usize; + + for path in cache.paths() { + self.emitter.execution_profile.assert_not_timed_out()?; + + let resolved = resolve(&self.range_pool, path); + covered = covered.saturating_add(self.emitter.emit_sampled(&resolved, window)?); + + if self.emitter.is_full() { + break; + } } + Ok(covered) + } +} + +impl Emitter { + #[inline] + fn is_full(&self) -> bool { + self.strings.len() >= self.limit + } + + /// Emits every combination of `ranges` that `offset` does not skip, in + /// ascending character order. Returns the number of combinations the path + /// holds. + fn emit_all(&mut self, ranges: &[&CharRange]) -> Result { let range_lengths: Vec = ranges .iter() .map(|r| r.get_cardinality() as usize) @@ -218,115 +619,384 @@ impl FastAutomaton { total_combinations = total_combinations.saturating_mul(len); } - if *offset >= total_combinations { - *offset -= total_combinations; - return Ok(()); + if self.offset >= total_combinations { + self.offset -= total_combinations; + return Ok(total_combinations); } - let mut current_str = String::with_capacity(ranges.len()); - Self::generate_combinations( - ranges, - &range_lengths, - 0, - &mut current_str, - strings, - count, - offset, - execution_profile, - ) - } - - #[allow(clippy::too_many_arguments)] - fn generate_combinations( - ranges: &Vec, + self.emit_combinations(ranges, &range_lengths)?; + Ok(total_combinations) + } + + /// Walks the combinations depth-first over an explicit stack of range + /// cursors, one per position: recursing per character would overflow the + /// stack on the paths thousands of transitions long. + fn emit_combinations( + &mut self, + ranges: &[&CharRange], range_lengths: &[usize], - depth: usize, - current_str: &mut String, - strings: &mut IndexSet, - count: usize, - offset: &mut usize, - execution_profile: &ExecutionProfile, ) -> Result<(), EngineError> { - if strings.len() >= count { + if ranges.is_empty() { + // A single-combination path: `emit_all` either skipped it whole or + // arrived here with nothing left of the offset. + debug_assert_eq!(0, self.offset); + self.strings.insert(String::new()); return Ok(()); } - if depth == ranges.len() { - if *offset > 0 { - *offset -= 1; + // Combinations under a single character at each position: the product + // of the range lengths past it. + let mut sub_combinations = vec![1usize; ranges.len()]; + for position in (0..ranges.len() - 1).rev() { + sub_combinations[position] = + sub_combinations[position + 1].saturating_mul(range_lengths[position + 1]); + } + + let mut current_str = String::with_capacity(ranges.len()); + let mut cursors = Vec::with_capacity(ranges.len()); + cursors.push(self.descend(ranges[0], sub_combinations[0])); + + while let Some(cursor) = cursors.last_mut() { + let next = cursor.next(); + let position = cursors.len() - 1; + + let Some(ch) = next else { + // The range is exhausted: back up to the previous position and + // move it to its next character. + cursors.pop(); + current_str.pop(); + continue; + }; + + self.execution_profile.assert_not_timed_out()?; + + current_str.push(ch.to_char()); + if position + 1 == ranges.len() { + // A full combination; whatever `offset` had left to skip was + // consumed by the descents, so this string is on the page. + self.strings.insert(current_str.clone()); + current_str.pop(); + + if self.is_full() { + break; + } } else { - strings.insert(current_str.clone()); + cursors.push(self.descend(ranges[position + 1], sub_combinations[position + 1])); } - return Ok(()); } - // Calculate combinations for the remaining suffix of ranges - let mut sub_combinations = 1usize; - for &len in &range_lengths[depth + 1..] { - sub_combinations = sub_combinations.saturating_mul(len); + Ok(()) + } + + /// A cursor over `range`, opened on the first combination `offset` does + /// not skip: the subtrees of `sub_combinations` strings each before it are + /// stepped over in one division, not walked character by character. + fn descend<'r>(&mut self, range: &'r CharRange, sub_combinations: usize) -> RangeCursor<'r> { + // Past the first emitted string the offset is zero and every cursor + // starts at its range's first character. `skip` stays within the + // range: the offset was left smaller than the previous position's + // subtree, which this whole range spans. (A saturated subtree count + // under-skips into the first character, never past the range.) + let skip = self.offset / sub_combinations; + self.offset -= skip * sub_combinations; + RangeCursor::new(range, skip as u32) + } + + /// Emits the combinations of `ranges` whose index falls inside `window`, + /// spread over the ranges by the sampling permutation. Returns how many of + /// them the window covered, the ones `offset` skipped included. + fn emit_sampled( + &mut self, + ranges: &[&CharRange], + window: &Range, + ) -> Result { + let range_lengths: Vec = ranges.iter().map(|r| r.get_cardinality() as u128).collect(); + + // `None` once the product stops fitting: such a path holds more + // combinations than a window will ever reach into. + let total_combinations = range_lengths + .iter() + .try_fold(1u128, |total, &len| total.checked_mul(len)); + let bound = + total_combinations.map_or(usize::MAX, |total| total.min(usize::MAX as u128) as usize); + + let covered = window.end.min(bound) - window.start.min(bound); + if self.offset >= covered { + self.offset -= covered; + return Ok(covered); } - for ch in ranges[depth].clone().iter() { - execution_profile.assert_not_timed_out()?; + // The permutation needs `index * multiplier` to stay within `u128`; + // beyond that the mixed-radix digits alone give enough variety, since + // consecutive indices already differ from their first character on. + let scramble = total_combinations + .filter(|&total| total <= u64::MAX as u128) + .map(|total| (scramble_multiplier(total), total)); - // If skipping this character's subtree fits within the remaining offset - if *offset >= sub_combinations { - *offset -= sub_combinations; - continue; + let first = window.start.min(bound) + self.offset; + self.offset = 0; + + for index in first..window.end.min(bound) { + self.execution_profile.assert_not_timed_out()?; + + let combination = match scramble { + Some((multiplier, total)) => (index as u128 * multiplier) % total, + None => index as u128, + }; + + let string = sample_string(ranges, &range_lengths, combination)?; + self.strings.insert(string); + + if self.is_full() { + break; } + } - current_str.push(ch.to_char()); - Self::generate_combinations( - ranges, - range_lengths, - depth + 1, - current_str, - strings, - count, - offset, - execution_profile, - )?; - current_str.pop(); + Ok(covered) + } +} + +/// Builds the combination of `ranges` at index `combination`, read as a +/// mixed-radix number whose least significant digit is the first character: +/// consecutive combinations then differ from their first character on, instead +/// of only in their last one. +fn sample_string( + ranges: &[&CharRange], + range_lengths: &[u128], + mut combination: u128, +) -> Result { + let mut string = String::with_capacity(ranges.len()); + + for (position, (&range, &length)) in ranges.iter().zip(range_lengths).enumerate() { + let index = (combination % length) as u32; + combination /= length; + string.push(sample_char(range, position, index)?.to_char()); + } + + Ok(string) +} + +/// Returns the character `range` holds at `index` in sampling order: the +/// [`SAMPLE_CHARS`] it contains first, rotated by `position` so that adjacent +/// characters of a sample differ, then the rest of the range in order. +/// +/// The mapping is a permutation of the range, which is what keeps the sampled +/// strings distinct and `offset` exact. +fn sample_char(range: &CharRange, position: usize, index: u32) -> Result { + let mut sampled = [0u32; SAMPLE_CHARS.len()]; + let mut count = 0; + + for rotation in 0..SAMPLE_CHARS.len() { + let ch = Char::new(SAMPLE_CHARS[(position + rotation) % SAMPLE_CHARS.len()]); + if !range.contains(ch) { + continue; + } + if count as u32 == index { + return Ok(ch); + } + sampled[count] = ordinal_of(range, ch); + count += 1; + } + + // Past the sample characters: take the `index - count`-th character of the + // range that is not one of them, so none is handed out twice. + let sampled = &mut sampled[..count]; + sampled.sort_unstable(); + let mut ordinal = index - count as u32; + for &taken in sampled.iter() { + if taken > ordinal { + break; + } + ordinal += 1; + } - if strings.len() >= count { + char_at(range, ordinal).ok_or(EngineError::InvalidCharacterInRegex) +} + +/// A cursor over the characters a [`CharRange`] holds, in order, opened at an +/// arbitrary ordinal: what the range's own iterator cannot do, and what lets +/// [`Emitter::descend`] skip an offset in one step. +struct RangeCursor<'a> { + /// The `(low, high)` bound pairs the range is made of. + bounds: &'a [Char], + /// Index of the current pair's low bound; past `bounds` once exhausted. + pair: usize, + /// [`scalar`] of the next character to hand out. + next_scalar: u32, +} + +impl<'a> RangeCursor<'a> { + /// A cursor whose first character is `range`'s `ordinal`-th; exhausted + /// from the start when `ordinal` is past the range's cardinality. + fn new(range: &'a CharRange, mut ordinal: u32) -> Self { + let bounds = range.0.as_slice(); + let mut pair = 0; + let mut next_scalar = 0; + + while pair < bounds.len() { + let (low, high) = (scalar(bounds[pair]), scalar(bounds[pair + 1])); + let length = high - low + 1; + if ordinal < length { + next_scalar = low + ordinal; break; } + ordinal -= length; + pair += 2; } - Ok(()) + RangeCursor { + bounds, + pair, + next_scalar, + } } - #[inline] - fn mix64(mut x: u64) -> u64 { - // splitmix64 - x = x.wrapping_add(0x9E3779B97F4A7C15); - let mut z = x; - z = (z ^ (z >> 30)).wrapping_mul(0xBF58476D1CE4E5B9); - z = (z ^ (z >> 27)).wrapping_mul(0x94D049BB133111EB); - z ^ (z >> 31) + fn next(&mut self) -> Option { + if self.pair >= self.bounds.len() { + return None; + } + + let ch = from_scalar(self.next_scalar); + if self.next_scalar == scalar(self.bounds[self.pair + 1]) { + // Past the current interval: on to the next one. + self.pair += 2; + if self.pair < self.bounds.len() { + self.next_scalar = scalar(self.bounds[self.pair]); + } + } else { + self.next_scalar += 1; + } + + ch } +} - #[inline] - fn path_mix(h: u64, x: u64) -> u64 { - h.wrapping_mul(0x9E3779B97F4A7C15).rotate_left(7) ^ x +/// The `(low, high)` scalar bounds of the intervals `range` is made of. +fn intervals(range: &CharRange) -> impl Iterator + '_ { + range + .0 + .chunks_exact(2) + .map(|bounds| (scalar(bounds[0]), scalar(bounds[1]))) +} + +/// The number of characters `range` holds before `target`, which it contains. +fn ordinal_of(range: &CharRange, target: Char) -> u32 { + // An absent target would underflow `target - low` below, silently in + // release builds. + debug_assert!(range.contains(target), "{target} is not in {range}"); + + let target = scalar(target); + let mut ordinal = 0; + + for (low, high) in intervals(range) { + if target <= high { + return ordinal + (target - low); + } + ordinal += high - low + 1; + } + + ordinal +} + +/// The character `range` holds at `ordinal`, `None` past its cardinality. +fn char_at(range: &CharRange, ordinal: u32) -> Option { + RangeCursor::new(range, ordinal).next() +} + +/// The index of `ch` among all the characters, the surrogate block excluded. +#[inline] +fn scalar(ch: Char) -> u32 { + let code = ch.to_u32(); + if code >= SURROGATES.end { + code - (SURROGATES.end - SURROGATES.start) + } else { + code } } +/// The inverse of [`scalar`]. +#[inline] +fn from_scalar(index: u32) -> Option { + Char::from_u32(if index >= SURROGATES.start { + index + (SURROGATES.end - SURROGATES.start) + } else { + index + }) +} + +/// A stride coprime with `total`, close to its golden-ratio fraction, so that +/// consecutive sample indices land far apart in the combination space instead +/// of walking it in order. Being coprime keeps `index * stride % total` a +/// permutation, so no combination comes up twice. +fn scramble_multiplier(total: u128) -> u128 { + if total < 3 { + return 1; + } + + let mut multiplier = ((total as f64) * 0.618_033_988_749_895) as u128; + multiplier = multiplier.clamp(1, total - 1); + + for _ in 0..64 { + if gcd(multiplier, total) == 1 { + return multiplier; + } + multiplier += 1; + if multiplier >= total { + multiplier = 1; + } + } + + 1 +} + +fn gcd(mut a: u128, mut b: u128) -> u128 { + while b != 0 { + (a, b) = (b, a % b); + } + a +} + #[cfg(test)] mod tests { - use crate::{cardinality::Cardinality, regex::RegularExpression}; + use super::{GenerationOptions, GenerationOrder, RangeCursor, char_at, sample_char}; + use crate::CharRange; + use crate::cardinality::Cardinality; + use crate::{fast_automaton::FastAutomaton, regex::RegularExpression}; use regex::Regex; + use regex_charclass::{CharacterClass, char::Char, irange::range::AnyRange}; + + const ORDERS: [GenerationOrder; 2] = [GenerationOrder::Exhaustive, GenerationOrder::Sampled]; + + /// Every set of options the generation tests run through: both orders, + /// each of them once unrestricted and once over printable ASCII, which is + /// narrow enough to close off transitions in most of the patterns. + fn all_options() -> Vec { + ORDERS + .into_iter() + .flat_map(|order| { + [ + GenerationOptions::from(order), + GenerationOptions::from(order).with_charset(printable_ascii()), + ] + }) + .collect() + } + + fn printable_ascii() -> CharRange { + CharRange::new_from_range_char(' '..='~') + } #[test] fn test_generate_strings_1() -> Result<(), String> { - let automaton = - RegularExpression::parse(".*ab.*c(de|fg).*dab.*c(de|fg).*ab.*c(de|fg).*dab.*c", true) - .unwrap() - .to_automaton() - .unwrap(); + let automaton = RegularExpression::parse("((aad|..e.*|e.z)*|q)", false) + .unwrap() + .to_automaton() + .unwrap(); let automaton = automaton.determinize().unwrap(); - automaton.generate_strings(30, 0).unwrap(); + for order in ORDERS { + println!("{:?}", automaton.generate_strings(30, 0, order).unwrap()); + } Ok(()) } @@ -339,11 +1009,13 @@ mod tests { .unwrap(); let automaton = automaton.determinize().unwrap(); - let strings = automaton.generate_strings(2, 0).unwrap(); - assert_eq!(2, strings.len()); + for order in ORDERS { + let strings = automaton.generate_strings(2, 0, order).unwrap(); + assert_eq!(2, strings.len()); - let strings = automaton.generate_strings(2, 2).unwrap(); - assert_eq!(2, strings.len()); + let strings = automaton.generate_strings(2, 2, order).unwrap(); + assert_eq!(2, strings.len()); + } Ok(()) } @@ -405,74 +1077,337 @@ mod tests { Ok(()) } - fn assert_generate_strings_offset(regex: &str) { - println!("regex: {regex}"); - let automaton = RegularExpression::parse(regex, false) - .unwrap() - .to_automaton() + /// The sampled order exists so that a pattern's *shapes* get covered: what + /// the exhaustive order spends a million strings on (one path, every + /// character of its last range) has to fit in a handful of them. + #[test] + fn test_generate_strings_sampled_covers_the_whole_pattern() { + let automaton = automaton_of(".*abc.*").determinize().unwrap().into_owned(); + + let exhaustive = automaton + .generate_strings(20, 0, GenerationOrder::Exhaustive) + .unwrap(); + assert!( + exhaustive.iter().all(|s| s.starts_with("abc")), + "the exhaustive order stays on the first path it finds: {exhaustive:?}" + ); + + let sampled = automaton + .generate_strings(20, 0, GenerationOrder::Sampled) + .unwrap(); + assert!( + sampled.iter().any(|s| !s.starts_with("abc")), + "the sampled order has to reach the strings with a prefix before `abc`: {sampled:?}" + ); + assert!( + sampled.iter().any(|s| !s.ends_with("abc")), + "the sampled order has to reach the strings with a suffix after `abc`: {sampled:?}" + ); + assert!( + sampled.iter().any(|s| s.len() > 5), + "the sampled order has to reach longer strings too: {sampled:?}" + ); + } + + /// Sampling still enumerates a finite language in full, given the room: + /// the passes dig deeper into every path until nothing is left to cover. + #[test] + fn test_generate_strings_sampled_is_exhaustive_in_the_limit() { + let automaton = automaton_of("[a-z][0-9]"); + + let mut sampled = automaton + .generate_strings(1000, 0, GenerationOrder::Sampled) + .unwrap(); + sampled.sort(); + + let mut expected: Vec = ('a'..='z') + .flat_map(|letter| ('0'..='9').map(move |digit| format!("{letter}{digit}"))) + .collect(); + expected.sort(); + + assert_eq!(expected, sampled); + } + + /// The sample characters are what a range is reached for first, so a + /// pattern made of wide ranges samples as readable text rather than as + /// control characters. + #[test] + fn test_generate_strings_sampled_prefers_representative_characters() { + let automaton = automaton_of(".{3}"); + + let sampled = automaton + .generate_strings(1, 0, GenerationOrder::Sampled) + .unwrap(); + + assert_eq!(vec!["a0A".to_string()], sampled); + } + + /// Sampling hands out each character of a range exactly once: anything else + /// and a path would repeat a string, or `offset` would drift off its page. + #[test] + fn test_sample_char_is_a_permutation_of_the_range() { + let ranges = [ + CharRange::new_from_range_char('a'..='z'), + // Several intervals, one of them straddling the surrogate hole. + CharRange::new_from_ranges(&[ + AnyRange::from(Char::new('0')..=Char::new('9')), + AnyRange::from(Char::new('\u{d7fe}')..=Char::new('\u{e001}')), + ]), + // Past the hole, around one of the sample characters. + CharRange::new_from_range_char('\u{1f5ff}'..='\u{1f601}'), + ]; + + for range in ranges { + let expected: Vec = range.iter().map(|ch| ch.to_char()).collect(); + + for position in 0..3 { + let mut sampled: Vec = (0..range.get_cardinality()) + .map(|index| sample_char(&range, position, index).unwrap().to_char()) + .collect(); + sampled.sort_unstable(); + + assert_eq!(expected, sampled, "range {range}, position {position}"); + } + } + } + + /// A charset rules out whole paths, not single characters: a path that + /// needs a ruled-out character is dropped even when it only needs it + /// several transitions in, and the strings around it still come out. + #[test] + fn test_generate_strings_charset_drops_the_paths_it_closes() { + let automaton = automaton_of("(a[0-9]b|xyz)"); + let letters = CharRange::new_from_range_char('a'..='z'); + + for order in ORDERS { + let options = GenerationOptions::from(order).with_charset(letters.clone()); + + assert_eq!( + vec!["xyz".to_string()], + automaton.generate_strings(10, 0, options).unwrap(), + "{order:?}" + ); + } + } + + /// A charset that leaves the pattern nothing is an empty page, not an + /// error: the search prunes the start state like any other dead end. + #[test] + fn test_generate_strings_charset_can_leave_nothing() { + let automaton = automaton_of("[0-9]+"); + let letters = CharRange::new_from_range_char('a'..='z'); + + for order in ORDERS { + let options = GenerationOptions::from(order).with_charset(letters.clone()); + + assert!( + automaton + .generate_strings(10, 0, options) + .unwrap() + .is_empty(), + "{order:?}" + ); + } + } + + /// The charset narrows the ranges the strings are built from, so what is + /// generated is the language the pattern and the charset agree on. + #[test] + fn test_generate_strings_charset_narrows_every_range() { + let automaton = automaton_of(".{2}"); + let vowels = CharRange::new_from_ranges(&[ + AnyRange::from(Char::new('a')..=Char::new('a')), + AnyRange::from(Char::new('e')..=Char::new('e')), + ]); + + for order in ORDERS { + let options = GenerationOptions::from(order).with_charset(vowels.clone()); + + let mut strings = automaton.generate_strings(10, 0, options).unwrap(); + strings.sort(); + + assert_eq!(vec!["aa", "ae", "ea", "ee"], strings, "{order:?}"); + } + } + + /// The offset steps over whole subtrees at once: a page from deep inside a + /// large language comes back without walking everything before it. + #[test] + fn test_generate_strings_offset_reaches_deep_pages() { + let automaton = automaton_of("[a-z]{5}"); + let total = 26usize.pow(5); + + let strings = automaton + .generate_strings(2, total - 2, GenerationOrder::Exhaustive) .unwrap(); - // Generate 30 strings at once - let all_strings = automaton.generate_strings(30, 0).unwrap(); + assert_eq!(vec!["zzzzy".to_string(), "zzzzz".to_string()], strings); + } - //println!("all_strings {:?}", all_strings); + /// The cursor hands out exactly the characters past its opening ordinal, + /// across intervals and over the surrogate hole, like the plain iterator. + #[test] + fn test_range_cursor_opens_at_any_ordinal() { + let range = CharRange::new_from_ranges(&[ + AnyRange::from(Char::new('0')..=Char::new('9')), + AnyRange::from(Char::new('\u{d7fe}')..=Char::new('\u{e001}')), + ]); + + for start in 0..=range.get_cardinality() { + let expected: Vec = range + .iter() + .skip(start as usize) + .map(|c| c.to_char()) + .collect(); + + let mut cursor = RangeCursor::new(&range, start); + let mut walked = vec![]; + while let Some(ch) = cursor.next() { + walked.push(ch.to_char()); + } - // Generate the same 30 strings in chunks of 10 - let chunk1 = automaton.generate_strings(10, 0).unwrap(); - let chunk2 = automaton.generate_strings(10, 10).unwrap(); - let chunk3 = automaton.generate_strings(10, 20).unwrap(); + assert_eq!(expected, walked, "start {start}"); + } + } - /* - println!("chunk1 {:?}", chunk1); - println!("chunk2 {:?}", chunk2); - println!("chunk3 {:?}", chunk3); - */ + /// A huge `limit` means "everything the language holds", not "reserve this + /// much memory": it must not size an allocation before generation starts. + #[test] + fn test_generate_strings_limit_does_not_preallocate() { + let automaton = automaton_of("[ab]{2}"); - assert_eq!(all_strings.len(), 30, "Should generate exactly 30 strings"); - assert_eq!(chunk1.len(), 10); - assert_eq!(chunk2.len(), 10); - assert_eq!(chunk3.len(), 10); + for order in ORDERS { + let mut strings = automaton.generate_strings(usize::MAX, 0, order).unwrap(); + strings.sort(); - // Combine the chunks - let mut combined = chunk1; - combined.extend(chunk2); - combined.extend(chunk3); + assert_eq!(vec!["aa", "ab", "ba", "bb"], strings, "{order:?}"); + } + } - // Prove that generating in chunks perfectly matches the bulk generation + /// Combination emission walks an explicit stack, not the call stack: a + /// path tens of thousands of transitions long emits without overflowing. + #[test] + fn test_generate_strings_very_long_string() { + let automaton = automaton_of("[ab]{20000}"); + + for order in ORDERS { + let strings = automaton.generate_strings(2, 0, order).unwrap(); + assert_eq!(2, strings.len(), "{order:?}"); + for string in &strings { + assert_eq!(20_000, string.len(), "{order:?}"); + } + } + } + + /// The surrogate block is not a character, so the whole alphabet is one + /// character shorter than its last code point suggests. + #[test] + fn test_char_at_walks_over_the_surrogate_block() { + let total = CharRange::total(); + + assert_eq!('\u{0}', char_at(&total, 0).unwrap().to_char()); + assert_eq!('\u{d7ff}', char_at(&total, 0xd7ff).unwrap().to_char()); + assert_eq!('\u{e000}', char_at(&total, 0xd800).unwrap().to_char()); + + let cardinality = total.get_cardinality(); assert_eq!( - all_strings, combined, - "Chunked generation did not match bulk generation" + '\u{10ffff}', + char_at(&total, cardinality - 1).unwrap().to_char() ); + assert!(char_at(&total, cardinality).is_none()); + } - let cardinality = automaton.cardinality().unwrap(); + fn automaton_of(regex: &str) -> FastAutomaton { + RegularExpression::parse(regex, false) + .unwrap() + .to_automaton() + .unwrap() + } - if let Cardinality::Integer(count) = cardinality { - let empty_chunk = automaton.generate_strings(10, count as usize).unwrap(); - assert!(empty_chunk.is_empty(), "Chunk past limits should be empty"); + fn assert_generate_strings_offset(regex: &str) { + println!("regex: {regex}"); + let automaton = automaton_of(regex); + + for options in all_options() { + // Generate 30 strings at once + let all_strings = automaton.generate_strings(30, 0, options.clone()).unwrap(); + + // Generate the same 30 strings in chunks of 10 + let chunk1 = automaton.generate_strings(10, 0, options.clone()).unwrap(); + let chunk2 = automaton.generate_strings(10, 10, options.clone()).unwrap(); + let chunk3 = automaton.generate_strings(10, 20, options.clone()).unwrap(); + + assert_eq!( + all_strings.len(), + 30, + "Should generate exactly 30 strings ({options:?})" + ); + assert_eq!(chunk1.len(), 10, "{options:?}"); + assert_eq!(chunk2.len(), 10, "{options:?}"); + assert_eq!(chunk3.len(), 10, "{options:?}"); + + // Combine the chunks + let mut combined = chunk1; + combined.extend(chunk2); + combined.extend(chunk3); + + // Prove that generating in chunks perfectly matches the bulk generation + assert_eq!( + all_strings, combined, + "Chunked generation did not match bulk generation ({options:?})" + ); + + let cardinality = automaton.cardinality().unwrap(); + + // A charset only ever leaves fewer strings than the automaton + // holds, so its count is past the end of the restricted language + // too. + if let Cardinality::Integer(count) = cardinality { + let empty_chunk = automaton + .generate_strings(10, count as usize, options.clone()) + .unwrap(); + assert!( + empty_chunk.is_empty(), + "Chunk past limits should be empty ({options:?})" + ); + } } } fn assert_generate_strings(regex: &str, number: usize) { println!(":{}", regex); - let automaton = RegularExpression::parse(regex, false) - .unwrap() - .to_automaton() - .unwrap(); + let automaton = automaton_of(regex); let re = Regex::new(&format!("(?s)^{}$", regex)).unwrap(); - // Modified to include an offset of 0 - let strings = automaton.generate_strings(number, 0).unwrap(); - println!("nb of strings: {}/{}", strings.len(), number); - assert!(number >= strings.len()); - for string in strings { - if !re.is_match(&string) { - for byte in string.as_bytes() { - print!("{:02x} ", byte); + for options in all_options() { + let strings = automaton + .generate_strings(number, 0, options.clone()) + .unwrap(); + println!("nb of strings ({options:?}): {}/{}", strings.len(), number); + assert!(number >= strings.len()); + + let distinct: std::collections::HashSet<_> = strings.iter().collect(); + assert_eq!( + distinct.len(), + strings.len(), + "the same string came up twice ({options:?})" + ); + + for string in strings { + if let Some(charset) = options.charset() { + assert!( + string.chars().all(|ch| charset.contains(Char::new(ch))), + "'{string}' uses characters outside the charset" + ); + } + if !re.is_match(&string) { + for byte in string.as_bytes() { + print!("{:02x} ", byte); + } + panic!("'{string}' ({options:?})") } - panic!("'{string}'") } - assert!(re.is_match(&string), "'{string}'"); } } } diff --git a/src/fast_automaton/mod.rs b/src/fast_automaton/mod.rs index 980ced7..8e04de1 100644 --- a/src/fast_automaton/mod.rs +++ b/src/fast_automaton/mod.rs @@ -26,6 +26,8 @@ mod operation; /// character ranges, over which transition conditions are defined. pub mod spanning_set; +pub use generate::{GenerationOptions, GenerationOrder}; + /// Represents a finite-state automaton. #[derive(Clone, Debug, PartialEq, Eq)] #[must_use = "non-`_mut` operations return a new automaton"] diff --git a/src/fast_automaton/operation/concat.rs b/src/fast_automaton/operation/concat.rs index b2fd16f..60d773f 100644 --- a/src/fast_automaton/operation/concat.rs +++ b/src/fast_automaton/operation/concat.rs @@ -9,7 +9,9 @@ use super::*; impl FastAutomaton { /// Computes the concatenation between `self` and `other`. pub fn concat(&self, other: &FastAutomaton) -> Result { - Self::concat_all([self, other]) + let mut new_automaton = self.clone(); + new_automaton.concat_mut(other)?; + Ok(new_automaton) } /// Computes the concatenation of all automata in the given iterator. @@ -17,9 +19,26 @@ impl FastAutomaton { pub fn concat_all<'a, I: IntoIterator>( automata: I, ) -> Result { + // Each operand's degenerate checks run once, on the operand: running + // them per fold step, on the growing result, made long concatenations + // quadratic. let mut new_automaton = FastAutomaton::new_empty_string(); + let mut seeded = false; for automaton in automata { - new_automaton.concat_mut(automaton)?; + if automaton.is_empty() { + // ∅ annihilates the whole concatenation. + return Ok(FastAutomaton::new_empty()); + } + if automaton.is_empty_string() { + // {""} is the identity. + continue; + } + if seeded { + new_automaton.concat_mut_nondegenerate(automaton, false)?; + } else { + new_automaton.apply_model(automaton); + seeded = true; + } } Ok(new_automaton) @@ -44,11 +63,7 @@ impl FastAutomaton { other: &FastAutomaton, force_no_merge: bool, ) -> Result<(), EngineError> { - let execution_profile = ExecutionProfile::get(); - execution_profile.assert_not_timed_out()?; - execution_profile.assert_max_number_of_states( - self.concat_state_count_heuristic(other, force_no_merge), - )?; + ExecutionProfile::get().assert_not_timed_out()?; if other.is_empty() { self.make_empty(); @@ -65,6 +80,25 @@ impl FastAutomaton { return Ok(()); } + self.concat_mut_nondegenerate(other, force_no_merge) + } + + /// The concatenation core: both operands must be neither the empty + /// language `∅` nor the empty-string language `{""}`. Callers looping over + /// a growing accumulator (`repeat_mut`, [`concat_all`](Self::concat_all)) + /// establish that invariant once and call this directly: the degenerate + /// checks of [`concat_mut_with`](Self::concat_mut_with) walk the whole + /// automaton, and re-running them on every iteration made those loops + /// quadratic. + pub(crate) fn concat_mut_nondegenerate( + &mut self, + other: &FastAutomaton, + force_no_merge: bool, + ) -> Result<(), EngineError> { + self.assert_nondegenerate_operation_fits(other, || { + self.concat_state_count_nondegenerate(other, force_no_merge) + })?; + let new_spanning_set = &self.spanning_set.merge(&other.spanning_set); self.apply_new_spanning_set(new_spanning_set)?; let condition_converter = ConditionConverter::new(&other.spanning_set, new_spanning_set)?; @@ -180,6 +214,16 @@ impl FastAutomaton { return other.number_of_states(); } + self.concat_state_count_nondegenerate(other, force_no_merge) + } + + /// [`concat_state_count_heuristic`](Self::concat_state_count_heuristic) + /// for operands already known to be non-degenerate: no emptiness walks. + fn concat_state_count_nondegenerate( + &self, + other: &FastAutomaton, + force_no_merge: bool, + ) -> usize { // Determine if we are forced to create a new state to avoid unintended loops let start_state_and_accept_states_not_mergeable = force_no_merge || (other.in_degree(other.start_state) > 0 diff --git a/src/fast_automaton/operation/mod.rs b/src/fast_automaton/operation/mod.rs index 9b558c6..f303f6e 100644 --- a/src/fast_automaton/operation/mod.rs +++ b/src/fast_automaton/operation/mod.rs @@ -11,6 +11,27 @@ mod repeat; mod union; impl FastAutomaton { + /// The shared preamble of the non-degenerate operation cores + /// (`concat_mut_nondegenerate`, `union_mut_nondegenerate`): a cheap + /// necessary condition for the caller-guaranteed invariant (the full + /// degenerate checks are exactly what the cores exist to avoid re-running), + /// the timeout check, and — only when a state limit is configured — the + /// predicted-size check. + fn assert_nondegenerate_operation_fits( + &self, + other: &FastAutomaton, + predicted_states: impl FnOnce() -> usize, + ) -> Result<(), crate::error::EngineError> { + debug_assert!(!self.accept_states.is_empty() && !other.accept_states.is_empty()); + + let execution_profile = crate::execution_profile::ExecutionProfile::get(); + execution_profile.assert_not_timed_out()?; + if execution_profile.limits_number_of_states() { + execution_profile.assert_max_number_of_states(predicted_states())?; + } + Ok(()) + } + /// Removes "dead" states (those that cannot reach any accept state), since /// they never contribute to the language. If the language is empty the whole /// automaton collapses to the canonical empty automaton. diff --git a/src/fast_automaton/operation/repeat.rs b/src/fast_automaton/operation/repeat.rs index 2fb63ca..24df7de 100644 --- a/src/fast_automaton/operation/repeat.rs +++ b/src/fast_automaton/operation/repeat.rs @@ -5,11 +5,8 @@ impl FastAutomaton { #[tracing::instrument(level = "debug", skip(self), fields(states = self.number_of_states(), deterministic = self.is_deterministic(), min = min, max_opt = tracing::field::debug(max_opt)))] pub fn repeat(&self, min: u32, max_opt: Option) -> Result { let mut automaton = self.clone(); - if let Err(error) = automaton.repeat_mut(min, max_opt) { - Err(error) - } else { - Ok(automaton) - } + automaton.repeat_mut(min, max_opt)?; + Ok(automaton) } pub(crate) fn repeat_mut(&mut self, min: u32, max_opt: Option) -> Result<(), EngineError> { @@ -93,9 +90,14 @@ impl FastAutomaton { return Ok(()); } + // From here on `self` and `automaton_to_repeat` are known to be + // neither ∅ nor {""} (checked above), and concatenating two such + // languages preserves that: the loops call the concatenation core + // directly, since re-checking the growing chain on every iteration is + // quadratic. let iter = if min == 0 { 0..0 } else { 0..min - 1 }; for _ in iter { - self.concat_mut(&automaton_to_repeat)?; + self.concat_mut_nondegenerate(&automaton_to_repeat, false)?; } if max_opt.is_none() { @@ -137,8 +139,10 @@ impl FastAutomaton { // clean accepting start instead of marking the looping start // accepting, which would otherwise accept partial copies // (e.g. `(a*b)+` matching "aaba"). + // The star of a non-degenerate language is non-degenerate: it + // keeps every string of `r` and gains "". let star = automaton_to_repeat.repeat(0, None)?; - self.concat_mut(&star)?; + self.concat_mut_nondegenerate(&star, false)?; } return Ok(()); @@ -158,7 +162,7 @@ impl FastAutomaton { let force_no_merge = automaton_to_repeat.in_degree(automaton_to_repeat.start_state) > 0; let mut end_states = self.accept_states.iter().cloned().collect::>(); for _ in cmp::max(min, 1)..max_opt.unwrap() { - self.concat_mut_with(&automaton_to_repeat, force_no_merge)?; + self.concat_mut_nondegenerate(&automaton_to_repeat, force_no_merge)?; end_states.extend(self.accept_states.iter()); } for end_state in end_states { @@ -297,6 +301,22 @@ impl FastAutomaton { #[cfg(test)] mod tests { + // Building a large bounded repetition must stay linear in the bound: the + // per-copy concatenations run against a growing chain, and re-checking + // that chain's emptiness on every copy made this quadratic (~10 s in + // debug builds at this size, milliseconds when linear). + #[test] + fn repeat_large_bounded_stays_linear() { + let automaton = crate::regex::RegularExpression::parse("[ab]{5000}", false) + .unwrap() + .to_automaton() + .unwrap(); + + assert_eq!(5001, automaton.number_of_states()); + assert!(automaton.is_match(&"ab".repeat(2500))); + assert!(!automaton.is_match(&"ab".repeat(2499))); + } + // Repeating an empty-language automaton must respect ∅* = {""} and // ∅ⁿ = ∅ even when the emptiness comes from unreachable accept states or // dead-but-reachable transitions (rather than an absent accept set): the diff --git a/src/fast_automaton/operation/union.rs b/src/fast_automaton/operation/union.rs index 00e3b6f..248ede1 100644 --- a/src/fast_automaton/operation/union.rs +++ b/src/fast_automaton/operation/union.rs @@ -11,7 +11,9 @@ use super::*; impl FastAutomaton { /// Computes the union between `self` and `other`. pub fn union(&self, other: &FastAutomaton) -> Result { - Self::union_all([self, other]) + let mut new_automaton = self.clone(); + new_automaton.union_mut(other)?; + Ok(new_automaton) } /// Computes the union of all automata in the given iterator. @@ -19,9 +21,27 @@ impl FastAutomaton { pub fn union_all<'a, I: IntoIterator>( automata: I, ) -> Result { + // Each operand's degenerate checks run once, on the operand: running + // them per fold step, on the growing result, made large alternations + // quadratic. let mut new_automaton = FastAutomaton::new_empty(); + let mut seeded = false; for automaton in automata { - new_automaton.union_mut(automaton)?; + if automaton.is_empty() { + // ∅ is the identity. + continue; + } + if automaton.is_total() { + // Σ* absorbs the whole union. + new_automaton.make_total(); + return Ok(new_automaton); + } + if seeded { + new_automaton.union_mut_nondegenerate(automaton)?; + } else { + new_automaton.apply_model(automaton); + seeded = true; + } } Ok(new_automaton) } @@ -187,9 +207,7 @@ impl FastAutomaton { * - the accept states can't be merged if they have outgoing edges */ pub(crate) fn union_mut(&mut self, other: &FastAutomaton) -> Result<(), EngineError> { - let execution_profile = ExecutionProfile::get(); - execution_profile.assert_not_timed_out()?; - execution_profile.assert_max_number_of_states(self.union_state_count_heuristic(other))?; + ExecutionProfile::get().assert_not_timed_out()?; if other.is_empty() || self.is_total() { return Ok(()); @@ -201,6 +219,21 @@ impl FastAutomaton { return Ok(()); } + self.union_mut_nondegenerate(other) + } + + /// The union core: neither operand may be the empty language `∅` or all + /// strings `Σ*`. Callers folding many operands + /// ([`union_all`](Self::union_all)) establish that invariant per operand + /// and call this directly: the degenerate checks of + /// [`union_mut`](Self::union_mut) walk the whole automaton, and re-running + /// them on the growing result at every fold step made large alternations + /// quadratic. + fn union_mut_nondegenerate(&mut self, other: &FastAutomaton) -> Result<(), EngineError> { + self.assert_nondegenerate_operation_fits(other, || { + self.union_state_count_nondegenerate(other) + })?; + let new_spanning_set = &self.spanning_set.merge(&other.spanning_set); self.apply_new_spanning_set(new_spanning_set)?; let condition_converter = ConditionConverter::new(&other.spanning_set, new_spanning_set)?; @@ -241,6 +274,10 @@ impl FastAutomaton { } /// Computes the expected number of states after calling `union_mut`. + /// Kept as the specification of the union's state growth; the exactness + /// tests validate it against `union_mut`, and the non-degenerate half + /// backs the state-limit check in the union core. + #[cfg(test)] fn union_state_count_heuristic(&self, other: &FastAutomaton) -> usize { // Edge cases if other.is_empty() || self.is_total() { @@ -249,6 +286,12 @@ impl FastAutomaton { return other.number_of_states(); } + self.union_state_count_nondegenerate(other) + } + + /// [`union_state_count_heuristic`](Self::union_state_count_heuristic) for + /// operands already known to be non-degenerate: no emptiness walks. + fn union_state_count_nondegenerate(&self, other: &FastAutomaton) -> usize { let v1 = self.number_of_states(); let v2 = other.number_of_states(); diff --git a/src/lib.rs b/src/lib.rs index d87120f..36a7319 100644 --- a/src/lib.rs +++ b/src/lib.rs @@ -59,7 +59,7 @@ use std::{ use cardinality::Cardinality; use error::EngineError; -use fast_automaton::FastAutomaton; +use fast_automaton::{FastAutomaton, GenerationOptions}; #[cfg(feature = "parallel")] use rayon::prelude::*; use regex::RegularExpression; @@ -133,6 +133,7 @@ pub type CharRange = RangeSet; /// ```rust /// use regexsolver::Term; /// use regexsolver::error::EngineError; +/// use regexsolver::fast_automaton::GenerationOrder; /// /// fn main() -> Result<(), EngineError> { /// // Create terms from regex @@ -168,7 +169,7 @@ pub type CharRange = RangeSet; /// /// // Generate examples /// let samples = Term::from_pattern("(x|y){1,3}")? -/// .generate_strings(5, 0)?; +/// .generate_strings(5, 0, GenerationOrder::Sampled)?; /// println!("Some matches: {:?}", samples); /// /// // Equivalence & subset @@ -566,14 +567,25 @@ impl Term { } } - /// Generates up to `limit` distinct strings matched by the term, skipping the first `offset` strings. + /// Generates up to `limit` distinct strings matched by the term under the + /// given [`GenerationOptions`], skipping the first `offset` strings. + /// + /// `options` is a [`GenerationOrder`](fast_automaton::GenerationOrder) on + /// its own, or a full [`GenerationOptions`] to also restrict the + /// characters used. + /// [`Exhaustive`](fast_automaton::GenerationOrder::Exhaustive) sweeps the + /// language, one path at a time; + /// [`Sampled`](fast_automaton::GenerationOrder::Sampled) spreads the + /// strings over the shapes the pattern allows, which is what you want to + /// derive test cases from a pattern. /// /// Strings are only guaranteed to be distinct **within a single call**: /// the offset fast-skips by counting paths, and in a non-deterministic /// automaton the same string can be reached through several paths, so /// calls with different offsets may repeat strings (or skip some). The /// enumeration order also depends on the automaton's structure, so - /// offsets are only consistent across calls made on the same term. + /// offsets are only consistent across calls made on the same term with + /// the same options. /// /// For pagination without repetition or skipped strings, make the term deterministic once and generate /// from it. To check if a term is deterministic use [`is_deterministic`](Self::is_deterministic). @@ -582,65 +594,86 @@ impl Term { /// # Examples /// /// ``` - /// use regexsolver::Term; + /// use regexsolver::{CharRange, Term, fast_automaton::{GenerationOptions, GenerationOrder}}; + /// use regexsolver::regex_charclass::char::Char; /// /// // Minimize once, then paginate with consistent offsets. /// let term = Term::from_pattern("(abc|de){2}").unwrap().minimize().unwrap(); /// - /// let batch = term.generate_strings(2, 0).unwrap(); + /// let batch = term.generate_strings(2, 0, GenerationOrder::Exhaustive).unwrap(); /// assert_eq!(2, batch.len()); // ["dede", "deabc"] /// - /// let batch = term.generate_strings(2, 2).unwrap(); + /// let batch = term.generate_strings(2, 2, GenerationOrder::Exhaustive).unwrap(); /// assert_eq!(2, batch.len()); // ["abcde", "abcabc"] + /// + /// // The exhaustive order works through one path at a time, so a limit + /// // spent on `.*abc.*` never leaves the strings starting with `abc`. + /// let term = Term::from_pattern(".*abc.*").unwrap().minimize().unwrap(); + /// + /// let batch = term.generate_strings(5, 0, GenerationOrder::Exhaustive).unwrap(); + /// assert!(batch.iter().all(|s| s.starts_with("abc"))); + /// + /// // The sampled order covers the pattern instead. + /// let batch = term.generate_strings(5, 0, GenerationOrder::Sampled).unwrap(); + /// assert!(batch.iter().any(|s| !s.starts_with("abc"))); + /// + /// // A charset keeps generation to the characters you can use. + /// let printable = CharRange::new_from_range(Char::new(' ')..=Char::new('~')); + /// let options = GenerationOptions::from(GenerationOrder::Sampled).with_charset(printable); + /// + /// let batch = term.generate_strings(5, 0, options).unwrap(); + /// assert!(batch.iter().all(|s| s.chars().all(|c| c.is_ascii_graphic() || c == ' '))); /// ``` - #[tracing::instrument(level = "debug", skip(self), fields(self_deterministic = self.is_deterministic(), limit = limit, offset = offset))] + #[tracing::instrument(level = "debug", skip(self, options), fields(self_deterministic = self.is_deterministic(), limit = limit, offset = offset))] pub fn generate_strings( &self, limit: usize, offset: usize, + options: impl Into, ) -> Result, EngineError> { - self.to_automaton()?.generate_strings(limit, offset) + self.to_automaton()? + .generate_strings(limit, offset, options) } - /// Returns a lazy iterator over the strings matched by the term, fetched in - /// batches behind the scenes so you can stop early without choosing a limit - /// up front. + /// Returns a lazy iterator over the strings matched by the term under the + /// given [`GenerationOptions`], fetched in batches behind the scenes so you + /// can stop early without choosing a limit up front. /// - /// The underlying automaton is computed once at construction time, not on + /// The underlying deterministic automaton is computed once at construction time, not on /// every batch. Each item is a `Result`: a construction or generation error /// (e.g. a timeout from the active [`ExecutionProfile`]) surfaces as an - /// `Err`, after which the iterator ends. The same determinism caveat as - /// [`generate_strings`](Self::generate_strings) applies: call - /// [`determinize`](Self::determinize) (or [`minimize`](Self::minimize)) - /// first for distinct, stable enumeration. + /// `Err`, after which the iterator ends. /// /// # Examples /// /// ``` - /// use regexsolver::Term; + /// use regexsolver::{Term, fast_automaton::GenerationOrder}; /// /// let term = Term::from_pattern("(abc|de){2}").unwrap().minimize().unwrap(); /// /// // Take the first three matches lazily. /// let first_three = term - /// .iter_strings() + /// .iter_strings(GenerationOrder::Exhaustive) /// .take(3) /// .collect::, _>>() /// .unwrap(); /// assert_eq!(3, first_three.len()); /// ``` - pub fn iter_strings(&self) -> StringGenerator<'_> { - match self.to_automaton() { + pub fn iter_strings(&self, options: impl Into) -> StringGenerator<'_> { + let options = options.into(); + match self.to_deterministic_automaton() { Ok(automaton) => StringGenerator { automaton: Some(automaton), pending_error: None, offset: 0, + options, buffer: VecDeque::new(), }, Err(e) => StringGenerator { automaton: None, pending_error: Some(e), offset: 0, + options, buffer: VecDeque::new(), }, } @@ -904,6 +937,14 @@ impl Term { }) } + fn to_deterministic_automaton(&self) -> Result, EngineError> { + let automaton = self.to_automaton()?; + if automaton.is_deterministic() { + return Ok(automaton); + } + Ok(Cow::Owned(automaton.determinize()?.into_owned())) + } + /// Converts the term to a [`RegularExpression`]. /// /// Returns a [`Cow`]: borrows the expression when the term is already @@ -980,6 +1021,7 @@ pub struct StringGenerator<'a> { automaton: Option>, pending_error: Option, offset: usize, + options: GenerationOptions, buffer: VecDeque, } @@ -1000,7 +1042,7 @@ impl Iterator for StringGenerator<'_> { return Some(Err(e)); } let automaton = self.automaton.as_ref()?; - match automaton.generate_strings(BATCH, self.offset) { + match automaton.generate(BATCH, self.offset, &self.options) { Ok(batch) => { if batch.len() < BATCH { self.automaton = None; @@ -1019,6 +1061,7 @@ impl Iterator for StringGenerator<'_> { #[cfg(test)] mod tests { + use crate::fast_automaton::GenerationOrder; use crate::regex::RegularExpression; use super::*; @@ -1217,8 +1260,13 @@ mod tests { .minimize() .unwrap(); - let eager = term.generate_strings(1000, 0).unwrap(); - let lazy = term.iter_strings().collect::, _>>().unwrap(); + let eager = term + .generate_strings(1000, 0, GenerationOrder::Exhaustive) + .unwrap(); + let lazy = term + .iter_strings(GenerationOrder::Exhaustive) + .collect::, _>>() + .unwrap(); assert_eq!(eager.len(), lazy.len()); assert_eq!(eager, lazy); @@ -1352,7 +1400,7 @@ mod tests { // Must not hang on an infinite language: take a finite prefix. let term = Term::from_pattern("a+").unwrap(); let first = term - .iter_strings() + .iter_strings(GenerationOrder::Exhaustive) .take(5) .collect::, _>>() .unwrap(); @@ -1371,7 +1419,7 @@ mod tests { .build(); profile.run(|| { - let mut it = term.iter_strings(); + let mut it = term.iter_strings(GenerationOrder::Exhaustive); assert!(matches!( it.next(), Some(Err(EngineError::AutomatonHasTooManyStates)) diff --git a/src/regex/analyze/number_of_states.rs b/src/regex/analyze/number_of_states.rs index e901a3b..5f33538 100644 --- a/src/regex/analyze/number_of_states.rs +++ b/src/regex/analyze/number_of_states.rs @@ -22,6 +22,14 @@ struct AbstractNFAMetadata { start: AbstractStateMetadata, accepted: Vec, number_of_states: usize, + /// Whether the language contains the empty string, i.e. the start state + /// is accepting. Exact for the constructions modeled here. + accepts_empty_string: bool, + /// Whether some accept state may sit one transition away from the start + /// state. Over-approximate (may be `true` when none does), never + /// under-approximate: `alternate` withholds a merge discount on it, so + /// erring towards `true` keeps the estimate an upper bound. + accept_adjacent_to_start: bool, } impl AbstractNFAMetadata { @@ -30,6 +38,8 @@ impl AbstractNFAMetadata { start: AbstractStateMetadata::new(false, true), accepted: vec![AbstractStateMetadata::new(true, false)], number_of_states: 2, + accepts_empty_string: false, + accept_adjacent_to_start: true, } } @@ -38,6 +48,8 @@ impl AbstractNFAMetadata { start: AbstractStateMetadata::new(false, false), accepted: vec![AbstractStateMetadata::new(false, false)], number_of_states: 1, + accepts_empty_string: true, + accept_adjacent_to_start: false, } } @@ -46,9 +58,20 @@ impl AbstractNFAMetadata { start: AbstractStateMetadata::new(false, false), accepted: vec![], number_of_states: 1, + accepts_empty_string: false, + accept_adjacent_to_start: false, } } + /// [`accept_adjacent_to_start`](Self::accept_adjacent_to_start) of the + /// concatenation `self · nfa`: the boundary attaches `nfa`'s structure to + /// `self`'s accept states, so an accept can only end up next to the start + /// through an accepting start on one side of the boundary. + fn concat_accept_adjacency(&self, nfa: &AbstractNFAMetadata) -> bool { + (nfa.accepts_empty_string && self.accept_adjacent_to_start) + || (self.accepts_empty_string && nfa.accept_adjacent_to_start) + } + pub(crate) fn concat(&self, nfa: &AbstractNFAMetadata) -> Self { let is_empty_string = |m: &AbstractNFAMetadata| { m.number_of_states == 1 && !m.accepted.is_empty() && !m.start.has_outgoing_edges @@ -68,12 +91,16 @@ impl AbstractNFAMetadata { start: self.start.clone(), accepted: nfa.accepted.clone(), number_of_states: self.number_of_states.saturating_add(nfa.number_of_states), + accepts_empty_string: self.accepts_empty_string && nfa.accepts_empty_string, + accept_adjacent_to_start: self.concat_accept_adjacency(nfa), } } else { AbstractNFAMetadata { start: self.start.clone(), accepted: nfa.accepted.clone(), number_of_states: self.number_of_states.saturating_add(nfa.number_of_states) - 1, + accepts_empty_string: self.accepts_empty_string && nfa.accepts_empty_string, + accept_adjacent_to_start: self.concat_accept_adjacency(nfa), } } } @@ -105,6 +132,11 @@ impl AbstractNFAMetadata { number_of_states: self .number_of_states .saturating_add((min as usize - 1).saturating_mul(appended_copy_cost)), + accepts_empty_string: self.accepts_empty_string, + // An accept of rᵐⁱⁿ can neighbour the start only when it is + // one copy deep, or when copies collapse over "" ∈ r. + accept_adjacent_to_start: self.accept_adjacent_to_start + && (min == 1 || self.accepts_empty_string), }; return mandatory.concat(&self.repeat(0, &None)); } @@ -139,6 +171,8 @@ impl AbstractNFAMetadata { start: return_start, accepted: return_accepted, number_of_states: (self.number_of_states - 1).max(1), + accepts_empty_string: true, + accept_adjacent_to_start: self.accept_adjacent_to_start, }; } @@ -197,6 +231,11 @@ impl AbstractNFAMetadata { start: return_start, accepted: return_accepted, number_of_states: return_number_of_states, + accepts_empty_string: min == 0 || self.accepts_empty_string, + // An accept can neighbour the start only when it is one copy deep + // (min <= 1), or when copies collapse over "" ∈ r. + accept_adjacent_to_start: self.accept_adjacent_to_start + && (min <= 1 || self.accepts_empty_string), } } @@ -221,6 +260,13 @@ impl AbstractNFAMetadata { if !self_accepted_not_mergeable && !nfa_accepted_not_mergeable + // A looping start (incoming edges) makes the union materialize + // the start's direct successors before accept states are merged, + // and an accept among those successors never merges (e.g. `a*a`, + // whose accept hangs directly off the looping start). Withhold the + // saving when an accept may sit there: an upper bound may + // overshoot, but never undershoot. + && !(nfa_start_state_not_mergeable && nfa.accept_adjacent_to_start) && !self.accepted.is_empty() && !nfa.accepted.is_empty() && self.number_of_states > 1 @@ -246,6 +292,9 @@ impl AbstractNFAMetadata { AbstractNFAMetadata { start: return_start, accepted: return_accepted, + accepts_empty_string: self.accepts_empty_string || nfa.accepts_empty_string, + // The union's entry state carries both operands' start edges. + accept_adjacent_to_start: self.accept_adjacent_to_start || nfa.accept_adjacent_to_start, // Both merge discounts can apply to two single-state {""} // operands (e.g. `a{0,0}|b{0,0}`); clamp so the count never // reaches zero (see `repeat`). diff --git a/tests/integration_tests.rs b/tests/integration_tests.rs index 9b66086..f025258 100644 --- a/tests/integration_tests.rs +++ b/tests/integration_tests.rs @@ -4,20 +4,24 @@ use std::{ }; use regex::Regex; -use regexsolver::regex::RegularExpression; +use regexsolver::{fast_automaton::GenerationOrder, regex::RegularExpression}; fn assert_regex(regex: &str) { let re = Regex::new(&format!("(?s)^{}$", regex)).unwrap(); let regex = RegularExpression::parse(regex, true).unwrap(); let automaton = regex.to_automaton().unwrap(); - let strings = automaton.generate_strings(500, 0).unwrap(); + let strings = automaton + .generate_strings(500, 0, GenerationOrder::Exhaustive) + .unwrap(); for string in strings { assert!(re.is_match(&string), "'{string}'"); } let determinized_automaton = automaton.determinize().unwrap(); - let strings = determinized_automaton.generate_strings(500, 0).unwrap(); + let strings = determinized_automaton + .generate_strings(500, 0, GenerationOrder::Exhaustive) + .unwrap(); for string in strings { assert!(re.is_match(&string), "'{string}'"); } diff --git a/tests/readme_examples.rs b/tests/readme_examples.rs index 58fb8f9..dfa4e9a 100644 --- a/tests/readme_examples.rs +++ b/tests/readme_examples.rs @@ -7,6 +7,7 @@ use regexsolver::Term; use regexsolver::error::EngineError; use regexsolver::execution_profile::ExecutionProfileBuilder; +use regexsolver::fast_automaton::GenerationOrder; #[test] fn readme_automaton_building_example() -> Result<(), EngineError> { @@ -44,7 +45,10 @@ fn readme_hero_example() -> Result<(), EngineError> { assert!(both.matches("abxy")?); // ...and sample them: - assert_eq!(both.generate_strings(2, 0)?, ["xyxy", "abxy"]); + assert_eq!( + both.generate_strings(2, 0, GenerationOrder::Exhaustive)?, + ["xyxy", "abxy"] + ); Ok(()) }