Skip to content

Optimize automaton intersection when all remaining byte suffixes match. - #16564

Open
vsop-479 wants to merge 11 commits into
apache:mainfrom
vsop-479:optimize-automaton-intersection
Open

Optimize automaton intersection when all remaining byte suffixes match.#16564
vsop-479 wants to merge 11 commits into
apache:mainfrom
vsop-479:optimize-automaton-intersection

Conversation

@vsop-479

@vsop-479 vsop-479 commented Aug 26, 2026

Copy link
Copy Markdown
Contributor

This PR is a simplified version of #13072.

When IntersectTermsEnum reaches an accepting automaton state that can match all remaining byte suffixes, it can stop running the automaton for the remaining bytes and enumerate terms and sub-blocks directly.

This PR only computes match-all-suffix states for the simple case where an accepting state has a full-alphabet self-loop ([0, alphabetSize - 1]). This covers PrefixQuery, TermRangeQuery, and custom binary automata.

The more complex match-all-suffix states produced by UTF32ToUTF8, such as those used by RegexpQuery and WildcardQuery, are intentionally left out of this PR and can be addressed separately.

Compared with #13072, this keeps the computation simple and avoids any query-specific or UTF-8-specific handling in RunAutomaton.

@vsop-479

Copy link
Copy Markdown
Contributor Author

@mikemccand Please take a look when you get a chance.

Performance on wikimediumall:

TaskQPS baseline      StdDevQPS my_modified_version      StdDev                Pct diff p-value
                          IntNRQ      153.28      (7.4%)      149.87      (7.2%)   -2.2% ( -15% -   13%) 0.333
                        Wildcard       47.94      (4.9%)       47.01      (4.1%)   -1.9% ( -10% -    7%) 0.171
                      OrHighHigh      206.47     (10.2%)      202.86     (11.2%)   -1.7% ( -21% -   21%) 0.606
                       MedPhrase      241.49      (3.9%)      237.40      (5.6%)   -1.7% ( -10% -    8%) 0.268
                          Fuzzy1      129.37      (3.5%)      127.47      (4.2%)   -1.5% (  -8% -    6%) 0.229
               HighTermMonthSort     1914.15      (4.9%)     1894.67      (5.8%)   -1.0% ( -11% -   10%) 0.552
            BrowseDateTaxoFacets        5.51     (12.0%)        5.45     (12.7%)   -1.0% ( -22% -   26%) 0.801
     BrowseRandomLabelTaxoFacets        4.78      (9.0%)        4.74      (7.4%)   -0.9% ( -15% -   17%) 0.737
       BrowseDayOfYearSSDVFacets        7.23     (11.0%)        7.16      (9.9%)   -0.8% ( -19% -   22%) 0.801
          OrHighMedDayTaxoFacets       18.96      (4.3%)       18.83      (4.5%)   -0.7% (  -9% -    8%) 0.609
                     LowSpanNear       84.86      (4.8%)       84.27      (3.9%)   -0.7% (  -8% -    8%) 0.614
       BrowseDayOfYearTaxoFacets        5.51     (11.6%)        5.47     (11.2%)   -0.7% ( -20% -   24%) 0.855
                       OrHighMed      381.36      (7.4%)      378.88      (6.9%)   -0.6% ( -13% -   14%) 0.774
                          Fuzzy2       98.88      (3.0%)       98.24      (4.6%)   -0.6% (  -8% -    7%) 0.602
                  AndMissingHigh     5463.33     (10.4%)     5429.54      (7.8%)   -0.6% ( -17% -   19%) 0.832
            HighTermTitleBDVSort       25.12      (3.4%)       25.03      (3.9%)   -0.4% (  -7% -    7%) 0.753
                         Respell       60.04      (2.4%)       59.87      (3.8%)   -0.3% (  -6% -    6%) 0.790
                     MedSpanNear      109.23      (4.0%)      108.96      (5.2%)   -0.2% (  -9% -    9%) 0.867
                        HighTerm     1131.28     (11.5%)     1128.83     (11.4%)   -0.2% ( -20% -   25%) 0.952
               HighTermTitleSort       95.97      (6.0%)       95.81      (4.8%)   -0.2% ( -10% -   11%) 0.924
                    HighSpanNear       44.85      (3.2%)       44.84      (3.0%)   -0.0% (  -6% -    6%) 0.973
                   OrHighNotHigh      483.97      (4.5%)      484.41      (4.7%)    0.1% (  -8% -    9%) 0.951
           BrowseMonthTaxoFacets        5.30      (5.4%)        5.31      (5.2%)    0.2% (  -9% -   11%) 0.905
           HighTermDayOfYearSort      292.41      (6.6%)      293.02      (5.9%)    0.2% ( -11% -   13%) 0.916
                   OrNotHighHigh      644.10      (6.2%)      645.45      (5.2%)    0.2% ( -10% -   12%) 0.907
                        BM25MSM2       31.96      (2.6%)       32.05      (2.9%)    0.3% (  -5% -    5%) 0.728
                           range     5141.84      (4.1%)     5159.70      (6.5%)    0.3% (  -9% -   11%) 0.840
                        PKLookup      441.24      (1.6%)      442.89      (1.6%)    0.4% (  -2% -    3%) 0.447
                          IntSet     1086.78      (9.4%)     1091.14      (8.4%)    0.4% ( -15% -   20%) 0.887
             LowIntervalsOrdered      162.34      (5.0%)      163.00      (5.5%)    0.4% (  -9% -   11%) 0.809
                 LowSloppyPhrase       24.47      (3.2%)       24.58      (3.7%)    0.5% (  -6% -    7%) 0.679
            HighIntervalsOrdered       14.86      (2.1%)       14.94      (3.5%)    0.5% (  -5% -    6%) 0.604
            MedTermDayTaxoFacets       45.17      (2.7%)       45.39      (3.1%)    0.5% (  -5% -    6%) 0.599
                    OrHighNotMed     1321.98      (6.3%)     1329.87      (7.0%)    0.6% ( -12% -   14%) 0.778
                      AndHighLow     2216.36      (6.4%)     2230.17      (5.9%)    0.6% ( -11% -   13%) 0.750
                HighSloppyPhrase      122.81      (4.2%)      123.64      (4.3%)    0.7% (  -7% -    9%) 0.615
            BrowseDateSSDVFacets        1.39      (8.2%)        1.40      (9.5%)    0.7% ( -15% -   20%) 0.794
     BrowseRandomLabelSSDVFacets        6.37     (13.9%)        6.42     (15.9%)    0.7% ( -25% -   35%) 0.875
                       ConstMSM2     1963.63     (11.4%)     1983.01     (11.3%)    1.0% ( -19% -   26%) 0.783
                 MedSloppyPhrase      147.03      (3.8%)      148.49      (3.6%)    1.0% (  -6% -    8%) 0.398
                         LowTerm     3648.41     (11.1%)     3685.97      (9.8%)    1.0% ( -17% -   24%) 0.756
         AndHighMedDayTaxoFacets       41.70      (1.9%)       42.19      (2.6%)    1.2% (  -3% -    5%) 0.098
                       OrHighLow     1714.28      (4.8%)     1736.93      (4.6%)    1.3% (  -7% -   11%) 0.372
                      HighPhrase      166.54      (5.4%)      168.79      (3.7%)    1.3% (  -7% -   11%) 0.360
                    OrNotHighMed      803.24      (4.4%)      815.33      (6.5%)    1.5% (  -9% -   13%) 0.393
        AndHighHighDayTaxoFacets       15.52      (2.8%)       15.76      (2.7%)    1.6% (  -3% -    7%) 0.070
                       LowPhrase      138.68      (5.5%)      141.52      (3.2%)    2.0% (  -6% -   11%) 0.148
                    OrNotHighLow     2011.80      (9.9%)     2057.04      (5.9%)    2.2% ( -12% -   19%) 0.381
                     AndHighHigh      170.59     (12.0%)      174.54     (11.8%)    2.3% ( -19% -   29%) 0.540
                    OrHighNotLow     1311.47      (8.6%)     1342.85      (8.7%)    2.4% ( -13% -   21%) 0.381
                         MedTerm     1555.75     (11.4%)     1594.72     (11.0%)    2.5% ( -17% -   28%) 0.480
                      TermDTSort      324.50      (8.4%)      332.95      (6.5%)    2.6% ( -11% -   19%) 0.275
             MedIntervalsOrdered      307.70      (5.4%)      316.12      (6.2%)    2.7% (  -8% -   15%) 0.138
           BrowseMonthSSDVFacets        7.80     (20.1%)        8.09     (21.4%)    3.6% ( -31% -   56%) 0.583
                      AndHighMed      571.81     (10.5%)      594.57      (6.4%)    4.0% ( -11% -   23%) 0.147
                         Prefix3      474.99      (7.7%)      534.56      (6.7%)   12.5% (  -1% -   29%) 0.000

@vsop-479
vsop-479 requested a review from mikemccand August 26, 2026 10:32
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant