mbe: match eligible arms directly on the flat token entries [tt-muncher -6%, clap_derive -1.5%] - #85
Draft
xmakro wants to merge 1 commit into
Draft
mbe: match eligible arms directly on the flat token entries [tt-muncher -6%, clap_derive -1.5%]#85xmakro wants to merge 1 commit into
xmakro wants to merge 1 commit into
Conversation
When no tracker observes matching and an arm's captures are all resolvable straight off the token entries (tt, ident, lifetime), the matcher now steps a lightweight flat input instead of driving a full Parser: advancing is an entry load instead of a parser bump, tt fragments become buffer slices without a black-box parser round trip, and ident and lifetime captures read the token directly. Arms with AST fragment captures keep the parser-driven path, selected by a flag computed once per rule at definition time; materializing parsers for those from the flat path measured as a net loss. The parser-driven input is also built lazily per invocation, so all-flat definitions never construct a Parser at all. The flat input caches its loaded start state so rewinding for the next arm is three copies, and parse_tt_inner and parse_tt_flat are inlined into their callers; both of these showed up as whole-crate regressions on many-arm macros before. tt-muncher check -6.05%, clap_derive check -1.48%, geomean -0.60% across the mbe suite, with html5ever check +0.26% as the residual per-arm dispatch cost on many-arm mixed definitions.
xmakro
marked this pull request as draft
August 2, 2026 21:26
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #57, stacked on #84. The matcher no longer drives a full
Parserover the input for arms it can handle directly.When no tracker observes matching and an arm's captures are all resolvable straight off the flat token entries (
tt,ident,lifetime), matching steps a lightweight flat input instead of a parser:Parser::bump(noprev_token/spacing bookkeeping)ttfragments become buffer slices directly, with no black-box parser round trip throughparse_nonterminalidentandlifetimecaptures read the token in place;nonterminal_may_begin_withalready vets the token with the same predicates, so these are infallibleParser::bumpapplies, and invisible-delimiter filtering matches the cursor exactlyArms that capture AST fragments (
expr,ty,pat, ...) keep the parser-driven path, selected by a flag computed once per rule at definition time. An earlier variant of this PR materialized parsers for those from the flat path; it measured as a net loss on fragment-heavy crates (html5ever +3.9% at worst) and was dropped. The parser-driven input is now also built lazily per invocation, so all-flat definitions never construct aParserat all, and all-parser definitions pay nothing for the flat machinery.Three codegen details that each showed up as whole-crate regressions during development, now fixed and documented in the code:
parse_tt_innermust stay inlined into both matchers, the flat input caches its loaded start state so rewinding for the next arm is three copies rather than a rescan, andparse_tt_flatis inlined into its single caller to drop per-arm call overhead.Performance (ThinLTO stage2, jemalloc, instructions:u, full scenario, on top of the perf stack; base includes #82+#83+#84):
One caveat: several of the fixes here are inlining-shaped, and this config is plain stage2 without PGO; upstream dist builds make their own inlining choices, so these numbers want a rust-timer confirmation before any upstream attempt.
Correctness: stage2 bootstraps std and the compiler entirely through the flat matcher; full tests/ui failure set byte-identical to the pre-existing baseline (21301 passed, zero new diffs);
x check compilergreen standalone on this branch; a targeted smoke test covering ident/lifetime/tt captures, vis-adjacent shapes, separators, and thety-fragment>>-splitting case produces byte-identical-Zunpretty=expandedoutput vs the parser-driven matcher.