mbe: cache metavar resolution across repetition iterations [tt-muncher -2.2%] - #82
Draft
xmakro wants to merge 6 commits into
Draft
mbe: cache metavar resolution across repetition iterations [tt-muncher -2.2%]#82xmakro wants to merge 6 commits into
xmakro wants to merge 6 commits into
Conversation
The parser consumes tokens from a pre-flattened entry buffer: advancing is an index increment, lookahead is direct indexing, and parser snapshots clone two Arcs instead of a cursor stack.
Lexing a source file now produces the flat entry buffer without building a token tree first. The few cold callers that still need a TokenStream rebuild it from the buffer.
mbe transcription writes open/close entries and match indices through FlatSink. Captured tt fragments splice as buffer slices instead of being rebuilt as trees and re-flattened on every expansion.
Macro invocation arguments flow to the matcher as views of the source buffer, materialized to trees only on first tree-level access. Debug output is bounded to the viewed range and the pre-expansion KeywordIdents lint scans flat entries, so neither defeats the laziness by materializing every invocation in the crate.
The parser runs entirely on FlatTokenCursor, so the old tree-walking TokenCursor and its TokenTreeCursor are unreferenced. Delete both and retarget the stale break_up_float FIXME at the cursor that now exists.
Resolving a metavar occurrence during transcription looked up the interpolation map and walked every repetition level on every iteration of the innermost repetition frame, although everything but the innermost index is invariant for the frame's lifetime. Cache the resolution prefix per frame and apply only the innermost index per occurrence. tt-muncher check instructions drop by 2.2%.
This was referenced Aug 1, 2026
xmakro
marked this pull request as draft
August 2, 2026 21:26
xmakro
force-pushed
the
perf/flat-token-cursor
branch
3 times, most recently
from
August 6, 2026 10:42
e9ee005 to
98ddb58
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #57. During transcription, resolving a metavar occurrence called
lookup_cur_matched, which does an interpolation-map lookup plus a walk over every repetition level, once per occurrence per iteration of the innermost repetition frame. Everything except the innermost repetition index is invariant for the lifetime of that frame.This caches the resolution prefix (all levels but the innermost) in a small per-frame cache keyed by normalized ident, and applies only the innermost index per occurrence. The cache is cleared whenever the repetition stack changes depth, and capped at 8 entries so lookups stay a short linear scan even for pathological rule bodies.
Performance (ThinLTO stage2, jemalloc, instructions:u, full scenario, measured on top of the perf stack; A/B over 10 crates x Check/Debug):
Correctness: stage2 build green,
x check compilergreen on this branch standalone, tests/ui macros subset 510 passed / 0 failed.The logic is independent of the flat token buffer and would apply to upstream
transcribe.rsas well; it is stacked on #57 here because both touch the same function.