perf(codegen): encode generated data tables as compact blobs - #360
Conversation
Generated recognizers embedded their lexer DFA tables, packed parser ATN, and serialized lexer ATN as decimal Rust integer arrays. Large grammars made rustc lex hundreds of thousands of integer tokens and produced multi-megabyte single source lines (the ANTLRv4 lexer carried one 2.72 MB line) before the runtime ever saw the already-packed data. All three artifacts now travel as one string literal each: LEB128 varints (zigzag-mapped for i32 values) armored as canonical unpadded base64, wrapped at 120 columns with backslash-newline continuations so the literal stays a single token. The new antlr4_runtime::encoded module defines the container - magic "AR4B", format version, element kind, and checked element count - and rejects corrupt, truncated, overflowing, overlong, padded, and unsupported inputs with targeted EncodedBlobError diagnostics. The decoded integer streams are byte-identical to the previous arrays, so the inner tag-guarded DFA stream and PATN-versioned packed ATN formats and all recognizer behavior are unchanged; the lexer DFA keeps its documented fallback of recompiling from the ATN when the embedded data comes from another runtime version, and the serialized lexer ATN (which has no rebuild path) panics with the decode failure and grammar name instead. Decoding happens once inside the existing OnceLock first-use points via CompiledLexerDfa::from_encoded, ParserAtn::from_encoded (structural tail-call validation, owned words), and GrammarMetadata::new_with_encoded_atn / serialized_atn(); the latter loses `const`, which no caller needed. New generated source calls those new runtime entry points, so the generated-code API revision increments to 15; revisions 12-14 stay accepted because the integer-array constructors and validators all remain. Checked-in recognizers (ANTLRv4 via the stage0 fixed point, Rust, TOML, and the runtime's own XPath lexer) are regenerated in the new format. Measured against main@5caa7d3e interleaved on one machine (full record in docs/issue-357-encoded-blob-benchmark.md): lexer modules shrink ~37-39% (antlr_v4_lexer.rs 2.75 MB -> 1.68 MB, kotlin_lexer.rs 0.92 MB -> 0.57 MB), parser modules ~5%; debug cargo check of the g4-parser crate drops 1.00 s -> 0.67 s wall and 293 MB -> 199 MB peak RSS; release rlibs shrink up to 2.6% while the runtime rlib grows 0.9% for the decoder. First use pays one decode per artifact (+2 transient allocations; 4 ms for the 401k-word ANTLRv4 lexer DFA after switching the hot loop to a table-driven four-character chunk decoder), and steady-state parse timings are within +/-2% on the Kotlin parity snippets. Conformance sweep passes 357/357 and Kotlin parity stays byte-identical to the Python oracle. Closes #357
Copy/Paste DetectionFound 11 duplication(s) across 19 changed non-generated Rust file(s) (threshold: 100 tokens). Show duplicationsFound a 26 line (142 tokens) duplication in the following files:
atn.set_end_state(1, 4).expect("block end state");
atn.add_transition(0, ParserTransitionSpec::Epsilon { target: 1 })
.expect("transition");
atn.add_transition(1, ParserTransitionSpec::Epsilon { target: 2 })
.expect("transition");
atn.add_transition(1, ParserTransitionSpec::Epsilon { target: 3 })
.expect("transition");
atn.add_transition(
2,
ParserTransitionSpec::Atom {
target: 4,
label: 1,
},
)
.expect("transition");
atn.add_transition(
3,
ParserTransitionSpec::Atom {
target: 4,
label: 2,
},
)
.expect("transition");
atn.add_transition(4, ParserTransitionSpec::Epsilon { target: 5 })
.expect("transition");
atn.add_decision_state(1).expect("decision state");
```rust
---
Found a 26 line (125 tokens) duplication in the following files:
* Starting at line 794 of crates/antlr-rust-runtime/src/generated.rs
* Starting at line 971 of crates/antlr-rust-runtime/src/generated.rs
```rust
$input: $crate::char_stream::CharStream,
$hooks: $crate::parser::SemanticHooks,
{
pub fn metadata() -> &'static $crate::generated::GrammarMetadata {
$metadata()
}
/// Adds a listener for lexer diagnostics.
pub fn add_error_listener<T>(&mut self, listener: T)
where
T: for<'a> $crate::errors::ErrorListener<dyn $crate::recognizer::Recognizer + 'a>
+ ::core::marker::Send
+ 'static,
{
$crate::recognizer::Recognizer::add_error_listener(&mut self.$base, listener);
}
/// Removes every lexer error listener, including the default console listener.
pub fn remove_error_listeners(&mut self) {
$crate::recognizer::Recognizer::remove_error_listeners(&mut self.$base);
}
/// Routes every token through ATN interpretation instead of the compiled
/// lexer DFA, so the learned-DFA trace (`lexer_dfa_string`) observes each
/// match.
pub fn set_force_interpreted(&mut self, force_interpreted: bool) {Found a 25 line (115 tokens) duplication in the following files:
atn.add_state(AtnStateKind::BlockStart, Some(0))
.expect("state")
.index(),
1
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
2
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
3
);
assert_eq!(
atn.add_state(AtnStateKind::BlockEnd, Some(0))
.expect("state")
.index(),
4
);
assert_eq!(
atn.add_state(AtnStateKind::RuleStop, Some(0))
```rust
---
Found a 22 line (112 tokens) duplication in the following files:
* Starting at line 4286 of crates/antlr-rust-codegen/src/generator/tests.rs
* Starting at line 4360 of crates/antlr-rust-codegen/src/generator/tests.rs
```rust
fn plus_loop_atn() -> ParserAtn {
let mut atn = ParserAtnBuilder::new(2);
assert_eq!(
atn.add_state(AtnStateKind::RuleStart, Some(0))
.expect("state")
.index(),
0
);
assert_eq!(
atn.add_state(AtnStateKind::PlusBlockStart, Some(0))
.expect("state")
.index(),
1
);
assert_eq!(
atn.add_state(AtnStateKind::Basic, Some(0))
.expect("state")
.index(),
2
);
assert_eq!(
atn.add_state(AtnStateKind::BlockEnd, Some(0))Found a 27 line (110 tokens) duplication in the following files:
decision: 0,
alts: (1, 2),
track_alt_number: false,
allow_semantic_context: true,
force_context: false,
plus_loop: false,
fast_path: None,
body: &body,
},
0,
GeneratedStepRenderContext {
current_rule_index: 0,
embedded: None,
portable_locals: None,
decision_routing: DecisionRoutingRender::default(),
inline_action_statements: &BTreeMap::new(),
track_alt_numbers: false,
track_context_alt_numbers: false,
direct_generated_rule_calls: &[],
atn_preferred_rule_calls: &[],
adaptive_atn_preferred_rule_slots: &[],
adaptive_atn_probe_rule_slots: &[],
},
);
// The whole rendered star-loop captures the leading-predicate-to-exit-alt filtering.
insta::assert_snapshot!(
```rust
---
Found a 17 line (105 tokens) duplication in the following files:
* Starting at line 183 of crates/antlr-rust-runtime/src/generated.rs
* Starting at line 280 of crates/antlr-rust-runtime/src/generated.rs
```rust
fn __from_node_with_invocation_states(
node: $crate::RuleNodeView<'a>,
invocation_states: Option<Vec<isize>>,
) -> Self {
$(
let __default = <$attrs>::default();
let __attrs = node.generated_attrs::<$attrs>().unwrap_or(&__default);
)?
Self {
__node: __GeneratedRuleContext::Stored(node),
__invocation_states: invocation_states,
__state: std::marker::PhantomData,
$(
$($field: __attrs.$field.clone(),)+
)?
}
}Found a 25 line (104 tokens) duplication in the following files:
state: 1,
decision: 0,
track_alt_number: false,
allow_semantic_context: false,
force_context: false,
fast_path: None,
alts: &alts,
},
0,
GeneratedStepRenderContext {
current_rule_index: 0,
embedded: None,
portable_locals: None,
decision_routing: DecisionRoutingRender::default(),
inline_action_statements: &BTreeMap::new(),
track_alt_numbers: false,
track_context_alt_numbers: false,
direct_generated_rule_calls: &[],
atn_preferred_rule_calls: &[],
adaptive_atn_preferred_rule_slots: &[],
adaptive_atn_probe_rule_slots: &[],
},
);
assert!(rendered.contains("ll1_decision_prediction(atn(), 1)"));
```rust
---
Found a 15 line (104 tokens) duplication in the following files:
* Starting at line 932 of crates/antlr-rust-runtime/src/atn/parser_atn.rs
* Starting at line 1792 of crates/antlr-rust-runtime/src/atn/parser_atn.rs
```rust
impl ParserTransitionData<'_> {
pub const fn target(self) -> usize {
match self {
Self::Epsilon { target }
| Self::Atom { target, .. }
| Self::Range { target, .. }
| Self::Set { target, .. }
| Self::NotSet { target, .. }
| Self::Wildcard { target }
| Self::Rule { target, .. }
| Self::Predicate { target, .. }
| Self::Action { target, .. }
| Self::Precedence { target, .. } => target,
}
}Found a 28 line (102 tokens) duplication in the following files:
state: 1,
decision: 0,
track_alt_number: false,
allow_semantic_context: true,
force_context: false,
fast_path: None,
alts: &alts,
},
0,
GeneratedStepRenderContext {
current_rule_index: 0,
embedded: None,
portable_locals: None,
decision_routing: DecisionRoutingRender::default(),
inline_action_statements: &BTreeMap::new(),
track_alt_numbers: false,
track_context_alt_numbers: false,
direct_generated_rule_calls: &[],
atn_preferred_rule_calls: &[],
adaptive_atn_preferred_rule_slots: &[],
adaptive_atn_probe_rule_slots: &[],
},
);
// One decision renders into a fresh String; snapshot the whole emitted control flow (the
// semantic-context gate, both predicate probes, the alt rewrite, the no-viable fallback)
// instead of six positive probes plus one negative guard.
insta::assert_snapshot!(
```rust
---
Found a 14 line (102 tokens) duplication in the following files:
* Starting at line 1684 of crates/antlr-rust-runtime/src/atn/lexer_dfa.rs
* Starting at line 1702 of crates/antlr-rust-runtime/src/atn/lexer_dfa.rs
```rust
let id = next_token_compiled(lexer, &mut sink, atn, dfa).expect("test token should fit");
let token = sink.view(id).expect("emitted token should exist");
TokenSnapshot {
token_type: token.token_type(),
text: token.text_or_empty().to_owned(),
channel: token.channel(),
start: token.start(),
stop: token.stop(),
start_byte: token.start_byte(),
stop_byte: token.stop_byte(),
line: token.line(),
column: token.column(),
}
}Found a 16 line (101 tokens) duplication in the following files:
atn.set_loop_back_state(3, 4).expect("loop back state");
atn.add_transition(0, ParserTransitionSpec::Epsilon { target: 1 })
.expect("transition");
atn.add_transition(1, ParserTransitionSpec::Epsilon { target: 2 })
.expect("transition");
atn.add_transition(1, ParserTransitionSpec::Epsilon { target: 3 })
.expect("transition");
atn.add_transition(
2,
ParserTransitionSpec::Atom {
target: 4,
label: 1,
},
)
.expect("transition");
atn.add_transition(4, ParserTransitionSpec::Epsilon { target: 1 })
```rust |
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: ASSERTIVE Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (6)
Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review. 📝 WalkthroughWalkthroughThe runtime adds a versioned encoded-blob format with validation. The generator emits encoded lexer DFA, lexer ATN, and parser ATN data. Runtime compatibility advances to API revision 15 while retaining legacy data formats and revisions 12–14. ChangesEncoded static data
Estimated code review effort: 4 (Complex) | ~45 minutes Merge Risk: ⚪ Minimal · up to This change replaces generated integer tables with compact encoded blobs while preserving decoded data and runtime behavior; no actionable merge-blocking risk remains beyond normal checks and review. Sequence Diagram(s)sequenceDiagram
participant GeneratedLexer
participant CompiledLexerDfa
participant GeneratedParser
participant ParserAtn
participant EncodedBlob
GeneratedLexer->>CompiledLexerDfa: Decode encoded DFA text
CompiledLexerDfa->>EncodedBlob: decode_u32_values
CompiledLexerDfa-->>GeneratedLexer: Return DFA or use ATN fallback
GeneratedParser->>ParserAtn: Decode encoded parser ATN text
ParserAtn->>EncodedBlob: decode_u32_values
EncodedBlob-->>ParserAtn: Return packed ATN words or error
ParserAtn-->>GeneratedParser: Return validated ParserAtn
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Code review — PR #360 (encoded blobs)
View job run · branch |
Codecov Report❌ Patch coverage is 📢 Thoughts on this report? Let us know! |
…construction CI clippy (Rust 1.98) promotes the new chunks_exact_to_as_chunks lint to an error; the base64 encode/decode loops now use as_chunks::<3>() / as_chunks::<4>(), which also drops the by_ref iterator dance. The three ParserAtn constructors shared a verbatim construction tail; hoist it into one from_validated helper so from_static, from_owned, and from_encoded only differ in the storage and tail-call validation they choose.
Responds to the Claude Code Review pass on #360; all items were non-blocking. - GrammarMetadata::serialized_atn keeps its borrowed SerializedAtn<'_> return: encoded-blob metadata now decodes once into a OnceLock cached for the metadata's lifetime, so repeat calls are allocation-free and the corrupt-blob panic is confined to first use. The now-unneeded SerializedAtn::from_owned is removed before it ships in a release. - ParserAtn::from_encoded documents its trust assumption: structural tail-call validation is justified for generator-emitted text only; other data belongs in from_owned, which recomputes the markers. - CompiledLexerDfa::from_encoded keeps degrading silently on foreign-version word streams, but a container-level decode failure can only be a generation bug, so that path now debug_asserts while release behavior is unchanged. - The base64 remainder unreachable! message names as_chunks::<4>() after fd6857c switched away from chunks_exact. - docs/migration.md notes that serialized_atn lost `const` and that the revision gate does not cover hand-written const-context callers. - The lexer_dfa test helper adopts slice::fill, one of the four pre-existing clippy 1.98 findings (the rest follow separately).
CI's stable toolchain rolled to 1.98, whose new manual_slice_fill and chunks_exact_to_as_chunks lints promote to errors under the workspace's -D warnings gate. All findings predate #360; this commit converts the constant-size chunks_exact(2) walks in the Unicode property tables and their ICU oracle tests (the lexer_dfa test-helper fill landed with the review feedback commit).
|
@claude Thanks for the deep pass — the independent byte-identity verification of every checked-in blob is exactly the evidence this change needed. All six observations are addressed in the two commits just pushed:
On the clippy note: confirmed — all four findings pre-existed this PR under the 1.98 lints ( |
|
Claude finished @tinovyatkin's task in 1m 56s —— View job Verification of the pushed fixesI reviewed the two new commits (
All six items are correctly and completely addressed.
Clippy commit ( I also re-diffed Note on tooling: No further action needed from me on these six items. |
The previous commit fixed only the sites the review sampled; running the freshly rolled 1.98 stable toolchain locally (matching CI) surfaced the rest, all pre-existing: five more constant-size chunks_exact walks (escape sequences, lexer Unicode-property expansion, Unicode include/exclude and union range math), one interval-intersection helper that is now provably const, and a trait import 1.98 newly reports as unused. No behavior changes; the workspace clippy gate is green on 1.98.
📊 Source Code Metrics (this PR vs
|
| File | Cognitive | ABC | MI | Hotspot | Churn |
|---|---|---|---|---|---|
| crates/antlr-rust-runtime/src/atn/parser_atn.rs | 355 (main: 353) 🔴 | 1381.73 (main: 1376.59) 🔴 | 0 ⚪ | 3905 (main: 2824) 🔴 | 1.04 (main: 1.03) 🔴 |
| crates/antlr-rust-codegen/src/generator/tests.rs | 48 ⚪ | 2132.27 ⚪ | 0 ⚪ | 816 (main: 768) 🔴 | 1.15 (main: 1.15) 🔴 |
| crates/antlr-rust-codegen/src/grammar/atn/interp_test.rs | 249 ⚪ | 1681.53 ⚪ | 0 ⚪ | 1245 (main: 996) 🔴 | 1.00 (main: 1.00) 🔴 |
| crates/antlr-rust-runtime/src/atn/lexer_dfa.rs | 286 ⚪ | 1207.19 (main: 1202.96) 🔴 | 0 ⚪ | 5720 (main: 5148) 🔴 | 1.27 (main: 1.27) 🔴 |
| crates/antlr-rust-codegen/src/grammar/atn/lexer.rs | 239 ⚪ | 842.48 (main: 841.59) 🔴 | 0 ⚪ | 1673 (main: 1434) 🔴 | 1.00 (main: 1.00) 🔴 |
| crates/antlr-rust-codegen/src/grammar/unicode.rs | 92 ⚪ | 527.91 (main: 526.04) 🔴 | 0 ⚪ | 552 (main: 368) 🔴 | 1.02 (main: 1.01) 🔴 |
| crates/antlr-rust-runtime/src/generated.rs | 34 (main: 33) 🔴 | 260.56 (main: 247.28) 🔴 | 0 ⚪ | 578 (main: 495) 🔴 | 1.04 (main: 1.02) 🔴 |
| crates/antlr-rust-runtime/src/atn/serialized.rs | 217 ⚪ | 465.07 ⚪ | 0 ⚪ | 1953 (main: 1519) 🔴 | 1.15 (main: 1.13) 🔴 |
| crates/antlr-rust-codegen/src/parser/surface/support_abi.rs | 95 ⚪ | 397.32 (main: 402.32) 🟢 | 0 ⚪ | 570 (main: 475) 🔴 | 1.12 (main: 1.10) 🔴 |
| crates/antlr-rust-codegen/src/grammar/escape_sequence.rs | 16 ⚪ | 77.50 ⚪ | 12.66 (main: 12.70) 🔴 | 80 (main: 64) 🔴 | 1.04 (main: 1.02) 🔴 |
| crates/antlr-rust-codegen/src/parser/decision.rs | 145 ⚪ | 340.04 ⚪ | 0 ⚪ | 580 (main: 435) 🔴 | 1.00 (main: 1.00) 🔴 |
| crates/antlr-rust-runtime/src/encoded.rs | 44 🆕 | 220.78 🆕 | 1.63 🆕 | 132 🆕 | 1.04 🆕 |
| crates/antlr-rust-codegen/src/pipeline.rs | 13 ⚪ | 72.15 ⚪ | 13.13 (main: 13.13) 🔴 | 52 (main: 39) 🔴 | 1.02 (main: 1.01) 🔴 |
| crates/antlr-rust-codegen/src/rust_output.rs | 38 (main: 37) 🔴 | 110.93 (main: 99.25) 🔴 | 15.30 (main: 17.10) 🔴 | 114 (main: 74) 🔴 | 1 ⚪ |
| crates/antlr-rust-codegen/src/lexer/render.rs | 78 (main: 77) 🔴 | 187.10 (main: 191.77) 🟢 | 0 ⚪ | 546 (main: 462) 🔴 | 1.53 (main: 1.45) 🔴 |
| crates/antlr-rust-codegen/src/grammar/unicode_icu_tests.rs | 18 ⚪ | 69.66 (main: 68.73) 🔴 | 23.49 (main: 23.69) 🔴 | 90 (main: 72) 🔴 | 1.06 (main: 1.04) 🔴 |
| crates/antlr-rust-codegen/src/lexer/render_model.rs | 1 ⚪ | 35.71 (main: 40.80) 🟢 | 26.01 (main: 25.36) 🟢 | 6 (main: 5) 🔴 | 2.55 (main: 2.36) 🔴 |
| crates/antlr-rust-codegen/src/parser/render/mod.rs | 41 ⚪ | 123.86 (main: 123.04) 🔴 | 9.23 (main: 9.33) 🔴 | 287 (main: 246) 🔴 | 2.48 (main: 2.47) 🔴 |
| crates/antlr-rust-runtime/src/lib.rs | 3 ⚪ | 7.68 ⚪ | 29.07 (main: 29.26) 🔴 | 111 (main: 108) 🔴 | 2.34 (main: 2.31) 🔴 |
Generated by mehen v1.10.0 — the code quality watcher.

Implements #357.
What changed
Generated recognizers embedded their three static data tables — the ahead-of-time compiled lexer DFA, the packed parser ATN, and the serialized lexer ATN inside
GrammarMetadata— as decimal Rust integer arrays. For large grammars that meant rustc lexed hundreds of thousands of integer tokens and the ANTLRv4 lexer carried a single 2.72 MB source line. Each artifact now travels as one string literal: LEB128 varints (zigzag fori32) armored as canonical unpadded base64, wrapped at 120 columns with\-newline continuations so the literal stays a single token and one-file generation is preserved (no sidecars).Format (
antlr4_runtime::encoded)AR4B, version byte (1), element-kind byte (1 =u32, 2 = zigzagi32), LEB128 element count, then the values; minimal-varint enforcement, 32-bit overflow checks, count-vs-payload bound check before allocation, and a trailing-bytes check. Host-independent; no struct memory, endianness, or alignment dependence.EncodedBlobErrordiagnostics for every failure shape (snapshot-tested).The decoded integer streams are byte-identical to the previous arrays, so the inner tag-guarded DFA stream and
PATN-versioned packed ATN formats and all recognizer behavior are unchanged.Decode paths (first-use
OnceLockclosures, as before)CompiledLexerDfa::from_encoded→Option, preserving the documented rebuild-from-ATN fallback for foreign/corrupt data.ParserAtn::from_encoded→ structural tail-call validation likefrom_static, owned words; newParserAtnError::EncodedBlobvariant.GrammarMetadata::new_with_encoded_atn(const) +serialized_atn()decoding on first use into aOnceLockcached for the metadata's lifetime, so repeat calls are allocation-free borrows and the return type staysSerializedAtn<'_>; the method losesconst(no caller needed it; documented indocs/migration.md). Corrupt metadata panics with the grammar name and decode failure — the serialized ATN has no rebuild path.Generated-code API revision 14 → 15
New generated source calls the new runtime entry points. Revisions 12–14 stay accepted (integer-array constructors and validators all remain); macro arms, mismatch diagnostic, CLI compatibility test, and snapshots updated per the repo policy.
docs/migration.mdand the README compatibility section document the new representation.Regenerated recognizers
ANTLRv4 (via
update-stage0.sh --update, Stage 1 → Stage 2 fixed point proven), Rust, TOML (update-generated.sh --update;--checkpasses), and the runtime's own XPath lexer.Measurements (full record:
docs/issue-357-encoded-blob-benchmark.md)Interleaved against
main@5caa7d3eon one machine (Xeon 8275CL, Rust 1.97.1):antlr_v4_lexer.rsbytes / longest data linekotlin_lexer.rs/kotlin_parser.rsbytescargo check(wall / peak RSS)Steady-state parse performance is within the ±2% protected-runtime bound everywhere measured; the one-time first-use decode cost is reported above, not hidden.
Testing
encodedmodule unit tests: round-trips over varint width and sign boundaries, empty streams, determinism, zigzag reference pairs, and a snapshot of every decode diagnostic (invalid byte, rejected padding, truncated/non-canonical base64, truncated header, bad magic, unsupported version, kind mismatch, invalid/overlong count, truncated payload, truncated/overflowing/overlong values, trailing bytes).cargo clippy --locked --workspace --all-targets --all-features -- -D warningsclean; full workspace test suite passes.antlr4-python3-runtime.Reviewer notes
encodedmodule, five small runtime integration points, and three codegen emission sites.Post-review updates
ParserAtn::from_encoded,OnceLock-cached metadata decode, migration note for theconstremoval, stale message fix,debug_assert!on container-level lexer-DFA decode failures with unchanged release fallback); its follow-up pass confirms no further action needed, and it independently verified every checked-in blob decodes byte-identically tomain.chunks_exact_to_as_chunks,manual_slice_fill,missing_const_for_fn, one newly detected unused import) were pre-existing and are fixed here without suppressions so the clippy gate is green.