Skip to content

perf(codegen): encode generated data tables as compact blobs - #360

Merged
tinovyatkin merged 5 commits into
mainfrom
issue-357-encoded-blobs
Aug 21, 2026
Merged

perf(codegen): encode generated data tables as compact blobs#360
tinovyatkin merged 5 commits into
mainfrom
issue-357-encoded-blobs

Conversation

@tinovyatkin

@tinovyatkin tinovyatkin commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Implements #357.

What changed

Generated recognizers embedded their three static data tables — the ahead-of-time compiled lexer DFA, the packed parser ATN, and the serialized lexer ATN inside GrammarMetadata — as decimal Rust integer arrays. For large grammars that meant rustc lexed hundreds of thousands of integer tokens and the ANTLRv4 lexer carried a single 2.72 MB source line. Each artifact now travels as one string literal: LEB128 varints (zigzag for i32) armored as canonical unpadded base64, wrapped at 120 columns with \-newline continuations so the literal stays a single token and one-file generation is preserved (no sidecars).

Format (antlr4_runtime::encoded)

  • Text layer: canonical unpadded RFC 4648 base64; padding, whitespace, foreign bytes, and non-zero trailing bits rejected — every byte payload has exactly one valid encoding.
  • Byte layer: magic AR4B, version byte (1), element-kind byte (1 = u32, 2 = zigzag i32), LEB128 element count, then the values; minimal-varint enforcement, 32-bit overflow checks, count-vs-payload bound check before allocation, and a trailing-bytes check. Host-independent; no struct memory, endianness, or alignment dependence.
  • Targeted EncodedBlobError diagnostics for every failure shape (snapshot-tested).

The decoded integer streams are byte-identical to the previous arrays, so the inner tag-guarded DFA stream and PATN-versioned packed ATN formats and all recognizer behavior are unchanged.

Decode paths (first-use OnceLock closures, as before)

  • CompiledLexerDfa::from_encodedOption, preserving the documented rebuild-from-ATN fallback for foreign/corrupt data.
  • ParserAtn::from_encoded → structural tail-call validation like from_static, owned words; new ParserAtnError::EncodedBlob variant.
  • GrammarMetadata::new_with_encoded_atn (const) + serialized_atn() decoding on first use into a OnceLock cached for the metadata's lifetime, so repeat calls are allocation-free borrows and the return type stays SerializedAtn<'_>; the method loses const (no caller needed it; documented in docs/migration.md). Corrupt metadata panics with the grammar name and decode failure — the serialized ATN has no rebuild path.

Generated-code API revision 14 → 15

New generated source calls the new runtime entry points. Revisions 12–14 stay accepted (integer-array constructors and validators all remain); macro arms, mismatch diagnostic, CLI compatibility test, and snapshots updated per the repo policy. docs/migration.md and the README compatibility section document the new representation.

Regenerated recognizers

ANTLRv4 (via update-stage0.sh --update, Stage 1 → Stage 2 fixed point proven), Rust, TOML (update-generated.sh --update; --check passes), and the runtime's own XPath lexer.

Measurements (full record: docs/issue-357-encoded-blob-benchmark.md)

Interleaved against main@5caa7d3e on one machine (Xeon 8275CL, Rust 1.97.1):

Metric Baseline This PR
antlr_v4_lexer.rs bytes / longest data line 2,749,090 / 2.72 MB 1,683,997 (-38.7%) / ≤120 cols
ANTLRv4 lexer DFA payload tokens ~802,281 1
kotlin_lexer.rs / kotlin_parser.rs bytes 921,517 / 1,702,212 566,068 / 1,607,071
g4-parser debug cargo check (wall / peak RSS) 0.99–1.01 s / 289–293 MB 0.66–0.68 s / 198–199 MB
g4-parser release build (wall / peak RSS) 6.90 s / 458 MB 6.08–6.13 s / 360–361 MB
g4-parser rlib / kotlin dumper binary 10,774,386 / 3,512,120 10,493,968 (-2.6%) / 3,429,432 (-2.4%)
First use, ANTLRv4 (largest artifact set) +1 decode per artifact, +2 transient allocations; 4 ms for the 401k-word lexer DFA
Cold first parse (ANTLRv4 / TOML) 4.2–5.2 ms / 1.3–1.4 ms 8.2–8.9 ms / 1.6–1.7 ms
Warm parse median (ANTLRv4 / TOML) 878–882 µs / 518–524 µs 790–792 µs / 518–526 µs
Kotlin parity snippets, parse-only min 0.200–1.150 ms within ±2% on every snippet

Steady-state parse performance is within the ±2% protected-runtime bound everywhere measured; the one-time first-use decode cost is reported above, not hidden.

Testing

  • New encoded module unit tests: round-trips over varint width and sign boundaries, empty streams, determinism, zigzag reference pairs, and a snapshot of every decode diagnostic (invalid byte, rejected padding, truncated/non-canonical base64, truncated header, bad magic, unsupported version, kind mismatch, invalid/overlong count, truncated payload, truncated/overflowing/overlong values, trailing bytes).
  • cargo clippy --locked --workspace --all-targets --all-features -- -D warnings clean; full workspace test suite passes.
  • ANTLR runtime-testsuite conformance sweep: 357 passed, 0 failed (run twice: after the format change and again after the table-driven base64 decoder optimization).
  • Kotlin parity harness: parse trees byte-identical to antlr4-python3-runtime.

Reviewer notes

  • The generated-module diff is dominated by the regenerated recognizers; the hand-written change surface is the runtime encoded module, five small runtime integration points, and three codegen emission sites.
  • The base64 hot loop is table-driven and processes four characters per iteration; the initial per-character bit-accumulator version doubled the large-grammar decode cost and was replaced before landing (measurement doc shows the final numbers).

Post-review updates

  • All six non-blocking items from the Claude Code Review pass are addressed (trust-assumption doc on ParserAtn::from_encoded, OnceLock-cached metadata decode, migration note for the const removal, stale message fix, debug_assert! on container-level lexer-DFA decode failures with unchanged release fallback); its follow-up pass confirms no further action needed, and it independently verified every checked-in blob decodes byte-identically to main.
  • CI's stable toolchain rolled to Rust 1.98 mid-review; all new-lint findings (chunks_exact_to_as_chunks, manual_slice_fill, missing_const_for_fn, one newly detected unused import) were pre-existing and are fixed here without suppressions so the clippy gate is green.

Generated recognizers embedded their lexer DFA tables, packed parser
ATN, and serialized lexer ATN as decimal Rust integer arrays. Large
grammars made rustc lex hundreds of thousands of integer tokens and
produced multi-megabyte single source lines (the ANTLRv4 lexer carried
one 2.72 MB line) before the runtime ever saw the already-packed data.

All three artifacts now travel as one string literal each: LEB128
varints (zigzag-mapped for i32 values) armored as canonical unpadded
base64, wrapped at 120 columns with backslash-newline continuations so
the literal stays a single token. The new antlr4_runtime::encoded
module defines the container - magic "AR4B", format version, element
kind, and checked element count - and rejects corrupt, truncated,
overflowing, overlong, padded, and unsupported inputs with targeted
EncodedBlobError diagnostics. The decoded integer streams are
byte-identical to the previous arrays, so the inner tag-guarded DFA
stream and PATN-versioned packed ATN formats and all recognizer
behavior are unchanged; the lexer DFA keeps its documented fallback of
recompiling from the ATN when the embedded data comes from another
runtime version, and the serialized lexer ATN (which has no rebuild
path) panics with the decode failure and grammar name instead.

Decoding happens once inside the existing OnceLock first-use points via
CompiledLexerDfa::from_encoded, ParserAtn::from_encoded (structural
tail-call validation, owned words), and
GrammarMetadata::new_with_encoded_atn / serialized_atn(); the latter
loses `const`, which no caller needed. New generated source calls those
new runtime entry points, so the generated-code API revision increments
to 15; revisions 12-14 stay accepted because the integer-array
constructors and validators all remain. Checked-in recognizers
(ANTLRv4 via the stage0 fixed point, Rust, TOML, and the runtime's own
XPath lexer) are regenerated in the new format.

Measured against main@5caa7d3e interleaved on one machine (full record
in docs/issue-357-encoded-blob-benchmark.md): lexer modules shrink
~37-39% (antlr_v4_lexer.rs 2.75 MB -> 1.68 MB, kotlin_lexer.rs 0.92 MB
-> 0.57 MB), parser modules ~5%; debug cargo check of the g4-parser
crate drops 1.00 s -> 0.67 s wall and 293 MB -> 199 MB peak RSS;
release rlibs shrink up to 2.6% while the runtime rlib grows 0.9% for
the decoder. First use pays one decode per artifact (+2 transient
allocations; 4 ms for the 401k-word ANTLRv4 lexer DFA after switching
the hot loop to a table-driven four-character chunk decoder), and
steady-state parse timings are within +/-2% on the Kotlin parity
snippets. Conformance sweep passes 357/357 and Kotlin parity stays
byte-identical to the Python oracle.

Closes #357
@github-actions

github-actions Bot commented Aug 20, 2026

Copy link
Copy Markdown

Copy/Paste Detection

Found 11 duplication(s) across 19 changed non-generated Rust file(s) (threshold: 100 tokens).

Show duplications

Found a 26 line (142 tokens) duplication in the following files:

  • Starting at line 4188 of crates/antlr-rust-codegen/src/generator/tests.rs
  • Starting at line 4411 of crates/antlr-rust-codegen/src/generator/tests.rs
    atn.set_end_state(1, 4).expect("block end state");
    atn.add_transition(0, ParserTransitionSpec::Epsilon { target: 1 })
        .expect("transition");
    atn.add_transition(1, ParserTransitionSpec::Epsilon { target: 2 })
        .expect("transition");
    atn.add_transition(1, ParserTransitionSpec::Epsilon { target: 3 })
        .expect("transition");
    atn.add_transition(
        2,
        ParserTransitionSpec::Atom {
            target: 4,
            label: 1,
        },
    )
    .expect("transition");
    atn.add_transition(
        3,
        ParserTransitionSpec::Atom {
            target: 4,
            label: 2,
        },
    )
    .expect("transition");
    atn.add_transition(4, ParserTransitionSpec::Epsilon { target: 5 })
        .expect("transition");
    atn.add_decision_state(1).expect("decision state");
```rust

---

Found a 26 line (125 tokens) duplication in the following files:
* Starting at line 794 of crates/antlr-rust-runtime/src/generated.rs
* Starting at line 971 of crates/antlr-rust-runtime/src/generated.rs

```rust
            $input: $crate::char_stream::CharStream,
            $hooks: $crate::parser::SemanticHooks,
        {
            pub fn metadata() -> &'static $crate::generated::GrammarMetadata {
                $metadata()
            }

            /// Adds a listener for lexer diagnostics.
            pub fn add_error_listener<T>(&mut self, listener: T)
            where
                T: for<'a> $crate::errors::ErrorListener<dyn $crate::recognizer::Recognizer + 'a>
                    + ::core::marker::Send
                    + 'static,
            {
                $crate::recognizer::Recognizer::add_error_listener(&mut self.$base, listener);
            }

            /// Removes every lexer error listener, including the default console listener.
            pub fn remove_error_listeners(&mut self) {
                $crate::recognizer::Recognizer::remove_error_listeners(&mut self.$base);
            }

            /// Routes every token through ATN interpretation instead of the compiled
            /// lexer DFA, so the learned-DFA trace (`lexer_dfa_string`) observes each
            /// match.
            pub fn set_force_interpreted(&mut self, force_interpreted: bool) {

Found a 25 line (115 tokens) duplication in the following files:

  • Starting at line 4159 of crates/antlr-rust-codegen/src/generator/tests.rs
  • Starting at line 4369 of crates/antlr-rust-codegen/src/generator/tests.rs
        atn.add_state(AtnStateKind::BlockStart, Some(0))
            .expect("state")
            .index(),
        1
    );
    assert_eq!(
        atn.add_state(AtnStateKind::Basic, Some(0))
            .expect("state")
            .index(),
        2
    );
    assert_eq!(
        atn.add_state(AtnStateKind::Basic, Some(0))
            .expect("state")
            .index(),
        3
    );
    assert_eq!(
        atn.add_state(AtnStateKind::BlockEnd, Some(0))
            .expect("state")
            .index(),
        4
    );
    assert_eq!(
        atn.add_state(AtnStateKind::RuleStop, Some(0))
```rust

---

Found a 22 line (112 tokens) duplication in the following files:
* Starting at line 4286 of crates/antlr-rust-codegen/src/generator/tests.rs
* Starting at line 4360 of crates/antlr-rust-codegen/src/generator/tests.rs

```rust
fn plus_loop_atn() -> ParserAtn {
    let mut atn = ParserAtnBuilder::new(2);
    assert_eq!(
        atn.add_state(AtnStateKind::RuleStart, Some(0))
            .expect("state")
            .index(),
        0
    );
    assert_eq!(
        atn.add_state(AtnStateKind::PlusBlockStart, Some(0))
            .expect("state")
            .index(),
        1
    );
    assert_eq!(
        atn.add_state(AtnStateKind::Basic, Some(0))
            .expect("state")
            .index(),
        2
    );
    assert_eq!(
        atn.add_state(AtnStateKind::BlockEnd, Some(0))

Found a 27 line (110 tokens) duplication in the following files:

  • Starting at line 3513 of crates/antlr-rust-codegen/src/generator/tests.rs
  • Starting at line 3625 of crates/antlr-rust-codegen/src/generator/tests.rs
            decision: 0,
            alts: (1, 2),
            track_alt_number: false,
            allow_semantic_context: true,
            force_context: false,
            plus_loop: false,
            fast_path: None,
            body: &body,
        },
        0,
        GeneratedStepRenderContext {
            current_rule_index: 0,
            embedded: None,
            portable_locals: None,
            decision_routing: DecisionRoutingRender::default(),
            inline_action_statements: &BTreeMap::new(),
            track_alt_numbers: false,
            track_context_alt_numbers: false,
            direct_generated_rule_calls: &[],
            atn_preferred_rule_calls: &[],
            adaptive_atn_preferred_rule_slots: &[],
            adaptive_atn_probe_rule_slots: &[],
        },
    );

    // The whole rendered star-loop captures the leading-predicate-to-exit-alt filtering.
    insta::assert_snapshot!(
```rust

---

Found a 17 line (105 tokens) duplication in the following files:
* Starting at line 183 of crates/antlr-rust-runtime/src/generated.rs
* Starting at line 280 of crates/antlr-rust-runtime/src/generated.rs

```rust
            fn __from_node_with_invocation_states(
                node: $crate::RuleNodeView<'a>,
                invocation_states: Option<Vec<isize>>,
            ) -> Self {
                $(
                    let __default = <$attrs>::default();
                    let __attrs = node.generated_attrs::<$attrs>().unwrap_or(&__default);
                )?
                Self {
                    __node: __GeneratedRuleContext::Stored(node),
                    __invocation_states: invocation_states,
                    __state: std::marker::PhantomData,
                    $(
                        $($field: __attrs.$field.clone(),)+
                    )?
                }
            }

Found a 25 line (104 tokens) duplication in the following files:

  • Starting at line 3253 of crates/antlr-rust-codegen/src/generator/tests.rs
  • Starting at line 3420 of crates/antlr-rust-codegen/src/generator/tests.rs
            state: 1,
            decision: 0,
            track_alt_number: false,
            allow_semantic_context: false,
            force_context: false,
            fast_path: None,
            alts: &alts,
        },
        0,
        GeneratedStepRenderContext {
            current_rule_index: 0,
            embedded: None,
            portable_locals: None,
            decision_routing: DecisionRoutingRender::default(),
            inline_action_statements: &BTreeMap::new(),
            track_alt_numbers: false,
            track_context_alt_numbers: false,
            direct_generated_rule_calls: &[],
            atn_preferred_rule_calls: &[],
            adaptive_atn_preferred_rule_slots: &[],
            adaptive_atn_probe_rule_slots: &[],
        },
    );

    assert!(rendered.contains("ll1_decision_prediction(atn(), 1)"));
```rust

---

Found a 15 line (104 tokens) duplication in the following files:
* Starting at line 932 of crates/antlr-rust-runtime/src/atn/parser_atn.rs
* Starting at line 1792 of crates/antlr-rust-runtime/src/atn/parser_atn.rs

```rust
impl ParserTransitionData<'_> {
    pub const fn target(self) -> usize {
        match self {
            Self::Epsilon { target }
            | Self::Atom { target, .. }
            | Self::Range { target, .. }
            | Self::Set { target, .. }
            | Self::NotSet { target, .. }
            | Self::Wildcard { target }
            | Self::Rule { target, .. }
            | Self::Predicate { target, .. }
            | Self::Action { target, .. }
            | Self::Precedence { target, .. } => target,
        }
    }

Found a 28 line (102 tokens) duplication in the following files:

  • Starting at line 3306 of crates/antlr-rust-codegen/src/generator/tests.rs
  • Starting at line 3466 of crates/antlr-rust-codegen/src/generator/tests.rs
            state: 1,
            decision: 0,
            track_alt_number: false,
            allow_semantic_context: true,
            force_context: false,
            fast_path: None,
            alts: &alts,
        },
        0,
        GeneratedStepRenderContext {
            current_rule_index: 0,
            embedded: None,
            portable_locals: None,
            decision_routing: DecisionRoutingRender::default(),
            inline_action_statements: &BTreeMap::new(),
            track_alt_numbers: false,
            track_context_alt_numbers: false,
            direct_generated_rule_calls: &[],
            atn_preferred_rule_calls: &[],
            adaptive_atn_preferred_rule_slots: &[],
            adaptive_atn_probe_rule_slots: &[],
        },
    );

    // One decision renders into a fresh String; snapshot the whole emitted control flow (the
    // semantic-context gate, both predicate probes, the alt rewrite, the no-viable fallback)
    // instead of six positive probes plus one negative guard.
    insta::assert_snapshot!(
```rust

---

Found a 14 line (102 tokens) duplication in the following files:
* Starting at line 1684 of crates/antlr-rust-runtime/src/atn/lexer_dfa.rs
* Starting at line 1702 of crates/antlr-rust-runtime/src/atn/lexer_dfa.rs

```rust
        let id = next_token_compiled(lexer, &mut sink, atn, dfa).expect("test token should fit");
        let token = sink.view(id).expect("emitted token should exist");
        TokenSnapshot {
            token_type: token.token_type(),
            text: token.text_or_empty().to_owned(),
            channel: token.channel(),
            start: token.start(),
            stop: token.stop(),
            start_byte: token.start_byte(),
            stop_byte: token.stop_byte(),
            line: token.line(),
            column: token.column(),
        }
    }

Found a 16 line (101 tokens) duplication in the following files:

  • Starting at line 4259 of crates/antlr-rust-codegen/src/generator/tests.rs
  • Starting at line 4411 of crates/antlr-rust-codegen/src/generator/tests.rs
    atn.set_loop_back_state(3, 4).expect("loop back state");
    atn.add_transition(0, ParserTransitionSpec::Epsilon { target: 1 })
        .expect("transition");
    atn.add_transition(1, ParserTransitionSpec::Epsilon { target: 2 })
        .expect("transition");
    atn.add_transition(1, ParserTransitionSpec::Epsilon { target: 3 })
        .expect("transition");
    atn.add_transition(
        2,
        ParserTransitionSpec::Atom {
            target: 4,
            label: 1,
        },
    )
    .expect("transition");
    atn.add_transition(4, ParserTransitionSpec::Epsilon { target: 1 })
```rust

@coderabbitai

coderabbitai Bot commented Aug 20, 2026

Copy link
Copy Markdown

Review Change Stack

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: ASSERTIVE

Plan: Pro Plus

Run ID: d78eaaec-529c-465d-89cd-78ae93498ade

📥 Commits

Reviewing files that changed from the base of the PR and between 0230ab6 and a0bc366.

⛔ Files ignored due to path filters (1)
  • docs/migration.md is excluded by !**/docs/**
📒 Files selected for processing (6)
  • crates/antlr-rust-codegen/src/grammar/unicode.rs
  • crates/antlr-rust-codegen/src/grammar/unicode_icu_tests.rs
  • crates/antlr-rust-runtime/src/atn/lexer_dfa.rs
  • crates/antlr-rust-runtime/src/atn/parser_atn.rs
  • crates/antlr-rust-runtime/src/encoded.rs
  • crates/antlr-rust-runtime/src/generated.rs

Included review availability: Your plan provides up to 1 included review per hour; 0 remain after this review.


📝 Walkthrough

Walkthrough

The runtime adds a versioned encoded-blob format with validation. The generator emits encoded lexer DFA, lexer ATN, and parser ATN data. Runtime compatibility advances to API revision 15 while retaining legacy data formats and revisions 12–14.

Changes

Encoded static data

Layer / File(s) Summary
Encoded blob format
crates/antlr-rust-runtime/src/encoded.rs
Adds canonical base64 encoding, AR4B headers, LEB128 values, zigzag encoding, decoding errors, and validation tests.
Runtime decoding and storage
crates/antlr-rust-runtime/src/atn/*, crates/antlr-rust-runtime/src/generated.rs
Adds encoded lexer DFA and parser ATN construction. GrammarMetadata decodes encoded ATN data lazily while preserving legacy integer-array data.
Encoded code generation
crates/antlr-rust-codegen/src/lexer/*, crates/antlr-rust-codegen/src/parser/*, crates/antlr-rust-codegen/src/rust_output.rs, crates/antlr-rust-codegen/src/pipeline.rs, crates/antlr-rust-codegen/src/generator/tests.rs
The generator encodes lexer and parser static data and renders segmented Rust string literals. Generated code uses the new decoding APIs.
API revision and compatibility updates
crates/antlr-rust-runtime/src/lib.rs, crates/antlr-rust-codegen/tests/antlr4_rust_gen_cli/cli.rs, README.md
Advances the code-generation API revision to 15. Compatibility diagnostics and documentation cover revisions 12–15 and legacy construction APIs.
Generated fixtures and helper maintenance
third_party/antlr-v4-grammar/self-hosted.sha256, crates/antlr-rust-codegen/src/grammar/*
Updates generated grammar checksums and replaces chunk-slice iteration with fixed-size range pairs in Unicode helpers.

Estimated code review effort: 4 (Complex) | ~45 minutes

Merge Risk: ⚪ Minimal · up to a0bc3

This change replaces generated integer tables with compact encoded blobs while preserving decoded data and runtime behavior; no actionable merge-blocking risk remains beyond normal checks and review.

Sequence Diagram(s)

sequenceDiagram
  participant GeneratedLexer
  participant CompiledLexerDfa
  participant GeneratedParser
  participant ParserAtn
  participant EncodedBlob
  GeneratedLexer->>CompiledLexerDfa: Decode encoded DFA text
  CompiledLexerDfa->>EncodedBlob: decode_u32_values
  CompiledLexerDfa-->>GeneratedLexer: Return DFA or use ATN fallback
  GeneratedParser->>ParserAtn: Decode encoded parser ATN text
  ParserAtn->>EncodedBlob: decode_u32_values
  EncodedBlob-->>ParserAtn: Return packed ATN words or error
  ParserAtn-->>GeneratedParser: Return validated ParserAtn
Loading
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed Docstring coverage is 84.31% which is sufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 51 functions across 15 files.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title clearly and concisely describes the main change: encoding generated data tables as compact blobs for code generation performance.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch issue-357-encoded-blobs

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@claude

claude Bot commented Aug 20, 2026

Copy link
Copy Markdown

Code review — PR #360 (encoded blobs)

Reviewing the new push (95158f3 clippy sweep) plus a full pass over the hand-written surface.

  • Gather context: diff origin/main...HEAD, identify hand-written vs generated changes
  • Read relevant CLAUDE.md guidance
  • Review new commit 95158f3 (clippy 1.98 sweep in codegen)
  • Parallel deep review: CLAUDE.md compliance ×2, bug hunt ×2
  • Validate each candidate finding independently
  • Post consolidated review

View job run · branch issue-357-encoded-blobs

@codspeed-hq

codspeed-hq Bot commented Aug 20, 2026

Copy link
Copy Markdown
Contributor

Merging this PR will not alter performance

✅ 17 untouched benchmarks


Comparing issue-357-encoded-blobs (95158f3) with main (5caa7d3)

Open in CodSpeed

@codecov

codecov Bot commented Aug 20, 2026

Copy link
Copy Markdown

…construction

CI clippy (Rust 1.98) promotes the new chunks_exact_to_as_chunks lint to
an error; the base64 encode/decode loops now use as_chunks::<3>() /
as_chunks::<4>(), which also drops the by_ref iterator dance. The three
ParserAtn constructors shared a verbatim construction tail; hoist it
into one from_validated helper so from_static, from_owned, and
from_encoded only differ in the storage and tail-call validation they
choose.
Responds to the Claude Code Review pass on #360; all items were
non-blocking.

- GrammarMetadata::serialized_atn keeps its borrowed SerializedAtn<'_>
  return: encoded-blob metadata now decodes once into a OnceLock cached
  for the metadata's lifetime, so repeat calls are allocation-free and
  the corrupt-blob panic is confined to first use. The now-unneeded
  SerializedAtn::from_owned is removed before it ships in a release.
- ParserAtn::from_encoded documents its trust assumption: structural
  tail-call validation is justified for generator-emitted text only;
  other data belongs in from_owned, which recomputes the markers.
- CompiledLexerDfa::from_encoded keeps degrading silently on
  foreign-version word streams, but a container-level decode failure
  can only be a generation bug, so that path now debug_asserts while
  release behavior is unchanged.
- The base64 remainder unreachable! message names as_chunks::<4>()
  after fd6857c switched away from chunks_exact.
- docs/migration.md notes that serialized_atn lost `const` and that the
  revision gate does not cover hand-written const-context callers.
- The lexer_dfa test helper adopts slice::fill, one of the four
  pre-existing clippy 1.98 findings (the rest follow separately).
CI's stable toolchain rolled to 1.98, whose new manual_slice_fill and
chunks_exact_to_as_chunks lints promote to errors under the workspace's
-D warnings gate. All findings predate #360; this commit converts the constant-size
chunks_exact(2) walks in the Unicode property tables and their ICU
oracle tests (the lexer_dfa test-helper fill landed with the review
feedback commit).
@tinovyatkin

Copy link
Copy Markdown
Contributor Author

@claude Thanks for the deep pass — the independent byte-identity verification of every checked-in blob is exactly the evidence this change needed. All six observations are addressed in the two commits just pushed:

  1. ParserAtn::from_encoded trust assumption — documented explicitly: the doc comment now states the text must be generator output, that structural tail-call validation trusts the generator's marking, and routes non-generator data to from_owned (which recomputes the markers).

  2. serialized_atn() per-call decode + panic — took your suggestion: encoded-blob metadata now decodes once into a OnceLock<Vec<i32>> on GrammarMetadata, so repeat calls are allocation-free borrows and the corrupt-blob panic is confined to first use. This also restored the borrowed SerializedAtn<'_> return type, and the now-unneeded SerializedAtn::from_owned is removed before it ships in any release.

  3. Signature change outside the revision gatedocs/migration.md now carries an explicit paragraph: serialized_atn lost const, the revision check only gates generated source, and hand-written const-context callers must move the call to runtime. (The lifetime widening from item 2's original shape is gone — the return type is SerializedAtn<'_> again.)

  4. Stale unreachable! message — now names as_chunks::<4>().

  5. Silent-fallback blind spot — split the failure shapes as you suggested: a word stream from another runtime version still degrades silently (that drift is the documented, legitimate fallback), but a container-level EncodedBlobError can only be a generation bug and now trips debug_assert! while release behavior is unchanged.

  6. debug_assert!(encoded.is_ascii()) nit — leaving as-is per your read; it documents that the non-ASCII → split_at panic case was considered, and the in-module encoder is the only producer.

On the clippy note: confirmed — all four findings pre-existed this PR under the 1.98 lints (manual_slice_fill, chunks_exact_to_as_chunks), and CI's toolchain has already rolled, which is why the clippy gate failed here while main hasn't rebuilt yet. Fixed all four in this branch (slice::fill in the lexer_dfa test helper; as_chunks::<2>() in the Unicode property walks) so the gate is green without any lint suppressions.

@claude

claude Bot commented Aug 21, 2026

Copy link
Copy Markdown

Claude finished @tinovyatkin's task in 1m 56s —— View job


Verification of the pushed fixes

I reviewed the two new commits (c321a28 "address encoded-blob review feedback", a0bc366 "fix pre-existing findings of clippy 1.98") against each of the six prior observations.

All six items are correctly and completely addressed.

  1. from_encoded trust doc (atn/parser_atn.rs:254-264) — doc comment now states the text must be generator output and points non-generator callers at from_owned. Matches the request precisely.

  2. serialized_atn() caching (generated.rs) — GrammarMetadata gained a decoded_serialized_atn: OnceLock<Vec<i32>> field, populated via get_or_init inside serialized_atn(), and the return type stays the borrowed SerializedAtn<'_> (not 'static as in the original draft). Clone correctly re-seeds the OnceLock only when already populated (.get().cloned().map_or_else(OnceLock::new, OnceLock::from)), so cloning an unresolved instance doesn't force an eager decode. The now-dead SerializedAtn::from_owned was removed — confirmed no remaining callers anywhere (SerializedAtn::from_owned — zero matches) and the Cow import is still used elsewhere (from_chars), so nothing is orphaned. This exactly resolves the original "decode + allocate every call" concern.

  3. Signature-change docdocs/migration.md now has an explicit paragraph: serialized_atn lost const, decodes on first use and caches, and the revision gate doesn't cover hand-written const-context callers.

  4. Stale unreachable! (encoded.rs:383) — now reads "as_chunks::<4>() leaves at most three remainder bytes".

  5. debug_assert! split (atn/lexer_dfa.rs:599-608) — foreign-version word streams still degrade silently to None (unchanged, correct — that's expected drift, not a bug), while a blob-container decode failure now hits debug_assert!(false, "embedded lexer DFA blob failed to decode: {error}") before returning None, so release behavior is byte-for-byte unchanged. The let _ = error; after the assert correctly suppresses the release-mode unused-variable warning once the macro compiles away. I checked for clippy::assertions_on_constants risk on debug_assert!(false, …) — this exact pattern already exists twice in semir.rs (pre-existing, presumably clippy-clean), so it's not a new risk this PR introduces.

  6. debug_assert!(encoded.is_ascii()) nit — left as-is with the stated rationale; reasonable, agreed it's not worth churning.

Clippy commit (a0bc366) — both unicode.rs and unicode_icu_tests.rs convert their chunks_exact(2) walks to .as_chunks::<2>().0.iter() / .as_chunks::<2>().0, consistent with the as_chunks::<4>() pattern already used in encoded.rs from fd6857c. lexer_dfa.rs's test helper switched a manual fill loop to scalar.ascii_runs.fill(AsciiRun::None). All four match the lint names cited (manual_slice_fill, chunks_exact_to_as_chunks).

I also re-diffed docs/migration.md, serialized.rs, and encoded.rs directly to confirm the removed/added code matches what the commit message claims — no discrepancies found.

Note on tooling: cargo clippy/cargo test/rustc --version all required approval I couldn't grant in this run (only read-only git subcommands were available), so I verified the two commits by direct code/diff inspection rather than a fresh build. Everything traced through cleanly by inspection, but if you want a build-verified confirmation, a maintainer-approved CI run or local cargo test --locked --workspace --all-features + the clippy command from CLAUDE.md is worth doing before merge — the earlier review job did run these successfully against this same code shape.

No further action needed from me on these six items.
· branch issue-357-encoded-blobs

The previous commit fixed only the sites the review sampled; running
the freshly rolled 1.98 stable toolchain locally (matching CI) surfaced
the rest, all pre-existing: five more constant-size chunks_exact walks
(escape sequences, lexer Unicode-property expansion, Unicode
include/exclude and union range math), one interval-intersection helper
that is now provably const, and a trait import 1.98 newly reports as
unused. No behavior changes; the workspace clippy gate is green on
1.98.
@github-actions

Copy link
Copy Markdown

📊 Source Code Metrics (this PR vs main)

File Cognitive ABC MI Hotspot Churn
crates/antlr-rust-runtime/src/atn/parser_atn.rs 355 (main: 353) 🔴 1381.73 (main: 1376.59) 🔴 0 ⚪ 3905 (main: 2824) 🔴 1.04 (main: 1.03) 🔴
crates/antlr-rust-codegen/src/generator/tests.rs 48 ⚪ 2132.27 ⚪ 0 ⚪ 816 (main: 768) 🔴 1.15 (main: 1.15) 🔴
crates/antlr-rust-codegen/src/grammar/atn/interp_test.rs 249 ⚪ 1681.53 ⚪ 0 ⚪ 1245 (main: 996) 🔴 1.00 (main: 1.00) 🔴
crates/antlr-rust-runtime/src/atn/lexer_dfa.rs 286 ⚪ 1207.19 (main: 1202.96) 🔴 0 ⚪ 5720 (main: 5148) 🔴 1.27 (main: 1.27) 🔴
crates/antlr-rust-codegen/src/grammar/atn/lexer.rs 239 ⚪ 842.48 (main: 841.59) 🔴 0 ⚪ 1673 (main: 1434) 🔴 1.00 (main: 1.00) 🔴
crates/antlr-rust-codegen/src/grammar/unicode.rs 92 ⚪ 527.91 (main: 526.04) 🔴 0 ⚪ 552 (main: 368) 🔴 1.02 (main: 1.01) 🔴
crates/antlr-rust-runtime/src/generated.rs 34 (main: 33) 🔴 260.56 (main: 247.28) 🔴 0 ⚪ 578 (main: 495) 🔴 1.04 (main: 1.02) 🔴
crates/antlr-rust-runtime/src/atn/serialized.rs 217 ⚪ 465.07 ⚪ 0 ⚪ 1953 (main: 1519) 🔴 1.15 (main: 1.13) 🔴
crates/antlr-rust-codegen/src/parser/surface/support_abi.rs 95 ⚪ 397.32 (main: 402.32) 🟢 0 ⚪ 570 (main: 475) 🔴 1.12 (main: 1.10) 🔴
crates/antlr-rust-codegen/src/grammar/escape_sequence.rs 16 ⚪ 77.50 ⚪ 12.66 (main: 12.70) 🔴 80 (main: 64) 🔴 1.04 (main: 1.02) 🔴
crates/antlr-rust-codegen/src/parser/decision.rs 145 ⚪ 340.04 ⚪ 0 ⚪ 580 (main: 435) 🔴 1.00 (main: 1.00) 🔴
crates/antlr-rust-runtime/src/encoded.rs 44 🆕 220.78 🆕 1.63 🆕 132 🆕 1.04 🆕
crates/antlr-rust-codegen/src/pipeline.rs 13 ⚪ 72.15 ⚪ 13.13 (main: 13.13) 🔴 52 (main: 39) 🔴 1.02 (main: 1.01) 🔴
crates/antlr-rust-codegen/src/rust_output.rs 38 (main: 37) 🔴 110.93 (main: 99.25) 🔴 15.30 (main: 17.10) 🔴 114 (main: 74) 🔴 1 ⚪
crates/antlr-rust-codegen/src/lexer/render.rs 78 (main: 77) 🔴 187.10 (main: 191.77) 🟢 0 ⚪ 546 (main: 462) 🔴 1.53 (main: 1.45) 🔴
crates/antlr-rust-codegen/src/grammar/unicode_icu_tests.rs 18 ⚪ 69.66 (main: 68.73) 🔴 23.49 (main: 23.69) 🔴 90 (main: 72) 🔴 1.06 (main: 1.04) 🔴
crates/antlr-rust-codegen/src/lexer/render_model.rs 1 ⚪ 35.71 (main: 40.80) 🟢 26.01 (main: 25.36) 🟢 6 (main: 5) 🔴 2.55 (main: 2.36) 🔴
crates/antlr-rust-codegen/src/parser/render/mod.rs 41 ⚪ 123.86 (main: 123.04) 🔴 9.23 (main: 9.33) 🔴 287 (main: 246) 🔴 2.48 (main: 2.47) 🔴
crates/antlr-rust-runtime/src/lib.rs 3 ⚪ 7.68 ⚪ 29.07 (main: 29.26) 🔴 111 (main: 108) 🔴 2.34 (main: 2.31) 🔴

Generated by mehen v1.10.0 — the code quality watcher.

@tinovyatkin
tinovyatkin merged commit a056c71 into main Aug 21, 2026
14 checks passed
@tinovyatkin
tinovyatkin deleted the issue-357-encoded-blobs branch August 21, 2026 04:59
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant