-
Notifications
You must be signed in to change notification settings - Fork 5.6k
Convert managed ilasm to use ANTLR actions instead of visiting the parse tree #132346
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Open
Changes from all commits
Commits
Show all changes
29 commits
Select commit
Hold shift + click to select a range
fd78232
Compile IL incrementally with ANTLR parser actions
jkoritzinsky e7381a8
Compile common IL instructions during parsing
jkoritzinsky 1bce930
Compile value instruction operands during parsing
jkoritzinsky 7e0f85a
Dispatch reference instructions from operand actions
jkoritzinsky 53a9bd6
Synthesize IL type and signature grammar values
jkoritzinsky 76da971
Synthesize IL marshalling grammar values
jkoritzinsky c9137e8
Compile method directives and exception regions during parsing
jkoritzinsky 8caacdb
Compile IL class members during parsing
jkoritzinsky 1ebe234
Compile IL declarations during parsing
jkoritzinsky 2d69195
Synthesize custom attributes and initializers
jkoritzinsky 24ad439
Compile shared IL directives during parsing
jkoritzinsky 9cd2e20
Compile assembly and manifest directives during parsing
jkoritzinsky 871d8f0
Stop generating ANTLR visitors for ILAssembler
jkoritzinsky 9511763
Remove obsolete parser-action scaffolding
jkoritzinsky 91a0662
Use typed semantic values in the IL parser
jkoritzinsky af64e3e
Use ASCII punctuation in ILAssembler comments
jkoritzinsky 37a1669
Fixes from rebasing
jkoritzinsky 4155e94
Fix action-model export diagnostics
jkoritzinsky 8e27b4a
Fix grammar action scope recovery
jkoritzinsky 0ff1f1f
Address remaining grammar action review feedback
jkoritzinsky 5127ab5
Re-enable ARM managed ilasm round trips
jkoritzinsky 2b1be92
Re-enable x86 managed ilasm round trips
jkoritzinsky ceaeef8
Re-enable HugeField1 managed ilasm round trip
jkoritzinsky 26238ca
Remove obsolete action localloc tracking
jkoritzinsky d92053e
Re-enable additional huge managed ilasm round trips
jkoritzinsky bc105fc
Address ILAssembler action review feedback
jkoritzinsky f2741b3
Preserve PDB language per document
jkoritzinsky 9eef034
Make parser token recovery non-throwing
jkoritzinsky edca868
Resolve typedef aliases in parser actions
jkoritzinsky File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -1,25 +1,112 @@ | ||
| # ILAssembler Build Workflow | ||
| # ILAssembler | ||
|
|
||
| This directory contains the ILAssembler tool and its build instructions. | ||
| ILAssembler compiles declarations while ANTLR parses the input. The parser uses | ||
| `UnbufferedTokenStream` with parse-tree construction disabled. Namespace, type, top-level, | ||
| class-member, method-body and shared-directive structure is action-driven. A complete document, | ||
| declaration body or method body is never retained. Rules such as `bytes` stream their content into | ||
| an accumulator instead of building a subtree at all. The generator emits neither listeners nor | ||
| visitors; parser actions own traversal. | ||
|
|
||
| ## Build Instructions | ||
| `GrammarActions` is a single `internal sealed partial class` split across | ||
| `src/ILAssembler/Actions/GrammarActions.*.cs`: | ||
|
|
||
| ### Regular Builds | ||
| For everyday development and regular builds, simply run: | ||
| | File | Contents | | ||
| | ---- | -------- | | ||
| | `GrammarActions.cs` | Per-document lifecycle. | | ||
| | `GrammarActions.BuildImage.cs` | PE and portable PDB construction. | | ||
| | `GrammarActions.Bytes.cs` | `bytearray` accumulation. | | ||
| | `GrammarActions.Conversions.cs` | Diagnostics and shared state. | | ||
| | `GrammarActions.CustomAttributes.Actions.cs` | Custom attribute descriptors, declarations and blob lists. | | ||
| | `GrammarActions.CustomAttributes.Sequences.cs` | Custom attribute scalar-array sequence synthesis. | | ||
| | `GrammarActions.CustomAttributes.Serialization.cs` | Serialized attribute values and field/parameter initializers. | | ||
| | `GrammarActions.Data.cs` | Streaming mapped-data declarations, labels and reference fixups. | | ||
| | `GrammarActions.Debug.cs` | Direct source-location, document and language directives. | | ||
| | `GrammarActions.Declarations.Actions.cs` | Direct top-level declarations and shared-directive dispatch. | | ||
| | `GrammarActions.Instructions.cs` | Tree-free value instruction and method-item actions. | | ||
| | `GrammarActions.Instructions.References.cs` | Reference and signature instruction actions. | | ||
| | `GrammarActions.Literals.cs` | Literals, names and strings. | | ||
| | `GrammarActions.Manifest.Assembly.cs` | Assembly definitions, identity, keys, security and attributes. | | ||
| | `GrammarActions.Manifest.ExportedTypes.cs` | Exported-type headers, implementations and attributes. | | ||
| | `GrammarActions.Manifest.Files.cs` | Assembly file declarations and entry points. | | ||
| | `GrammarActions.Manifest.References.cs` | Assembly references, identities, keys and hashes. | | ||
| | `GrammarActions.Manifest.Resources.cs` | Embedded and external manifest resources. | | ||
| | `GrammarActions.Manifest.Typedefs.cs` | Type, member and custom-attribute aliases. | | ||
| | `GrammarActions.Manifest.VTable.cs` | Vtable fixup declarations and flags. | | ||
| | `GrammarActions.Marshalling.Actions.cs` | Synthesized native type and marshalling descriptor actions. | | ||
| | `GrammarActions.Members.Class.cs` | Class directives, generic parameter annotations and method overrides. | | ||
| | `GrammarActions.Members.Fields.cs` | Field declarations, attributes, layout, constants, marshalling and RVA data. | | ||
| | `GrammarActions.Members.PropertiesEvents.cs` | Property and event headers, bodies and accessors. | | ||
| | `GrammarActions.MethodHeaders.cs` | Method definition and signature materialization. | | ||
| | `GrammarActions.MethodHeaders.Actions.cs` | Method header, attribute, P/Invoke and generic parser actions. | | ||
| | `GrammarActions.MethodHeaders.Generics.cs` | Generic parameter and constraint synthesis and materialization. | | ||
| | `GrammarActions.MethodBodies.cs` | Label validation and method-name parsing. | | ||
| | `GrammarActions.MethodBodies.Directives.cs` | Direct method-body directives and parameter ownership. | | ||
| | `GrammarActions.MethodBodies.ExceptionHandling.cs` | Lexical scopes and synthesized exception regions. | | ||
| | `GrammarActions.Security.cs` | Synthesized declarative-security values and permission sets. | | ||
| | `GrammarActions.Signatures.cs` | Member and type signature materialization helpers. | | ||
| | `GrammarActions.Signatures.Actions.cs` | Signature grammar actions and typed aggregation helpers. | | ||
| | `GrammarActions.Signatures.References.cs` | Member-reference synthesis and materialization. | | ||
| | `GrammarActions.Signatures.Types.cs` | Type-signature materialization and encoding. | | ||
| | `GrammarActions.Types.cs` | Namespace and type scope ownership and shared type conversion. | | ||
| | `GrammarActions.Types.Headers.cs` | Namespace and type-header materialization. | | ||
| | `GrammarActions.Types.Headers.Actions.cs` | Namespace, type attribute, base and interface parser actions. | | ||
| | `GrammarActions.Types.References.cs` | Type-name synthesis and resolution. | | ||
|
|
||
| The hand-written `public partial CILParser` semantic model is split by feature: | ||
|
|
||
| | File | Contents | | ||
| | ---- | -------- | | ||
| | `CILParser.SemanticValues.CustomAttributes.cs` | Custom attribute, serialization and initializer values. | | ||
| | `CILParser.SemanticValues.Declarations.cs` | Type/member headers and their context-owned builders. | | ||
| | `CILParser.SemanticValues.Manifest.cs` | Assembly, file, exported-type, resource and typedef values. | | ||
| | `CILParser.SemanticValues.Marshalling.cs` | Native, variant and marshalling values and builders. | | ||
| | `CILParser.SemanticValues.MethodBodies.cs` | Debug, data, security, exception and instruction values. | | ||
| | `CILParser.SemanticValues.Signatures.cs` | Managed types, signatures, names, owners and member references. | | ||
|
|
||
| These types are public because ANTLR emits public rule-context return and local fields, but they are | ||
| implementation details and are not a supported API contract. | ||
|
|
||
| ## Rules for grammar actions | ||
|
|
||
| Parser actions in `src/ILAssembler/gen/CIL.g4` must remain thin. They pass concrete child-rule | ||
| values to `GrammarActions`; mechanical assignments and typed builder additions may happen directly | ||
| in the grammar. Compilation orchestration belongs in the `GrammarActions` partial-class files. | ||
|
|
||
| `DocumentCompiler` disables parse-tree construction when it creates the parser, and no parser action | ||
| changes that setting. ANTLR generates neither listeners nor visitors. All semantics come from parser | ||
| actions and concrete synthesized values. Rule contexts own their typed builders and pass finalized | ||
| child values to their parents; parser semantic data is never erased to `object`. | ||
|
|
||
| All namespace, type-header, top-level, type, signature, reference, marshalling, class-member, | ||
| method-header, method-body directive, exception-handling, data, security, source, language, | ||
| assembly, manifest, vtable and typedef structure is action-driven. `scopeBlock` records offsets | ||
| under its context key without inspecting children. `BuildParseTree` remains disabled throughout. | ||
|
|
||
| Rule-local builders are finalized from that rule's `finally` clause when error recovery requires a | ||
| value. Semantic roots capture the initial syntax-error count in a context local instead of a global | ||
| frame stack. The remaining stacks model active compiler nesting: namespaces, types, declaration | ||
| owners and lexical method scopes. Their owning declaration or scope releases them from `finally` | ||
| because ANTLR skips `@after` actions after a syntax error. | ||
|
|
||
| Only parser actions process structural rules. There is no parse-tree walker or mode toggling. | ||
|
|
||
| ## Build | ||
|
|
||
| ``` | ||
| ./dotnet.sh build src/tools/ilasm/src/ILAssembler | ||
| ``` | ||
|
|
||
| ### Updating Generated Files | ||
| If you modify any `.g4` grammar files (rare), you must regenerate the parser and related files: | ||
| On Windows, use `.\dotnet.cmd` instead of `./dotnet.sh`. | ||
|
|
||
| ## Updating generated parser files | ||
|
|
||
| After modifying `CIL.g4`, regenerate the checked-in ANTLR output before building ILAssembler: | ||
|
|
||
| ``` | ||
| ./dotnet.sh build src/tools/ilasm/src/ILAssembler/gen | ||
| ./dotnet.sh build src/tools/ilasm/src/ILAssembler | ||
| ``` | ||
|
|
||
| This will update the generated files before building the main project. | ||
|
|
||
| --- | ||
|
|
||
| For more details, see the main repository README or contact the maintainers. | ||
| Do not edit generated `CIL*.cs` or `.interp` files manually. | ||
| Regeneration produces `CILLexer.cs` and `CILParser.cs`; it does not produce visitor or listener | ||
| types. | ||
Oops, something went wrong.
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Is it intentional that
genwas dropped here?