diff --git a/CHANGELOG.md b/CHANGELOG.md index a8e6ef5..3dd6e63 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -4,6 +4,12 @@ All notable changes to this project will be documented in this file. The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/), and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html). +## [3.1.0] - 2026-07-29 +### Added +- Analysis passes: `ASTTransform::AbstractAnalysis`, the read-only counterpart of `AbstractTransformation`. Both leaves share the extracted traversal core `ASTTransform::AbstractProcessor` (sealed `process`, `process_node` hook, thunk descent, `TransformationHelper`) and differ only in what `run` returns — the rebuilt tree for transformations, the analysis instance (result readers) for analyses. +- Public parse seam: `ASTTransform::SourceParser` (`#parse(source, file_path:)` / `#parse_file(path)`) extracts the framework's Prism-backed parsing from `Transformer` so analysis-only consumers that never emit can instantiate it directly. `Transformer#build_ast` / `#build_ast_from_file` delegate to it (unchanged public behavior). +- Pass-taxonomy documentation on `AbstractTransformation`, `Transformation`, and the README: structural (`on_*` handlers, pattern matched anywhere), positional (node builders with the bare `run(node) → node` duck type, caller owns traversal), and sibling-annotation (marker statement + next sibling, matched in `process_node` where the child list is visible — `transform!` itself). + ## [3.0.0] - 2026-07-24 ### Added - Line-aligned emission: transformed code is emitted with every loc-carrying statement on its original source line, making backtraces, breakpoints, and debugger display correct by construction (`LineAlignedEmitter`). diff --git a/Gemfile.lock b/Gemfile.lock index 6a40419..327d529 100644 --- a/Gemfile.lock +++ b/Gemfile.lock @@ -1,7 +1,7 @@ PATH remote: . specs: - ast_transform (3.0.0) + ast_transform (3.1.0) parser (>= 3.3) prism (>= 1.5) unparser (>= 0.8) diff --git a/README.md b/README.md index a505849..e170df6 100644 --- a/README.md +++ b/README.md @@ -130,6 +130,16 @@ class MyTransformation < ASTTransformation::AbstractTransformation end ``` +#### Choosing a pass shape + +Passes come in three shapes, distinguished by how the rewrite site is found: + +* **Structural** — the pattern alone identifies the site. Subclass `AbstractTransformation` and write `on_*` handlers; the pass rewrites the pattern wherever it occurs in the tree. +* **Positional (node builders)** — the *caller* owns traversal and has already located the site. Implement a plain `run(node) → node` object (include `TransformationHelper` for the `s(...)` vocabulary); the builder maps a single node to its replacement subtree and never walks. Anything with that duck type composes with `Transformer` and other passes. +* **Sibling-annotation** — the pattern is a marker statement plus its NEXT sibling. Match in `process_node`, where a node's child list is visible: an `on_*` handler sees one node, never its siblings, and cannot delete itself from its parent. `ASTTransform::Transformation` (the `transform!` detector) is the canonical example. + +For read-only passes that harvest information instead of rewriting, see [Analysis passes](#analysis-passes). + #### Transformation discoverability ASTTransform automatically loads your transformations at compile time. As such, we expect your files to be located at a known path. @@ -162,6 +172,48 @@ In the above, `node#updated` allows updating the node, either its type or its ch The [ast gem](https://github.com/whitequark/ast) uses a pattern in which a Transformation may implement a method matching a node type, i.e. `on_class`, `on_send`, `on_lvar`, etc... This is very useful when transformations should process all nodes of this type. +### Analysis passes + +Not every pass rewrites. An analysis walks the tree and harvests information — `Parser::AST::Processor` has no read-only mode (every walk functionally rebuilds the tree), so an analysis is simply a walk whose rebuilt tree is discarded. Derive from `ASTTransform::AbstractAnalysis`: it shares the traversal engine with `AbstractTransformation` (including thunk descent and the `s(...)` matching vocabulary) and differs only in what `run` returns — the analysis instance, so callers chain result readers off the run. + +```ruby +require 'ast_transform/abstract_analysis' + +class TypeAliasRanges < ASTTransform::AbstractAnalysis + attr_reader :ranges + + def initialize + @ranges = [] + super + end + + def on_block(node) + send_node = node.children.first + @ranges << (node.loc.expression.first_line..node.loc.expression.last_line) if type_alias?(send_node) + super + end + + private + + def type_alias?(send_node) + send_node == s(:send, s(:const, nil, :T), :type_alias) + end +end +``` + +Harvest state in `on_*` handlers and always call `super` so traversal continues. Node equality ignores source locations, so `s(...)` patterns match structurally. + +Analysis-only consumers don't need a `Transformer`; instantiate the parsing seam directly (`Transformer` uses the same class internally) and keep the instance for as many parses as you need: + +```ruby +parser = ASTTransform::SourceParser.new + +parser.parse(source) # => Parser::AST::Node +parser.parse_file(file_path) # => Parser::AST::Node + +TypeAliasRanges.new.run(parser.parse(source)).ranges +``` + ### Line-aligned emission and the authoring contract ASTTransform owns text and lines; transform authors own semantics and execution order. The contract: diff --git a/lib/ast_transform.rb b/lib/ast_transform.rb index e07cf69..643030d 100644 --- a/lib/ast_transform.rb +++ b/lib/ast_transform.rb @@ -4,6 +4,7 @@ require "ast_transform/instruction_sequence" require "ast_transform/instruction_sequence/mixin" require "ast_transform/instruction_sequence/bootsnap_mixin" +require "ast_transform/source_parser" module ASTTransform DEFAULT_OUTPUT_PATH = Pathname.new("").join("tmp", "ast_transform").to_s diff --git a/lib/ast_transform/abstract_analysis.rb b/lib/ast_transform/abstract_analysis.rb new file mode 100644 index 0000000..63d7086 --- /dev/null +++ b/lib/ast_transform/abstract_analysis.rb @@ -0,0 +1,38 @@ +# frozen_string_literal: true + +require "ast_transform/abstract_processor" + +module ASTTransform + # Base class for read-only analysis passes: subclass, harvest state in +on_*+ handlers (always call +super+ so + # traversal continues), and expose results through readers. The walk still functionally rebuilds the tree — + # Parser::AST::Processor has no read-only mode — but +run+ discards the rebuilt tree, so handlers never need to + # care what they return. + # + # class SendCounter < ASTTransform::AbstractAnalysis + # attr_reader :count + # + # def initialize + # @count = 0 + # super + # end + # + # def on_send(node) + # @count += 1 + # super + # end + # end + # + # SendCounter.new.run(SourceParser.new.parse(source)).count + class AbstractAnalysis < AbstractProcessor + # Runs this analysis on +node+, discarding the rebuilt tree. + # Note: If you want to add one-time setup or result finalization, override this, then call super. + # + # @param node [Parser::AST::Node] The node to be analyzed. + # + # @return [ASTTransform::AbstractAnalysis] self, so callers can chain result readers off the run. + def run(node) + process(node) + self + end + end +end diff --git a/lib/ast_transform/abstract_processor.rb b/lib/ast_transform/abstract_processor.rb new file mode 100644 index 0000000..bea4c6d --- /dev/null +++ b/lib/ast_transform/abstract_processor.rb @@ -0,0 +1,39 @@ +# frozen_string_literal: true + +require "ast_transform/transformation_helper" + +module ASTTransform + # Shared traversal core for tree passes. Parser::AST::Processor is a rewriting walker — every visit functionally + # rebuilds the tree — so transformation and analysis share one engine and differ only in what they keep: + # AbstractTransformation's +run+ returns the rebuilt tree, AbstractAnalysis's +run+ discards it and returns the + # harvested results. Subclass one of those leaves rather than this class. + class AbstractProcessor < Parser::AST::Processor + include TransformationHelper + + # Used internally by Parser::AST::Processor to process each node. DO NOT OVERRIDE. + def process(node) + return node unless node.is_a?(Parser::AST::Node) + + process_node(node) + end + + # Thunks are framework-owned IR: descend into the body so passes that don't know about thunks still process the + # wrapped statements. Without this, Processor's handler_missing default would pass the node through opaquely, + # hiding the body from every later pass. The token (first child) is not a node and passes through untouched. + def on_ast_thunk(node) + node.updated(nil, [node.children[0], *process_all(node.children.drop(1))]) + end + + private + + # Processes the given +node+. + # Note: If you want to do processing on each node, override this. + # + # @param node [Parser::AST::Node] The node being visited. + # + # @return [Parser::AST::Node] The rebuilt node. + def process_node(node) + method(:process).super_method.call(node) + end + end +end diff --git a/lib/ast_transform/abstract_transformation.rb b/lib/ast_transform/abstract_transformation.rb index 56fd0c7..faf7b12 100644 --- a/lib/ast_transform/abstract_transformation.rb +++ b/lib/ast_transform/abstract_transformation.rb @@ -1,12 +1,24 @@ # frozen_string_literal: true -require "ast_transform/transformation_helper" +require "ast_transform/abstract_processor" module ASTTransform - class AbstractTransformation < Parser::AST::Processor - include TransformationHelper - - # Runs this transformation on +node+. + # Base class for structural transformations: subclass and write +on_*+ handlers to rewrite a pattern wherever it + # occurs in the tree. This is one of three authoring shapes — pick by how the rewrite site is found: + # + # - Structural (this class): the pattern alone identifies the site, so +on_*+ handlers rewrite it anywhere in + # the tree. + # - Positional (node builders): a plain object with +run(node) → node+, including only TransformationHelper. The + # caller owns traversal and applies the builder at a site it already located; the builder never walks. Anything + # with that duck type composes with Transformer and other passes — deriving from this class is just one way to + # implement it. + # - Sibling-annotation: the pattern is a marker statement plus its NEXT sibling, matched in +process_node+ where + # the child list is visible — an +on_*+ handler sees one node, never its siblings, and cannot delete itself + # from its parent. See ASTTransform::Transformation (the +transform!+ detector) for the canonical example. + # + # For read-only passes that harvest information instead of rewriting, see ASTTransform::AbstractAnalysis. + class AbstractTransformation < AbstractProcessor + # Runs this transformation on +node+ and returns the rebuilt tree. # Note: If you want to add one-time checks to the transformation, override this, then call super. # # @param node [Parser::AST::Node] The node to be transformed. @@ -15,32 +27,5 @@ class AbstractTransformation < Parser::AST::Processor def run(node) process(node) end - - # Used internally by Parser::AST::Processor to process each node. DO NOT OVERRIDE. - def process(node) - return node unless node.is_a?(Parser::AST::Node) - - process_node(node) - end - - # Thunks are framework-owned IR: descend into the body so passes that don't know about thunks still process the - # wrapped statements. Without this, Processor's handler_missing default would pass the node through opaquely, - # hiding the body from every later transformation. The token (first child) is not a node and passes through - # untouched. - def on_ast_thunk(node) - node.updated(nil, [node.children[0], *process_all(node.children.drop(1))]) - end - - private - - # Processes the given +node+. - # Note: If you want to do processing on each node, override this. - # - # @param node [Parser::AST::Node] The node to be transformed. - # - # @return [Parser::AST::Node] The transformed node. - def process_node(node) - method(:process).super_method.call(node) - end end end diff --git a/lib/ast_transform/kwargs_builder.rb b/lib/ast_transform/kwargs_builder.rb index a0a02be..cb54b70 100644 --- a/lib/ast_transform/kwargs_builder.rb +++ b/lib/ast_transform/kwargs_builder.rb @@ -3,22 +3,21 @@ require "prism/translation/parser" module ASTTransform - # Extends the default Prism parser builder to distinguish keyword arguments from hash literals in the AST. + # The framework's parser builder: the stock builder with the parser gem's `emit_kwargs` opt-in, which + # distinguishes keyword arguments from hash literals in the AST. # - # The upstream builder always emits :hash nodes for both `foo(bar: 1)` and `foo({ bar: 1 })`. Unparser uses the - # node type to decide whether to emit braces: :hash gets `{}`, :kwargs does not. Since Ruby 3.0+ treats these as - # semantically different (strict keyword/positional separation), we need the AST to preserve the distinction. + # With the flag off (the gem's compatibility default) both `foo(bar: 1)` and `foo({ bar: 1 })` emit :hash nodes. + # Unparser uses the node type to decide whether to emit braces: :hash gets `{}`, :kwargs does not. Since Ruby 3.0+ + # treats these as semantically different (strict keyword/positional separation), the AST must preserve the + # distinction. `emit_kwargs` rewrites at the call sites where kwargs semantics live (method calls, index, + # super/yield); the flag is a class-level ivar, so setting it here opts in this builder only — the global + # Parser::Builders::Default stays untouched. # # NOTE: parsed nodes deliberately stay plain Parser::AST::Node. Custom node classes exist only for registered # custom types (see ASTTransform::Node), which are IR and never reach Unparser: AST::Node#eql? compares class, and # Unparser verifies dynamic-string emission by re-parsing and comparing eql? against the freshly parsed # (plain-class) node — custom-class nodes of standard types would fail that verification. class KwargsBuilder < Prism::Translation::Parser::Builder - def associate(begin_t, pairs, end_t) - node = super - return node unless begin_t.nil? && end_t.nil? - - node.updated(:kwargs) - end + self.emit_kwargs = true end end diff --git a/lib/ast_transform/source_parser.rb b/lib/ast_transform/source_parser.rb new file mode 100644 index 0000000..d3be302 --- /dev/null +++ b/lib/ast_transform/source_parser.rb @@ -0,0 +1,59 @@ +# frozen_string_literal: true + +require "prism" +require "prism/translation/parser" +require "ast_transform/kwargs_builder" + +module ASTTransform + # Owns source → AST parsing: Prism's C parser through its whitequark translation layer, so every consumer gets the + # node vocabulary Parser::AST::Processor understands. Transformer parses through this seam; analysis-only + # consumers that never emit instantiate it directly and keep the instance for as many parses as they need. + class SourceParser + # Constructs a new SourceParser instance. + # + # KwargsBuilder is a required implementation detail, not an injection seam: the framework's emission depends + # on the kwargs/hash distinction it preserves, so every parse goes through it. + def initialize + @builder = KwargsBuilder.new + end + + # Parses the given +source+. + # + # @param source [String] The input source code. + # @param file_path [String] The file path recorded on source locations. This is important for source mapping + # in backtraces. + # + # @return [Parser::AST::Node] The AST. + def parse(source, file_path: "tmp") + # A fresh parser per parse: parser instances accumulate per-run state (lexer position, diagnostics), and + # constructing one is trivial next to the parse itself. + parser = Prism::Translation::Parser.new(@builder) + parser.parse(create_buffer(source, file_path, parser.default_encoding)) + end + + # Parses the source in the given +file_path+. + # + # @param file_path [String] The input file path. + # + # @return [Parser::AST::Node] The AST. + def parse_file(file_path) + parse(File.read(file_path), file_path: file_path) + end + + private + + # Builds a source buffer over +source+ in the given +encoding+. + # + # @param source [String] The input source code. + # @param file_path [String] The file path recorded on the buffer. + # @param encoding [Encoding] The encoding the buffer's source is coerced to. + # + # @return [Parser::Source::Buffer] The buffer. + def create_buffer(source, file_path, encoding) + buffer = Parser::Source::Buffer.new(file_path) + buffer.source = source.dup.force_encoding(encoding) + + buffer + end + end +end diff --git a/lib/ast_transform/transformation.rb b/lib/ast_transform/transformation.rb index 1fe4caa..90d4239 100644 --- a/lib/ast_transform/transformation.rb +++ b/lib/ast_transform/transformation.rb @@ -6,6 +6,11 @@ require "unparser" module ASTTransform + # The +transform!+ detector — the canonical sibling-annotation pass (see AbstractTransformation for the taxonomy). + # A +transform!(...)+ statement is a pragma on the NEXT sibling: the transformations it names are applied to the + # following class definition or constant assignment, and the marker itself is deleted from the child list. Both + # effects need the parent's child list in view, which is why matching happens in +process_node+ — an +on_send+ + # handler would see the marker node alone, with no access to its next sibling and no way to remove itself. class Transformation < ASTTransform::AbstractTransformation TRANSFORM_AST = s(:send, nil, :transform!) diff --git a/lib/ast_transform/transformer.rb b/lib/ast_transform/transformer.rb index 567aaf4..16d9248 100644 --- a/lib/ast_transform/transformer.rb +++ b/lib/ast_transform/transformer.rb @@ -1,10 +1,7 @@ # frozen_string_literal: true -require 'prism' -require 'prism/translation/parser' -require 'unparser' -require 'ast_transform/kwargs_builder' require 'ast_transform/line_aligned_emitter' +require 'ast_transform/source_parser' module ASTTransform class Transformer @@ -15,6 +12,7 @@ class Transformer def initialize(*transformations, emitter: LineAlignedEmitter.new) @transformations = transformations @emitter = emitter + @source_parser = SourceParser.new end # Builds the AST for the given +source+. @@ -24,8 +22,7 @@ def initialize(*transformations, emitter: LineAlignedEmitter.new) # # @return [Parser::AST::Node] The AST. def build_ast(source, file_path: "tmp") - buffer = create_buffer(source, file_path) - parser.parse(buffer) + @source_parser.parse(source, file_path: file_path) end # Builds the AST for the given +file_path+. @@ -34,8 +31,7 @@ def build_ast(source, file_path: "tmp") # # @return [Parser::AST::Node] The AST. def build_ast_from_file(file_path) - source = File.read(file_path) - build_ast(source, file_path: file_path) + @source_parser.parse_file(file_path) end # Transforms the given +source+. @@ -89,19 +85,5 @@ def transform_ast(ast) transformation.run(ast) end end - - private - - def create_buffer(source, file_path) - buffer = Parser::Source::Buffer.new(file_path) - buffer.source = source.dup.force_encoding(parser.default_encoding) - - buffer - end - - def parser - @parser&.reset - @parser ||= Prism::Translation::Parser.new(ASTTransform::KwargsBuilder.new) - end end end diff --git a/lib/ast_transform/version.rb b/lib/ast_transform/version.rb index 2cfdbce..3370827 100644 --- a/lib/ast_transform/version.rb +++ b/lib/ast_transform/version.rb @@ -1,5 +1,5 @@ # frozen_string_literal: true module ASTTransform - VERSION = "3.0.0" + VERSION = "3.1.0" end diff --git a/test/ast_transform/abstract_analysis_test.rb b/test/ast_transform/abstract_analysis_test.rb new file mode 100644 index 0000000..d9044d7 --- /dev/null +++ b/test/ast_transform/abstract_analysis_test.rb @@ -0,0 +1,62 @@ +# frozen_string_literal: true + +require "test_helper" +require "transformation_helper" +require "ast_transform/abstract_analysis" + +module ASTTransform + class AbstractAnalysisTest < Minitest::Test + extend ASTTransform::Declarative + include ASTTransform::Helpers::TransformationHelper + + class SendNameCollector < ASTTransform::AbstractAnalysis + attr_reader :names + + def initialize + @names = [] + super + end + + def on_send(node) + @names << node.children[1] + super + end + end + + test "#run returns the analysis instance" do + analysis = SendNameCollector.new + + assert_same analysis, analysis.run(s(:send, nil, :a)) + end + + test "#run harvests from every matching node in the tree" do + ast = ASTTransform::SourceParser.new.parse(<<~CODE) + foo + bar(baz) + CODE + + assert_equal [:foo, :bar, :baz], SendNameCollector.new.run(ast).names + end + + test "#run returns the analysis instance for non-node input without harvesting" do + analysis = SendNameCollector.new + + assert_same analysis, analysis.run(Object.new) + assert_empty analysis.names + end + + test "#run descends into thunk bodies" do + ast = s(:begin, thunk(s(:send, nil, :hidden))) + + assert_equal [:hidden], SendNameCollector.new.run(ast).names + end + + test "#run leaves the analyzed tree equal to a fresh parse" do + source_parser = ASTTransform::SourceParser.new + ast = source_parser.parse("foo(bar)") + SendNameCollector.new.run(ast) + + assert_equal source_parser.parse("foo(bar)"), ast + end + end +end diff --git a/test/ast_transform/source_parser_test.rb b/test/ast_transform/source_parser_test.rb new file mode 100644 index 0000000..bad5d9b --- /dev/null +++ b/test/ast_transform/source_parser_test.rb @@ -0,0 +1,44 @@ +# frozen_string_literal: true + +require "tmpdir" +require "test_helper" +require "transformation_helper" +require "ast_transform/source_parser" + +module ASTTransform + class SourceParserTest < Minitest::Test + extend ASTTransform::Declarative + include ASTTransform::Helpers::TransformationHelper + + def setup + @source_parser = ASTTransform::SourceParser.new + end + + test "#parse returns the expected AST" do + assert_equal s(:send, nil, :method_call), @source_parser.parse("method_call") + end + + test "#parse records the file path on source locations" do + ast = @source_parser.parse("method_call", file_path: "some/file.rb") + + assert_equal "some/file.rb", ast.loc.expression.source_buffer.name + end + + test "#parse can be called repeatedly on one instance" do + assert_equal s(:send, nil, :first), @source_parser.parse("first") + assert_equal s(:send, nil, :second), @source_parser.parse("second") + end + + test "#parse_file parses the file's source and records its path" do + Dir.mktmpdir do |dir| + file_path = File.join(dir, "input.rb") + File.write(file_path, "method_call\n") + + ast = @source_parser.parse_file(file_path) + + assert_equal s(:send, nil, :method_call), ast + assert_equal file_path, ast.loc.expression.source_buffer.name + end + end + end +end