Trivia is modelled two different ways, and the leading-trivia path is unreliable #261

Summary

Comments and whitespace attached to a node are handled by two unrelated mechanisms depending on
which side of the node they sit on. Trailing trivia is a static per-node integer supplied by the
adapter; leading trivia is reconstructed at rewrite time by scanning raw bytes for hardcoded comment
delimiters. The second mechanism is language-agnostic by construction and therefore cannot be
correct, and it already fails on ordinary Java input.

Current behaviour

There is no extended_start_offset. _RewriteActions.__correct_for_comments_and_whitespace
compensates by searching backwards through the buffer between the preceding sibling (or parent) and
the node.

Symptoms

1. // wins over /* regardless of which is closer to the node

get_comment_location tests the // branch first and returns unconditionally, so a block comment is
only ever found when no line comment exists in the search window.

/** Does the thing. */
// FIXME later
void f() {}

The scan stops at // FIXME later and the Javadoc is left behind when the function is removed.
Swapping the two comment lines produces the opposite outcome: the range then starts at the // line
and swallows the Javadoc incidentally. The same input shape yields two different answers depending
on comment order.

2. Multi-line comment blocks are truncated

Known and marked in the code:

• # TODO: the // and # branches below only find the single closest comment line
  (get_comment_location)
• # TODO: same single-line limitation as get_comment_location (__get_comment_after_location)

Covered by two xfail(strict=True) cases in test/syntax_tree/test_ast_rewriter.py:
multiline_hash_comment and multiline_slash_comment. A conventional multi-line Javadoc is exactly
this case.

3. The core hardcodes comment syntax it has no way to know

//, /* and # are baked into the language-agnostic rewriter. It cannot distinguish # as a
Python comment from # as a C preprocessor directive, nor recognise that the # in
x = "# not a comment" is inside a string literal. rfind(b"#") will match it and start the
replacement range there.

Relationship to ADR 11

docs/developer/architecture/adr/11_parser_with_space_and_comment.md already decided this:

Comments and whitespace are made part of the AST node itself (as leading/trailing trivia attached
to the node), rather than stored in a separate data structure.

For non-Python languages, attach leading and trailing trivia directly to each AST node, following
the pattern used by Roslyn (C#) and tree-sitter.

Leading trivia is part of the accepted decision. The implementation delivered the trailing half and
substituted byte-scanning for the rest.

Options

A. Add extended_start_offset, mirroring extended_end_offset

Symmetric and cheap, but inadequate:

• Two scalars cannot express the shape. Leading trivia is routinely non-contiguous: Javadoc,
  blank line, // FIXME, annotation. A single integer collapses this to all-or-nothing, with the
  cut-point chosen by whoever wrote the adapter.
• Attachment is policy, not parser fact. Whether the blank line above a Javadoc belongs to the
  function depends on the operation: when deleting, yes, otherwise a double blank line is left
  behind; when copying, no. The node cannot know the caller's intent. That include_whitespace and
  include_comments already exist as per-call flags shows this is a parameter, not a property.
• Adapters would each invent a policy. libcst attaches leading_lines to the node, tree-sitter
  exposes comments as ordinary siblings, clang largely discards them. Requiring all three to produce
  one integer guarantees they disagree.

B. Adapters expose trivia as text segments; the core composes a replacement range

• adapters expose leading_trivia / trailing_trivia as sequences of TextSegment
• the core composes them under an explicit policy, e.g.
  expand(node, include_comments=..., include_whitespace=...) -> TextSegment
• the rewriter replaces the single resulting segment

This matches what TextSegment already documents about itself:

Instances include comments, whitespace (incl. empty lines), and syntax nodes.

extended_end_offset then disappears into trailing_trivia, and PythonCstNode's
"libcst attaches trailing trivia to the node itself, so there is nothing to extend" becomes an
honest empty sequence. Siblings is a natural return type for node-plus-trivia, since it already
models a contiguous range of segments including the empty case.

Removing the heuristics deletes get_comment_location, __get_comment_after_location,
__extend_with_whitespace and __correct_for_comments_and_whitespace - roughly 100 lines carrying
two xfails and two TODOs.

Cost: clang is the hard case. libclang exposes only doc-comments, via
clang_Cursor_getRawCommentText. Scanning may still be needed there, but inside the clang adapter
where the comment syntax is known, rather than in a core that has to guess.

Proposed direction

Option B, as the concrete implementation of the leading/trailing trivia decision already recorded in
ADR 11.

Acceptance criteria

• Leading and trailing trivia are obtained through one mechanism, not two.
• The core contains no hardcoded comment delimiters.
• Removing a Java method removes its Javadoc, independently of whether a line comment also
  precedes it, and independently of the order of the two.
• A multi-line comment block is captured in full.
• A comment delimiter inside a string literal is not treated as a comment.
• The existing multiline_hash_comment and multiline_slash_comment xfails in
  test/syntax_tree/test_ast_rewriter.py flip to passing, or are replaced by equivalent
  coverage against the new mechanism.

Related

• ADR 11 - Parser with Space and Comment
• docs/developer/architecture/position-consistency.md
• src/renaissance/syntax_tree/text_segment.py, siblings.py
• issue 189