Skip to content

fix: source lookup for exec'd agenda functions - #1200

Open
olemke wants to merge 1 commit into
atmtools:mainfrom
olemke:fix-pyarts-source-from-string-agendas
Open

olemke wants to merge 1 commit into
atmtools:mainfrom
olemke:fix-pyarts-source-from-string-agendas

Conversation

@olemke

@olemke olemke commented Oct 23, 2026

Copy link
Copy Markdown
Member

Problem

@arts_agenda, @callback_operator and @in_parallel parse the decorated
function's body with inspect.getsource. That only works through
linecache, so it fails for any function defined by executing a file as a
string — exactly how matplotlib's plot directive (and Sphinx-Gallery style
tooling) runs ARTS example files. The code objects of such functions carry
the filename <string> and no linecache entry, and the decorators raised
OSError: could not get source code.

What changes

Adds pyarts3.workspace.utility._source_and_location(func), returning
(source, filename, first_line) in one step, and switches the three
decorators to it.

  • Fast path: a single inspect.getsourcelines + getfile — strictly
    fewer inspect walks than the old code, which did getsource +
    getfile + getsourcelines (two full source searches per agenda).
  • Fallback path: when inspect fails, reads candidate files in order
    (co_filename, __file__, sys.argv[0] — the latter is where the plot
    directive keeps the original example path), locates the definition at
    co_firstlineno, and takes the block via inspect.getblock.
  • Validation: a located block is only accepted if recompiling it
    matches the live code object — argument counts, local variable names,
    used names, bytecode (when no free variables are involved) and
    constants. The constants check (_consts_match) is needed because
    bytecode alone pins the position of literals but not their values:
    two same-line agendas differing only in species=["H2O"] vs
    species=["O2"] compile to identical co_code.
  • Unwrapping: inspect.unwrap first, matching getsourcelines.
    Without it, a functools.wraps wrapper below the decorator would hand
    out the wrapper's source — and the validation would not notice, since a
    wrapper's own source matches the wrapper's code object perfectly.
  • _arts_agenda now accepts an already-resolved (source, fn, ln) triple,
    so in_parallel no longer looks the source up twice.

Adds tests/core/agenda/source_from_exec.py: exec-string agendas,
callbacks and in_parallel; indented (nested) functions; decoy rejection
for an unrelated file (same name, and separately same lines with different
literals); fast/slow path agreement for decorated and wraps-decorated
functions in readable files.

Advantages

  • Decorators now work for code executed from strings — the class of failure
    that breaks agenda creation in doc/example builds.
  • Failure mode is explicit: if no located block provably is the requested
    function, an OSError naming the qualname is raised (from the original
    inspect error) rather than silently parsing an unrelated same-named
    function that happens to sit at that line of some candidate file.
  • Fewer inspect walks per decoration than before (one lookup instead of
    three), and no behavioral change for normal file-based functions: the
    fast path returns exactly what the old code returned.

Disadvantages / limitations

  • The fallback is heuristic with a verified core. A same-line block is
    only accepted after re-compilation matching; but for functions with
    free variables the bytecode comparison is skipped (free variables
    resolve as globals in the stand-alone block), so for closures the
    guarantee reduces to names/args/locals/consts — a contrived unrelated
    closure at the same line could still be accepted. Documented in
    _block_compiles_to.
  • Constants comparison is skipped when the function's constants contain
    code objects
    (lambdas/inner defs compare by identity across
    compilations); those functions keep a weaker guarantee. Documented.
  • Line-number alignment assumption: the fallback only finds the right
    block if the executed string has the same line layout as the candidate
    file. Misaligned sources fail loudly (validation rejects), not silently.
  • sys.argv[0] is consulted as a candidate, which is only meaningful while
    the code is being executed by such a tool; the validation bounds the risk
    (see above).
  • New private helper is imported across three modules (_source_and_location).

Performance

Measured microbenchmarks (µs, best-of-3 loops, two processes each, Python
3.14.7, HEAD baseline vs this branch):

Operation HEAD This PR
@arts_agenda applied (full parse) 597 / 609 575 / 595 (~2% faster)
_arts_agenda internals 614 / 618 581 / 605
@callback_operator applied 15.3 / 15.3 15.5 / 15.7 (+0.3 µs, one extra getfile)
raw source lookup (fast path) 39.0 / 39.2 19.8 / 20.1 (~2× faster)
exec-string decorated agenda fails (OSError) succeeds, ~1.4 ms

Net: normal (file-based) decoration is neutral-to-slightly-faster; the only
measurable regression is ~0.3 µs on callback_operator. The ~1.4 ms
fallback cost applies only where the old code raised outright, so it is a
new capability's cost, not a fast-path tax. Full suite timing unchanged
(tests.core: 237 s, 155/155 passed).

The agenda and callback decorators parsed function bodies with
inspect.getsource, which fails when a file is executed as a string:
matplotlib's plot directive runs example files that way, and those
code objects carry the filename <string> with no linecache entry.

Add utility._source_and_location(), returning source, filename and
first line number in one step.  It tries inspect first - a single
getsourcelines walk instead of getsource, getfile and
getsourcelines - and otherwise reads candidate files (co_filename,
__file__, sys.argv[0]), locating the function at co_firstlineno.  A
located block is accepted only if recompiling it matches the live
code object in argument counts, variable names, used names and
constants, plus bytecode when no free variables are involved, so a
decoy definition in an unrelated file cannot silently produce a
wrong agenda.  Functions are unwrapped first, or functools.wraps
would hand out the wrapper's source, which matches the wrapper's own
code object exactly.

_arts_agenda, in_parallel and _callback_operator use the new
helper; _arts_agenda now takes the location from its caller instead
of looking it up a second time.

Add tests/core/agenda/source_from_exec.py covering exec-string
agendas, callbacks and in_parallel, indented functions, decoy
rejection, fast/slow path agreement and wrapped functions.
@olemke
olemke requested a review from riclarsson October 23, 2026 09:11
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant