For-each parsing options

[liamhuber]

What the recipe needs

• What should be scattered vs broadcast
• Whether scattering is zipped or nested form
• What to do in the body
• What list-like outputs should be produced

The state of main

Empty var = [] declarations are used in the proximate scope of a for ...: command (or nested sequence of these) in conjuction with var.append(...) in order to build up outputs. Nested/zipped/broadcast can be assessed by what is used inside the for-context and what appears in the for-header. Accumulator variables give port names for output, while the body of the for-loop is grouped into a workflow node whose IO can be automatically named based on what

In #285, in the context of the dev branch #277

With #270, declaring symbols as JSONable constants became allowed, e.g. var = [5]. That left exactly one JSONable constant in an awkward position of not rendering as a constant -- []. So a simple sentinel

def accumulator():
    return []

is used instead.

Pros:

• Minimal change
• Explicit is better than implicit
• No negative impact on running workflows that use it as plain python functions
• Removes constant/accumulator conflict

Cons:

• Reads less python-y
• Workflow designers using @workflow need to remember this rule

@XzzX suggests list comprehensions

ys = [some_func(x) for x in xs]

Pros:

• AST parses clearly as a ast.ListComp, so easy to identify
• No conflict with constants
• Easy to read and highly pythonic
• Handles nesting and zipping fine, e.g.:

def prod(a, b, c, d):
    return a*b*c*d
    
d = 1
ys = [
    prod(a, b, c, d)
    for a in range(4) 
    for b, c in zip(
        range(3), range(3), strict=True
    )
]

Cons:

• By construction, only allows a single listified output
  • Particularly bad if later steps need to
  • Possibly circumvented by the body outputing structured data that itself captures the dependent and independent variable in its own substructure, but IMO this is ugly
• More verbose if the body is complex
  • In a for ...: body we can do multiple steps and parse that into a workflow macro automatically; in a list comprehension complex steps need to be explicitly pre-bundled into a separate body macro by the author

The pin to a single output is the bit I find most troublesome. I find it perfectly reasonable that a user is going to want multiple axes of data coming out of a for-loop, and so the only way to get this is a list[MyDataStructure]. To then strip off a single axis of this, you effectively need another loop for each axis, like

def some_func(x):
    foo = f(x)
    bar = g(x, foo)
    baz = h(x, bar)
    return foo, bar, baz

def wf(xs):
    ys =  [some_func(x) for x in xs]
    bars = [y[1] in ys]  
    # Repeat for each attribute field/tuple element/whatever that you to turn into a data axis
    result = smelt_gold(bars)
    return result

With decent @workflow-parsing sugar for item and attribute access this is not too bad, but it is painful enough to encourage me to think twice and look for alternatives. @XzzX, @samwaseda, if having seen these concerns you're both strongly in favour of the list comprehension route, then this no hill I'll die on and we could proceed. Even though the parser is limited to single-output ForEach recipes, I would probably leave the recipe un-touched.

@XzzX suggests explicit map:

In theory our for-loop could be a map-reduce pattern even though I have not seen it being used like that. Then I prefer: y = reduce(operator.add, map(some_func, xs), 0). I know, not the first thing one learns in a programming class and many of our users won't be aware of this option. But, hey, who is writing code themselves anyway right now?

I hardly ever use map and reduce, so I did indeed chat with Claude on this topic. So that's where my snippets here are coming from, and I'll quote Claude directly at the end.

First, note that to get listy output back out like ForEachRecipe intends, you need something more like reduce(operator.add, map(lambda x: [some_func(x)], xs), []).

Pros:

• No conflict with constants
• Handles zipping natively (loses the option for strict=True)
• Capable of returning multiple listy results, e.g.

a, b, c = reduce(
    lambda acc, x: (acc[0] + [f(x)], acc[1] + [g(x)], acc[2] + [h(x)]),
    xs,
    ([], [], []),  # the initializer *is* your `a, b, c = [], [], []`
)

Maybe there is a more elegant way of doing it than Claude landed on in that snippet, but as-is I find it quite ugly.

Cons:

• Forget difficulty writing, it won't be readable for many of our users
• Nesting requires explicit use of itertools.product
• Using ast to identify that we're trying to generate a ForEachRecipe will be annoying, and extracting port names is difficult or perhaps not universally possible, and here I quote Claude directly:

reduce(reducer, map(body, xs), init) → a bare ast.Call. To extract "what are the output ports," you have to inspect and essentially interpret an arbitrary reducer lambda. (acc[0] + [f(x)], acc[1] + [g(x)], ...) only means "three accumulators" if you decode the subscripts and concatenations — there's no structural guarantee, and a user could write a reducer that folds in ways your parser can't recover.

Honestly, getting this robustly parsed seems like an absolute nightmare.

My take

I like the readability of the list comprehensions, so my best-case scenario would be that one of you sees a clever solution to accessing multiple result axes. We can't ever truly get multiple results from a single comprehension, so I'm a little uncomfortable that we're stuck either digging into ForEachRecipe to constrain it, or with ForEachRecipes that don't compile back into python (not a total deal breaker -- this is already the case for some very special made-by-hand recipes). But still, as long as we have easy attribute and item access, I can live with it.

Of course, it's not binary. val = fr.accumulator; for ... and ys = [... for ...] trigger different AST hits -- we can have both, at the cost of having two sets of for-parsing. But they're completely independent and happen to both result in a ForEachRecipe, so this is not a huge cost.

I would thus suggest to proceed merging #285 down to #277 so that we can remove conflicts with some_constant = []. Between now and merging #277 to main, if we come up with a solution that is clearly superior to declaring and using accumulators, then we can drop fr.accumulator; otherwise, we're free to add list comprehension parsing at any time in the future. Since accumulators and comprehensions can live side by side, we could choose to deprecate the accumulators in the future and have a safe deprecation period (not that this is needed now, since we're v0, but if we decide later.)

[XzzX]

@XzzX suggests explicit map:

In theory our for-loop could be a map-reduce pattern even though I have not seen it being used like that. Then I prefer: y = reduce(operator.add, map(some_func, xs), 0). I know, not the first thing one learns in a programming class and many of our users won't be aware of this option. But, hey, who is writing code themselves anyway right now?

I hardly ever use map and reduce, so I did indeed chat with Claude on this topic. So that's where my snippets here are coming from, and I'll quote Claude directly at the end.

This line is already the deciding line. If you have troubles, non of our users will manage to do it.

Nevertheless, since I like functional programming and map-reduce I will write a few words :D

First, note that to get listy output back out like ForEachRecipe intends, you need something more like reduce(operator.add, map(lambda x: [some_func(x)], xs), []).

list(map(some_func, xs))

This shows one of the problems. map returns a generator which we do not want to support (I think). So we should not support map as it always has to be preceded by list. Just support [some_func(x) for x in xs] which is equivalent. We loose lazy evaluation, however.

Pros:

• No conflict with constants

👍

• Handles zipping natively (loses the option for strict=True)

Not true, map(function, iterable, /, *iterables, strict=False)

• Capable of returning multiple listy results, e.g.

a, b, c = reduce(
lambda acc, x: (acc[0] + [f(x)], acc[1] + [g(x)], acc[2] + [h(x)]),
xs,
([], [], []), # the initializer is your a, b, c = [], [], []
)
Maybe there is a more elegant way of doing it than Claude landed on in that snippet, but as-is I find it quite ugly.

Yes there is:

a = [f(x) for x in xs]
b = [g(x) for x in xs]
c = [h(x) for x in xs]

or

def func(x):
    return x, x**2, x**3

items = [1, 2, 3, 4]

results = [func(i) for i in items] # apply function with multiple returns
firsts, seconds, thirds = [list(x) for x in zip(*results)] # transpose and unpack

Cons:

• Forget difficulty writing, it won't be readable for many of our users

Unfortunately, true. But not because it inherently is difficult. It simply isn't taught in programming classes and, therefore, most of our users will be unfamiliar with it. It requires a different way of thinking. Fun fact, a large potion of the numpy API follows this principle and there it is pretty accepted.

• Nesting requires explicit use of itertools.product

Yes, and I like it. Nesting is a cartesian product that extends your parameter space by a lot. Making this fact explicit is not bad in my opinion.

• Using ast to identify that we're trying to generate a ForEachRecipe will be annoying, and extracting port names is difficult or perhaps not universally possible, and here I quote Claude directly:

reduce(reducer, map(body, xs), init) → a bare ast.Call. To extract "what are the output ports," you have to inspect and essentially interpret an arbitrary reducer lambda. (acc[0] + [f(x)], acc[1] + [g(x)], ...) only means "three accumulators" if you decode the subscripts and concatenations — there's no structural guarantee, and a user could write a reducer that folds in ways your parser can't recover.

I have not checked the ast. But we only have to look for calls to reduce, map and list comprehension, don't we?

Honestly, getting this robustly parsed seems like an absolute nightmare.

I do not know. I have not checked the AST. But for me map-reduce isn't more complex than for-each.

A note about reduce. Reduce is a n->1 operation. Of course, individual elements -> list, is a n->1 operation but the usage here is quite odd.

[liamhuber]

Yes there is:

a = [f(x) for x in xs]
b = [g(x) for x in xs]
c = [h(x) for x in xs]

Ah, but that's cheating you just went back to comprehensions 😉

firsts, seconds, thirds = [list(x) for x in zip(*results)] # transpose and unpack

And we don't support parsing this at all right now, not by a long shot. You could definitely just wrap this operation inside an atomic node, but that is just side-stepping the question of how to make an unambiguous, readable, and succinct ForEachRecipe-parseable statement that is powerful enough to cover our use cases.

I do like list comprehensions though, and I'd like to support them at some point.

reduce(reducer, map(body, xs), init) → a bare ast.Call. To extract "what are the output ports," you have to inspect and essentially interpret an arbitrary reducer lambda. (acc[0] + [f(x)], acc[1] + [g(x)], ...) only means "three accumulators" if you decode the subscripts and concatenations — there's no structural guarantee, and a user could write a reducer that folds in ways your parser can't recover.

I have not checked the ast. But we only have to look for calls to reduce, map and list comprehension, don't we?

Honestly, getting this robustly parsed seems like an absolute nightmare.

I do not know. I have not checked the AST. But for me map-reduce isn't more complex than for-each.

We need to be able to parse a body node from the lambda, which I would naively think is either quite difficult or requires placing constraints on what sort of lambdas can be written that the users then need to learn.

I guess we'd need to also know how to identify parts like itertools.product. There might be more, but for me this is a case of not knowing what I don't know.

I'm not at all philosophically opposed to parsing map-reduce statements into ForEachRecipe, I am only opposed to it being the only parsable option we provide because I think many users would struggle with it. It sounds like we're on close enough to the same page there. I also won't start working on an implementation of parsing them any time soon -- I would first need to spend some time to really grok their implementation independent of parsing it into a for-node. If you want to take a crack at it though, there's zero objections from me!