A control-variable port on every operator: design and prototype

[aglinxinyuan]

Summary

Every operator gets one more input port, the control-variable port, drawn in purple. Whatever arrives on it becomes the operator's control variables, and a property that holds $name takes the value of the variable name. An existing operator can then run once per loop iteration on a value computed upstream. Today File Scan needs a second "from input" operator for this.

A prototype that runs end to end is on aglinxinyuan/texera:control-variable-port, with draft PR #8917.

Example: read every file of a folder

Text Input ──► Loop Start ══(purple)══► File Scan ──► Loop End
(one path       init:   i = 0                fileName = $file   update:    i += 1
 per row)       output: file = D.line[i]                        condition: i < len(D)

File Scan is a source. Its only change is that it resolves the file name at run time, once $file is bound. In each iteration, Loop Start's Output sets file to that iteration's path and Loop Start sends {i, file} into File Scan's port. File Scan binds $file, opens that file, and its rows flow on to Loop End.

Design

1. One port per operator, no new edge type. Any output port may link to it, and the port does the conversion.
2. No schema. A control message merges into the operator's control variables. A tuple's columns become variables of the same name.
3. Binding is by name. A name that never arrives is a run-time error.
4. The port's input ends before the operator starts. The edge into the port is materialized. The receiver's data edges can still be pipelined.
5. The later value wins. Several edges may enter the port. Everywhere, if two values of one name reach an operator at the same loop depth before it first uses them, the later one replaces the earlier one. This replaces today's conflict error.
6. Port-fed values travel with the block. They are forwarded through the rest of the loop block, like any other control variable.
7. Loop Start's Output is a Python cell. Every name it assigns becomes a control variable of that iteration. The value of a trailing bare expression is the data sent to the body; without one, the body gets no data. A single expression such as table.iloc[[i]] works as before.

Prototype

• The port is the reserved PortIdentity(-1). The compiler adds it only when a link targets it.
• The worker starts the port's reader first and holds data channels until the port's input ends.
• Loop Start's Initialization and Output can read D. Output runs once per iteration, before Loop Start sends its variables, so the body and the port see what it assigns.
• The frontend draws the port and its links in purple and sends each link as port -1. The property panel offers the names a Loop Start's Initialization and Output assign.

PASTA's search needs no change. The prototype adds the port link to getBlockingAndDependeeLinks, so the sender's region runs before the receiver's. The receiver's own data edge can still be pipelined.

End-to-end test. ControlVariablePortIntegrationSpec runs the example over three files that hold the numbers 1 to 2, 3 to 5 and 6, one per line: once with file set in Output as above, and once with it set in Initialization and Update. Loop End receives 1 through 6 both times, in CI on Linux and macOS. The workflow also runs in a local deployment.

Limits

• Among file readers, only File Scan. The CSV reader infers its schema from the file at compile time, and with $file there is no file yet.
• No receivers that run code (UDFs, Python-based operators, Loop Start, Loop End). They are rejected at compile time.
• Only the first tuple that reaches the port is converted. Later tuples, from any edge, are dropped.
• One output port cannot feed both the port and a data port of one operator. The compiler rejects this.
• The branch includes open PRs feat(frontend): accept $variable references in loop blocks #8640 and feat(pyamber): register the state before process_state #8642. The prototype itself is these commits.

[sshiv012]

How do you decide which operator properties can resolve at runtime, especially when properties like CSV file paths affect compile time schema inference? And how are the bound values checked against the property’s expected type?

[aglinxinyuan]

1. A property can take $name unless the plan is built from it, because the plan is compiled before any variable has a value. fix(amber): reject $i in settings the plan is built from #8750 marks those properties with @NoLoopVariable (Projection's columns, group-by keys, join keys), and a $name there is a compile error. File Scan works with $file because its schema comes from its properties, not from the file. CSV infers its schema by reading the file at compile time, so $file doesn't work there yet. The plan is to take the schema from an example file or a declared schema instead.

2. Values are checked at run time, right before the operator first sees data. StateReferenceBinding converts each value to the field's Java type: an integer field accepts "7" but rejects 7.5. The value then goes through the descriptor's own Jackson deserialization. A wrong type or a missing name shows up as an operator error, and the operator pauses.

[bobbai00]

Can you give more examples on what are control-variables ?

[aglinxinyuan]

Control variables are values that can be passed between operators but are not part of the data itself. For example, you might want to filter records where price < x, with x supplied by an upstream operator as a control variable.

[bobbai00]

I see. So if we change it to control ports, does that change the user experience ? Or the change is only at the engine level ?

e.g., for a Filter operator, to set price < 20, can I still do it via the operator parameter panel ?

[aglinxinyuan]

It doesn't change user experience for people who don't need control variable.

Even for user who needs control variable, they assign the value in operator property panel:

[bobbai00]

I like this design. It makes workflow's expressiveness better.

In this proposal, is this report required for all operators , or optional ? It it is required, I would worry about the backward compatibility issues.

[aglinxinyuan]

I like this design. It makes workflow's expressiveness better.

In this proposal, is this report required for all operators , or optional ? It it is required, I would worry about the backward compatibility issues.

It's optional.

[zyratlo]

I like this. Having one optional port on every operator is a lot cleaner than adding a "from input" version of each operator that needs a run-time value, and the folder example makes the use case easy to see. Since the port is optional, existing workflows and anyone who doesn't need control variables won't notice a difference.

One suggestion I have is about the limit where only the first tuple reaching the port is converted and the rest are dropped. If someone accidentally connects a multi-row table, they'd silently get results from the first row. Could we show a warning when more than one tuple arrives on the port?

[chenlica]

I support this design. It's based on many offline discussions with @aglinxinyuan and @Xiao-zhen-Liu in the context of adding "control block" to the system.