Skip to content

feat(workflow-operator): leave explanatory comments out of exported Python - #9154

Open
kz930 wants to merge 1 commit into
apache:mainfrom
kz930:feat/export-no-comments
Open

kz930 wants to merge 1 commit into
apache:mainfrom
kz930:feat/export-no-comments

Conversation

@kz930

@kz930 kz930 commented Oct 26, 2026

Copy link
Copy Markdown
Contributor

What changes were proposed in this PR?

The Python that "Export to Python" writes carried notes meant for whoever maintains the generator: why a frame is copied, why a row is dropped before scoring, why an empty cell keeps its row. Someone running the exported script does not need them, and they made the script longer to read. This PR takes those comments out of the exported code in 24 operators, and keeps a one or two line note in Scala above the string where the reason is worth keeping.

  • Machine learning: the sklearn estimators' shared training, testing and fit-and-score code, Linear Regression, and the Machine Learning Scorer.
  • Hugging Face: Iris Logistic Regression, Sentiment Analysis, Spam SMS Detection, Text Summarization.
  • Visualization: Bar, Box/Violin, Bubble, Bullet, Choropleth, Contour, Dendrogram, Gauge, Hierarchy, Histogram 2D, Icicle, Radar, Tree Plot, URL Visualizer.
  • Unnest String.

Only comments go. The translator's own markers stay: the # [Operator] heading over each section, # --- Output ---, and the # TODO it writes for an operator it cannot export yet. The Python that operators run inside the engine is not touched.

Any related issues, documentation, discussions?

The comments came in with the export PRs #8365, #8367, #8368, #8504 and #8340.

#9035 also changes Unnest String's export. Whichever of the two merges second keeps the other's change.

How was this PR tested?

No generated statement changes, only comment lines. Running the verification suite before and after, the 1,098 scripts it generates for 142 operators are identical once comment lines and temporary paths are set aside, and none of them has an explanatory comment left. All of WorkflowOperator/test passes (2,912 tests), as do scalafmtCheck and scalafixAll --check.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 5.5)

🤖 Generated with Claude Code

…ython

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

  • Contributors with relevant context: @aglinxinyuan
    You can notify them by mentioning @aglinxinyuan in a comment.

@github-actions

Copy link
Copy Markdown
Contributor

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 5 worse · ⚪ 10 noise (<±5%) · 0 without baseline

Compared against main 8fbd4f2 benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

config throughput MB/s latency max Δ latest / 7d
🔴 bs=10 sw=10 sl=64 519 0.317 18,855/23,785/23,785 us 🔴 +16.3% / 🔴 +94.6%
🔴 bs=100 sw=10 sl=64 1,217 0.743 78,252/120,539/120,539 us 🔴 +13.9% / 🔴 +38.3%
⚪ bs=1000 sw=10 sl=64 1,439 0.878 692,198/775,517/775,517 us ⚪ within ±5% / 🟢 -11.3%
Baseline details

Latest main 8fbd4f2 from same runner

config metric PR latest main 7d avg Δ latest Δ 7d
bs=10 sw=10 sl=64 throughput 519 tuples/sec 568 tuples/sec 1,019 tuples/sec -8.6% -49.1%
bs=10 sw=10 sl=64 MB/s 0.317 MB/s 0.347 MB/s 0.622 MB/s -8.6% -49.0%
bs=10 sw=10 sl=64 p50 18,855 us 16,213 us 10,144 us +16.3% +85.9%
bs=10 sw=10 sl=64 p95 23,785 us 24,119 us 12,225 us -1.4% +94.6%
bs=10 sw=10 sl=64 p99 23,785 us 24,119 us 15,527 us -1.4% +53.2%
bs=100 sw=10 sl=64 throughput 1,217 tuples/sec 1,232 tuples/sec 1,371 tuples/sec -1.2% -11.2%
bs=100 sw=10 sl=64 MB/s 0.743 MB/s 0.752 MB/s 0.837 MB/s -1.2% -11.2%
bs=100 sw=10 sl=64 p50 78,252 us 78,450 us 79,910 us -0.3% -2.1%
bs=100 sw=10 sl=64 p95 120,539 us 105,791 us 87,155 us +13.9% +38.3%
bs=100 sw=10 sl=64 p99 120,539 us 105,791 us 106,643 us +13.9% +13.0%
bs=1000 sw=10 sl=64 throughput 1,439 tuples/sec 1,442 tuples/sec 1,415 tuples/sec -0.2% +1.7%
bs=1000 sw=10 sl=64 MB/s 0.878 MB/s 0.88 MB/s 0.864 MB/s -0.2% +1.6%
bs=1000 sw=10 sl=64 p50 692,198 us 693,654 us 780,407 us -0.2% -11.3%
bs=1000 sw=10 sl=64 p95 775,517 us 749,780 us 817,814 us +3.4% -5.2%
bs=1000 sw=10 sl=64 p99 775,517 us 749,780 us 848,683 us +3.4% -8.6%
Raw CSV
config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,385.09,200,128000,519,0.317,18854.51,23785.19,23785.19
1,100,10,64,20,1642.87,2000,1280000,1217,0.743,78251.84,120539.34,120539.34
2,1000,10,64,20,13900.10,20000,12800000,1439,0.878,692197.93,775516.62,775516.62

@codecov-commenter

codecov-commenter commented Oct 26, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 91.81%. Comparing base (8fbd4f2) to head (b816be7).
⚠️ Report is 2 commits behind head on main.

Additional details and impacted files
@@            Coverage Diff            @@
##               main    #9154   +/-   ##
=========================================
  Coverage     91.81%   91.81%           
- Complexity     5069     5070    +1     
=========================================
  Files          1271     1271           
  Lines         55208    55207    -1     
  Branches       6987     6987           
=========================================
+ Hits          50688    50689    +1     
+ Misses         2800     2799    -1     
+ Partials       1720     1719    -1     
Flag Coverage Δ *Carryforward flag
access-control-service 77.44% <ø> (ø)
agent-service 99.16% <ø> (ø) Carriedforward from 8fbd4f2
amber 85.97% <100.00%> (+<0.01%) ⬆️
computing-unit-managing-service 61.84% <ø> (ø)
config-service 87.50% <ø> (ø)
file-service 81.92% <ø> (ø)
frontend 96.50% <ø> (ø) Carriedforward from 8fbd4f2
notebook-migration-service 83.77% <ø> (ø)
pyamber 98.54% <ø> (ø) Carriedforward from 8fbd4f2
workflow-compiling-service 72.61% <ø> (ø)

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@aglinxinyuan aglinxinyuan left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!

@aglinxinyuan
aglinxinyuan added this pull request to the merge queue Oct 26, 2026
@kz930

kz930 commented Oct 26, 2026

Copy link
Copy Markdown
Contributor Author

@carloea2 This removes the explanatory comments that the merged export PRs left in exported Python, and keeps a short note in Scala where the reason matters. The generated statements are unchanged across all 1,098 verification scripts. Could you take a look?

@aglinxinyuan
aglinxinyuan removed this pull request from the merge queue due to a manual request Oct 26, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants