[carloea2]

What changes were proposed in this PR?

Add a typed Cloudpickle codec for values crossing generated Python workflow boundaries. A boundary payload records the selected field contract separately from the fields present on a particular execution path, while a workflow envelope groups canonical boundary payloads under one execution key.

The codec validates boundary identities, field ordering, payload shape, duplicate boundaries, field-contract mismatches, and cross-key fan-in. It serializes only explicitly supplied values and preserves aliases and cycles within a boundary payload.

Any related issues, documentation, discussions?

Related to #8319.

How was this PR tested?

Python 3.12 on Windows in a fresh detached worktree from upstream main at fdd860d270. The two wide-input regression cases fail before the set-reuse change: 2 failed and 35 passed. Afterward, all 37 codec tests and all 197 SDK tests pass. Coverage includes empty and partial contracts, malformed subsets, Unicode, aliases, cycles, and metadata errors.

Generate protobufs with bash bin/python-proto-gen.sh, then run from amber:

python -m pytest src/test/python/pytexera/workflow/test_codec.py -q --tb=short -p no:cacheprovider
python -m pytest src/test/python/pytexera -m "not integration" -q --tb=short -p no:cacheprovider
python -m ruff check src/main/python src/test/python
python -m ruff format --check src/main/python src/test/python

Both Ruff checks pass across all 219 Python files. Fresh CI is running for the rebased one-commit branch.

Was this PR authored or co-authored using generative AI tooling?

Generated-by: OpenAI Codex

[github-actions]

Automated Reviewer Suggestions

Based on the git blame history of the changed files, we recommend the following reviewers:

• No candidates found from git blame history.

[codecov-commenter]

Codecov Report

❌ Patch coverage is 86.48649% with 10 lines in your changes missing coverage. Please review.
✅ Project coverage is 91.80%. Comparing base (fdd860d) to head (c0c6d4d).

Files with missing lines	Patch %	Lines
amber/src/main/python/pytexera/workflow/codec.py	86.48%	5 Missing and 5 partials ⚠️

Additional details and impacted files

@@             Coverage Diff              @@
##               main    #8433      +/-   ##
============================================
- Coverage     91.80%   91.80%   -0.01%     
  Complexity     5066     5066              
============================================
  Files          1270     1271       +1     
  Lines         55193    55267      +74     
  Branches       6987     7006      +19     
============================================
+ Hits          50672    50736      +64     
- Misses         2800     2805       +5     
- Partials       1721     1726       +5     

Flag	Coverage Δ		*Carryforward flag
access-control-service	77.44% <ø> (ø)		Carriedforward from fdd860d
agent-service	99.16% <ø> (ø)		Carriedforward from fdd860d
amber	85.96% <ø> (ø)		Carriedforward from fdd860d
computing-unit-managing-service	61.84% <ø> (ø)		Carriedforward from fdd860d
config-service	87.50% <ø> (ø)		Carriedforward from fdd860d
file-service	81.92% <ø> (ø)		Carriedforward from fdd860d
frontend	96.49% <ø> (ø)		Carriedforward from fdd860d
notebook-migration-service	83.77% <ø> (ø)		Carriedforward from fdd860d
pyamber	98.35% <86.48%> (-0.19%)	⬇️
workflow-compiling-service	72.61% <ø> (ø)		Carriedforward from fdd860d

*This pull request uses carry forward flags. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[github-actions]

⚠️ Benchmark changes need a look

🟢 0 better · 🔴 5 worse · ⚪ 10 noise (<±5%) · 0 without baseline

Compared against main fdd860d benchmarked on this same runner, so the delta is largely free of cross-runner hardware noise. The "7d avg" column still reflects the gh-pages dashboard. Treat <±5% as noise unless repeated.

Dashboard · Run

	config	throughput	MB/s	latency	max Δ latest / 7d
🔴	bs=10 sw=10 sl=64	376	0.229	25,187/40,005/40,005 us	🔴 +14.9% / 🔴 +255.5%
⚪	bs=100 sw=10 sl=64	812	0.496	122,178/160,123/160,123 us	⚪ within ±5% / 🔴 +113.9%
⚪	bs=1000 sw=10 sl=64	961	0.587	1,034,533/1,106,625/1,106,625 us	⚪ within ±5% / 🔴 +56.6%

Baseline details

Latest main fdd860d from same runner

config	metric	PR	latest main	7d avg	Δ latest	Δ 7d
bs=10 sw=10 sl=64	throughput	376 tuples/sec	410 tuples/sec	1,119 tuples/sec	-8.3%	-66.4%
bs=10 sw=10 sl=64	MB/s	0.229 MB/s	0.25 MB/s	0.683 MB/s	-8.4%	-66.5%
bs=10 sw=10 sl=64	p50	25,187 us	22,090 us	8,992 us	+14.0%	+180.1%
bs=10 sw=10 sl=64	p95	40,005 us	34,831 us	11,255 us	+14.9%	+255.5%
bs=10 sw=10 sl=64	p99	40,005 us	34,831 us	13,808 us	+14.9%	+189.7%
bs=100 sw=10 sl=64	throughput	812 tuples/sec	828 tuples/sec	1,518 tuples/sec	-1.9%	-46.5%
bs=100 sw=10 sl=64	MB/s	0.496 MB/s	0.505 MB/s	0.927 MB/s	-1.8%	-46.5%
bs=100 sw=10 sl=64	p50	122,178 us	116,388 us	68,504 us	+5.0%	+78.4%
bs=100 sw=10 sl=64	p95	160,123 us	161,154 us	74,866 us	-0.6%	+113.9%
bs=100 sw=10 sl=64	p99	160,123 us	161,154 us	89,939 us	-0.6%	+78.0%
bs=1000 sw=10 sl=64	throughput	961 tuples/sec	971 tuples/sec	1,569 tuples/sec	-1.0%	-38.7%
bs=1000 sw=10 sl=64	MB/s	0.587 MB/s	0.593 MB/s	0.957 MB/s	-1.0%	-38.7%
bs=1000 sw=10 sl=64	p50	1,034,533 us	1,023,639 us	669,501 us	+1.1%	+54.5%
bs=1000 sw=10 sl=64	p95	1,106,625 us	1,067,426 us	706,487 us	+3.7%	+56.6%
bs=1000 sw=10 sl=64	p99	1,106,625 us	1,067,426 us	737,560 us	+3.7%	+50.0%

Raw CSV

config_idx,batch_size,schema_width,string_len,num_batches,total_ms,total_tuples,total_bytes,tuples_per_sec,mb_per_sec,lat_p50_us,lat_p95_us,lat_p99_us
0,10,10,64,20,532.40,200,128000,376,0.229,25187.29,40004.63,40004.63
1,100,10,64,20,2463.03,2000,1280000,812,0.496,122178.27,160122.93,160122.93
2,1000,10,64,20,20803.20,20000,12800000,961,0.587,1034533.06,1106625.38,1106625.38

[sshiv012]

Some minor changes required, looks good to me otherwise!

[carloea2]

@sshiv012 All seven comments are addressed in a90fcd2, with replies on each thread. Could you take another look?

[carloea2]

@sshiv012 Following up: the required checks are now passing on a90fcd2, including PyAmber on Python 3.11–3.13 and Amber integration on Linux and macOS. Your review comments have been addressed; could you take another look when you have time?

[sshiv012]

LGTM, comments have been addressed :)

[carloea2]

/request-review @mengw15

[carloea2]

@mengw15, Surya approved the current revision and the required checks pass. Could you give the committer review and merge if satisfied? I also corrected the title scope to pyamber.

[Copilot]

🟡 Changes recommended

Repeated set construction makes present-field validation quadratic across boundary and envelope processing.

1 open issue

• Avoid rebuilding the present-field set for every contract field · New

What changed in this PR

Adds a typed Cloudpickle transport layer for generated Python workflows, supporting the script and notebook compiler planned in #8319.

Changes:

• Introduces boundary payloads and execution-key envelopes with metadata validation and fan-in merging.
• Separates selected fields from path-present values while preserving aliases and cycles within each payload.
• Adds codec tests and pins the Cloudpickle dependency.

File	Description
amber/​src/​test/​python/​pytexera/​workflow/​test_codec.py	Tests round trips, object identity, metadata validation, and merge rejection.
amber/​src/​main/​python/​pytexera/​workflow/​codec.py	Implements typed boundary serialization and envelope handling.
amber/​requirements.txt	Adds Cloudpickle 3.1.2.

_{Review effort: Balanced}

[carloea2]

/request-review @mengw15

The quadratic validation comment is addressed with a failing regression first. All 197 SDK tests and Ruff checks pass locally. The branch is rebased onto current main, with fresh CI running.