Skip to content

Fix IcebergIO Managed Integration Tests on Dataflow workflow #40726

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
Amar3tto wants to merge 2 commits into from
Closed

Fix IcebergIO Managed Integration Tests on Dataflow workflow #40726

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
1e43b21
Test legacy worker
Amar3tto Oct 22, 2026
0d8b42c
Add flag for enableManagedTransforms
Amar3tto Oct 22, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions sdks/java/io/iceberg/build.gradle
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -235,6 +235,7 @@ task dataflowIntegrationTest(type: Test) {
]
if (project.hasProperty('enableManagedTransforms')) {
args.add("--experiments=enable_managed_transforms")
systemProperty 'beam.test.enableManagedTransforms', 'true'
}
systemProperty "beamTestPipelineOptions", JsonOutput.toJson(args)

Expand Down
3 changes: 2 additions & 1 deletion ...o/iceberg/src/test/java/org/apache/beam/sdk/io/iceberg/catalog/IcebergCdcWriteBaseIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -855,7 +855,8 @@ private void roundTripThroughSource(boolean upsert, boolean changeTypeColumn) th
}
metadataColumns.add(IcebergCdcMetadataColumns.COMMIT_SNAPSHOT_SEQUENCE_NUMBER);
PipelineOptions options = TestPipeline.testingPipelineOptions();
if (!changeTypeColumn) {
// Use the legacy worker for managed transforms until the Runner v2 CDC issue is resolved.
if (!changeTypeColumn || Boolean.getBoolean("beam.test.enableManagedTransforms")) {

@Abacn Abacn Oct 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's supposed to run with managed, and we should fix that cc: @chamikaramj @ahmedabu98

@Abacn Abacn Oct 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Checked that both tests were added in #40230. Since it's never worked. We can add skip them from Dataflow suite for now, to make the test suite back green and avoid further breakage get in unnoticed.

@chamikaramj chamikaramj Oct 22, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Skip only runner v2 tests till the Dataflow service is updated to use Beam 2.77.0 for managed transforms please. Runner v1 tests should work and we can re-enable Runner v2 tests in a week or so when the service is updated.

CdcSinkTestUtils.useLegacyDataflowWorker(options);
}
Map<String, Object> readConfig = new HashMap<>(managedIcebergConfig(sourceId.toString()));
Expand Down
2 changes: 1 addition & 1 deletion ...ceberg/src/test/java/org/apache/beam/sdk/io/iceberg/cdc/sink/DeltaLakeToIcebergCdcIT.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -311,7 +311,7 @@ private void applyChangeFeed(boolean upsert, boolean changeTypeColumn) throws Ex

// The native ValueKind does not survive Dataflow Runner v2 yet, so that variant runs on the
// legacy worker there; the change-type column travels on any runner.
if (!changeTypeColumn) {
if (!changeTypeColumn || Boolean.getBoolean("beam.test.enableManagedTransforms")) {
CdcSinkTestUtils.useLegacyDataflowWorker(p.getOptions());
}
Map<String, Object> readConfig = new HashMap<>();
Expand Down
Loading