[Bug]: Iceberg managed connector upgrade failed in GCP Dataflow

What happened?

We are getting an error running our Apache Beam pipeline in Dataflow (v2
runners) using the managed transform for iceberg in python (cross-language).

The dataflow job fails to start with error:

Dataflow is unable to process the managed transform(s)
ref_AppliedPTransform_WriteOutput-SchemaAwareExternalTransform-beam-transform-managed-v1_10(beam:schematransform:org.apache.beam:iceberg_write:v1)
due to an internal error. Because the upgrade failed, the new job or
the update request has been rejected. To troubleshoot, see
https://cloud.google.com/dataflow/docs/guides/common-errors#managed-transforms-upgrade.

We are using Python 3.12, Apache beam SDK 2.67.0.
The Iceberg connector we are using is:
https://cloud.google.com/dataflow/docs/guides/managed-io-iceberg

We are using the Java version in Python through the managed write transforms like this:

apache_beam.managed.Write(beam.managed.ICEBERG, config=iceberg_config)

The pipeline runs fine without the connector, but fails early on the job setup stage if added. No pipeline graph or worker logs are generated at this stage.

Issue Priority

Priority: 2 (default / most bugs should be filed as P2)

Issue Components

@alxmrs

@alxmrs @TheNeuralBit my colleague @marianobrc is hitting the above error trying to write to Iceberg using the x-language managed connector. This seems like it should be supported and we're a bit at a loss for next steps. Tagging you in case you happen to know, or know someone who knows, what may be happening here. 🙏

@ahmedabu98

@ahmedabu98 @chamikaramj

Thanks for filing this.

Can you check if the the Dataflow Log Explorer logs managed-transforms-worker and managed-transforms-worker-startup mentioned in [1] has additional information regarding the error ?

Also, can you mention your Iceberg config ?

If you can file a Google Cloud support ticket, the support team take a look at your exact job.

[1] https://cloud.google.com/dataflow/docs/guides/common-errors#managed-transforms-upgrade

@chamikaramj

Thanks for filing this.

Can you check if the the Dataflow Log Explorer logs managed-transforms-worker and managed-transforms-worker-startup mentioned in [1] has additional information regarding the error ?

Also, can you mention your Iceberg config ?

If you can file a Google Cloud support ticket, the support team take a look at your exact job.

[1] https://cloud.google.com/dataflow/docs/guides/common-errors#managed-transforms-upgrade

Thanks for looking into this @chamikaramj
The iceberg config is:

iceberg_config = {
    "table": "earthranger_catalog.warehouse.observations",
    "catalog_name": "earthranger_catalog",
    "catalog_properties": {
        "warehouse": "gs://dwh_dev/iceberg-warehouse/",
        "catalog-impl": "org.apache.iceberg.hadoop.HadoopCatalog",
    },
    "triggering_frequency_seconds": 60
}

There aren't any logs from the worker or worker-startup in the logs explorer, nor from harness or harness-startup, only the ones I shared from job-message.
Unfortunately, we cannot file a Google Cloud support ticket since we don't have a GCP Support plan.

Lemme try to run a job using a similar config.

BTW the logs you have to check are managed-transforms-worker and managed-transforms-worker-startup not the worker etc. you mentioned. These have to be explicitly selected in the Log Explorer UI with the correct time range. It might also help to unselect other logs. See the attached screenshot below.

@marianobrc

Can you share the job id? @marianobrc

@liferoad

Lemme try to run a job using a similar config.

BTW the logs you have to check are managed-transforms-worker and managed-transforms-worker-startup not the worker etc. you mentioned. These have to be explicitly selected in the Log Explorer UI with the correct time range. It might also help to unselect other logs. See the attached screenshot below.

Thank you for your quick response. For some reason, the managed- ones don't show up as an option for me:

If I clear all the filters (All log names), then those job logs are all I can see:

@liferoad here is the job id and other details in case are useful:

Job name
beamapp-dev-0929124506-405713-ih4cgl5j

Job ID
2025-09-29_05_47_09-6071847335088109426

Job type
Streaming

Job status
 Failed

SDK version
Apache Beam Python 3.12 SDK 2.67.0
 A newer version of the SDK family exists and updating is recommended. [Learn more ](https://cloud.google.com/dataflow/docs/support/sdk-version-support-status)

Job region
us-west1

Service zones
us-west1-b

Worker location 
us-west1

Thank you

Thanks. We are looking into this.

As a workaround, you can specify the following pipeline option when you start the job which prevents managed transforms expansion.

--experiments=use_pipeline_version_for_managed_transforms="beam:schematransform:org.apache.beam:iceberg_write:v1"

@chamikaramj

Thanks. We are looking into this.

As a workaround, you can specify the following pipeline option when you start the job which prevents managed transforms expansion.

--experiments=use_pipeline_version_for_managed_transforms="beam:schematransform:org.apache.beam:iceberg_write:v1"

@chamikaramj this workaround worked great, and our pipeline runs ok now. Would it be safe for us to use this workaround in prod or do you foresee any potential issues in doing that?

Thanks

It should be OK to use it in prod but you will not be getting Dataflow managed I/O service features: https://cloud.google.com/dataflow/docs/guides/managed-io

I'll let you know here when you can try without the experiment.

@chamikaramj

@chamikaramj quick question, does the iceberg connector (through the managed i/o service or not) support passing any configs to enable the upsert mode? We'd like data to be updated when processing rows with the same id (instead of inserting new rows).

Thanks

@ahmedabu98

@chamikaramj quick question, does the iceberg connector (through the managed i/o service or not) support passing any configs to enable the upsert mode? We'd like data to be updated when processing rows with the same id (instead of inserting new rows).

Thanks

I don't think this is currently supported but we can probably support this in the future.

cc: @ahmedabu98 @liferoad

We did several updates related to this (and more improvements are coming). Can you let us know if you are still running into this issue.

@chamikaramj

We did several updates related to this (and more improvements are coming). Can you let us know if you are still running into this issue.

Thank you @chamikaramj. We've switched to our own implementation of an Iceberg writer since the upsert mode support was key for our use case. Do you know if that is supported in the connector now?

@damccorm

We did several updates related to this (and more improvements are coming). Can you let us know if you are still running into this issue.

Thank you @chamikaramj. We've switched to our own implementation of an Iceberg writer since the upsert mode support was key for our use case. Do you know if that is supported in the connector now?

Not yet, but I can mention here when the Iceberg upsert mode is available.

cc: @damccorm @ahmedabu98 @rezarokni

@marianobrc

@marianobrc Upserts aren't supported yet. We're working on a few things but it's on the roadmap (might get to it around mid-2026)

This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.

This issue has been closed due to lack of activity. If you think that is incorrect, you can reopen the issue at any time.

@marianobrc

Hey @marianobrc! We just finished adding merge-on-read support to the Iceberg sink, so it can now apply CDC rows to Iceberg tables: #39979

There's an option to enable upsert mode. Feel free to check it out! It's in HEAD and should be coming out with the next release (2.77)

@marianobrc

Hey @marianobrc! We just finished adding merge-on-read support to the Iceberg sink, so it can now apply CDC rows to Iceberg tables: #39979

Thanks for letting me know @ahmedabu98. We have pivoted away from beam/dataflow, to a spark/dataproc job running on a schedule and doing merge-on-write to optimize reads, since it works better for our use case. But I'm sure this feature will be useful for someone else.

Navigation Menu

Repository navigation

[Bug]: Iceberg managed connector upgrade failed in GCP Dataflow #36340

What happened?

Issue Priority

Issue Components

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Bug]: Iceberg managed connector upgrade failed in GCP Dataflow #36340

Description

What happened?

Issue Priority

Issue Components

Activity

marianobrc commented on Sep 30, 2025

cisaacstern commented on Sep 30, 2025

liferoad commented on Oct 1, 2025

chamikaramj commented on Oct 1, 2025

marianobrc commented on Oct 1, 2025

chamikaramj commented on Oct 1, 2025

liferoad commented on Oct 1, 2025

marianobrc commented on Oct 1, 2025

chamikaramj commented on Oct 1, 2025

marianobrc commented on Oct 2, 2025

chamikaramj commented on Oct 2, 2025

marianobrc commented on Oct 2, 2025

chamikaramj commented on Oct 2, 2025

chamikaramj commented on Dec 3, 2025

marianobrc commented on Dec 4, 2025

chamikaramj commented on Dec 4, 2025

ahmedabu98 commented on Dec 6, 2025

github-actions commented on May 5, 2026

github-actions commented on Jun 4, 2026

ahmedabu98 commented on Sep 28, 2026

marianobrc commented on Sep 28, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions