Repository navigation
[Task]: Setup a GCS based delta lake repo for integration testing #38556
Description
What needs to happen?
For context, please see https://s.apache.org/beam-delta-lake-source.
Issue Priority
Priority: 2 (default / most normal work should be filed as P2)
Issue Components
- Component: Python SDK
- Component: Java SDK
- Component: Go SDK
- Component: Typescript SDK
- Component: IO connector
- Component: Beam YAML
- Component: Beam examples
- Component: Beam playground
- Component: Beam katas
- Component: Website
- Component: Infrastructure
- Component: Spark Runner
- Component: Flink Runner
- Component: Prism Runner
- Component: Twister2 Runner
- Component: Hazelcast Jet Runner
- Component: Google Cloud Dataflow Runner
Activity
utkarshparekh commentedon Jun 25, 2026
@chamikaramj I can take this. I noticed the ignored test in DeltaIOTest still points at a local test_table_1_gb path. Do you have a preferred bucket/path and table set for the shared GCS repo, and should this complement the Dataflow IT in #38954? Happy to put together a PR either way.
chamikaramj commentedon Jul 9, 2026
Actually, we might not need this. We can setup repo in GCS easily when running a given test and delete at the end.
chamikaramj commentedon Aug 15, 2026
We already have Dataflow integration tests that setup a Delta Lake repo in GCS.
https://github.com/apache/beam/actions/workflows/beam_PostCommit_Java_Delta_IO_Dataflow.yml
Closing this.