[Amar3tto]

Adds YAML support for reading Google Cloud Bigtable change streams.

This PR:

• adds a Bigtable CDC SchemaTransform provider
• converts ChangeStreamMutation records to Beam Rows
• supports SetCell, DeleteCells, DeleteFamily, AddToCell, and MergeToCell entries
• registers ReadFromBigtableCDC in standard_io.yaml
• adds unit tests for configuration and mutation entry conversion

Tested with:

./gradlew :sdks:java:io:google-cloud-platform:test \
  --tests org.apache.beam.sdk.io.gcp.bigtable.changestreams.BigtableChangeStreamReadSchemaTransformProviderTest

Successful run: https://github.com/apache/beam/actions/runs/35729484761

---

Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:

• Mention the appropriate issue in your description (for example: addresses #123), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment fixes #<ISSUE NUMBER> instead.
• Update CHANGES.md with noteworthy changes.
• If this contribution is large, please file an Apache Individual Contributor License Agreement.

See the Contributor Guide for more tips on how to make review process smoother.

To check the build health, please visit https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md

GitHub Actions Tests Status (on master branch)

See CI.md for more information about GitHub Actions CI or the workflows README to see a list of phrases to trigger workflows.

[github-actions]

Reminder, please take a look at this pr: @Abacn

[damccorm]

Load balancing out some reviews.

R: @derrickaw

[github-actions]

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

[derrickaw]

R: @ahmedabu98, can you take a quick peek at this to see if anything else is missing? Thanks

[ahmedabu98]

Looks good from a skim, just consider adding a flattened output option.

Also please add a YAML IT

[ahmedabu98]

This output schema looks complex, which will be noticed when using it with YAML.

In the regular BigtableReadSchemaTransformProvider, we added an option to use a flattened schema (one output per entry, instead of per mutation). Consider doing something similar here

[Amar3tto]

Done

[ahmedabu98]

cc @mutianf might be relevant for you

[Amar3tto]

please add a YAML IT

I tried adding a Bigtable CDC YAML IT using a temporary change-stream-enabled table and a bounded end time, but the pipeline consistently times out in the YAML Direct/FnApi integration-test harness.

Do you have a recommendation for how this integration test should be structured for a streaming CDC source? ReadFromSpannerCDC seems to have a similar setup, but I couldn't find a YAML IT for it either.

[mutianf]

@jackdingilian

[derrickaw]

LGTM for yaml, @ahmedabu98 - did you want to take one more peek? Thanks

[ahmedabu98]

but the pipeline consistently times out in the YAML Direct/FnApi integration-test harness

Hmm why is it timing out? does this mean the end_at_timestamp config option doesn't work?