Skip to content

Add Prism integration tests for Delta IO - #39764

Open
goutamadwant wants to merge 4 commits into
apache:masterfrom
goutamadwant:beam-38557-prism-delta-tests
Open

goutamadwant wants to merge 4 commits into
apache:masterfrom
goutamadwant:beam-38557-prism-delta-tests

Conversation

@goutamadwant

@goutamadwant goutamadwant commented Aug 16, 2026 •

Copy link
Copy Markdown
Contributor

Adds Prism coverage by running four existing Managed Delta IO tests: snapshot reads, version and timestamp reads, and change-data reads.

The dedicated Gradle task builds the local Prism binary and runs these tests in LOOPBACK mode using their existing local fixtures. No new test class or cloud credentials are needed. The existing DirectRunner and Dataflow test selections are unchanged.

The Delta IO precommit workflow runs the Prism task, and the trigger file is updated to run the suite for this PR.

Fixes #38557

Testing

  • Delta module build: 39 tests passed, 1 existing manual test skipped.
  • :sdks:java:io:delta:prismIntegrationTest: 4 tests passed with Java 17 and the locally built Prism binary.
  • :beam-test-gha:preCommit: passed.

Checklist

  • Linked the associated issue.
  • Added integration coverage and ran the affected module build.
  • CHANGES.md is not needed because this change only adds test and CI coverage.
  • This is not a large contribution requiring an ICLA.

@codecov

codecov Bot commented Aug 16, 2026 •

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 58.17%. Comparing base (a8d6afd) to head (56a26a5).
⚠️ Report is 75 commits behind head on master.

Additional details and impacted files
@@              Coverage Diff              @@
##             master   #39764       +/-   ##
=============================================
- Coverage     80.04%   58.17%   -21.88%     
- Complexity        0    13101    +13101     
=============================================
  Files           490     2523     +2033     
  Lines         82916   265755   +182839     
  Branches          0    10795    +10795     
=============================================
+ Hits          66373   154612    +88239     
- Misses        16543   105332    +88789     
- Partials          0     5811     +5811     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@github-actions

Copy link
Copy Markdown
Contributor

Assigning reviewers:

R: @ahmedabu98 for label java.
R: @liferoad for label build.

Note: If you would like to opt out of this review, comment assign to next reviewer.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

The PR bot will only process comments in the main thread (not review comments).

@github-actions

Copy link
Copy Markdown
Contributor

Reminder, please take a look at this pr: @ahmedabu98 @liferoad

@github-actions

Copy link
Copy Markdown
Contributor

Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment assign to next reviewer:

R: @Abacn for label java.
R: @damccorm for label build.

Available commands:

  • stop reviewer notifications - opt out of the automated review tooling
  • remind me after tests pass - tag the comment author after tests pass
  • waiting on author - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)

@damccorm

Copy link
Copy Markdown
Contributor

R: @chamikaramj

@github-actions

Copy link
Copy Markdown
Contributor

Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment assign set of reviewers

@chamikaramj chamikaramj left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks!

@@ -0,0 +1,141 @@
/*
* Licensed to the Apache Software Foundation (ASF) under one

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Instead of adding new tests, can we update the config to run existing tests with the Prism runner ?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

reused the four existing Managed IO tests and removed this new test class @chamikaramj

- name: run Delta IO Prism integration tests
uses: ./.github/actions/gradle-command-self-hosted-action
with:
gradle-command: :sdks:java:io:delta:prismIntegrationTest

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please also do a small update to the trigger file so that the test suite get triggered for this PR.
https://github.com/apache/beam/blob/master/.github/trigger_files/beam_PreCommit_Java_Delta_IO_Direct.json

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done @chamikaramj the Delta suite ran and passed, including the Prism step.

Comment thread .github/workflows/beam_PreCommit_Java_Delta_IO_Direct.yml Fixed
@chamikaramj

Copy link
Copy Markdown
Contributor

waiting on author

@github-actions github-actions Bot added the size/s S: 10-49 lines changed label Oct 25, 2026
@goutamadwant

Copy link
Copy Markdown
Contributor Author

@chamikaramj review fixes are pushed. can you PTAL when you get a chance? thanks!

@goutamadwant

Copy link
Copy Markdown
Contributor Author

@chamikaramj also could you rerun the failed Dataflow Examples check? WordCountIT hit a GCS read timeout while staging a JAR. unrelated to my change and I don't have access for it.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Task]: Add prism-based integration tests

4 participants