Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
114 commits
Select commit Hold shift + click to select a range
2b4ecc5
add WRITE_TRUNCATE_DATA write_disposition to the bigquery python sdk
GaryPWhite Oct 16, 2026
6f9745d
changes.md
GaryPWhite Oct 16, 2026
a5a8a77
revert changes to bigquery_v2_messages
GaryPWhite Oct 16, 2026
d233c0c
Update CHANGES.md
GaryPWhite Oct 16, 2026
c43e2ea
trim doc line length
GaryPWhite Oct 16, 2026
b502882
Merge branch 'master' of https://github.com/garypwhite/beam
GaryPWhite Oct 16, 2026
9335f60
add WRITE_TRUNCATE_DATA support to bigqueryIO in java
GaryPWhite Oct 16, 2026
86504f7
Bump google.golang.org/api from 0.301.0 to 0.302.0 in /sdks (#40627)
dependabot[bot] Oct 16, 2026
0072521
[Java] Fix GRPC channel leak (#39663)
shunping Oct 16, 2026
3336b15
[Prism] Guard ClearOrderedListState against uninitialized state maps …
prodriguezdefino Oct 16, 2026
f3a5f4f
Add stale GCS test bucket cleaner (#40386)
goutamadwant Oct 16, 2026
6572166
[Java] Make non-extensible classes final instead of suppressing CT_CO…
mxtymoshyk Oct 16, 2026
1f76c9c
Fix Python unit tests broken by google-api-core arriving via cloud-sq…
shunping Oct 16, 2026
084fb32
Restore Java 11 support in setJavaVerOptions (#40634)
shunping Oct 16, 2026
d5f6472
Enable throttling logic in RemoteInference. (#40598)
tvalentyn Oct 16, 2026
964d15b
[Python] Document the import-time default of PeriodicImpulse start_ti…
raams95 Oct 16, 2026
a3f3c8e
Fix handling of cancelled tickets. (#40588)
tvalentyn Oct 16, 2026
1fc615f
[typescript] - fix typescript failures (#40635)
derrickaw Oct 16, 2026
a17e73a
Clarify known restrictions for model updates. (#40600)
tvalentyn Oct 16, 2026
a74f328
[yaml] - fix post commit yaml test (#40633)
derrickaw Oct 16, 2026
e41f0ce
[Python] Add examples for AWS credentials from a secret manager (#40592)
shunping Oct 16, 2026
8eda31b
[Python] Fix max_models_per_worker_hint not being enforced in KeyedMo…
schizophrenicmaniac Oct 16, 2026
2be5254
[Prism] Finalize bundles of DoFns fused after another transform (#40640)
prodriguezdefino Oct 17, 2026
8ef7176
Bump minimum cryptography bounds beyond known severe CVEs (#40591)
jrmccluskey Oct 19, 2026
40df757
Support attributing BigQuery quota and billing to a custom GCP projec…
shahar1 Oct 19, 2026
707b951
Preserve HTTP errors when quota response is not JSON (#40631)
Gaurav598 Oct 19, 2026
5fc82a6
[Spanner Change Streams] Match single-RPC deadline to total timeout i…
stankiewicz Oct 19, 2026
10d3b8d
Fix temp table cleanup for continuous read in BigQueryIO (#37535)
stankiewicz Oct 19, 2026
49d41ad
Bump google.golang.org/api from 0.302.0 to 0.304.0 in /sdks (#40650)
dependabot[bot] Oct 19, 2026
e4bc725
Bump zizmorcore/zizmor-action from 0.6.4 to 0.7.1 (#40652)
dependabot[bot] Oct 19, 2026
5fa8c2c
Bump github/codeql-action from 4.38.3 to 4.39.0 (#40651)
dependabot[bot] Oct 19, 2026
8648b89
Update users.yml (#40654)
stankiewicz Oct 19, 2026
590abac
[Python] Avoid ZeroDivisionError in ConcatRangeTracker for zero weigh…
raams95 Oct 19, 2026
14575dc
Update api key for tour of beam (#40655)
stankiewicz Oct 19, 2026
2c60b19
Fix postgres database for grafana (#40656)
Amar3tto Oct 19, 2026
bd10258
Run portable ValidatesRunner tests one at a time (#40632)
aIbrahiim Oct 19, 2026
6235796
Remove a race with another file write that is not essential to test p…
tvalentyn Oct 19, 2026
a8ee2b8
[Spark 4] Group the streaming stateful ParDo by encoded key bytes to …
tkaymak Oct 19, 2026
59a8647
Update nlp_tensorflow_streaming.ipynb (#40641)
ryanmcginnis Oct 19, 2026
5dc1270
[Prism] Keep a stateful key in progress while a split primary holds i…
prodriguezdefino Oct 19, 2026
05c4879
allow dropping pk or watermark column (#40457)
ahmedabu98 Oct 19, 2026
3bef0c0
[Python] Fail bundles on state write errors (#39992)
bvolpato Oct 19, 2026
3ab18d0
[Go SDK] Add portable logical type registration and standard date and…
Eliaaazzz Oct 19, 2026
f8b1e8c
[Python] Add typed decision questions and model adapters (#40220)
bvolpato Oct 19, 2026
75d7e2c
Process deletes before adds when committing OrderedList state updates…
acrites Oct 19, 2026
b4ecb26
Modernize legacy DoFn in BatchElementsExample (#40557) (#40560)
CH-Abhinav Oct 19, 2026
18db489
[Multikey] [Dataflow Streaming] Replace KeyGroupWorkQueue with a shar…
arunpandianp Oct 19, 2026
68e762e
Default to parquet in AddFiles for extension less files. (#40614)
claudevdm Oct 19, 2026
0c37058
Remove Beam SQL Data Catalog provider (#40663)
aIbrahiim Oct 19, 2026
d37ad24
[Python] Support max_batch_weight in BatchElements and honor element_…
tvalentyn Oct 15, 2026
86d036c
Apply element_size_fn and batch_length_fn to examples in keyed RunInf…
tvalentyn Oct 15, 2026
a8d6afd
Finish max_batch_weight support for RunInference: issue number in CHA…
Eliaaazzz Oct 20, 2026
ba412b0
Use with_output_types
Amar3tto Oct 20, 2026
f40b716
Bump distlib from 0.4.3 to 0.4.4 in /sdks/python (#40679)
dependabot[bot] Oct 20, 2026
41df604
Add missing properties (#40684)
Amar3tto Oct 20, 2026
4dbf0d7
Fix Release NightlySnapshot after enabling errorprone for Java17+ bui…
Abacn Oct 20, 2026
d746d74
[Python] Fix GCS client creation in non-default universes (#40674)
shunping Oct 20, 2026
62f901f
Upgrade google-api-client to 2.7.2 and use bomupgrade to manage (#40670)
shunping Oct 20, 2026
cdf9c3a
Update google-adk dependency
jrmccluskey Oct 20, 2026
bf57625
Revert errant google-adk bump on master (#40687)
jrmccluskey Oct 20, 2026
d8ef6a9
Bump github.com/klauspost/compress in /playground/backend (#40688)
dependabot[bot] Oct 20, 2026
45e6dd2
Fix chown
Amar3tto Oct 20, 2026
a412e69
enable canceling SQL streaming pipeline (#40666)
ahmedabu98 Oct 20, 2026
2fdcfb8
Render ARRAY and ITERABLE fields that are not backed by a List (#39750)
PDGGK Oct 20, 2026
f014f5a
Update test case for numpy 2.4.6 (#40692)
jrmccluskey Oct 20, 2026
9ea8253
Add -R flag
Amar3tto Oct 20, 2026
7bdc6ff
Fix Release Nightly Snapshot job (#40685)
Amar3tto Oct 20, 2026
14be1c0
Update google-adk dependency range (#40689)
jrmccluskey Oct 20, 2026
8a9afdc
Fix bigquery_tools_test to mock gcp_bigquery library instead... (#40676)
chamikaramj Oct 20, 2026
9146ff2
Merge pull request #40693 from apache/fix-publish-website
Amar3tto Oct 20, 2026
d50c0a8
[RunInference] Reject use_model_manager with a multi-model KeyedModel…
schizophrenicmaniac Oct 20, 2026
0724ef9
[Spark 4] Clear the watermark of keys without timers or state in the …
schizophrenicmaniac Oct 20, 2026
11e1ce2
Add an option to set the Dataflow container base repository (Python, …
shunping Oct 20, 2026
ceeb5fa
Merge pull request #40682 from Eliaaazzz/fix/runinference-max-batch-w…
tvalentyn Oct 21, 2026
f1ec3ac
[Go] Support non-default universes for GCS and Dataflow clients (#40686)
shunping Oct 21, 2026
e18c217
[Python] Infer the right type for slices of fixed-size tuples (#40680)
raams95 Oct 21, 2026
bbd5e28
[RunInference] Honor model updates in also in the first batch. (#40667)
tvalentyn Oct 21, 2026
3988ad7
Apply before set image
Amar3tto Oct 21, 2026
fa6c858
Merge pull request #40706 from apache/fix-publish-beam-metrics
Amar3tto Oct 21, 2026
8cae499
[Java] Close GcsUtilV2's small-file performance gaps with GcsUtilV1 (…
shunping Oct 21, 2026
b54c607
move pipeline examples from blueprint to pipelines (#40615)
derrickaw Oct 21, 2026
63d8acc
[BEAM-19954] Resolve string annotations in callable type hints (#40625)
Gaurav598 Oct 21, 2026
5b9c5ba
test: allow users to set custom Neo4j config (#40700)
fbiville Oct 21, 2026
06e4102
[yaml] - fix spanner configs and examples (#40587)
derrickaw Oct 21, 2026
b4f9f3e
Update xgboost dependency, execute in CI (#40694)
jrmccluskey Oct 21, 2026
f3f7ad1
Fix prober community metrics (#40707)
Amar3tto Oct 21, 2026
5b6aa70
Merge pull request #40683 from apache/fix-spanner-flatmap
Amar3tto Oct 21, 2026
f5f6006
rename TestRecord (#40728)
aIbrahiim Oct 22, 2026
1484637
drop transformers 428 (#40719)
aIbrahiim Oct 22, 2026
9811046
[DebeziumIO] Add opt-in full metadata (op and Debezium source fields)…
tkaymak Oct 22, 2026
e261c9f
[#36841] Spark 4 streaming: windowed GroupByKey and Combine via trans…
tkaymak Oct 22, 2026
cae975b
Update FlareDB runner doc (#40576)
ganesh-skumar Oct 22, 2026
248548d
Merge pull request #40303 from reuvenlax/merge_validates_runner_phase2
reuvenlax Oct 22, 2026
3d5375a
Expand Beam SQL TPC-DS query support and add Python TPC-DS suite (#40…
Abacn Oct 22, 2026
9f5f83e
[Python] Keep BigQuery load jobs distinct for table aliases (#40001)
bvolpato Oct 22, 2026
6e5fdbf
Temporarily skip Iceberg cdc tests for managed service (#40731)
ahmedabu98 Oct 22, 2026
94b84b6
[Java] Use GcsUtil V2 by default (#40713)
shunping Oct 22, 2026
1fcf426
Upgrade allowed version of pillow (#40708)
damccorm Oct 22, 2026
e333810
Only feed total memory readings to the ModelManager resource estimato…
schizophrenicmaniac Oct 22, 2026
d593d38
[Python] Support custom BigQuery endpoints and universe domains (#40675)
shunping Oct 22, 2026
d4ba44d
[Java] Honor GOOGLE_CLOUD_UNIVERSE_DOMAIN for the Dataflow endpoint (…
shunping Oct 22, 2026
c8b5acf
Scope RunInference model_identifier annotation to BeamML_RunInference…
AMOOOMA Oct 22, 2026
e4c2f8c
Honor allowMissingRequiredFields when building Storage Write API mess…
khaledh Oct 22, 2026
932dbe6
[Java][BigQueryIO] Fix BigQueryIO streaming insert silently dropping …
damjad Oct 23, 2026
6848f84
Handle null split initialization results in Flink lazy source enumera…
ddebowczyk92 Oct 23, 2026
1ea3175
include WRITE_TRUNCATE_DATA as a write disposition in java impl
GaryPWhite Oct 23, 2026
8acf928
add WRITE_TRUNCATE_DATA write_disposition to the bigquery python sdk
GaryPWhite Oct 16, 2026
88f0966
changes.md
GaryPWhite Oct 16, 2026
c276cc5
revert changes to bigquery_v2_messages
GaryPWhite Oct 16, 2026
6e55f62
trim doc line length
GaryPWhite Oct 16, 2026
a8030f0
Update CHANGES.md
GaryPWhite Oct 16, 2026
62b9965
add WRITE_TRUNCATE_DATA support to bigqueryIO in java
GaryPWhite Oct 16, 2026
55d8dae
include WRITE_TRUNCATE_DATA as a write disposition in java impl
GaryPWhite Oct 23, 2026
caf01f7
changes.md
GaryPWhite Oct 23, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
2 changes: 1 addition & 1 deletion .github/trigger_files/IO_Iceberg_Integration_Tests.json
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{
"comment": "Modify this file in a trivial way to cause this test suite to run.",
"modification": 11
"modification": 1
}
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"comment": "Modify this file in a trivial way to cause this test suite to run.",
"modification": 2,
"modification": 1,
"https://github.com/apache/beam/pull/39990": "removing dead code from FnApiDoFnRunner"
}
2 changes: 1 addition & 1 deletion .github/trigger_files/beam_PostCommit_Java.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"https://github.com/apache/beam/pull/39893": "Fix nullness in BigQueryIO",
"comment": "Modify this file in a trivial way to cause this test suite to run",
"modification": 6
"modification": 7
}
2 changes: 1 addition & 1 deletion .github/trigger_files/beam_PostCommit_Java_DataflowV1.json
Original file line number Diff line number Diff line change
Expand Up @@ -4,6 +4,6 @@
"https://github.com/apache/beam/pull/34902": "Introducing OutputBuilder",
"https://github.com/apache/beam/pull/35177": "Introducing WindowedValueReceiver to runners",
"comment": "Modify this file in a trivial way to cause this test suite to run",
"modification": 9,
"modification": 10,
"https://github.com/apache/beam/pull/35159": "moving WindowedValue and making an interface"
}
2 changes: 1 addition & 1 deletion .github/trigger_files/beam_PostCommit_Java_DataflowV2.json
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"https://github.com/apache/beam/pull/39893": "Fix nullness in BigQueryIO",
"modification": 9,
"modification": 10,
"https://github.com/apache/beam/pull/35159": "moving WindowedValue and making an interface",
"https://github.com/apache/beam/pull/39990": "removing dead code from FnApiDoFnRunner"
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"comment": "Modify this file in a trivial way to cause this test suite to run",
"modification": 1
}
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,6 @@
"https://github.com/apache/beam/pull/36138": "Cleanly separating v1 worker and v2 sdk harness container image handling",
"https://github.com/apache/beam/pull/34902": "Introducing OutputBuilder",
"comment": "Modify this file in a trivial way to cause this test suite to run",
"modification": 2,
"modification": 3,
"https://github.com/apache/beam/pull/39990": "removing dead code from FnApiDoFnRunner"
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"comment": "Modify this file in a trivial way to cause this test suite to run",
"modification": 1
}
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{
"comment": "Modify this file in a trivial way to cause this test suite to run",
"modification": 5
"modification": 6
}
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{
"comment": "Modify this file in a trivial way to cause this test suite to run",
"modification": 4,
"modification": 5,
}
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"comment": "Modify this file in a trivial way to cause this test suite to run",
"https://github.com/apache/beam/pull/32440": "test new datastream runner for batch",
"modification": 2,
"modification": 8,
"https://github.com/apache/beam/pull/39990": "removing dead code from FnApiDoFnRunner"
}
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
{
"comment": "Modify this file in a trivial way to cause this test suite to run",
"modification": 2,
"modification": 3,
"https://github.com/apache/beam/pull/32440": "test new datastream runner for batch",
"https://github.com/apache/beam/pull/39990": "removing dead code from FnApiDoFnRunner"
}
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
{}
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{
"comment": "Modify this file in a trivial way to cause this test suite to run",
"modification": 1
"modification": 2
}
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{
"comment": "Modify this file in a trivial way to cause this test suite to run",
"modification": 1
"modification": 2
}
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"comment": "Modify this file in a trivial way to cause this test suite to run!",
"modification": 7,
"modification": 14,
}

Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{
"comment": "Modify this file in a trivial way to cause this test suite to run",
"modification": 3
"modification": 4
}
Original file line number Diff line number Diff line change
Expand Up @@ -12,5 +12,5 @@
"https://github.com/apache/beam/pull/40103": "noting that PR #40103 should run this test",
"https://github.com/apache/beam/issues/40427": "Spark ValidatesRunner tests run serially",
"https://github.com/apache/beam/pull/40281": "noting that PR #40281 should run this test",
"https://github.com/apache/beam/issues/36841": "Spark 4 streaming watermark lag"
"https://github.com/apache/beam/issues/36841": "Spark 4 streaming GroupByKey"
}
2 changes: 1 addition & 1 deletion .github/trigger_files/beam_PostCommit_Python.json
Original file line number Diff line number Diff line change
@@ -1,5 +1,5 @@
{
"comment": "Modify this file in a trivial way to cause this test suite to run.",
"pr": "40241",
"modification": 65
"modification": 66
}
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{
"comment": "Modify this file in a trivial way to cause this test suite to run",
"modification": 4
"modification": 8
}
2 changes: 1 addition & 1 deletion .github/trigger_files/beam_PostCommit_Python_Versions.json
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{
"comment": "Modify this file in a trivial way to cause this test suite to run",
"revision": 5
"revision": 6
}
2 changes: 1 addition & 1 deletion .github/trigger_files/beam_PostCommit_XVR_Direct.json
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{
"modification": 2,
"modification": 3,
"https://github.com/apache/beam/pull/39990": "removing dead code from FnApiDoFnRunner"
}
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{
"comment": "Modify this file in a trivial way to cause this test suite to run",
"revision": 1
"revision": 2
}
4 changes: 4 additions & 0 deletions .github/trigger_files/beam_PreCommit_Java_IOs_Direct.json
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
{
"comment": "Modify this file in a trivial way to cause this test suite to run.",
"modification": 1
}
2 changes: 1 addition & 1 deletion .github/trigger_files/beam_PreCommit_Python_ML.json
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{
"comment": "Modify this file in a trivial way to cause this test suite to run",
"revision": 7
"revision": 23
}
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
{
"comment": "Modify this file in a trivial way to cause this test suite to run",
"revision": 2
"revision": 3
}
1 change: 1 addition & 0 deletions .github/workflows/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -372,6 +372,7 @@ PostCommit Jobs run in a schedule against master branch and generally do not get
| [ PostCommit Python Examples Spark ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Examples_Spark.yml) | ['3.9','3.11','3.12'] |`beam_PostCommit_Python_Examples_Spark.json`| [![.github/workflows/beam_PostCommit_Python_Examples_Spark.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Examples_Spark.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Examples_Spark.yml?query=event%3Aschedule) |
| [ PostCommit Python MongoDBIO IT ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_MongoDBIO_IT.yml) | N/A |`beam_PostCommit_Python_MongoDBIO_IT.json`| [![.github/workflows/beam_PostCommit_Python_MongoDBIO_IT.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_MongoDBIO_IT.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_MongoDBIO_IT.yml?query=event%3Aschedule) |
| [ PostCommit Python Nexmark Direct ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Nexmark_Direct.yml) | N/A |`beam_PostCommit_Python_Nexmark_Direct.json`| [![.github/workflows/beam_PostCommit_Python_Nexmark_Direct.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Nexmark_Direct.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Nexmark_Direct.yml?query=event%3Aschedule) |
| [ PostCommit Python Tpcds Prism ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Tpcds_Prism.yml) | N/A |`beam_PostCommit_Python_Tpcds_Prism.json`| [![.github/workflows/beam_PostCommit_Python_Tpcds_Prism.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Tpcds_Prism.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_Tpcds_Prism.yml?query=event%3Aschedule) |
| [ PostCommit Python ValidatesContainer Dataflow ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesContainer_Dataflow.yml) | ['3.9','3.10','3.11','3.12'] |`beam_PostCommit_Python_ValidatesContainer_Dataflow.json`| [![.github/workflows/beam_PostCommit_Python_ValidatesContainer_Dataflow.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesContainer_Dataflow.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesContainer_Dataflow.yml?query=event%3Aschedule) |
| [ PostCommit Python ValidatesContainer Dataflow With RC ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesContainer_Dataflow_With_RC.yml) | ['3.9','3.10','3.11','3.12'] |`beam_PostCommit_Python_ValidatesContainer_Dataflow_With_RC.json`| [![.github/workflows/beam_PostCommit_Python_ValidatesContainer_Dataflow_With_RC.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesContainer_Dataflow_With_RC.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesContainer_Dataflow_With_RC.yml?query=event%3Aschedule) |
| [ PostCommit Python ValidatesRunner Dataflow ](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Dataflow.yml) | ['3.9','3.11','3.12'] |`beam_PostCommit_Python_ValidatesRunner_Dataflow.json`| [![.github/workflows/beam_PostCommit_Python_ValidatesRunner_Dataflow.yml](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Dataflow.yml/badge.svg?event=schedule)](https://github.com/apache/beam/actions/workflows/beam_PostCommit_Python_ValidatesRunner_Dataflow.yml?query=event%3Aschedule) |
Expand Down
8 changes: 4 additions & 4 deletions .github/workflows/beam_PostCommit_Java_Tpcds_Dataflow.yml
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,7 @@ name: PostCommit Java Tpcds Dataflow

on:
schedule:
- cron: '30 4/6 * * *'
- cron: '30 4/8 * * *'
pull_request:
paths: ['release/trigger_all_tests.json', '.github/trigger_files/beam_PostCommit_Java_Tpcds_Dataflow.json']
workflow_dispatch:
Expand Down Expand Up @@ -54,8 +54,8 @@ env:
--dataSize=1GB
--sourceType=PARQUET
--dataDirectory=gs://beam-tpcds/datasets/parquet/nonpartitioned
--resultsDirectory=gs://beam-tpcds/results/dataflow/
--tpcParallel=1
--resultsDirectory=gs://temp-storage-for-perf-tests/tpcds/results/dataflow
--tpcParallel=4
INFLUXDB_USER: ${{ secrets.INFLUXDB_USER }}
INFLUXDB_USER_PASSWORD: ${{ secrets.INFLUXDB_USER_PASSWORD }}
tpcdsBigQueryArgs: |
Expand All @@ -71,7 +71,7 @@ env:
--baseInfluxMeasurement=tpcds
--exportSummaryToInfluxDB=true
--influxRetentionPolicy=forever
tpcdsQueriesArg: 3,7,10,25,26,29,35,38,40,42,43,52,55,69,79,83,84,87,93,96
tpcdsQueriesArg: supported

jobs:
beam_PostCommit_Java_Tpcds_Dataflow:
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/beam_PostCommit_Java_Tpcds_Flink.yml
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ env:
--dataSize=1GB
--sourceType=PARQUET
--dataDirectory=gs://beam-tpcds/datasets/parquet/nonpartitioned
--resultsDirectory=gs://beam-tpcds/results/flink/
--resultsDirectory=gs://temp-storage-for-perf-tests/tpcds/results/flink
--tpcParallel=1
INFLUXDB_USER: ${{ secrets.INFLUXDB_USER }}
INFLUXDB_USER_PASSWORD: ${{ secrets.INFLUXDB_USER_PASSWORD }}
Expand All @@ -68,7 +68,7 @@ env:
--baseInfluxMeasurement=tpcds
--exportSummaryToInfluxDB=true
--influxRetentionPolicy=forever
tpcdsQueriesArg: 3,7,10,25,26,29,35,38,40,42,43,52,55,69,79,83,84,87,93,96
tpcdsQueriesArg: supported

jobs:
beam_PostCommit_Java_Tpcds_Flink:
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/beam_PostCommit_Java_Tpcds_Spark.yml
Original file line number Diff line number Diff line change
Expand Up @@ -49,7 +49,7 @@ env:
--dataSize=1GB
--sourceType=PARQUET
--dataDirectory=gs://beam-tpcds/datasets/parquet/nonpartitioned
--resultsDirectory=gs://beam-tpcds/results/spark3-rdd/
--resultsDirectory=gs://temp-storage-for-perf-tests/tpcds/results/spark3-rdd
--tpcParallel=1
INFLUXDB_USER: ${{ secrets.INFLUXDB_USER }}
INFLUXDB_USER_PASSWORD: ${{ secrets.INFLUXDB_USER_PASSWORD }}
Expand All @@ -66,7 +66,7 @@ env:
--baseInfluxMeasurement=tpcds
--exportSummaryToInfluxDB=true
--influxRetentionPolicy=forever
tpcdsQueriesArg: 3,7,10,25,26,29,35,38,40,42,43,52,55,69,79,83,84,87,93,96
tpcdsQueriesArg: supported

jobs:
beam_PostCommit_Java_Tpcds_Spark:
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ jobs:
uses: ./.github/actions/gradle-command-self-hosted-action
with:
gradle-command: :runners:google-cloud-dataflow-java:validatesRunner
max-workers: 12
max-workers: 24
- name: Archive JUnit Test Results
uses: actions/upload-artifact@v7
if: ${{ !success() }}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -81,7 +81,7 @@ jobs:
arguments: |
-PtestJavaVersion=${{ matrix.java_version }} \
-Pjava${{ matrix.java_version }}Home=$JAVA_HOME_${{ matrix.java_version }}_X64 \
max-workers: 12
max-workers: 24
- name: Archive JUnit Test Results
uses: actions/upload-artifact@v7
if: ${{ !success() }}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -74,7 +74,7 @@ jobs:
uses: ./.github/actions/gradle-command-self-hosted-action
with:
gradle-command: :runners:google-cloud-dataflow-java:validatesRunnerV2
max-workers: 12
max-workers: 24
- name: Archive JUnit Test Results
uses: actions/upload-artifact@v7
if: ${{ !success() }}
Expand Down
75 changes: 75 additions & 0 deletions .github/workflows/beam_PostCommit_Python_Tpcds_Prism.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,75 @@
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements. See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License. You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

name: PostCommit Python Tpcds Prism

on:
schedule:
- cron: '30 4/6 * * *'
pull_request:
paths: ['release/trigger_all_tests.json', '.github/trigger_files/beam_PostCommit_Python_Tpcds_Prism.json']
workflow_dispatch:

# Setting explicit permissions for the action
permissions:
actions: write
pull-requests: read
checks: read
contents: read
deployments: read
id-token: none
issues: read
discussions: read
packages: read
pages: read
repository-projects: read
security-events: read
statuses: read

# This allows a subsequently queued workflow run to interrupt previous runs
concurrency:
group: '${{ github.workflow }} @ ${{ github.event.pull_request.number || github.sha || github.head_ref || github.ref }}-${{ github.event.schedule || github.event.sender.login }}'
cancel-in-progress: true

env:
DEVELOCITY_ACCESS_KEY: ${{ secrets.BEAM_DEVELOCITY_ACCESS_KEY }}

jobs:
beam_PostCommit_Python_Tpcds_Prism:
if: |
github.event_name == 'workflow_dispatch' ||
github.event_name == 'pull_request' ||
(github.event_name == 'schedule' && github.repository == 'apache/beam')
runs-on: [self-hosted, ubuntu-24.04, main]
timeout-minutes: 60
name: ${{ matrix.job_name }}
strategy:
matrix:
job_name: ["beam_PostCommit_Python_Tpcds_Prism"]
steps:
- uses: actions/checkout@v7
with:
persist-credentials: false
- name: Setup repository
uses: ./.github/actions/setup-action
- name: Setup environment
uses: ./.github/actions/setup-environment-action
with:
python-version: default
go-version: default
- name: Run full TPC-DS suite on PrismRunner with regression check
uses: ./.github/actions/gradle-command-self-hosted-action
with:
gradle-command: :sdks:python:apache_beam:testing:benchmarks:tpcds:checkAllQueries
2 changes: 1 addition & 1 deletion .github/workflows/beam_PreCommit_GHA.yml
Original file line number Diff line number Diff line change
Expand Up @@ -96,7 +96,7 @@ jobs:
- name: Validate GHA Allowlist
uses: apache/infrastructure-actions/allowlist-check@main # zizmor: ignore[unpinned-uses]
- name: Run zizmor
uses: zizmorcore/zizmor-action@cc914d7f3750a2d13d75c7f184a1060aa0e9d482 # v0.6.4
uses: zizmorcore/zizmor-action@85b2b9fb8e3325e7a23d7d0a9d7ba5ae0f999e3e # v0.7.1
with:
advanced-security: true
- name: run GHA PreCommit script
Expand Down
4 changes: 2 additions & 2 deletions .github/workflows/codeql.yml
Original file line number Diff line number Diff line change
Expand Up @@ -154,7 +154,7 @@ jobs:

# Initializes the CodeQL tools for scanning.
- name: Initialize CodeQL
uses: github/codeql-action/init@v4.38.3
uses: github/codeql-action/init@v4.39.0
with:
languages: ${{ matrix.language }}
build-mode: ${{ matrix.build-mode }}
Expand Down Expand Up @@ -193,6 +193,6 @@ jobs:
fi

- name: Perform CodeQL Analysis
uses: github/codeql-action/analyze@v4.38.3
uses: github/codeql-action/analyze@v4.39.0
with:
category: "/language:${{matrix.language}}"
10 changes: 4 additions & 6 deletions .test-infra/metrics/build.gradle
Original file line number Diff line number Diff line change
Expand Up @@ -121,19 +121,17 @@ task deploy {
exec {
commandLine './build_and_publish_containers.sh', 'true', "${project.lastCommitId}"
}

exec {
executable 'sh'
args '-c', 'kubectl --namespace default apply -f kubernetes/beamgrafana-deploy.yaml'
}
exec {
executable 'sh'
args '-c', "kubectl --namespace default set image deployment/beamgrafana \
beamgrafana=gcr.io/apache-beam-testing/beamgrafana:${project.lastCommitId} \
beammetricssyncjenkins=gcr.io/apache-beam-testing/beammetricssyncjenkins:${project.lastCommitId} \
beammetricssyncgithub=gcr.io/apache-beam-testing/beammetricssyncgithub:${project.lastCommitId}"
}

exec {
executable 'sh'
args '-c', 'kubectl --namespace default apply -f kubernetes/beamgrafana-deploy.yaml'
}
} else {
println 'Nothing to do, exiting.'
}
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -29,8 +29,8 @@ datasources:
secureJsonData:
password: ${PSQL_DB_DBPWD}
user: ${PSQL_DB_DBUSERNAME}
database: ${PSQL_DB_DBNAME}
jsonData:
database: ${PSQL_DB_DBNAME}
sslmode: disable
editable: false

Loading
Loading