Repository navigation

Added Filter and Projection Pushdown support to ParquetIO and Beam SQL's ParquetTable - #35589

Closed

talatuyarer wants to merge 499 commits into

apache:masterfrom

talatuyarer:parquet-filter-pushdown-support

Closed

Added Filter and Projection Pushdown support to ParquetIO and Beam SQL's ParquetTable#35589
talatuyarer wants to merge 499 commits into
apache:masterfrom
talatuyarer:parquet-filter-pushdown-support

This pull request is big! We're only showing the most recent 250 commits

Commits on Aug 27, 2025
Commits on Aug 28, 2025
Commits on Aug 29, 2025
Commits on Aug 30, 2025
- [Prism] Surface log format option and improve logging presentation (#36008)
  shunping
  authored
  258b8d1
- python: replace <=3.12 with <3.13 for google-apitools marker (semantics unchanged) (#35929)
  
  diegofhdz
  and
  Diego Hernandez
  authored
  4bb6e5a
- update description and logs for timeout (#36003)
  
  LEEKYE
  and
  tvalentyn
  authored
  6bdfcb3
- Pass update compat through as_deterministic_coder and use cloudpickle for deterministic special types. (#35725)
  
  authored
  e8fab26
- Fix PostCommit Python ValidatesContainer Dataflow With RC job (#36016)
  Amar3tto
  authored
  f1a0855
Commits on Sep 1, 2025
- Bump github.com/aws/aws-sdk-go-v2/config from 1.31.4 to 1.31.6 in /sdks (#36025)
  dependabot[bot]
  authored
  17d5039
Commits on Sep 2, 2025
- Upgrade Errorprone to 2.31.0 (#35684)
  Abacn
  authored
  2ab70ff
- Bump cloud.google.com/go/bigquery from 1.69.0 to 1.70.0 in /sdks (#36024)
  dependabot[bot]
  authored
  266de26
- Exit the poll loop in ReadFromKafkaDoFn after consumerPollingTimeout has elapsed (#36029)
  sjvanrossum
  authored
  f69fd72
- refactor(build): update changelog formatting in gradle task (#36021)
  liferoad
  authored
  a4ad5cd
- Add validation to check for non-ints from partitionfns (#35881)
  hjtran
  authored
  9f3b160
- Cache deterministic types (#36032)
  
  claudevdm
  and
  Claude
  authored
  523d833
- Update CHANGES.md (#36033)
  flpablo
  authored
  f725e85
- Allow stateful exception handling (#35965)
  damccorm
  authored
  4289ea2
- [Prism] Fix two kinds of errors when using json log format. (#36020)
  shunping
  authored
  7d6de9a
- Introduce more abstract handling of throttling signaling (#35984)
  
  jrmccluskey
  and
  damccorm
  authored
  35969b3
Commits on Sep 3, 2025
- Bump cloud.google.com/go/spanner from 1.83.0 to 1.85.0 in /sdks (#36005)
  dependabot[bot]
  authored
  faacf70
- Moving to 2.69.0-SNAPSHOT on master branch.
  
  Amar3tto
  authored and
  Amar3tto
  committed
  f3441bd
- Update CHANGES.md to have fields for 2.69.0 release
  Amar3tto
  committed
  6280b39
- Cloudpickle deterministic coder changes.md. (#36037)
  
  authored
  230f3ea
- Support Protobuf 6.x. (#35477)
  yilei
  authored
  9adc8dc
- Only retrieve the shared caches once during setup in ReadFromKafkaDoFn (#36028)
  sjvanrossum
  authored
  cccaa75
- feat(build): add validation task for CHANGES.md formatting (#36041)
  liferoad
  authored
  d28fd64
- Fix jdbc logical type not found when python sdk worker running in docker env (#36014)
  shunping
  authored
  67eaded
Commits on Sep 4, 2025
- fix(Partition): support numpy integer types in partition validation (#36044)
  liferoad
  authored
  0a536e7
- Comment send email action
  Amar3tto
  committed
  20b52a1
- Bump github.com/aws/aws-sdk-go-v2/service/s3 in /sdks (#36042)
  dependabot[bot]
  authored
  07ea0f8
- Comment send email action (#36050)
  Amar3tto
  authored
  1982627
- Update notebook to use embedding gemma (#36035)
  damccorm
  authored
  32d53ab
- Add Iceberg to AlloyDB Blueprint (#36036)
  tarun-google
  authored
  c367bd3
- [YAML] Fraud Detection MLOps Workflow Example (#36012)
  charlespnh
  authored
  626be06
Commits on Sep 5, 2025
Commits on Sep 6, 2025
- Add YAML Editor and Visualization Panel (#35947)
  
  Chenzo1001
  and
  gemini-code-assist[bot]
  authored
  d9ddcd4
- YAML example suite minor updates (#36074)
  charlespnh
  authored
  0ba89bb
- Update iceberg to 1.9.2 (#35981)
  ahmedabu98
  authored
  32f5be6
- Only update size metrics once per batch
  sjvanrossum
  committed
  520b456
Commits on Sep 8, 2025
Commits on Sep 9, 2025
Commits on Sep 10, 2025
Commits on Sep 11, 2025
Commits on Sep 12, 2025
- Fix flaky GrpcDirectGetWorkStreamTest.testConsumedWorkItems_itemsSplitAcrossResponses (#36129)
  scwhittle
  authored
  abbaab8
- Update import gpg action (#36130)
  Amar3tto
  authored
  ab2db45
- Update action (#36131)
  Amar3tto
  authored
  f296000
- Fix ml gradle (#36134)
  damccorm
  authored
  55bbbb3
- workflows: add `ubuntu-latest` to `beam_PreCommit_Python_ML`, `beam_PostCommit_Python`, and `beam_PreCommit_Python_Coverage` (#35816)
  moawnallah
  authored
  1c74621
- Managed Bigquery Java Read&Write Transforms
  tarun-google
  committed
  6d5af4b
- switch to bigquery
  tarun-google
  committed
  469132a
- Make dill optional (#36093)
  
  claudevdm
  and
  Claude
  authored
  1211cc8
Commits on Sep 13, 2025
- [Prism] Add previousInput watermark and use it in bundleReady (#36137)
  shunping
  authored
  fe5c288
- Log an error if parsing a meta option file fails. (#36060)
  baeminbo
  authored
  aa42205
Commits on Sep 14, 2025
- Sort output before comparison in enrichment test (#36143)
  liferoad
  authored
  6077034
Commits on Sep 15, 2025
Commits on Sep 16, 2025
Commits on Sep 17, 2025
Commits on Sep 18, 2025
Commits on Sep 19, 2025
Commits on Sep 20, 2025
- Install go for python unit tests to use prism runner. (#36221)
  shunping
  authored
  1d30d0c
- Revert "Install go for python unit tests to use prism runner. (#36221)" (#36223)
  shunping
  authored
  65dfd30
Commits on Sep 21, 2025
- Infra iam enforcer service accounts (#36215)
  ksobrenat32
  authored
  3659832
- Fix panic in teststream. (#36227)
  shunping
  authored
  354a387
Commits on Sep 22, 2025
- Bump github.com/docker/docker in /sdks (#36230)
  dependabot[bot]
  authored
  7b22d61
- [Java] Change precondition checks to use lazy formatting (#36065)
  scwhittle
  authored
  a5c9850
- Bump github.com/testcontainers/testcontainers-go in /sdks (#36231)
  dependabot[bot]
  authored
  6d80ed6
- Revert "Cleanly separate v1 worker and v2 sdk harness container image handling in DataflowRunner" (#36216)
  kennknowles
  authored
  8fd07a4
- Remove Vertex LLM Inference Integration Test (#36172)
  
  jrmccluskey
  and
  gemini-code-assist[bot]
  authored
  11f7568
- Bump cloud.google.com/go/storage from 1.56.1 to 1.56.2 in /sdks (#36204)
  dependabot[bot]
  authored
  df255a3
- Add support for PROTO format in YAML Pub/Sub transform (#36185)
  liferoad
  authored
  d26dbac
- Remove (inactive) key from example (#36234)
  damccorm
  authored
  350537f
- Touch trigger files for integration tests for OutputBuilder
  kennknowles
  committed
  8ca2a0a
- Move WindowMatchers to main component of runners-core
  kennknowles
  committed
  aa27e9e
- Add OutputBuilder to the Java SDK and use in runners
  kennknowles
  committed
  2cff4cc
- Fix a deadlock situation for BlockOnSplit test case. (#36226)
  shunping
  authored
  9146029
- Fix Null Pointer Exception in BigQueryIO (#36209)
  tarun-google
  authored
  cf55fee
- [python] Fix output pcollections of composite transforms that return DoOutputsTuple (#36220)
  hjtran
  authored
  c7b6576
- Update Beam website to release 2.68.0
  Amar3tto
  committed
  85b2703
- Update dates
  Amar3tto
  committed
  e4142f4
- Update links
  Amar3tto
  committed
  16b3137
- Update import gpg action version
  Amar3tto
  committed
  4c8fdf6
- Adding release-2.68.0-postrelease to protected branches in .asf.yaml
  
  Amar3tto
  authored and
  Amar3tto
  committed
  8057963
- blog: GSoC 2025 - Beam YAML, Kafka and Iceberg User Accessibility (#36233)
  charlespnh
  authored
  42aed71
- Fix bug when execute DataframeTransform a dictionary of Pcoll (#35893)
  DKER2
  authored
  08b0572
- Merge pull request #36136 from apache/release-268-website
  Amar3tto
  authored
  6ec4678
- Add Charles to the list of Website authors
  chamikaramj
  committed
  5ba054e
- [Prism] Support injecting triggered bundle for a batch of elements. (#36219)
  shunping
  authored
  990b5ff
Commits on Sep 23, 2025
Commits on Sep 24, 2025
Commits on Sep 25, 2025
- Bump github.com/aws/aws-sdk-go-v2/config from 1.31.8 to 1.31.10 in /sdks (#36279)
  dependabot[bot]
  authored
  8bcd766
- Bump google-github-actions/setup-gcloud from 2 to 3 (#36280)
  dependabot[bot]
  authored
  b8f0d5f
- Require dill for cloudml (#36269)
  Amar3tto
  authored
  39b1b26
- Update Beam Protobuf Schema (Java) (#35150)
  baeminbo
  authored
  c3ef7f3
- Fix PostCommit Java Hadoop Versions job (#36282)
  Amar3tto
  authored
  432479c
- Fix changes whitespace (#36286)
  damccorm
  authored
  8cb3d44
- Fix JUnit 5 module (#36258)
  Abacn
  authored
  e31079b
Commits on Sep 26, 2025
Commits on Sep 27, 2025
- Initial SQL ParquetTable predicate support
  talatuyarer
  committed
  a8cfece
- Initial SQL ParquetTable predicate support
  talatuyarer
  committed
  e1f4ecd
- Resolved merge conflict
  talatuyarer
  committed
  35b1d85
- Addressed PR review comments
  talatuyarer
  committed
  854febc
- Merge remote-tracking branch 'talat/parquet-filter-pushdown-support' into parquet-filter-pushdown-support
  talatuyarer
  committed
  5b8161e

Commits

Commits on Aug 27, 2025

Commits on Aug 28, 2025

Commits on Aug 29, 2025

Commits on Aug 30, 2025

Commits on Sep 1, 2025

Commits on Sep 2, 2025

Commits on Sep 3, 2025

Commits on Sep 4, 2025

Commits on Sep 5, 2025

Commits on Sep 6, 2025

Commits on Sep 8, 2025

Commits on Sep 9, 2025

Commits on Sep 10, 2025

Commits on Sep 11, 2025

Commits on Sep 12, 2025

Commits on Sep 13, 2025

Commits on Sep 14, 2025

Commits on Sep 15, 2025