Skip to content

feat(connectors): add Google BigQuery sink connector - #4394

Open
kparisa wants to merge 4 commits into
masterfrom
feat/bigquery-sink
Open

kparisa wants to merge 4 commits into
masterfrom
feat/bigquery-sink

Conversation

@kparisa

@kparisa kparisa commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

Which issue does this PR address?

Closes #4393

Rationale

Streams Iggy messages into BigQuery through the Storage Write API _default stream with Arrow-encoded rows. Supports mapped and raw modes, Iggy metadata columns, per-row rejection with offset logging, request splitting below the AppendRows limit, and retries for transient gRPC failures. Verified against a live BigQuery table; CI coverage uses an in-process fake of tables.get and BigQueryWrite.

What changed?

New sink connector under /core/connectors/sinks

Local Execution

  • Passed
  • Pre-commit hooks ran

AI Usage

  • Leveraged ChatGPT, Claude for planning, test scenarios, implementation.
  • Ran local tests, publishing payloads to iggy, configured GCP test project, created the BigQuery (BQ) dataset "iggy" and events table.
  • Verified the records in the BQ table.
  • Tested failure scenarios as well for mis-configs and missing table definitions etc

Streams Iggy messages into BigQuery through the Storage Write API
_default stream with Arrow-encoded rows. Supports mapped and raw
modes, Iggy metadata columns, per-row rejection with offset logging,
request splitting below the AppendRows limit, and retries for
transient gRPC failures. Verified against a live BigQuery table;
CI coverage uses an in-process fake of tables.get and BigQueryWrite.

Closes #4393
@github-actions github-actions Bot added the S-waiting-on-review PR is waiting on a reviewer label Oct 3, 2026
@codecov

codecov Bot commented Oct 3, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.29619% with 219 lines in your changes missing coverage. Please review.
✅ Project coverage is 87.86%. Comparing base (2d7fddb) to head (a32b3af).

Files with missing lines Patch % Lines
core/connectors/sinks/bigquery_sink/src/encode.rs 91.83% 43 Missing and 32 partials ⚠️
core/connectors/sinks/bigquery_sink/src/client.rs 74.60% 55 Missing and 10 partials ⚠️
core/connectors/sinks/bigquery_sink/src/schema.rs 90.68% 37 Missing and 4 partials ⚠️
core/connectors/sinks/bigquery_sink/src/sink.rs 79.31% 20 Missing and 4 partials ⚠️
core/connectors/sinks/bigquery_sink/src/lib.rs 96.34% 4 Missing and 4 partials ⚠️
core/connectors/sinks/bigquery_sink/src/error.rs 93.81% 4 Missing and 2 partials ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master    #4394      +/-   ##
============================================
- Coverage     87.96%   87.86%   -0.11%     
+ Complexity     1579     1578       -1     
============================================
  Files          1290     1296       +6     
  Lines        230145   232191    +2046     
  Branches     193480   195529    +2049     
============================================
+ Hits         202440   204007    +1567     
- Misses        22980    23367     +387     
- Partials       4725     4817      +92     
Components Coverage Δ
Rust Core 89.04% <89.29%> (-0.03%) ⬇️
Java SDK 68.72% <ø> (-0.02%) ⬇️
C# SDK 77.61% <ø> (-0.02%) ⬇️
Python SDK 91.24% <ø> (ø)
PHP SDK 85.67% <ø> (ø)
Node SDK 94.91% <ø> (-1.69%) ⬇️
Go SDK 70.29% <ø> (+0.10%) ⬆️
Files with missing lines Coverage Δ
core/connectors/sinks/bigquery_sink/src/error.rs 93.81% <93.81%> (ø)
core/connectors/sinks/bigquery_sink/src/lib.rs 96.34% <96.34%> (ø)
core/connectors/sinks/bigquery_sink/src/sink.rs 79.31% <79.31%> (ø)
core/connectors/sinks/bigquery_sink/src/schema.rs 90.68% <90.68%> (ø)
core/connectors/sinks/bigquery_sink/src/client.rs 74.60% <74.60%> (ø)
core/connectors/sinks/bigquery_sink/src/encode.rs 91.83% <91.83%> (ø)

... and 43 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@hubcio

hubcio commented Oct 3, 2026

Copy link
Copy Markdown
Contributor

/skill team-review-slim

@github-actions github-actions Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Summary: The review found a rustdoc link error that fails the doctest job, a token error classified as permanent so open() never retries it, a missing integration test, and hot-path encoding costs. Two simplifications cover a hand-rolled hex helper and two Option fields that always move together.

Counts: critical 0, warning 5, nit 2, simplification 2


This review was generated by Claude Code 2.1.284 on deepseek-flash[1m]. Review the output before you act on it.

Comment thread core/connectors/sinks/bigquery_sink/src/lib.rs Outdated
Comment thread core/connectors/sinks/bigquery_sink/src/error.rs Outdated
Comment thread core/connectors/sinks/bigquery_sink/src/encode.rs Outdated
Comment thread core/connectors/sinks/bigquery_sink/src/encode.rs Outdated
Comment thread core/connectors/sinks/bigquery_sink/tests/bigquery_sink.rs
Comment thread core/connectors/sinks/bigquery_sink/src/sink.rs Outdated
Comment thread core/connectors/sinks/bigquery_sink/src/encode.rs Outdated
Comment thread core/connectors/sinks/bigquery_sink/src/encode.rs Outdated
Comment thread core/connectors/sinks/bigquery_sink/src/lib.rs Outdated
@github-actions github-actions Bot added S-waiting-on-author PR is waiting on author response and removed S-waiting-on-review PR is waiting on a reviewer labels Oct 3, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

S-waiting-on-author PR is waiting on author response

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add Google BigQuery Sink Connector

2 participants