Repository navigation
feat: add the bigquery backend, with portable per-row refusal reasons - #14
Open
crisfiorini-oss wants to merge 14 commits into
Open
crisfiorini-oss wants to merge 14 commits into
crisfiorini-oss wants to merge 14 commits into
Conversation
crisfiorini-oss
force-pushed
the
feat/bigquery-backend
branch
from
October 11, 2026 18:45
e8dfaf7 to
c7ced09
Compare
crisfiorini-oss
force-pushed
the
feat/bigquery-backend
branch
from
October 15, 2026 15:38
c7ced09 to
31ca287
Compare
crisfiorini-oss
marked this pull request as draft
October 27, 2026 21:31
InsertRows reported a refused row as an index and free text, so a caller that must set aside exactly the rows a backend permanently refused, and retry the rest, could only read the text. That is the question codefly-dev#12 asks: can per-row refusal detail survive the error normalization? RowError now carries a RowRefusal: unspecified, invalid value, row too large, or schema mismatch. The names are the ones every warehouse can mean the same thing by, and schema mismatch is the one that says the table, not the row, is what disagrees, so a caller does not set a row aside for a table's fault. A failure that says nothing against the row (throttle, quota, timeout) stays a status code and is never a row refusal. The Go-native backend.RowError gets the same field and the server maps it in one place; a reason the server does not know stays unspecified instead of being guessed. mem still returns Unsupported for InsertRows, now pinned by a test, and the wire mapping is covered with a backend that refuses one row per reason. gen/ was regenerated through `codefly generate proto`; the generator version banners are unchanged. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The normalized error model had no code for "the query ran past its time limit", so a backend that enforces timeout_ms or SWH_QUERY_TIMEOUT could only report it as Internal, which tells a client the gateway is broken when the query was merely too slow. Add serr.DeadlineExceeded and map it to the gRPC code of the same name, so a client can branch on it and retry with a longer limit. The service comment in the proto lists the status codes a client may see; it now names DEADLINE_EXCEEDED too, and gen/ is regenerated through `codefly generate proto` (comment-only change, generator banners unchanged). A test pins every normalized code to its gRPC code. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
… DSN A backend bound to a database was refused with "SWH_DATABASE (or SWH_DSN) is required" whatever it was. For a backend with no connection string, such as one bound to a cloud project, the message sends the operator to set a variable the backend then refuses, and a DSN given alone passed the check only to fail later. Registration gains DSN, which says whether the backend reads SWH_DSN. Only such a backend may stand a DSN in for the database, and only it is told it may. The two kinds are covered by test backends, and the unregistered-kind test now uses snowflake so it does not name a backend this repository implements. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
A BigQuery backend behind the uniform contract, so a client sends SQL and reads Arrow without linking the BigQuery SDK. It is registered as `bigquery` and compiled into the one binary. Implemented: the catalog and DDL plane (list/get/create/drop of datasets and tables, with partitioning, clustering and CTAS), Query with named parameters, a dry run, a byte cap and a timeout and Arrow-IPC batches, GetJob and CancelJob, and InsertRows from Arrow-IPC batches with streaming inserts. Load, Unload and Native return serr.Unsupported and Capabilities says so. - Every BigQuery failure is mapped to a serr.Code in one place (errors.go). A client is told a fixed sentence per code; the one vendor text it can read is the diagnosis of its own invalid SQL, and the vendor's own error goes to the operator's log and is never wrapped. - The type mapping is explicit in both directions and refuses what it cannot carry (a RANGE result, an Arrow map, a nanosecond timestamp) with Unsupported or an invalid argument, never a lookalike type or a rounded value. - InsertRows is not atomic: BigQuery is asked to skip invalid rows and each refused row is reported with its index across the whole stream and a portable reason. A row whose content BigQuery refuses is INVALID_VALUE, a row over the request limit is ROW_TOO_LARGE, and a batch whose columns disagree with the table's, found by comparing it with the table's metadata and not by reading vendor text, refuses every row as SCHEMA_MISMATCH without sending any. A failure that says nothing against a row fails the call. - Names that become request paths are checked first: the client library expands them without escaping, so a dataset named "d/tables/t" would address a table. - SWH_DATABASE, SWH_DATASET, SWH_LOCATION, SWH_CREDENTIALS_FILE, SWH_MAX_QUERY_BYTES and SWH_QUERY_TIMEOUT already existed and cover what BigQuery needs, so no variable is added; settings it never reads (a DSN, a host, a login) are refused at startup instead of ignored. Credentials are Application Default Credentials, or the service-account file. Adds cloud.google.com/go/bigquery and github.com/apache/arrow-go/v18, the maintained Arrow module (v18.8.0 is its latest; apache/arrow/go/v15 is the old one). grpc moves to v1.83.2, which the BigQuery client requires. Unit tests run the real client against a fake of BigQuery's REST API served from the test process: they check what is sent and how answers are read, not what BigQuery does. One integration test runs against a real project when SWH_TEST_BIGQUERY_PROJECT is set and is skipped with a message otherwise; it has not been run. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The backend matrix marked bigquery as planned and said mem was the only wired backend. It now lists what each wired backend serves, per RPC, and the BigQuery notes a client needs and cannot guess: the parameter syntax and value forms, the Arrow framing (one Schema message, then RecordBatch messages), the type mapping, what a listing costs, and InsertRows semantics: the per-row refusal reasons, the stream-wide row index, and at-least-once delivery after a failed call. The env table says which existing variables mean what to BigQuery and that settings it never reads are refused. AGENTS.md said there was no fake for a cloud backend, which is no longer true of bigquery, so its "say what you did not verify" rule now says what the fake shows and what only a real project does. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…very backend A Query header carries the result schema as an Arrow IPC schema message, each message after it carries one record-batch message, and InsertRows takes the same two parts. Joined, they are a valid Arrow IPC stream, but arrow-go writes and reads whole streams, so the wire form needs its own code: a writer that hands back the schema and each batch as separate messages, and a reader that joins a header schema and a source of batch messages back into a stream. Every backend that produces or ingests Arrow needs exactly this, so it lives in its own package instead of in the first backend that wants it. Dictionary-encoded fields are refused by the encoder: the wire carries no dictionary messages. Tested against arrow-go's own reader: schema plus batches round-trip, the first batch message does not repeat the schema, a self-describing stream (no header schema) reads, an empty stream reads, a source error surfaces as itself. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…and read a header that ends the stream Two backends read InsertRows through this package, and each needs the same three things from it, which the first version left to the caller. A header schema that arrives terminated by the end-of-stream marker was read as the end of the stream: the batches after it were never reached and every row was dropped without an error. The marker is now removed from the header before it is read. A failure to read was one undifferentiated error, so a caller could not return the transport's own error as itself and report only the client's malformed bytes as an invalid argument. NewReader now returns a Reader whose errors are the source's own error when the source failed, and wrap ErrMalformed when the bytes did. The Reader has the read methods of arrow-go's ipc.Reader, so it is a drop-in for a caller that only reads. An empty message is refused as malformed. Skipping it would hide a client that sent a batch it never filled in. Tested: a terminated header still yields its rows; each kind of bad input (not a schema, a batch cut short, bytes that are not a message, a second schema where a batch belongs, an empty message, nothing at all) is ErrMalformed; a failed source is not. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…d's own copy The bigquery backend carried its own Arrow IPC framing, a writer that cut the schema and each batch out of a stream and a reader that rejoined them, and internal/arrowipc now does exactly that for every backend. Two copies of how Arrow travels on the wire is one too many: the next backend would have picked whichever it read first. Query writes its header schema with arrowipc.SchemaMessage and each batch with an arrowipc.Encoder. InsertRows reads with arrowipc.NewReader and returns a malformed stream as InvalidArgument and a source that failed as itself, which is what the backend's own reader did. One thing narrows, to what the proto already says. InsertRows used to also accept a header schema together with batches that each repeat their own schema; the contract is the header's schema and then bare batches, or an empty header and a first message that carries the schema, and that is what is read now. A second schema where a batch belongs is InvalidArgument. The framing tests that belonged to the copy are gone with it; arrowipc's own tests cover the same cases, and the backend's insert tests still drive the whole path, now with a test that a failed transport comes back as itself. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
The Arrow a client sends to InsertRows states its own sizes. arrow-go documents size limits for its message reader, but ipc.NewReader builds its message reader without applying them, so a message's stated metadata and body sizes were taken at face value. arrowipc.NewReader now builds the message reader with the limits and hands it to ipc.NewReaderFromMessageReader: metadata over 1 MiB (a constant; a schema or a buffer layout is a few bytes a column) and a body over SWH_MAX_ARROW_MESSAGE_BYTES (default 64 MiB, positive, malformed or zero rejected at startup like the other limits) are refused as ErrMalformed, which the backends return as InvalidArgument. A bound that is not positive means the default, never no bound, because arrow-go reads it as unbounded. The backends pass the configured value through backend.Config. A test fails if any other file in the repository, test files included, builds an Arrow reader directly; the two bigquery tests that did now go through arrowipc. Tested: a batch message whose metadata states a body of 16 TiB is refused with under 8 MiB allocated, by default, with a bound of zero and with a negative one; metadata that states 2 GiB, and a stream of text, likewise; a body over a small configured bound is refused and one under it read; bigquery InsertRows with a small SWH_MAX_ARROW_MESSAGE_BYTES returns InvalidArgument and sends nothing. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…its own bytes before decoding it STOPGAP: a defensive check, to be removed if the decoder validates its own input. A schema message states counts (fields, children, metadata entries, type ids), lengths and offsets, and the decoder sizes what it builds from them. The message size limits do not bound those, so arrowipc now checks them against the message's own bytes first. Before arrow-go sees the first message of a stream, and only when it is a schema, arrowipc walks its flatbuffer with the public flatbuffers table API: the schema's fields, each field's children, every custom metadata vector, a union's type ids and a timestamp's time zone, which is what the decoder reads from a schema. It refuses as ErrMalformed any vector that states more elements than the bytes after it can hold (at 4 bytes an element, the smallest there is), any offset that points outside the message or to or before itself (which also rules out a cycle), and any walk that would visit more structure than the message has bytes, counting a table that is referenced more than once at each reference. A schema that is a tree costs no more than its size, so a producer that writes an ordinary schema never reaches the last rule. What passes is bounded by the 1 MiB metadata cap, and so is what the decoder builds from it. Nothing else changes for a message that is not a schema. Tested: a nested schema (struct and list children, custom metadata on the schema and its fields, a union, a timestamp with a zone), a 4,000-column schema with metadata on every field and an empty one all still read and equal the schema that was sent; thirteen schemas, each the nested one with a count, a length or an offset changed to state more than the bytes hold, are refused with under 8 MiB allocated, in the header and in a first message that carries the schema; a 400 KB message whose 50,000 fields share 50,000 children is refused; setting any single byte of the nested schema message to any of five values never costs 8 MiB. The fuzz target FuzzSchemaReader runs its seeds in the ordinary suite; 5.5 million executions of `go test -fuzz` in a container found no input over the bound. Not changed: a message may still make the reader allocate the body it states, up to SWH_MAX_ARROW_MESSAGE_BYTES, before it finds the bytes are not there. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
…eceive limit An Arrow message reaches InsertRows inside one gRPC message and cannot be larger than the message carrying it, so a bound above the gRPC receive limit refuses nothing a client can send. The default is now 4 MiB, grpc-go's own; the knob stays, for a deployment that raises the gRPC server's limit. The two bigquery tests that send a batch of more than 4 MiB straight to the backend, to exercise its own 8 MiB row limit, raise the bound for themselves. Co-Authored-By: Claude Sonnet 5.5 <noreply@anthropic.com>
crisfiorini-oss
force-pushed
the
feat/bigquery-backend
branch
from
October 27, 2026 21:34
b6cc6fc to
b0420db
Compare
crisfiorini-oss
marked this pull request as ready for review
October 27, 2026 21:40
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implement the planned BigQuery backend behind the existing warehouse contract. The branch is current with upstream main
7aacb3f; headb0420dbe2e72c7291eb235ba2e0c8d5de01cf5ea.Behavior
Classification: fix, at the existing warehouse owner; the bounded Arrow preflight remains an explicitly documented stopgap.
The existing implementation's scope exception and reuse decision remain recorded in README. This review retains the documented catalog, DDL, query, job and streaming-insert behavior; it adds no new source code.
Validation
All four fresh checks on this exact rebased head PASS: build/vet/race, runtime end-to-end, image/SBOM and agent conformance. Marked ready only after all four passed. Module verification and whitespace checks pass. Range-diff preserves every backend feature patch; the only changed patch context is main's Go 1.27.3 floor. The redundant toolchain commit is dropped. No full local suite was repeated.
Complete owner generation reproduces the committed protobuf tree; generated files were not hand-edited or selectively retained. Tests use the real BigQuery client against an in-process REST server.
Not qualified
Live BigQuery remains unrun: the explicit project/credentials are unset, so TestAgainstRealBigQuery skips. No cloud or composed audit-delivery result is claimed. The current contract has no Scan API; Warehouse AuditStorage and the corresponding admin page remain dependent work. No release or deployment is performed by this PR preparation.