From c76c712ad2765be6c70a14edee26c3f9123c0fcb Mon Sep 17 00:00:00 2001 From: lloyd tabb Date: Wed, 12 Aug 2026 20:06:22 -0700 Subject: [PATCH 1/3] docs(config): document supplying a BigQuery service account key from the environment MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit `serviceAccountKey` is a `json` property, and json properties take their value literally — an `{"env": "..."}` inside one is never resolved. A server holding its key in an environment variable therefore cannot use that property, and the failure arrives as an error from BigQuery about a missing client_email field rather than anything naming the config. Document `serviceAccountKeyJson` and `serviceAccountKeyJsonBase64` (malloydata/malloy#3039), which are `secret` strings and do take references, with the shell commands that produce each form. Also state the json-property exception in the Environment Variables section, which until now claimed without qualification that any property value can be replaced with a reference. Signed-off-by: lloyd tabb --- src/documentation/setup/config.malloynb | 34 +++++++++++++++++++++++++ 1 file changed, 34 insertions(+) diff --git a/src/documentation/setup/config.malloynb b/src/documentation/setup/config.malloynb index c129bf4a..a019e666 100644 --- a/src/documentation/setup/config.malloynb +++ b/src/documentation/setup/config.malloynb @@ -146,12 +146,44 @@ Malloy exposes two parameters that let you choose how a connection participates | `projectId` | string | GCP project ID | | `serviceAccountKeyPath` | file | Path to service account JSON key | | `serviceAccountKey` | json | Service account key as a JSON object (alternative to file path) | +| `serviceAccountKeyJson` | secret | Service account key as a JSON **string** | +| `serviceAccountKeyJsonBase64` | secret | Service account key as base64-encoded JSON | | `location` | string | Dataset location | | `maximumBytesBilled` | string | Byte billing cap | | `timeoutMs` | string | Query timeout in ms | | `billingProjectId` | string | Billing project (if different) | | `setupSQL` | text | Connection setup SQL ([see below](#setup-sql)) | +With no key configured at all, the connection uses [application default credentials](https://cloud.google.com/docs/authentication/application-default-credentials) — the usual choice for local development, where `gcloud auth application-default login` has already run. + +**Supplying the key from the environment.** On a server the key normally arrives as an environment variable rather than as a file on disk. Use `serviceAccountKeyJson`, which holds the entire key file as a string: + +```json +{ + "connections": { + "my_bigquery": { + "is": "bigquery", + "projectId": "my-project", + "serviceAccountKeyJson": {"env": "BIGQUERY_CREDENTIALS_JSON"} + } + } +} +``` + +Set the variable to the key file's contents on a single line — the key's own `\n` escapes are preserved by JSON, and quoting the value keeps the shell out of it: + +```bash +export BIGQUERY_CREDENTIALS_JSON="$(jq -c . service-account-key.json)" +``` + +`serviceAccountKeyJsonBase64` takes the same key base64-encoded, for pipelines and secret stores that mangle the quoting or the embedded newlines of raw JSON: + +```bash +export BIGQUERY_CREDENTIALS_JSON_B64="$(base64 < service-account-key.json | tr -d '\n')" +``` + +Note that `serviceAccountKey` — the `json`-typed property — **cannot** take an environment variable reference. Like every `json` property, it treats `{"env": "..."}` as literal data, so that object itself becomes the credentials and BigQuery rejects it with `The incoming JSON object does not contain a client_email field`. Use one of the two `secret` properties above instead. If several are set, the order of precedence is `serviceAccountKey`, `serviceAccountKeyJson`, `serviceAccountKeyJsonBase64`. + ### `databricks` — Databricks | Parameter | Type | Description | @@ -343,3 +375,5 @@ The `{"env": "VAR_NAME"}` syntax looks up the value from the named environment v You can also provide a plain string value directly — this is useful for testing but not recommended for shared or committed config files. +**One exception: `json` properties.** Properties whose type is `json` — `serviceAccountKey`, `ssl`, `session`, `extraCredential`, `extraHeaders` — hold structured data, and they take that data literally. A `{"env": "..."}` written inside one is not resolved; it is passed along to the driver as the object it looks like, which usually surfaces later as a confusing error from the driver rather than as a missing-variable message. Where a secret needs to come from the environment, use the string-typed property meant for it (for BigQuery, [`serviceAccountKeyJson`](#bigquery-google-bigquery)). + From a930b4d62226274e1b97bb1257fa409f15b2f824 Mon Sep 17 00:00:00 2001 From: lloyd tabb Date: Fri, 14 Aug 2026 07:34:11 -0700 Subject: [PATCH 2/3] docs(config): one BigQuery key property, with the encoding detected MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Follows the change in malloydata/malloy#3039 to a single `serviceAccountKeyJson` that takes JSON or base64 and detects which. Drops the separate base64 property, and states the true reason base64 is offered: quoting, not newlines — a downloaded key file carries its newlines as `\n` escapes, so the JSON is one line with none to mangle. Signed-off-by: lloyd tabb --- src/documentation/setup/config.malloynb | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/src/documentation/setup/config.malloynb b/src/documentation/setup/config.malloynb index a019e666..d4c61a5b 100644 --- a/src/documentation/setup/config.malloynb +++ b/src/documentation/setup/config.malloynb @@ -146,8 +146,7 @@ Malloy exposes two parameters that let you choose how a connection participates | `projectId` | string | GCP project ID | | `serviceAccountKeyPath` | file | Path to service account JSON key | | `serviceAccountKey` | json | Service account key as a JSON object (alternative to file path) | -| `serviceAccountKeyJson` | secret | Service account key as a JSON **string** | -| `serviceAccountKeyJsonBase64` | secret | Service account key as base64-encoded JSON | +| `serviceAccountKeyJson` | secret | Service account key as a **string** — JSON or base64-encoded JSON | | `location` | string | Dataset location | | `maximumBytesBilled` | string | Byte billing cap | | `timeoutMs` | string | Query timeout in ms | @@ -170,19 +169,21 @@ With no key configured at all, the connection uses [application default credenti } ``` -Set the variable to the key file's contents on a single line — the key's own `\n` escapes are preserved by JSON, and quoting the value keeps the shell out of it: +Set the variable to the key file's contents. Quoting the value keeps the shell out of it: ```bash export BIGQUERY_CREDENTIALS_JSON="$(jq -c . service-account-key.json)" ``` -`serviceAccountKeyJsonBase64` takes the same key base64-encoded, for pipelines and secret stores that mangle the quoting or the embedded newlines of raw JSON: +The property also accepts the key base64-encoded, which is one unquoted token and so travels through shells, CI secret editors, and `.env` files more reliably than a blob of JSON braces and quotes: ```bash -export BIGQUERY_CREDENTIALS_JSON_B64="$(base64 < service-account-key.json | tr -d '\n')" +export BIGQUERY_CREDENTIALS_JSON="$(base64 < service-account-key.json | tr -d '\n')" ``` -Note that `serviceAccountKey` — the `json`-typed property — **cannot** take an environment variable reference. Like every `json` property, it treats `{"env": "..."}` as literal data, so that object itself becomes the credentials and BigQuery rejects it with `The incoming JSON object does not contain a client_email field`. Use one of the two `secret` properties above instead. If several are set, the order of precedence is `serviceAccountKey`, `serviceAccountKeyJson`, `serviceAccountKeyJsonBase64`. +No flag distinguishes the two — the encoding is detected, since JSON always begins with `{` and `{` is not a base64 character. Surrounding whitespace is ignored, so a trailing newline from `$(cat …)` or a here-doc is harmless. + +Note that `serviceAccountKey` — the `json`-typed property — **cannot** take an environment variable reference. Like every `json` property, it treats `{"env": "..."}` as literal data, so that object itself becomes the credentials and BigQuery rejects it with `The incoming JSON object does not contain a client_email field`. Use `serviceAccountKeyJson` instead. If both are set, `serviceAccountKey` wins. ### `databricks` — Databricks From d01541397359839db10c4236bb4319e9b4dcac10 Mon Sep 17 00:00:00 2001 From: lloyd tabb Date: Sat, 15 Aug 2026 09:23:26 -0700 Subject: [PATCH 3/3] docs(config): drop the encoding-detection and json-exception paragraphs MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Review feedback on #339. The detection rule (JSON starts with `{`, base64 can't) is implementation detail nobody reading this page needs — both encodings work, which the examples already show. The json-property exception in the Environment Variables section is dropped too; the note in the BigQuery section covers the case a reader actually hits, and postgres and trino already carry their own. Signed-off-by: lloyd tabb --- src/documentation/setup/config.malloynb | 4 ---- 1 file changed, 4 deletions(-) diff --git a/src/documentation/setup/config.malloynb b/src/documentation/setup/config.malloynb index d4c61a5b..bc65e724 100644 --- a/src/documentation/setup/config.malloynb +++ b/src/documentation/setup/config.malloynb @@ -181,8 +181,6 @@ The property also accepts the key base64-encoded, which is one unquoted token an export BIGQUERY_CREDENTIALS_JSON="$(base64 < service-account-key.json | tr -d '\n')" ``` -No flag distinguishes the two — the encoding is detected, since JSON always begins with `{` and `{` is not a base64 character. Surrounding whitespace is ignored, so a trailing newline from `$(cat …)` or a here-doc is harmless. - Note that `serviceAccountKey` — the `json`-typed property — **cannot** take an environment variable reference. Like every `json` property, it treats `{"env": "..."}` as literal data, so that object itself becomes the credentials and BigQuery rejects it with `The incoming JSON object does not contain a client_email field`. Use `serviceAccountKeyJson` instead. If both are set, `serviceAccountKey` wins. ### `databricks` — Databricks @@ -376,5 +374,3 @@ The `{"env": "VAR_NAME"}` syntax looks up the value from the named environment v You can also provide a plain string value directly — this is useful for testing but not recommended for shared or committed config files. -**One exception: `json` properties.** Properties whose type is `json` — `serviceAccountKey`, `ssl`, `session`, `extraCredential`, `extraHeaders` — hold structured data, and they take that data literally. A `{"env": "..."}` written inside one is not resolved; it is passed along to the driver as the object it looks like, which usually surfaces later as a confusing error from the driver rather than as a missing-variable message. Where a secret needs to come from the environment, use the string-typed property meant for it (for BigQuery, [`serviceAccountKeyJson`](#bigquery-google-bigquery)). -