From e3b95e4917471276dea434a045eebee1c7f8753d Mon Sep 17 00:00:00 2001 From: Lloyd Tabb Date: Thu, 20 Aug 2026 17:17:42 +0000 Subject: [PATCH 1/2] docs(config): the BigQuery server pattern and how to run it locally MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Lead the BigQuery section with the shape a deployment actually uses — a projectId plus a serviceAccountKeyJson reference to an environment variable — and document that the same file needs no local variant: an unset {"env": ...} reference is omitted, so the connection falls back to application default credentials. Also warn against the near-miss that fallback invites: base64-ing ~/.config/gcloud/application_default_credentials.json into the variable. That file is an authorized_user credential, and the connector rejects it for having no client_email and private_key. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01APyHsXjok5JD79hH4sjvbH Signed-off-by: Lloyd Tabb --- src/documentation/setup/config.malloynb | 23 ++++++++++++++++------ src/documentation/setup/extension.malloynb | 2 ++ 2 files changed, 19 insertions(+), 6 deletions(-) diff --git a/src/documentation/setup/config.malloynb b/src/documentation/setup/config.malloynb index bc65e724..a3a80257 100644 --- a/src/documentation/setup/config.malloynb +++ b/src/documentation/setup/config.malloynb @@ -155,32 +155,43 @@ Malloy exposes two parameters that let you choose how a connection participates With no key configured at all, the connection uses [application default credentials](https://cloud.google.com/docs/authentication/application-default-credentials) — the usual choice for local development, where `gcloud auth application-default login` has already run. -**Supplying the key from the environment.** On a server the key normally arrives as an environment variable rather than as a file on disk. Use `serviceAccountKeyJson`, which holds the entire key file as a string: +**The server pattern.** On a server the key arrives as an environment variable rather than as a file on disk, so a deployed BigQuery config is usually just a project and a reference to that variable: ```json { "connections": { - "my_bigquery": { + "malloy_bq": { "is": "bigquery", "projectId": "my-project", - "serviceAccountKeyJson": {"env": "BIGQUERY_CREDENTIALS_JSON"} + "serviceAccountKeyJson": {"env": "MALLOY_BQ_JSON"} } } } ``` -Set the variable to the key file's contents. Quoting the value keeps the shell out of it: +`serviceAccountKeyJson` holds the entire key file as a **string**. Set the variable to the key file's contents; quoting the value keeps the shell out of it: ```bash -export BIGQUERY_CREDENTIALS_JSON="$(jq -c . service-account-key.json)" +export MALLOY_BQ_JSON="$(jq -c . service-account-key.json)" ``` The property also accepts the key base64-encoded, which is one unquoted token and so travels through shells, CI secret editors, and `.env` files more reliably than a blob of JSON braces and quotes: ```bash -export BIGQUERY_CREDENTIALS_JSON="$(base64 < service-account-key.json | tr -d '\n')" +export MALLOY_BQ_JSON="$(base64 < service-account-key.json | tr -d '\n')" ``` +Which encoding arrived is detected, not declared, so switching between the two needs no config change. + +**Running that same config locally.** An `{"env": "..."}` reference to a variable that isn't set is omitted rather than an error, and a BigQuery connection with no key falls back to application default credentials. So the file above needs no local variant: leave `MALLOY_BQ_JSON` unset on your laptop, log in once, and the same `malloy_bq` connection runs against the same project through your own credentials. + +```bash +gcloud auth login --update-adc +gcloud config set project my-project --installation +``` + +Don't try to fill `MALLOY_BQ_JSON` from `~/.config/gcloud/application_default_credentials.json`. That file holds an `authorized_user` credential — a refresh token, not a key — and the connection rejects it up front with `serviceAccountKeyJson parsed but is not a service account key`. The property takes a service account key or an [external account](https://cloud.google.com/iam/docs/workload-identity-federation) (workload identity federation) config; ADC reaches BigQuery through the unset-variable fallback above instead. To exercise the keyed path itself before deploying, point the variable at a real service account key. + Note that `serviceAccountKey` — the `json`-typed property — **cannot** take an environment variable reference. Like every `json` property, it treats `{"env": "..."}` as literal data, so that object itself becomes the credentials and BigQuery rejects it with `The incoming JSON object does not contain a client_email field`. Use `serviceAccountKeyJson` instead. If both are set, `serviceAccountKey` wins. ### `databricks` — Databricks diff --git a/src/documentation/setup/extension.malloynb b/src/documentation/setup/extension.malloynb index 7b526ba4..403cef3f 100644 --- a/src/documentation/setup/extension.malloynb +++ b/src/documentation/setup/extension.malloynb @@ -155,6 +155,8 @@ Then add a BigQuery connection via **Malloy: Edit Connections**. Leave the servi In **Malloy: Edit Connections**, click "Pick file" to select your service account JSON key. +To develop against the same config a server will run — a `projectId` plus a `serviceAccountKeyJson` reference to an environment variable — check that `malloy-config.json` in and leave the variable unset locally, so the connection falls back to the gcloud credentials above. See [BigQuery configuration](config.malloynb#bigquery-google-bigquery). + ### Snowflake Both password and RSA key authentication are supported via **Malloy: Edit Connections**. From c25a053f7a40b2d374cd341f8d8723373cfe6195 Mon Sep 17 00:00:00 2001 From: Lloyd Tabb Date: Thu, 20 Aug 2026 17:28:16 +0000 Subject: [PATCH 2/2] docs(config): condense the BigQuery server setup to four steps Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01APyHsXjok5JD79hH4sjvbH Signed-off-by: Lloyd Tabb --- src/documentation/setup/config.malloynb | 26 ++++++++++------------ src/documentation/setup/extension.malloynb | 2 +- 2 files changed, 13 insertions(+), 15 deletions(-) diff --git a/src/documentation/setup/config.malloynb b/src/documentation/setup/config.malloynb index a3a80257..dcaf2f14 100644 --- a/src/documentation/setup/config.malloynb +++ b/src/documentation/setup/config.malloynb @@ -153,9 +153,11 @@ Malloy exposes two parameters that let you choose how a connection participates | `billingProjectId` | string | Billing project (if different) | | `setupSQL` | text | Connection setup SQL ([see below](#setup-sql)) | -With no key configured at all, the connection uses [application default credentials](https://cloud.google.com/docs/authentication/application-default-credentials) — the usual choice for local development, where `gcloud auth application-default login` has already run. +With no key configured at all, the connection uses [application default credentials](https://cloud.google.com/docs/authentication/application-default-credentials). -**The server pattern.** On a server the key arrives as an environment variable rather than as a file on disk, so a deployed BigQuery config is usually just a project and a reference to that variable: +**Setting up a server.** Keep the key out of the config file and read it from the environment. + +1. Point the config at the variable: ```json { @@ -169,30 +171,26 @@ With no key configured at all, the connection uses [application default credenti } ``` -`serviceAccountKeyJson` holds the entire key file as a **string**. Set the variable to the key file's contents; quoting the value keeps the shell out of it: +2. In your test environment, fill it from your own login: ```bash -export MALLOY_BQ_JSON="$(jq -c . service-account-key.json)" +gcloud auth login --update-adc +export MALLOY_BQ_JSON="$(cat ~/.config/gcloud/application_default_credentials.json)" ``` -The property also accepts the key base64-encoded, which is one unquoted token and so travels through shells, CI secret editors, and `.env` files more reliably than a blob of JSON braces and quotes: +3. In production, set the same variable to a service account key: ```bash -export MALLOY_BQ_JSON="$(base64 < service-account-key.json | tr -d '\n')" +export MALLOY_BQ_JSON="$(cat service-account-key.json)" ``` -Which encoding arrived is detected, not declared, so switching between the two needs no config change. - -**Running that same config locally.** An `{"env": "..."}` reference to a variable that isn't set is omitted rather than an error, and a BigQuery connection with no key falls back to application default credentials. So the file above needs no local variant: leave `MALLOY_BQ_JSON` unset on your laptop, log in once, and the same `malloy_bq` connection runs against the same project through your own credentials. +4. Base64 is also accepted, and is one unquoted token — handy for CI secret editors and `.env` files: ```bash -gcloud auth login --update-adc -gcloud config set project my-project --installation +export MALLOY_BQ_JSON="$(base64 < service-account-key.json | tr -d '\n')" ``` -Don't try to fill `MALLOY_BQ_JSON` from `~/.config/gcloud/application_default_credentials.json`. That file holds an `authorized_user` credential — a refresh token, not a key — and the connection rejects it up front with `serviceAccountKeyJson parsed but is not a service account key`. The property takes a service account key or an [external account](https://cloud.google.com/iam/docs/workload-identity-federation) (workload identity federation) config; ADC reaches BigQuery through the unset-variable fallback above instead. To exercise the keyed path itself before deploying, point the variable at a real service account key. - -Note that `serviceAccountKey` — the `json`-typed property — **cannot** take an environment variable reference. Like every `json` property, it treats `{"env": "..."}` as literal data, so that object itself becomes the credentials and BigQuery rejects it with `The incoming JSON object does not contain a client_email field`. Use `serviceAccountKeyJson` instead. If both are set, `serviceAccountKey` wins. +Use `serviceAccountKeyJson` instead. If both are set, `serviceAccountKey` wins. ### `databricks` — Databricks diff --git a/src/documentation/setup/extension.malloynb b/src/documentation/setup/extension.malloynb index 403cef3f..129125a7 100644 --- a/src/documentation/setup/extension.malloynb +++ b/src/documentation/setup/extension.malloynb @@ -155,7 +155,7 @@ Then add a BigQuery connection via **Malloy: Edit Connections**. Leave the servi In **Malloy: Edit Connections**, click "Pick file" to select your service account JSON key. -To develop against the same config a server will run — a `projectId` plus a `serviceAccountKeyJson` reference to an environment variable — check that `malloy-config.json` in and leave the variable unset locally, so the connection falls back to the gcloud credentials above. See [BigQuery configuration](config.malloynb#bigquery-google-bigquery). +To read the key from an environment variable instead, see [BigQuery configuration](config.malloynb#bigquery-google-bigquery). ### Snowflake