[Feature] Iceberg REST catalog: use vended GCS credentials (gcs.oauth2.token) #68671

Search before asking

• I had searched in the issues and found no similar issues.

Description

With an Iceberg REST catalog backed by a native GCS warehouse (e.g. Lakekeeper or Polaris with type: gcs), the catalog vends short-lived, table-scoped credentials as gcs.oauth2.token / gcs.oauth2.token-expires-at, per the Iceberg GCPProperties convention. Java Iceberg, iceberg-rust and iceberg-go all consume them.

Doris (checked on 4.1.4.1 and master) can't use them:

1. CredentialUtils.CLOUD_STORAGE_PREFIXES keeps gs. but not gcs., so the vended token is dropped.
2. Even if kept, the BE reaches GCS only through the S3-compatible endpoint with an HMAC key (gs.access_key / gs.secret_key), so there is nothing that can send a bearer token.

The only working setup is iceberg.rest.vended-credentials-enabled=false plus a long-lived, bucket-wide HMAC key in the catalog properties, which defeats per-table credential vending.

Proposal:

• keep gcs.-prefixed vended properties and map gcs.oauth2.token (and its expiry) to the storage properties;
• have the BE send it as Authorization: Bearer to the GCS XML API. [feature](vault) Support keyless GCS vaults with GKE Workload Identity #66484 adds exactly this bearer path to the shared object-storage client for storage vaults (tokens from the GKE metadata server); the Iceberg path could reuse it with the vended token as the source.

Use case

Doris on GKE querying and writing Iceberg tables in a GCS bucket through Lakekeeper, without handing Doris a long-lived HMAC key.

Related issues

#66484 (keyless GCS storage vaults with GKE Workload Identity), #65432

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Once #66484 is merged: this builds on its bearer-token path, so the PR will follow it.

Code of Conduct

• I agree to follow this project's Code of Conduct