[cyriltovena]

What problem does this PR solve?

Issue Number: close #65432

Problem:

GCP storage vaults require long-lived HMAC access keys. GKE deployments using Workload Identity Federation therefore cannot authenticate the vault data path with the pod identity and short-lived Google credentials.

Solution:

Add gcp_workload_identity, a GCP-only storage-vault credentials provider. The shared C++ object-storage client used by BE and recycler obtains OAuth2 access tokens from the GKE/GCE metadata server and applies Authorization: Bearer ... to every GCS XML API request while AWS SigV4 signing remains anonymous.

CREATE STORAGE VAULT gcs_vault PROPERTIES (
    "type" = "S3",
    "s3.endpoint" = "storage.googleapis.com",
    "s3.region" = "us-central1",
    "s3.bucket" = "my-bucket",
    "s3.root.path" = "doris",
    "provider" = "GCP",
    "s3.credentials_provider_type" = "gcp_workload_identity"
);

The implementation:

• validates the provider, endpoint, and credential combination at FE, Meta Service, BE, and recycler boundaries;
• canonicalizes the endpoint to https://storage.googleapis.com before bearer authentication;
• caches tokens process-wide, serializes refreshes, refreshes before expiry, and retains a still-valid token across transient refresh failures;
• fails locally when no valid token exists instead of issuing an anonymous storage request;
• supports HMAC-to-Workload-Identity and Workload-Identity-to-HMAC ALTER transitions, including stale-enum recovery after rollback;
• serializes storage-vault refreshes and rechecks connectivity when keyless credentials change; and
• returns an explicit NotSupported result for presigned URLs.

Upgrade compatibility:

The persisted credential-provider enum is new. Do not create or alter a Workload Identity vault during a mixed-version rollout; upgrade every BE and recycler consumer before enabling it. Explicit HMAC or role credentials take precedence if a stale unknown enum value survives a rollback transition.

Security Impact:

This removes the need to distribute long-lived HMAC secrets on GKE. Metadata acquisition uses a fixed metadata hostname with the required Metadata-Flavor: Google header, disables AWS IMDS discovery, redirects, and proxies, and uses bounded timeouts. Storage requests are restricted to the canonical GCS HTTPS endpoint, and missing tokens fail closed.

Testing:

• Added FE tests for storage-vault-only scope, aliases, credential/provider validation, endpoint canonicalization, partial ALTER mapping, and SHOW CREATE output.
• Added Meta Service tests for creation paths, invalid configurations, credential transitions, rollback recovery, and credential cleanup.
• Added BE/shared-client tests for token caching, refresh, transient failure, concurrent single-flight refresh, fail-closed behavior, bearer headers, client reset, and presigned-URL rejection.
• Added recycler configuration and stale-enum coverage.
• git diff --check origin/master passes.
• protoc --proto_path=gensrc/proto --descriptor_set_out=/dev/null gensrc/proto/cloud.proto passes.
• A focused syntax check of the metadata provider against Doris's pinned AWS SDK 1.11.219 passes.
• Full FE/BE/cloud unit execution and C++ formatting are pending PR CI because this checkout lacks the Doris third-party build bundle and LLVM 16.
• End-to-end validation on GKE with Workload Identity is required before merge.

Release note

Support keyless GCS storage vault authentication on GKE with Workload Identity Federation using "s3.credentials_provider_type" = "gcp_workload_identity".

Check List (For Author)

• Test

  • Regression test
  • Unit tests added; execution pending PR CI
  • Manual test
  • No need to test or manual test

• Behavior changed:

  • Yes. Adds GKE Workload Identity authentication for GCP storage vaults.

• Does this need documentation?

  • Yes. A doris-website follow-up is required.

Check List (For Reviewer who merge this PR)

• Confirm the release note
• Confirm test cases
• Confirm document
• Add branch pick label

[hello-stephen]

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
3. What features were added. Why was this function added?
4. Which code was refactored and why was this part of the code refactored?
5. Which functions were optimized and what is the difference before and after the optimization?

[cyriltovena]

/review

[cyriltovena]

/review

[SLoeuillet]

+1, we need this for Doris on GKE with Workload Identity: today our storage vault only works with a long-lived HMAC key. The same bearer-token path would also let Iceberg REST catalogs use vended GCS tokens (#68671); we'll send that PR once this one lands.

[SLoeuillet]

Superseeded by #68540 ?

[morningman]

Hi @cyriltovena thanks for you contribution.
This maybe superseeded by #68540.
I will check it later

[sollhui]

Thanks @cyriltovena for working on this. I compared the current implementations: #68540 covers the core keyless GCS storage-vault use case here, including GKE Workload Identity.

In #68540, provider=GCP with gs.credential_provider_type=DEFAULT uses Google Application Default Credentials. On a GKE workload configured for Workload Identity, ADC can obtain short-lived credentials through the GKE metadata server, without HMAC keys. This is the documented GKE credential flow. The vault credential is carried through FE/MetaService to both BE and recycler, which use the shared OAuth bearer-authenticated GCS XML client. COMPUTE_ENGINE and optional service-account impersonation are also supported.

So the configuration corresponding to this PR's s3.credentials_provider_type=gcp_workload_identity is gs.credential_provider_type=DEFAULT in #68540, with no s3.access_key / s3.secret_key or AWS credential-provider properties.

This is functional overlap, rather than identical changes: this PR also has endpoint restrictions and vault-refresh/connectivity hardening that #68540 does not reproduce in the same way. Any still-needed pieces could be carried over separately.

[sollhui]

We will move this work forward in #68540. Any remaining issues can be addressed in follow-up fixes after #68540 is merged. Thanks for your contribution!